introduction 20to 20 education 20design research

An Introduction to Educational Design Research

SLO • Netherlands institute for curriculum developmentSLO

PO box 20417500 CA EnschedeThe Netherlands

T +31(0)53 484 08 40F +31(0)53 430 76 92E [email protected]

www.slo.nl

SLO is the Netherlands institute for curriculum development. We are bridging the contexts of policy, research, and practice. Our expertise focuses on the development of curricular goals and content for various educational levels, from national policy to classroom practices. We closely collaborate with many different stakeholders from policy circles, schools (boards, principals, teachers), research, civic organizations, and the society at large. This allows us to design and validate relevant curriculum frameworks, to elaborate exemplary materials and to evaluate these in school practices. Our products and services support both policy makers and schools and teachers in making substantive curricular decisions and in elaborating these into relevant, inspiring and effective education.

ISBN: 978 90 329 2329 7

An In

trod

uctIo

n to

Edu

cAtIon

Al dEsIg

n rEsEArch

Editors: tjeerd Plomp &

nienke n

ieveenEditors:Tjeerd Plomp & Nienke Nieveen


SLO •Netherlands institute for curriculum development

Tjeerd Plomp & Nienke Nieveen (editors)

Proceedings of the seminar conducted at the East China Normal University, Shanghai (PR China), November 23-26, 2007

an introduction to educational design research2

Authors: JanvandenAkker BrendaBannan AnthonyE.Kelly NienkeNieveen TjeerdPlompEditors: TjeerdPlomp NienkeNieveen

Design: AxisMedia-ontwerpers,Enschede

Production: Netzodruk,Enschede

AN: 7.5115.183

ISBN: 9789032923297

Orderaddress: SLO P.O.Box2041 7500Enschede theNetherlands

www.slo.nl/organisatie/international/publications

Enschede,3rdprintMarch2010

Colophon

an introduction to educational design research 3

Contents

Preface 5

1. EducationalDesignResearch:anIntroduction 9 Tjeerd Plomp

2. CurriculumDesignResearch 37 Jan van den Akker

3. TheIntegrativeLearningDesignFramework:AnIllustratedExamplefrom theDomainofInstructionalTechnology 53 Brenda Bannan

4 WhenisDesignResearchAppropriate? 73 Anthony E. Kelly

5 FormativeEvaluationinEducationalDesignResearch 89 Nienke Nieveen

6 ReferencesandSourcesonEducationalDesignResearch 103 Tjeerd Plomp and Nienke Nieveen

Authorbiographies 125


PrefaceThisbookistheresultofaseminaron‘educationaldesignresearch’organizedfromNovem-ber23-26,2007,byProfZhuZhiting(DepartmentofEducationalTechnology)oftheCollegeofEducationalSciencesattheEastChinaNormalUniversityinShanghai(PRChina).Theprimarygoaloftheseminarwasto introduce a group postgraduate students and lectu-ring staff in China to educational design research as a research approach.Thesecondgoaloftheseminarwastoprepare,basedonthecontributionsofanumberinternationalexperts,proceedings of the seminarwritteninsuchawaythattheycanbeusedinpostgraduateseminarsoneducationaldesignresearchacrossChina.

About75peoplewithbackgroundsmainlyininstructionaltechnology,curriculumandinstructionaldesignparticipatedintheseminar.Mostofthemwereworkinginteachereducation,inschoolsasinstructionaltechnologistand/orindistanceeducation.Althoughparticipantshad(throughtheirstudies)alreadyknowledgeandsomeexperienceinin-structionalorcoursedesignandinresearchmethods,theywereeagertobeintroducedtodesignresearchasarelativelynewresearchapproachforaddressingcomplexproblemsineducationalpractice.

TheseminarstaffconsistedofProfsBrenda BannanandEamonn Kelly(bothGeorgeMasonUniversity,Fairfax,VA,USA)andProfJan van den Akker(UniversityofTwenteandNationalInstituteforCurriculumDevelopment[SLO],Enschede,TheNetherlands),andthetwoedi-torsofthisbookDrNienke Nieveen(NationalInstituteforCurriculumDevelopment[SLO],Enschede)andProfTjeerd Plomp(UniversityofTwente,Enschede,TheNetherlands).Ascanbeseenfromthetableofcontentofthisbook,theyarereflectingthebackgroundofthepar-ticipants,astheyrepresentedexperienceinconductingdesignresearchinthedomainsofcurriculumdevelopment,instructionaltechnologyandmathematicsandscienceeducation.ExpertswereconsciouslyinvitedfrombothEurope(TheNetherlands)aswellastheUSA,soastoascertainthatvariationinbackgroundandperspectiveondesignresearchwasrepre-sentedinconductingtheseminar.

Thechaptersinthisbookarebasedonthepresentationsandthesmallgroupdiscussionsduringthisseminar.Althoughthebookdoesnotprovidea‘howtodoguide’fordesigningandconductingdesignresearch,thechaptershavebeenwritteninsuchawaythattheyreflectboththeconceptualunderpinningandpracticalaspectsofthe‘what’and‘how’ofdoingdesignresearch(chaptersbyPlomp,KellyandNieveen),aswellasprovidethereaderaninsightinthespecificsofdoingdesignresearchinthedomainofcurriculum(chapterbyVandenAkker)andinstructionaltechnology(chapterbyBannan).Toassistthereadersinfindingtheirwayintheabundanceofliteratureondesignresearch,wehaveaddedachapterwithreferencesandsourcesoneducationaldesignresearch.This


bibliographyisfarfromcompleteandreflectsverymuchthebackgroundandthebiasesoftheeditorsofthisbook.Yetwetrustthatthischapterwillassisttheinterestedreaderingettingintroducedtothisexcitingandpromisingresearchapproach.

WewanttothankProfZhuZhitingfromtheEastChinaNormalUniversityfortakingtheinitiativeforthisseminar.Similarlywewanttothankourcolleaguesforcontributingtothisbook.

Butaboveall,weliketoexpressourhopethatthisbookwillstimulateandsupportmany(future)researcherstoengagethemselvesineducationaldesignresearch.

JanvandenAkkerDirector General SLO

TjeerdPlompandNienkeNieveenEditors


1. EducationalDesignResearch: anIntroduction Tjeerd Plomp

Introduction

Thepurposeofthischapteristoprovideanintroductiontoeducationaldesignresearchasaresearchapproachsuitabletoaddresscomplexproblemsineducationalpracticeforwhichnoclearguidelinesforsolutionsareavailable.Educationaldesignresearchisperceivedasthesystematicstudyofdesigning,developingandevaluatingeducationalinterventions,-suchasprograms,teaching-learningstrategiesandmaterials,productsandsystems-assolutionstosuchproblems,whichalsoaimsatadvancingourknowledgeaboutthecharacteristicsoftheseinterventionsandtheprocessestodesignanddevelopthem.

Theneedforaresearchapproachthataddressescomplexproblemsineducationalpracticehasbeenarguedbyresearchersinvarious‘corners’ofthedomainofeducationfromthelackofrelevanceofmucheducationalresearchforeducationalpractice.Forexample,theDesign-BasedResearchCollective(2003:5)arguesthateducationalresearchisoftendivorcedfromtheproblemsandissuesofeverydaypractice–asplitthatresultedinacredibilitygapandcreatesaneedfornewresearchapproachesthatspeakdirectlytoproblemsofpracticeandthatleadtothedevelopmentof‘usableknowledge’.Fromhisbackgroundinresearchinthedomainofcurriculum development and implementation,VandenAkker(1999:2)arguesthatmany‘traditional’researchapproachessuchasexperiments,surveys,correlationalanalyses,withtheiremphasisondescriptionhardlyprovideprescriptionsthatareusefulfordesignanddevelopmentproblemsineducation.Heclaimsthatanimportantreasonfordesignresearch1stemsfromthecomplexnatureoftheeducationalreformsworldwide.Ambitiousreformscannotbedevelopedatthedrawingtablesingovernmentoffices,butcallforsystematicresearchsupportingthedevelopmentandimplementationprocessesinavarietyofcontexts.Inhisreviewofthestateofeducationalresearchandmorespecificallyeducationaltechnologyresearch,Reeves(2006:57)concludesthatthereis“alegacyofill-conceivedandpoorlyconductedresearchthatresultsinnosignificantdifferencesor,atbest,inmodesteffectsizes”.Healsoarguesforthedomainofeducational technologythateducationaltechnologists,insteadofdoingmore(media)comparisonstudies,shouldundertaketypesofdesignresearch.Inotherwords,ReevesarguesthatinsteadofdoingmorestudiescomparingwhetherinacertaincontextmethodAisbetterthanmethodB,itisbetterto

1) whichhecalls‘developmentresearch’inhis1999publication


undertakedesignresearchaimedatdevelopinganoptimalsolutionforaproblemincontext.Inthefieldoflearning sciences,thebeliefthatcontextmattersleadtotheconclusionthatresearchparadigmsthatsimplyexamineslearningprocessesasisolatedvariableswithinlaboratorysettingswillnecessarilyleadtoanincompleteunderstandingoftheirrelevanceinmorenaturalisticsettings(Barab&Squire,2004;withreferencetoBrown,1992).Inthisfield,design-basedresearchwasintroducedwiththeexpectationthatresearcherswouldsystematicallyadjustvariousaspectsofthedesignedcontextsothateachadjustmentservedasatypeofexperimentationthatallowedtheresearcherstotestandgeneratetheoryinnaturalisticcontexts(Barab&Squire,2004:3).

Thesesourcesillustratetheneedfordesignresearchasanalternativeresearchapproach.Beforeelaboratingondesignresearchthispaperwillfirstdiscussmoregenerallypossiblefunctionsofresearchandhowresearchfunctionsarerelatedtoresearchapproaches.Thendesignresearchwillbedefinedandcharacterizedfromvariousperspectives,suchasthetypeofknowledgethedesignresearchersaimfor,thetypeofresearchquestionsthatcanbeaddressed,andtheoutputsofdesignresearch.Thiswillbefollowedbyasectioninwhichdifferentapproachestodesignresearchareintroducedandsectionsdiscussinghowdesignresearchcanorshouldbeconducted,withamorein-depthdiscussionofformativeevaluationasthemostprominentresearchactivityindesignresearch.Conductingdesignresearchputsresearchersinasituationinwhichtheyhavetofaceanumberofdilemmas.Thesewillbediscussedbeforeendingthechapterwithafewconcludingremarks.

Afinalnoteonterminology,followingVandenAkkeretal.(2006:4)weusedesign researchasacommonlabelfora‘family’ofrelatedresearchapproacheswhomayvarysomewhatingoalsandcharacteristics–examplesaredesignstudies,designexperiments,design-basedresearch,developmentalresearch,formativeresearch,engineeringresearch.

Research functions – research approaches

Beforeelaboratingonthemeaningofdesignresearch,itisimportanttopositiondesignresearchasaresearchapproachnexttootherresearchapproaches,whichisthepurposeofthissection.

Thekeyfocusinallscientificresearchisthesearchfor‘understanding’orfor‘knowing’withtheaimofcontributingtothebodyofknowledgeoratheoryinthedomainofresearch.Otherbroad aimsofdoingeducationalresearcharetoprovideinsightsandcontributionsforimprovingpractice,andtoinformdecisionmakingandpolicydevelopmentinthedomainofeducation.


ResearchfunctionsIngeneral,wecandistinguishvariousresearch functions,eachreflectingcertaintypesofresearchquestions.Examplesofresearchfunctions(withexemplaryresearchquestionsfittingthefunction)are:1. to describe: e.g.whatistheachievementofChinesegrade8pupilsinmathematics;

whatbarriersdostudentsexperienceinthelearningofmathematicalmodelling2. to compare:e.g.whatarethedifferencesandsimilaritiesbetweentheChineseandthe

Netherlandscurriculumforprimaryeducation;whatistheachievementinmathematicsofChinesegrade8pupilsascomparedtothatincertainothercountries

3. to evaluate:e.g.howwelldoesaprogramfunctionintermsofcompetencesofgraduates;whatarethestrengthsandweaknessesofacertainapproach;etc

4. to explain or to predict:e.g.whatarethecausesofpoorperformanceinmathematics(i.e.insearchofa‘theory’predictingaphenomenonwhencertainconditionsorcharacteristicsaremet)

5. to design and develop:e.g.whatarethecharacteristicsofaneffectiveteachingandlearningstrategyaimedatacquiringcertainlearningoutcomes;howcanweimprovethemotivationoflearners.

Inmanyresearchprojectstheresearchquestionsaresuchthatinfactvariousresearchfunctionsdoapply.Forexample,iftheresearchquestionpertainstocomparingthemathematicsachievementofChinesegrade8pupilsascomparedtothatincertainothercountries,thenaspartofcomparingtheresearcherswillevaluatetheachievementofgrade8pupilsineachofthecountriesinvolved.Or,asanotherexample,ifonewantstodesign and developateaching-learningstrategyforacquiringthecompetencyofmathematicalmodelling(ingrade11&12),thenresearchersmayfirstwanttounderstandandcarefullydescribewhatbarriersstudentsexperiencewithmathematicalmodelling,whilstalsotheevaluationfunctionisimportantindeterminingwhethertheteaching-learningstrategythathasbeendevelopediseffective.Bothexamplesillustratethatusuallyaresearchprojecthasaprimaryresearchfunction,butthatotherresearchfunctionsarebeingappliedto‘serve’theprimaryresearchfunction.

Atthelevelofaresearch project,startingfromaresearchproblemorquestion,wearesupposedtohavethefollowingsequence:Research question => (primary) research function =>choice of research approach.Inthischapterwefocusonresearchwhichhasdesign and developastheprimaryresearchfunction.

ResearchapproachesMosttextbooksonresearchmethodologypresentanddiscussanumberofresearch approachesorstrategies(seee.g.Denscombe,2007).Usuallyeachresearchapproachcanbe


usedforrealizingmorethanoneresearchfunction.Withoutgoingintodetailhere,examplesofresearchapproachesandtheirpossibleresearchfunctionsare:• survey: todescribe,tocompare,toevaluate• case studies:todescribe,tocompare,toexplain• experiments:toexplain,tocompare• action research:todesign/developasolutiontoapracticalproblem• ethnography:todescribe,toexplain• correlational research:todescribe,tocompare• evaluation research:todeterminetheeffectivenessofaprogramTextbooksonresearchmethodologyusuallydonotpresentanddiscussdesignresearch:• design research: todesign/developanintervention(suchasprograms,teaching-learning

strategiesandmaterials,productsandsystems)withtheaimtosolveacomplexeducationalproblemandtoadvanceourknowledgeaboutthecharacteristicsoftheseinterventionsandtheprocessestodesignanddevelopthem.

Inlinewiththeremarkthatmorethanoneresearchfunctionmayhavetobeappliedtoaddressaresearchquestion,itshouldbenoticedthatinaresearchprojectmorethanoneresearchapproachmayhavetobeapplied.Forexample,ifthereisaneedtocomparehowwellChinesegrade8pupilsperforminmathematicsascomparedtoanumberofothercountries,theprimaryresearchfunctionistocompare,leadinginthiscasetoasurveyasthebestresearchapproach.However,aspartofthedevelopmentofavalidandreliablemathematicstest,theresearchersmaydocorrelationalresearchtodeterminewhetherthetestbeingdevelopedisvalid,i.e.correlateswithothermeasuresofmathematicsachievement.

Asafinalremark,itisimportantthatdesignresearchers,likeallresearchers,keepinmindthatalsofortheirresearchtheguidingprinciplesforscientificresearch(Shavelson&Towne,2002)apply,viz:• Posesignificantquestionsthatcanbeinvestigated• Linkresearchtorelevanttheory• Usemethodsthatpermitdirectinvestigationofthequestion• Provideacoherentandexplicitchainofreasoning• Replicateandgeneralizeacrossstudies• Discloseresearchtoencourageprofessionalscrutinyandcritique


What is design research?

Asstatededucationaldesignresearchisthe systematic study of designing, developing and evaluating educational interventions (such as programs, teaching-learning strategies and materials, products and systems) as solutions for complex problems in educational practice, which also aims at advancing our knowledge about the characteristics of these interventions and the processes of designing and developing them. Thetwofoldyieldofdesignresearch,viz.researchbasedinterventionsaswellasknowledgeaboutthem,canalsofoundindefinitionsofdesignresearchbyotherauthors.Forexample,thebroaddefinitionofBarabandSquire(2004)alsoencompassesmostvariationsofeducationaldesignresearch:“aseriesofapproaches,withtheintentofproducingnewtheories,artefacts,andpracticesthataccountforandpotentiallyimpactlearningandteachinginnaturalisticsetting.

Byitsnature,designresearchisrelevantforeducationalpractice(andthereforealsoforeducationalpolicy)asitaimstodevelopresearch-basedsolutionsforcomplexproblemsineducationalpractice.Startingpointfordesignresearchareeducationalproblemsforwhichnooronlyafewvalidatedprinciples(‘howtodo’guidelinesorheuristics)areavailabletostructureandsupportthedesignanddevelopmentactivities2.Informedbypriorresearchandreviewofrelevantliterature,researchersincollaborationwithpractitionersdesignanddevelopworkableandeffectiveinterventionsbycarefullystudyingsuccessiveversions(orprototypes)ofinterventionsintheirtargetcontexts,andindoingsotheyreflectontheirresearchprocesswiththepurposetoproducedesignprinciples.Manyexamplesoftheneedforinnovativeinterventionscanbegivenatsystemlevelandinstitutionallevel.Atsystemlevel,forexample,onemaywanttodevelopasystemfore-learningtoserveaspecifictargetgroupofstudentsinhighereducation,andatthelevelofschoolorclassroomonemaywant,forexample,toaddressthequestionofwhatareeffectivemethodsforcollaborativelearning.SeealsoGustafson&Branch(2002)whodevelopedataxonomyofinstructionaldevelopmentmodelsbasedonaselectedcharacteristics;theydistinguishbetweenmodelswithaclassroomorientation,productorientationandsystemorientation.

Theresearchprocessindesignresearchencompasseseducationaldesignprocesses.Itis–likeallsystematiceducationalandinstructionaldesignprocesses-thereforecyclicalincharacter:analysis,design,evaluationandrevisionactivitiesareiterateduntilasatisfyingbalancebetweenideals(‘theintended’)andrealizationhasbeenachieved.

2) seealsothechapterofKellyinthisbookwherehediscusseswhendesignresearchisappropriate.


Thisprocesscanbeillustratedinvariousways.Justafewexamplesarepresentedheretoshowhowdifferentauthorshavevisualizedtheresearchprocess.

Reeves(2006)depictsthedesignresearchapproachasfollows:

Figure 1: Refinement of Problems, Solutions, Methods, and Design Principles (Reeves, 2000, 2006)

McKenney(2001)illustratesinherstudythiscyclicalprocessasfollows:

Figure 2: Display of the CASCADE-SEA study (McKenney, 2001)

Identify and analyse problems by

researchers & practitioners in

collaboration

Development of prototype solutions: informed by state-

of-art theory, existing design principals & technology innova-

tions

Iterative cycles of testing & refinement

of solutions in practice

Reflection to produce ‘design principles’ &

enhance solution implementation in

practice

160

140

120

100

80

60

40

20

0

num

ber o

f par

ticip

ants

needs & contextanalysis

design, development & formative evaluation semi-summativeevaluation

literaturereview &concept

validation

sitevisits

prototype1

prototype2

cycle width is proportional to timeschale: = circa 6 months

prototype3

prototype4

finalevaluation

query


The‘query’asthelastphaseinMcKenney’sdisplaycanbeinterpretedasthereflectionboxinthemodelofReeves(Figure1).AnotherexampleistheIntegrativeLearningDesignFrameworkthatBannan-Ritlandpresentsinchapter5ofthisbook(seealsoBannan-Ritland,2003).

Authorsmayvaryinthedetailsofhowtheypicturedesignresearch,buttheyallagreethatdesignresearchcomprisesofanumberofstagesorphases:• preliminary research:needsandcontextanalysis,reviewofliterature,developmentofa

conceptualortheoreticalframeworkforthestudy• prototyping phase:iterativedesignphase3consistingofiterations,eachbeingamicro-

cycleofresearch4withformativeevaluationasthemostimportantresearchactivityaimedatimprovingandrefiningtheintervention

• assessment phase:(semi-)summativeevaluationtoconcludewhetherthesolutionorinterventionmeetsthepre-determinedspecifications.Asalsothisphaseoftenresultsinrecommendationsforimprovementoftheintervention,wecallthisphasesemi-summative.

Throughoutalltheseactivitiestheresearcherorresearchgroupwilldosystematic reflection and documentationtoproducethetheoriesordesignprinciples(aconcepttakenfromVandenAkker,1999–seealsochapter2)asthescientificyieldfromtheresearch.Onemaystatethatthissystematicreflectionanddocumentationmakesthatsystematicdesignanddevelopmentofaninterventionbecomesdesignresearch.

Authorsaboutdesignresearchalsoagreeanumberofcharacteristicsofthistypeofresearch.ThesearesummarizedbyVandenAkkeretal.(2006:5):• Interventionist:theresearchaimsatdesigninganinterventioninarealworldsetting;• Iterative:theresearchincorporatescyclesofanalysis,designanddevelopment,

evaluation,andrevision;• Involvement of practitioners:activeparticipationofpractitionersinthevariousstages

andactivitiesoftheresearch• Process oriented:thefocusisonunderstandingandimprovinginterventions(ablack

boxmodelofinput–outputmeasurementisavoided);• Utility oriented:themeritofadesignismeasured,inpartbyitspracticalityforusersin

realcontexts;and• Theory oriented:thedesignis(atleastpartly)basedonaconceptualframeworkand

upontheoreticalpropositions,whilstthesystematicevaluationofconsecutiveprototypesoftheinterventioncontributestotheorybuilding.

3) itispossiblethatthedesign/developmentcomponentinasucharesearchprojectwillnotbeginfromscratchbutwiththeevaluationofanexistinginterventionwiththeaimofidentifyingtheneedforimprovement,whichthenisfollowedbyre-designandanumberofdesigncycles.

4) termtakenfromBannan-Ritland,chapter5


ThefeaturesandcharacteristicsofdesignresearcharenicelycapturedbyWademan(2005)inwhathecallstheGenericDesignResearchModel(Figure3).Hismodelclearlyillustratesthatthe‘successiveapproximationofpracticalproducts’(whatwecall‘interventions’)isgoinghandinhandwiththe‘successiveapproximationoftheory’(whichhealsocalls‘designprinciples’).

Figure 3: Generic Design Research Model (Wademan, 2005)

Itisimportanttonotethatdesignresearchfollowsaholisticapproach,anddoesnotemphasizeisolatedvariables.VandenAkkeretal.(2006:5)pointtoitthatyetdesignresearchersdofocusonspecificobjectsandprocesses(interventions)inspecificcontexts,buttheytrytostudythoseasintegralandmeaningfulphenomena.Thiscontextboundnatureofmuchdesignresearchalsoexplainswhyitusuallydoesnotstrivetowardscontext-freegeneralizations.Ifanefforttogeneralizingismade,thenitisananalyticalgeneralization(incontrasttostatisticalgeneralizationwheretheresearchermaygeneralizefromsampletopopulation).(Seealsothesection‘outputsofdesignresearch’)

Practitioner and User Participation

Researchers

Other Sources

Collaboratives

Practitioners

ConsultExperts &

Practitioners

FocusedLiterature

Review

AnalyzePromisingExamples

AnalyzePracticalContext

Tentative Product

Approaches

Tentative Design

Principles

Reflection

FormativeEvaluation

PracticalProduct/Results

Contributionto

Theory

Refinement of Problem, Solution and Method

Refinement of Design Theory

Preliminary investigation of Problem, Context,

& Approaches

Problem in Context

Phases

Prototyping & Assessment of Preliminary Products & Theories

Identification of Tentative Products & Design Principes

Problem Identification Tentative Products& Theories

Problem Resolution & Advancing Theory

Successive Approximation of Theory

Successive Approximation of Product

Redesign & Refinementof Products & Theories


A closer look at design research

Asstatedkeycharacteristicsofdesignresearcharethatitisresearchfocusedondesigninginterventionsintherealcontextofeducationortraining(interventionistcharacteristic)combinedwitheffortstounderstandandimproveinterventions(process orientation),utilizingstateofthearttheorieswhilstthefieldtestingandtheevaluationoftheconsecutiveprototypesshouldcontributetotheorybuilding(theory orientation).Inthissectionwewillhavealookatwhatitmeansthatresearchsupportseducationaldesignprocesses,andreverselythateducationaldesignprocessessupportresearch.Thisisfollowedbyabriefdiscussionofthetypeofresearchquestionindesignresearch.Possibleoutputsofdesignresearchwillbediscussedinthenextsection.

Aswealreadystated,oneoftheaimsofdesignresearchisdesigninganddevelopinganinterventionasan(innovative)solutiontoacomplexproblem,andthereforethestartingpointfordesignresearchareeducationalproblemsforwhichnooronlyafewvalidatedprinciples(‘howtodo’guidelines)areavailabletostructureandsupportthedesignanddevelopmentactivities.Ontheotherhand,designresearchisresearchandthereforetheappropriateyieldfordesignresearch(apartfromausableandeffectiveintervention)isempiricallyfoundedtheory,i.e.thechallengefordesignresearchistocaptureandmakeexplicittheimplicitdecisionsassociatedwithadesignprocess,andtotransformthemintoguidelinesforaddressingeducationalproblems(seeEdelson,2006;101;alsoBarab&Squire(2003),andmanyotherauthors).Thisaspectreferstothetheory orientation,mentionedaboveasoneofthecharacteristicsofdesignresearch.VandenAkker(1999,2006,alsochapter2),Reeves(2006;seefigure1)andWademan(2005;seefigure3)usetheconceptof‘designprinciples’whentheyrefertothetheoreticalyieldsofdesignresearch,whereothersspeakofnewtheories(e.g.Barab&Squire,2003;Edelson,2006).

However,itisnotself-evidenthowthedesignofinterventionsmaycontributetotheorybuilding.WithreferencetothegenericmodelofWademen(Figure3)andtheexemplaryschemesofReeves(2006)inFigure1andMcKenney(2001)inFigure2,onemaystatethattheresearcher(orbetter:thecollectiveofresearchersandpractitioners)-basedonanalysisoftheproblemincontext,andutilizingrelevant,state-of-the-arttheories–designsanddevelops(inaniterativeway)theinterventionwiththeaimthatafteranumberofcyclestheintendedoutcomesarerealized,i.e.asatisfyingsolutiontotheproblemidentified.Eachiterationorcycleisamicro-cycleofresearch,i.e.astepintheprocessofdoingresearchandwillincludesystematic reflectiononthetheoreticalaspectsordesignprinciplesinrelationshiptothestatusoftheintervention,resultingintheendindesignprinciplesortheoreticalstatements.


Inotherwords,intheendtheresearcher(orresearchgroup)willconcludeabouthisintervention:

Given my context, if I do <intervention (theory based) > then I expect <intended outcomes>.

Thiscanbedisplayedschematicallyas:

Twopointsareimportantinthisscheme:• theoutcomesoftheinterventionareindicatedasY1,Y2,…,Yn,becauseoftenan

interventionisdesignedtorealizemultipleoutcomes(e.g.betterachievement,improvedstudentattitude,increasedteachersatisfaction,etc).

• theinterventionispresentedas‘inputprocess’,becausedesigningaprocess(e.g.learningenvironment)hastotakeintoaccountalsotheinputsnecessarytomaketheprocessfunction(e.g.certaininstructionallearningmaterials,teacherdevelopment).

Sointheend,theresearchgrouphasnotonlyatitsdisposaltheinterventionresultinginthedesiredoutcomes,butalsobasedonasystematicreflectionandanalysisofthedatacollectedduringthiscyclicalprocessanunderstandingofthe‘howandwhy’ofthefunctioningoftheinterventionintheparticularcontextwithinitwasdeveloped.Thedesignresearcherwillsummarizethisunderstandingofthe‘howandwhy’oftheinterventioninoneormore‘designprinciples’ifwewouldusetheterminologyofVandenAkker(1999,2006)andReeves(2000,2006).Asotherauthors,e.g.Barak&Squire(2004)andEdelson(2006),useof‘theory’astheyieldofdesignresearch,onemayalsospeakof‘interventiontheory’or‘designtheory’(Wademan,2005;Figure3)asasecondgenerictermtorefertotheknowledgegeneratedfromthisresearchendeavour(seebelowforspecificexamples).

Indesignresearch,interventionsaredevelopedinacyclical processofsuccessiveprototypes:

Intervention XInput Process

OutcomesY1, Y2, ...., Yn

Intervention XInput Process

OutcomesY1, Y2, ...., Yn

design principles or intervention theory

Akeyideaisthatwheninacertaincycletheprototypeoftheinterventiondoesnotresultinthedesiredoutcomes,onemayconcludethatthedesignprinciples(orinterventiontheory)appliedarenot(yet)effective(or,inotherwords,thattheinterventiontheory‘fails’).Thishastoresultinare-designorrefinementoftheintervention,whichgoeshand-in-handwiththerefinementoftheinterventiontheoryordesigntheory.Whenafteranumberofiterationstheresearcher(orresearchgroup)concludesthatbasedontheanalysisoftheevaluationdatathe‘realizedoutcomes’arecloseenoughtothe‘intendedoutcomes’thenhecanbesatisfied:thedesignprinciplesappeartobeeffective.Or,inotherwords,theresearcher(orresearchgroup)hasdevelopeda‘local’(intervention)theory(i.e.forthecontextinwhichhe/sheworks):in context Z the intervention X (with certain characteristics) leads to outcomes Y1, Y2, …, Yn.

Twoexamplesaregiventoillustratethis–ratherabstract–phrasingoftheyieldofdesignresearch.TheDesign-BasedLearningResearchCollective(2003:5)statethat“thedesignofinnovationsenablesustocreateleaningconditionsthatlearningtheorysuggestsareproductive,butthatarenotcommonlypracticedorarenotwellunderstood”–inotherwordsincludedintheinnovationsisknowledgeabouthowtocreateconditionsforlearning.Thesecondexampleistakenfromscienceeducation.Lijnse(1995:192)arguesthatdesignresearch(hecallsitdevelopmentalresearch)is“acyclicprocessoftheoreticalreflections,conceptualanalysis,small-scalecurriculumdevelopment,andclassroomresearchoftheinteractionofteaching-learningprocesses.Thefinal,empiricallybaseddescriptionandjustificationoftheseinterrelatedprocessesandactivitiesconstituteswhatwecallapossible“didacticalstructure”forthetopicunderconsideration.”Inotherwords,thelocaltheoryconsistsofadidacticalstructureforteaching-learningprocessesforacertaintopic.

Theresearchquestionindesign/developmentresearchBynowitisclearthatdesigninganddevelopinganinterventionisinitselfnotyetdesignresearch.Butonemayconductadesign/developmentprojectasaresearchprojectbyemployingrigorouslysocialscienceresearchmethodology.Astheresearcherisstrivingtofinddesignprinciples(oranintervention theory)thatarevalidinacertaincontext,theresearchquestioncanbephrasedas:

what are the characteristics of an <intervention X> for the purpose/outcome Y (Y1, Y2, …, Yn) in context Z

Design/develop Implement/try-0ut

Evaluate (formative)



Examplesofresearchquestionsare:(i) whatarethecharacteristicsofaneffectivein-serviceprogrammeformathematics

teachersthroughwhichtheydeveloptheabilitytoapplystudent-centredpedagogicalmethods,and

(ii) whatarethecharacteristicsofanin-servicearrangementthatfacilitatestheimplementationofMBL5-supportedlessonactivitiesinphysicseducation(Tecle,2003)?

Obviously,notallresearchersareusingthistypeofphrasing,butthewordingofthemainresearchquestionindesignresearchalwaysimpliesasearchforcharacteristics.Anexampleis:Whatisanadequatelearningandteachingstrategyforgeneticsinuppersecondarybiologyeducationinordertocopewiththemaindifficultiesinlearningandteachinggenetics,andtopromotetheacquisitionofameaningfulandcoherentunderstandingofhereditaryphenomena?(Knippels,2002)

The outputs of design research

Wealreadyconcludedthatdesignresearchresultsininterventions(programs,products,processes)andindesignprinciplesorinterventiontheory.Athirdoutputofdesignresearchisprofessionaldevelopmentoftheparticipantsinvolvedintheresearch.Eachoftheseoutputsisbrieflydiscussed.

OndesignprinciplesorinterventiontheoryDesignresearchaimsatproducingknowledgeaboutwhetherandwhyaninterventionworksinacertaincontext.Intheprevioussectionthistypeofoutputhasbeencalleddesignprinciplesorinterventiontheory.Otherauthorsusetermslikedomainspecifictheories(Gravemeijer&Cobb,2006),designtheory(Wademan,2005;Figure3),heuristicsorjustlessonslearned(seeVandenAkkeretal.2006).Wewillusethetermdesign principlesintheremainingofthispaper.

Designprinciplesareheuristic statementsforwhichVandenAkker(1999)developedthefollowingformat:

“If you want to design intervention X for the purpose/function Y in context Z, then you are best advised to give that intervention the characteristics A, B, and C [substantive emphasis], and to do that via procedures K, L, and M [procedural

emphasis], because of arguments P, Q, and R.” (Van den Akker, 1999)

5) MBL=MicrocomputerBasedLaboratory.

Theheuristicprinciplesaremeanttosupportdesignersintheirtasks,butcannotguaranteesuccess-theyareintendedtoassist(inotherprojects)inselectingandapplyingthemostappropriate(substantiveandprocedural)knowledgeforspecificdesignanddevelopmenttasks.Substantiveknowledgeisknowledgeaboutessentialcharacteristicsofaninterventionandcanbeextracted(partly)fromaresultinginterventionitself.Proceduralknowledgereferstothesetofdesignactivitiesthatareconsideredmostpromisingindevelopinganeffectiveandworkableintervention.Asknowledgeisincorporatedininterventions,itisprofitablefordesignresearchersintheearlystageoftheirresearchtosearchforalreadyavailableinterventionsthatcanbeconsideredusefulexamplesorsourcesofinspirationfortheproblematstake.Carefulanalysisofsuchexamplesincombinationwithreviewingrelevantliterature)willgenerateideasforthenewdesigntask.Thevalueofknowledgeresultingfromadesignresearchprojectwillstronglyincreasewhenitisjustifiedbytheoreticalarguments,well-articulatedinprovidingdirections,andconvincinglybacked-upwithempiricalevidenceabouttheimpactofthoseprinciples.Itisforthisreasonthatauthors(e.g.VandenAkker1999,2006;Reeves,2000,2006)statethatthefinalstageofeachdesignresearchprojectshouldconsistofsystematicreflectionanddocumentationtoproducedesignprinciples.

GeneralizabilityindesignresearchHeuristicdesignprincipleswillbeadditionallypowerfuliftheyhavebeenvalidatedinthesuccessfuldesignofmoresimilarinterventionsinvariouscontexts.Chancesforsuchknowledgegrowthwillincreasewhendesignresearchisconductedintheframeworkofresearchprograms,becausethenprojectscanbuildupononeanother.Herewetouchonthequestiontowhatextentdesignprinciplescanbegeneralizedfromonecontexttoothers.ItisinthiscontextthatEdelson(2006)statesthatdesignresearchshouldresultingeneralizabletheory.Indesignresearch,likeincasestudiesandexperimentalstudies,thefindingscannotbegeneralizedtoalargeruniverse–thereisnostatisticalgeneralizationfromsampletopopulation,likecanbethecaseinsurveyresearch.Yin(2003)pointstoitthatincasestudiesandexperimentalstudies,theinvestigatorisstrivingtogeneralizeaparticularsetofresultstoabroadertheory.Thisisalsothecaseindesignresearch,theresearchershouldstrivetogeneralize‘designprinciples’tosomebroadertheory.Yin(2003:37)pointstoitthatgeneralizationisnotautomatic.Designprinciplesmustbetestedthroughreplicationsofthefindingsinasecond,thirdormorecasesinvariouscontextswiththepurposethatthesameresultsshouldoccur.Oncesuchreplicationshavebeenmade,theresultsmightbeacceptedforamuchlargernumberofsimilarcontexts,eventhoughfurtherreplicationshavenotbeenperformed.Thisreplication logicisthe



samethatunderliestheuseofexperimentsandallowsexperimentalscientiststogeneralizefromoneexperiment‘toanother’:Yin(2003)callsthisanalyticalgeneralizability.Butawarningshouldbephrasedhere.Wheredesignprinciplesmayhavebeensupportedbyanumberofreplications,andanewcontextmaybesimilartotheonesfromwhichdesignprincipleshaveemerged,yeteachcontexthasuniquecharacteristicsthatjustifiesthatthedesignprinciplesshouldbeusedas‘heuristic’statements:theyprovideguidanceanddirection,butdonotgive‘certainties’.ItisinthiscontextthatReeves(2006)citesLeeCronbachoneofthemostinfluentialresearchersofthe20thcentury:“When we give proper weight to local conditions, any generalization is a working hypothesis, not a conclusion.”(Cronbach,1975:125)

OninterventionsDesignresearchbyitscharacteraimstobepracticallyrelevant.Itisinitiatedtodesignanddevelopinnovativeinterventionstomeetaneedfeltinacomplex,practicalsituationforwhichnoready-madesolutionsorguidelinesareavailable.Thereforedesignresearchersaimatdevelopinginterventions(suchasprograms,teaching-learningstrategiesandmaterials,productsandsystems)thatcanbeusedinpracticeandareempiricallyunderpinnedsolutionstotheproblemsidentified.

OnprofessionaldevelopmentOneofthefeaturesofdesignresearchisthecollaborationofresearchersandpractitioners.Thiscollaborationincreasesthechancethattheinterventionwillindeedbecomepracticalandrelevantfortheeducationalcontextwhichincreasestheprobabilityforasuccessfulimplementation.Buttheparticipationofpractitionersshouldalsobeseenasanimportantformofprofessionaldevelopment.Anextraspin-offmaybethatpractitionerswilldevelopanawarenessofhowresearchmaycontributetoimprovingtheirprofessionalcontext.

Design research differentiation

Designresearchisconductedthroughanumberofcyclesofdesignanddevelopmentresultingintheinitialimplementationoftheinterventioninalimitednumberofcontexts.Asstatedabove,designresearchhasusuallyanumberofstagesorphases(seealsoFigures1,2and3):• needsandcontentanalysis• prototypingphase(iterativecyclesofdesignandformativeevaluation)• assessmentphase(semi-summativeevaluation)

Nieveenetal.(2006)suggestthatdesignresearchthathasresultedinavalidatedandeffectiveintervention(asasolutionfortheproblemunderstudy),andindesignprinciplescanbefollowedbyeffect studies(notnecessarilypartofthesameresearchproject)withan


emphasisonupscalingtheinterventiontoawidercontext,andindoingsoaimingatdesignprinciplestestedinawiderdomain.Effectstudiesmayrangefromsmall-scalelearningexperimentstolarge-scalecomparativetestingofimpact(e.g.viarandomizedcontrolledtrials).

Afurtherdifferentiationindesignstudiesispossibleonthebasisofvariationsingoalsofdesignresearchvizvalidationstudiesversusdevelopmentstudies(seeVandenAkker,Gravemeijeretal.,2006;chapters5and10).

Validation studieshaveafocusondesigninglearningenvironmentsortrajectorieswiththepurposetodevelopandvalidatetheoriesabouttheprocessoflearningandhowlearningenvironmentscanbedesigned.Validationstudiesaimatadvancinglearningandinstructiontheories,suchas(Gravemeijer&Cobb,2006):• micro-theories:atthelevelofinstructionalactivities• localinstructiontheories:atthelevelofinstructionalsequence;• domain-specificinstructiontheories:atthelevelofpedagogicalcontentknowledge.Invalidationstudies,researchersdonotworkincontrolled(laboratoryorsimulated)settings,buttheychoosethenaturalsettingofclassroomas‘testbeds’(althoughtheytendtoworkwithabove-averagenumberofteachingstaff).Usually,thestagesinvalidationstudiesare(Gravemeijer&Cobb,2006):• environment preparation:elaboratingapreliminaryinstructionaldesignbasedonan

interpretativeframework;• classroom experiment:testingandimprovingtheinstructionaldesignorlocal

instructionaltheoryanddevelopinganunderstandingofhowitworks;• retrospective analysis:studyingtheentiredatasettocontributetothedevelopmentofa

localinstructionaltheoryand(improvementof)theinterpretativeframework.DiSessaandCobb(2004:83)warnthat“designresearchwillnotbeparticularlyprogressiveinthelongrunifthemotivationforconductingexperimentsisrestrictedtothatofproducingdomainspecificinstructionaltheories”.Butthepracticalcontributionliesindevelopingandimplementingspecificlearningtrajectoriesthatwereimplementedtotestthetheoreticalbasisofthedesign.(Nieveenetal,2006:153)

Development studiesaimtowardsdesignprinciplesfordevelopinginnovativeinterventionsthatarerelevantforeducationalpractice.“Developmentstudiesintegratestate-of-the-artknowledgefrompriorresearchinthedesignprocessandfine-tuneeducationalinnovationsbasedonpilotinginthefield.…Byunpackingthedesignprocess,designprinciplesthatcaninformfuturedevelopmentandimplementationdecisionsarederived.”(Nieveenetal.,2006:153).Twomaintypesofdesignprinciplescanbedistinguished(VandenAkker,1999):1. proceduraldesignprinciples:characteristicsofthedesignapproach;2. substantivedesignprinciples:characteristicsofthedesign(=intervention)itself.


Figurethreesummarizesthecharacteristicsofaresearchcycleconsistingofdesignstudiesandeffectstudies(asdevelopedbyNieveenetal.;2006:155):

Design researchEffectiveness research

Validation studies Development studies

Design aim To elaborate and validate theories

To solve educational problems

-

Quality focus of design

Theoretical quality of design

Practicality of intervention

Effectiveness of intervention

Knowledge claim/ scientific output

Domain-specific instruction theories

Broadly applicable design principles

Evidence of impact of intervention

Methodological emphasis

Iterative design with small scale testing in research setting

Iterative development with formative evaluation in various user settings

Large scale, comparative field experiments

Practical contribution Specific learning trajectories for a specific classroom

Implemented interventions in several contexts/classrooms

Evidence-basedChange at large scale

Figure 4: Educational engineering research cycle (from Nieveen et al., 2006)

Itisimportanttonotethatthisdistinctionbetweenvalidationanddevelopmentstudiesisconceptuallyimportant,butthatinpracticemanyresearchprojecthaveaimsthatareacombinationofsolvingproblemsineducationalpracticeandelaboratingandvalidatingtheories(designprinciples).

Afurtherdifferentiationofdesignresearchisconceivable.Forexample,onecanimaginethatthedisseminationandimplementationofaparticularprogramissupportedbydesignresearch–theresultinginterventionisthesuccessfullydisseminatedandimplementedprogram,whilstthesystematicreflectionanddocumentationoftheprocessleadstoasetofproceduresandconditionsforsuccessfuldisseminationandimplementation(thedesignprinciples).

Asafinalnote,thedifferentiationbetweentypesofdesignresearch,suchasvalidationstudiesversusdevelopmentstudies,servesmainlyconceptualpurposes.Inpractice,designresearchersmaycombinethetwoorientationsintheirresearch.Forexample,startingfromacomplexandpersistentproblemine.g.scienceeducation,theresearchgroupmaydecidetoapplythedesignprinciples(localtheories)resultingfromotherstudiesintheirresearch.Indoingsotheyarenotonlydevelopinganintervention,butatthesametimeexploringthevalidityofdesignprinciples(theory)developedinanothercontextfortheirownproblemcontext.


Howisdesignresearchconducted?Designresearchisconductediterativelyasacollaborationofresearchersandpractitionersinareal-worldsetting.Onlythenthetwoprincipaloutputs(designprinciplesandempiricallyunderpinnedinnovativeinterventions)canberealized.Doingresearchinsuchasettingischallenginganddemandsacarefulresearchdesign.Itisthereforeimportanttoreflectnotonlyonthecyclical,iterativecharacterofthesystematicdesignoftheintervention,butalso–becauseitisresearch-tomakeexplicitthetenetsthatformthefoundationofthistypeofresearch(McKenneyetal.,2006)

McKenneyetal.(2006:77)definethreetenetstoshapedesignresearchforthecurriculumdomain(butthetenetsalsoapplytootherdomains):• Rigor–fordesignresearchtobeabletoresultinvalidandreliabledesignprinciples,the

researchhastomeetrigorousstandardsandapplytheguidingprinciplesforscientificresearchasmentionedbyShavelson&Towne(2002;mentionedabove).Muchliteratureisavailabletoguideresearchinnaturalsettingsthatofferssupporttoissueslikeinternalandexternalvalidity,reliabilityandutililizationoftheresearch.

• Relevance:Designresearchaimstoberelevantforeducationalpractice(andpolicy).Anecessaryconditionforthisisthattheresearchgroupmusthaveagoodworkingknowledgeofthetargetsettingandbeinformedbyresearchanddevelopmentsactivitiestakingplaceinnaturalsettings(ortestbeds).

• Collaboration:fordesignresearchtoberelevantforeducationalpractice,thedesignanddevelopmentactivitiesmustbeconductedincollaborationwithandnotjustforprofessionalsfromeducationalpractice.

Asexplainedinthebeginningofthischapter,designresearchiscyclicalandeachiterationorcyclecontributestosharpeningtheaimsandtobringingtheinterventionsclosertothedesireddesignoutcomesandresearchoutputs.AsisillustratedinFigures1-3,designresearchusuallygoesthroughseveralstageswhichNieveenetal.(2006:154)phraseasfollows(seealsop.15):• preliminary research:thoroughcontextandproblemanalysisalongwiththe

developmentofaconceptualframeworkbasedonliteraturereview;• prototyping stage:settingoutdesignguidelines,optimizingprototypesofthe

interventionthroughcyclesofdesign,formativeevaluation,andrevision–itisimportanttonotethateachcycleinthestudyisapieceofresearchinitself(i.e.havingitsresearchorevaluationquestiontobeaddressedwithaproperresearchdesign);

• assessment stage (summative evaluation):oftenexplorestransferabilityandscaling,alongwith(usuallysmall-scaleevaluationof)effectiveness;and

• systematic reflection and documentation:thisarecontinuousactivities(asillustratedinFigure3)thattakesplaceduringallcyclesintheresearch–however,attheendthe


researcherportraystheentirestudytosupportretrospectiveanalysis,followedbyspecificationofdesignprinciplesandarticulationoftheirlinkstotheconceptualframework.

Itisbeyondthescopeofthischaptertodiscussindetailhowtoperformthesestages.Butanexceptionismadeforformativeevaluation,becausethisisthekeyresearchactivityindesignresearchaimedatimprovingthequalityoftheconsecutiveprototypesoftheintervention.

Formativeevaluationindevelopmentresearch6

BasedonpriorworkNieveen(1999;seealsoChapter5)proposesfourgenericcriteriaforhighqualityinterventions(seeTable1).Sheexplainsthesecriteriaasfollows:Thecomponentsoftheinterventionshouldbebasedonstate-of-the-artknowledge(content validity)andallcomponentsshouldbeconsistentlylinkedtoeachother(construct validity).Iftheinterventionmeetstheserequirementsitisconsideredtobevalid.Anothercharacteristicofhigh-qualityinterventionsisthatend-users(forinstancetheteachersandlearners)considertheinterventiontobeusableandthatitiseasyforthemtousethematerialsinawaythatislargelycompatiblewiththedevelopers’intentions.Iftheseconditionsaremet,wecalltheseinterventionspractical.Athirdcharacteristicofhighqualityinterventionsisthattheyresultinthedesiredoutcomes,i.e.thattheinterventioniseffective.

Criterion

Relevance (also referred to as content validity)

There is a need for the intervention and its design is based on state-of-the-art (scientific) knowledge.

Consistency (also referred to as construct validity)

The intervention is ‘logically’ designed.

Practicality The intervention is realistically usable in the settings for which it has been designed and developed.

Effectiveness Using the intervention results in desired outcomes.

Table 1: Criteria for high quality interventions (from Nieveen, 1999; Chapter 5)

Giventhecharacterofdesignresearch,thesefourcriteriamaygetdifferentemphasisindifferentstagesoftheresearchasisillustratedbyFigure5.Forexample,duringthepreliminaryresearchwheretheemphasisisonanalyzingtheproblemandreviewingtheliterature,thecriterionofrelevance(contentvalidity)isthemostdominant,withsomeattentionforconsistency(constructvalidity)andpracticality,whilstinthatstatenoattentionisyetgiventoeffectiveness.Ontheotherhand,intheprototypingstagemuch

6) SeealsoNieveen’schapter5inthisbookinwhichshediscusseshowtodotheformativeevaluationindesignresearch


attentionhastobepaidintheformativeevaluationtothecriterionofpracticality,whilsteffectivenesswillbecomeincreasinglyimportantinlateriterations.Finally,inassessmentstageofsummativeevaluation,thefocuswillbeonpracticalityandeffectiveness(seeFigure5,andFigure2forthestages).

Stage Criteria Short description of activities

1 Preliminary research Emphasis mainly on content validity, not much on consistency and practicality

Review of the literature and of (passed and/or present) projects addressing questions similar to the ones in this study. This results in (guidelines for) a framework and first blueprint for the intervention.

2 Prototyping stage Initially: consistency (construct validity) and practicality. Later on mainly practicality and gradually attention for efficiency.

Development of a sequence of prototypes that will be tried out and revised on the basis of formative evaluations. Early prototypes can be just paper-based for which the formative evaluation takes place via expert judgments.

3 Assessment phase practicality and efficiency

Evaluate whether target users can work with intervention (practicality) and are willing to apply it in their teaching (relevance & sustainability). Also whether the intervention is effective.

Figure 5: Evaluation criteria related to stages in design research

Formativeevaluationtakesplaceinallphasesanditerativecyclesofdesignresearch.AsillustratedbyFigure5,formativeevaluationservesdifferentfunctions,or-inotherwords-isaimedatdifferentcriteria(orcombinationsofthese)inthevariousdevelopmentcycles,eachbeingamicro-cycleofresearchwithitsspecificresearch/evaluationquestionandrelatedresearch/evaluationdesign.OnemaysaythatformativeevaluationhasvariouslayersinadesignresearchprojectasisillustratedinFigure6,takenfromTessmer(1993):frommoreinformalintheearlystagesofaproject(self-evaluation,one-to-oneevaluation,expertreview)tosmallgroupevaluationaimedattestingthepracticalityandeffectiveness,toafullfieldtest(ifapplicable).Theresearch/evaluationdesignforeachcycleshouldreflectthespecificfocusandcharacterofthecycle–seeChapter5byNieveenformoredetails.


Figure 6: Layers of formative evaluation (taken from Tessmer, 1993)

Figure6alsoillustratesthatmanypossiblemethodsofformativeevaluationcanbechosen,suchas7

• expertreviewand/orfocusgroups(importanttoconsider‘expertsinwhat’)• self-evaluationorscreening(usingchecklistofimportantcharacteristicsordesign

specifications)• one-to-oneevaluationorwalkthrough(withrepresentativeoftargetaudience)• smallgroupormicro-evaluation• fieldtestortry-outDesignresearchersshouldchooseforeachphaseandforeachprototypeformativeevaluationapproachesthataresuitableforthepurposeofthatparticularstageoftheresearch.Designresearchhastomeetcriteriaforgoodresearch.Itisthereforeimportantthatforeachdevelopmentcycletheresearcher(orresearchgroup)appliesthemethodological‘rules’fordoingresearch,i.e.foridentifyingthetargetaudienceandsampling,forinstrumentdevelopmentandapplytriangulationtoobtaingoodqualityinformation.But

7) seealsoChapter5byNieveen

HighResistanceto Revision

LowResistanceto Revision

Revise

Revise

Revise

Field TestUser Acceptance, Implementability

Organizational Acceptance

Small GroupEffectiveness, Appeal

Implementability

Expert ReviewContent, Design,Techical Quality

One-to-OneCiarity, AppealObvious Errors

Self-Evaluationobvious errors


giventhelayersofformativeevaluationindesignresearch,intheearlycyclesofdevelopmenttheevaluationdesigncanbelessrigorousthaninlaterphases.Figure7adaptedfromNieveen(1999)presentsanexamplethatillustrateshowvariousformativeevaluationmethodsareusedfortherespectiveprototypesinaprojectaimedatdevelopingacomputerassistedsupportsystemforcurriculumdevelopers.

prelimcomp.based

paper-based computer-based versions

final version

Users (n=5)

experts (n=3)

users (n=5)

experts (n=6)

users (n=4)

users (n=4)

users (n=17)

Validity content *) √ ea √ ea

interface √ ea

Practicality content √ wt √ wt √ ea √ me √ to √ ft

interface √ wt √ wt √ ea √ me √ to √ ft

Effectiveness entire system √ to √ ft

*): Content refers to the content of the support system

√ = primary attention of prototype and of formative evaluation

Methods of formative evaluation: me = micro evaluation; wt = walk through; ea = expert appraisal;

ft = field trial; to = try-out

Figure 7: Focus of design and formative evaluation of the prototypes for computer assisted support system for curriculum development (adapted from Nieveen, 1999)

Afinalnoteonthecriteriaofpracticalityandeffectiveness.Itmayoccurincertainstudiesthattheresearcher(orresearchcollaborative)cannotdoafinalfieldtrialoftheinterventionwiththefull(orasampleofthe)targetgroup,buthastorestricthimselftoexpertappraisaland/ormicro-evaluationofthefinalprototypeoftheintervention.Itisobviousthatinsuchasituationtheactual practicalityandtheactual effectivenessoftheinterventioncannotbedemonstrated,butonlyconclusionsabouttheexpected practicalityandtheexpected effectivenesscanbedrawn.Moreevaluationwillthenbeneededtodemonstratetheactual practicalityandtheactual effectiveness.ThiscanbeillustratedwithanexampleadaptedfromMafumiko(2006)whoconducteddesignresearchtoinvestigatewhethermicro–scaleexperimentationcancontributetoimprovingthechemistrycurriculuminTanzania.HisresearchdesignhasbeensummarizedinFigure8.


Figure 8: Example of research research design (adapted from Mafumiko, 2006)

SupposearesearcherwouldrestricthimselftothedevelopmentofprototypesoftheinterventionasillustratedinFigure8,anddoesnotplantoinvestigatewhetherVersionIVworksinthetargetcontext.Insuchasituationthemosthecanconcludeiswhetherhisinterventionisexpectedtobepracticalandeffectiveforthetargetcontext.Onlywhenhewouldconductafieldtest,hewillbeinthepositiontodecideuponactualpracticalityandactualeffectiveness(whichiswhatMafumikodid).

Design research dilemmas

Designresearchisconductedinclosecollaborationwitheducationalpractice.Notonlytheproblemaddressedissituatedineducationalpractice,butakeyfeatureofthisresearchisthateducationalpractitionersareactivelyinvolved,oftenasmembersoftheresearchteam.Thisleadstoanumberofchallengesthataretypicalforthistypeofresearch.McKenneyetal.(2006:83,84)havediscussedsomeoftheseandprovidesuggestionsforhowtoaddressthem.Theirpointsarebrieflysummarizedhere.

1. theresearcherisdesignerandoftenalsoevaluatorandimplementer.Severalmeasurescanbetakentocompensateforthispotentialconflictofinterest:• makeresearchopentoprofessionalscrutinyandcritiquebypeopleoutsidetheproject• theresearcherappliesthefollowingruleofthumb:shiftfromadominanceof‘creative

designer’perspectiveintheearlystage,towardsthe‘criticalresearcher’perspectiveinlaterstages(thisisreflectedinTessmer’slayersofformativeevaluation,Figure6)

Summative

evaluation

Appraisel by3 experts

Tryout in3 classrooms

Panel session with experts

Field test infour schools

Tryout with teacher educ students

Version I Version II Version III Version IV

Development of prototypes

Design guidelines &specifications


• haveagoodqualityofresearchdesign,e.g. - strong chain of reasoning(Krathwohl,1998)-themetaphorexpressestheideathat

eachpartoftheresearchdesignisequallyimportant - triangulation–toincreasethequalityofdataandofanalysistriangulationofdata

sourcesanddatacollectionmethodsshouldbeapplied,aswellasinvestigatortriangulationtoavoidtheinfluenceofanyspecificresearcher(seee.g.Denscombe,2007;136)

- empirical testingofboththeusabilityandtheeffectivenessoftheintervention - systematic documentation, analysisandreflectionofthedesign,development,

evaluationandimplementationprocessandtheirresults - haveattentionforvalidityandreliabilityofdataandinstruments - applyavarietyofmethodsandtactics:e.g.usepractitionersandotherresearchersas

‘criticalfriends’;usemultipleobservers/ratersandcalculateinter-observer/raterreliability,etc.

2. real-worldsettingsbringreal-worldcomplicationsDesignresearchisconductedinreal-worldsettingsbecauseitaddressescomplexproblemsineducationalpractice.Oneoftheproblemsisthattheresearchercanbea‘culturalstranger’(Thijs,1999)inthesettingoftheresearchandthatparticipants(e.g.principals,teachersnotinvolvedintheresearch,etc)arehesitanttobecompletelyopentoaresearchercomingfromtheoutside.McKenneyetal.(2006:84)pointstotheimportanceofcollaborationandmutualbeneficialactivitiestogainparticipants’trustandthoroughunderstandingofthecontext(i.e.insiderperspective).Ontheotherhand,theyalsopointtotheadvantagestobeanoutsiderasthismayallowtheresearchertodevelopadegreeofobjectivityand“freedom(orforgiveness)forhonestythatisnotpermittedtothosewithinaparticulargroup”(o.c.85)

3. adaptabilityDesignresearchiscyclicalandtakesplaceinreal-worldsettings.Eachcyclehastotakethefindingsofthepreviousonesintoaccount.Soontheonehandtheresearchdesignhastochange(ordevelop)fromonecycletotheother,whilstontheotherhandanever-changingresearchdesigncanbeweak.Inthiscontext,McKenneyetal.(2006:84)refertothenotionofevolutionaryplanning,i.e.“aplanningframeworkthatisresponsivetofielddataandexperiencesasacceptablemomentsduringthecourseofthestudy”.Thisisalreadyalludedtointhediscussionofformativeevaluation(seeFigure6fromTessmerandtheexampletakenfromNieveen,1999).Theneedforadaptabilitypertainsalsototheroleoftheresearcher.AccordingtoVandenAkker(2005,inMcKenneyetal.,2006),thesynergybetweenresearchandpracticecanbemaximizedwhenresearchersdemonstrateadaptabilityby:


(i) beingprepared,wheredesirable,totakeontheadditionalroleofdesigner,advisor,andfacilitator,withoutlosingsightoftheirprimaryroleasresearcher,

(ii) beingtolerantwithregardtotheoftenunavoidablyblurredroledistinctionsandremainingopentoadjustmentsintheresearchdesignifprojectprocesssodictates,

(iii)allowingthestudytobeinfluenced,inpart,bytheneedsandwishesofthepartners,duringwhatisusuallyalong-termcollaborativerelationship.

Suchadaptabilityrequiresstrongorganizationalandcommunicativecapabilitiesonbehalfoftheresearcher,aswellassoundunderstandingtheresearchprocesssothatcarefulchangesandchoicesthatmaximizevalueandminimizethreatstoqualityaremade.(McKenneyetal.,2006:84).

Toaddressthechallengesmentioned,McKenneyetal.(2006:85,86)presentafewguidelines for conducting design researchthatmayhelpresearchersmonitoringthescientificcharacterofhis/herresearch:- haveanexplicitconceptualframework(basedonreviewofliterature,interviewsof

experts,studyingotherinterventions)- developcongruentstudydesign,i.e.applyastrongchainofreasoningwitheachcycle

havingitsresearchdesign- usetriangulation(ofdatasource,datatype,method,evaluatorandtheory)toenhance

thereliabilityandinternalvalidityofthefindings- applybothinductiveanddeductivedataanalysis- usefull,context-richdescriptionsofthecontext,designdecisionsandresearchresults- membercheck,i.e.takedataandinterpretationsbacktothesourcetoincreasethe

internalvalidityoffindings.Itisbeyondthescopeofthispapertoelaborateontheseguidelinesfurther–seeMcKenneyetal.(2006;85,86)andresearchmethodologybooks.

Concluding remarks

Inthefieldofeducationthereismuchneedforresearchrelevantforeducationalpractice.Wehavearguedthatforcomplexpracticalproblemsandforresearchquestion(s)callingforthedesignanddevelopmentofaninterventiondesignresearchistheappropriateresearchapproach.

Givenitsfocusonpracticalproblemsanditsnatureofconductingtheresearchinareal-worldsettingwithactiveinvolvementofpractitioners,designresearchmaylooklikeactionresearch.Soonemaywonderhowdesignresearchisrelatedtoactionresearch.Indeed,actionresearchisalsodealingwithreal-worldproblems,aimingatimprovingpractice,cyclicalinnatureandparticipative(Denscombe,2007),buttheessentialdifferenceisthat


actionresearchisnotaimedatgeneratingdesignprinciples–ithasaparticularnicheamongprofessionalswhowanttouseresearchtoimprovetheirpractices(o.c.:122).

Wediscussedhowdesignresearchersshouldstriveforgeneralizabledesignprinciplesinthemeaningofgeneralizingtoabroadertheory.Whendesignresearchisconductedwithintheframeworkofaprogramofresearchaddressingfundamentalproblemsineducationalpractice,itwillresultinaspecificbodyofknowledge,vizsubstantiveandproceduraldesignprinciplesthatmaycontributetoimproveeducation.Ontheotherhandmanyquestionsarestilltobeaddressedastherearemanytypesofpracticalproblemsandthereforemanytypesofresearchgoalsforwhichdesignresearchmaybethebestapproach(e.g.,Reeves(2000)mentionssixdifferenttypesofgoals).

VandenAkker,Gravemeijer,McKenneyandNieveen(2006)reportthepresentationsanddiscussionsataseminardedicatedtoeducationaldesignresearch.Theirbookpoints-nexttodiscussinganumberofapproachestodesignresearchbyGravemeijerandCobb(2006),Reeves(2006)andMcKenneyetal.(2006)-toissueslikeassessingthequalityofdesignresearchproposals(chaptersbyPhillips,2006,andbyEdelson,2006)andthequalityofdesignresearch(chapterbyKelly,2006)whichneedfurtherreflectionandelaboration.

Finally,anumberofresearchreportsanddissertationshavebeenpublishedwhichareexemplaryforhowdesignresearchcanbeconducted(seechapter6forexamples).Butfordesignresearchtomaturefurthermoreresearchprojectsinavarietyofcontextsshouldnotonlybeconducted,butalsoreportedanddiscussedinresearchjournalsandatconferences.

OurhopeisthatthecommunityofeducationaltechnologistsinChinawillembarkonthisresearchendeavorandwillactivelycontributetothefurtherdevelopmentofeducationaldesignresearch.

Acknowledgement: in preparing this chapter, much use has been made of Van den Akker, Gravemeijer, McKenney and Nieveen (2006).


References

Bannan-Ritland,B.(2003).TheRoleofDesigninResearch:TheIntegrativeLearningDesignFrameworkEducational Researcher, 32(1),21-24.

Barab,S.&Squire,K.(2004).Design-basedresearch:puttingastakeintheground.Journal of the Learning Sciences, 13(1),1-14.

Brown,A.L.(1992).Designexperiments:Theoreticalandmethodologicalchallengesincreatingcomplexinterventionsinclassroomsettings.The Journal of The Learning Sciences, 2(2),141-178.

Cronbach,L.J.(1975).Beyondthetwodisciplinesofscientificpsychology.American Psychologist, 30(2),116-27.

Denscombe,M.(2007,3rded)).The good research guide for small-scale social research projects. Buckingham(UK):OpenUniversityPress.

Design-BasedResearchCollective(2003).Design-basedresearch:Anemergingparadigmforeducationalinquiry.Educational Researcher, 32(1),5-8.

diSessa,A.A.&Cobb,P.(2004).Ontologicalinnovationandtheroleoftheoryindesignexperiments.Educational Researcher, 32(1),77-103.

Edelson,D.C.(2006).Balancinginnovationandrisk:assessingdesignresearchproposals.In:VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research.London:Routledge,100-106.

Gravemeijer,K.&Cobb,P.(2006).Designresearchfromalearningdesignperspective.In:VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research. London:Routledge,17-51.

Gustafson,K.L.&Branch,R.M.(2002).Survey of instructional development models(4thed).Syracuse(NY,USA):ERICClearinghouseonInformation&Technology.

Kelly,A.E.(2006).Qualitycriteriafordesignresearch:evidenceandcommitments.In:VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research.London:Routledge,107-118.

Knippels,M.C.P.J.(2002).Coping with the abstract and complex nature of genetics in biology education.Utrecht(TheNetherlands):UniversityofUtrecht–CD-BètaPress.

Krathwohl,D.R.(1998,2nded).Methods of Educational and Social Science Research: An integrated approach.NewYork:Longman

Lijnse,P.L.(1995).“DevelopmentalResearch”asawaytoanempiricallybased“DidacticalStructure”ofScience.Science Education, 29(2),189-199.

Mafumiko,FideliceSimbagungileMbaruku(2006).Micro-scale experimentation as a catalyst for improving the chemistry curriculum in Tanzania.Doctoralthesis.Enschede(TheNetherlands)UniversityofTwente.Availablefrom:http://purl.org/utwente/55448

McKenney,S.(2001).Computer-based support for science education materials developers in Africa: exploring potentials.Doctoraldissertation.Enschede:UniversityofTwente.


McKenney,S.,Nieveen,N.&VandenAkker,J.(2006).Designresearchfromacurriculumperspective.In:VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research. London:Routledge,62-90.

Nieveen,N.(1999).Prototypingtoreachproductquality.InJ.vandenAkker,R.M.Branch,K.Gustafson,N.Nieveen,&T.Plomp(Eds),Design approaches and tools in education and training.Boston:KluwerAcademic,125-136.

Nieveen,N.,McKenney,S.&VandenAkker,J.(2006).Educationaldesignresearch:thevalueofvariety.In:VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research.London:Routledge,151-158.

Phillips,D.C.(2006).Assessingthequalityofdesignresearchproposals:somephilosophicalperspectives.In:VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research.London:Routledge,93-99.

Reeves,T.(2000).Enhancingtheworthofinstructionaltechnologyresearchthrough“designexperiments”andotherdevelopmentalstrategies.AERA.AERA.RetrievedOct.20,2006fromhttp://it.coe.uga.edu/~treeves/AERA2000Reeves.pdf

Reeves,T.C.(2006).Designresearchfromatechnologyperspective..In:VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research.London:Routledge,52-66.

Shavelson,R.&Towne,L.(2002).Scientific research in education.WashingtonDC:NationalAcademicPress(website:http://books.nap.edu/books)

Tecle,A.T.(2006).The potential of a professional development scenario for supporting biology teachers in Eritrea.Doctoraldissertation.Enschede(TheNetherlands):UniversityofTwente.

Tessmer,M.(1993).Planning and conducting formative evaluations.London:KoganPage.Thijs,A.(1999).Supporting science curriculum reform in Botswana: The potential of peer

coaching.Doctoraldissertation.Enschede:UniversityofTwente.VandenAkker,J.(1999).PrinciplesandMethodsofDevelopmentResearch.InJ.vanden

Akker,R.M.Branch,K.Gustafson,N.Nieveen,&T.Plomp(Eds),Design approaches and tools in education and training.Boston:KluwerAcademic,1-14.

VandenAkker,J.(2005).Hoekansamenwerkingleidentotmeersuccesenwijsheidinonderwijsontwikkeling?[Howcancollaborationleadtomoresuccessandwisdomineducationdevelopment?].Pedagogische Studiën, 82(4),343-7.

VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research.London:Routledge.ISBN10:0-415-39635-2(pbk)

Wademan,M.(2005).Utilizing Development Research to Guide People Capability Maturity Model Adoption Considerations.Unpublisheddoctoraldissertation,SyracuseUniversity(NewYork,USA).

Yin,R.K.(2003,3eded).Case study research: design and methods.NewburyPark(CA,USA):Sage–AppliedSocialResearchMethodsSeries,volume5.


2. CurriculumDesignResearch Jan van den Akker

Introduction

Thetitleofthischapter (Curriculum Design Research)intentionallycombinestwofields:‘curriculumdesign’and‘designresearch’.Itsymbolizestheaimofthistexttodiscussthefunctionandformsofdesignresearchfromacurricularperspective.Inparticular,itfocusesonhowdesignresearchcanincreasethequalityofcurriculumdesignanddevelopment.Also,itillustrateshowtherelevanceofeducationalresearch-awidelydebatedissue-canbenefitfromaconnectiontocurriculumpoliciesandpractices.Giventhisaimithelpstohaveanumberofbasicconceptsandanalyticalperspectivesavailablethatcanstructurecurriculardeliberationsandreducethecomplexityofcurriculumtasks.Thusmyinitialfocusinthischapter(buildingonvandenAkker,2003)isonsummarizingasetofconceptsandperspectivesthathelptoincreasethetransparencyandbalanceofcurriculumanalysis,developmentanddiscourse.Then,thefocuswillshifttowards(curriculum)designresearch(buildingonvandenAkker,1999,2006,andonvandenAkker,Gravemeijer,McKenneyandNieveen,2006).First,Iwillsketchthepotentialandcharacteristicsofdesignresearchinaddressingcomplexcurriculumchallenges.Second,Iwilladdressanumberofmethodologicalissues.Finally,Iwillpayattentiontoaclassicprobleminalleducationalresearch:generalizationoffindings.

Curriculum, what’s in a name?

Whenthereisamyriadofdefinitionsofaconceptintheliterature(aswithcurriculum),itisoftendifficulttokeepaclearfocusonitsessence.Inthosecasesitoftenhelpstosearchfortheetymologicaloriginoftheconcept.TheLatinword‘curriculum’(relatedtotheverb‘currere’i.e.running)referstoa‘course’or‘track’tobefollowed.Inthecontextofeducation,wherelearningisthecentralactivity,themostobviousinterpretationofthewordcurriculumisthentoviewitasacourse,trajectory,or‘plan for learning’(cf.Taba,1962).Thisveryshortdefinition(reflectedinrelatedtermsinmanylanguages)limitsitselftothecoreofallotherdefinitions,permittingallsortsofelaborationsforspecificeducationallevels,contexts,andrepresentations.Obviously,contextualspecificationisalwaysneededincurriculumconversationstoclarifytheperspective.Giventhissimpledefinition,adifferentiationbetweenvariouslevelsofthecurriculumhasproventobeveryusefulwhentalkingaboutcurricularactivities(policy-making;designanddevelopment;evaluationandimplementation).Thenextdistinctionappearstobehelpful:• International/comparative(orsupralevel)• System/society/nation/state(ormacro)level(e.g.nationalsyllabiorcoreobjectives)


• School/institution(ormeso)level(e.g.school-specificcurriculum)• Classroom(ormicro)level(e.g.textbooks,instructionalmaterials)• Individual/personal(ornano)level.

Thesupralevelusuallyreferstointernationaldebatesoragreementsonaimsandqualityofeducation,sometimesfuelledbyoutcomesofinternationallycomparativestudies(cf.PISAorTIMSS1).Curriculumdevelopmentatthesupralevelisusuallyofa‘generic’nature,while‘site-specific’approachesaremoreapplicableforthelevelsclosertoschoolandclassroompractice.Moreover,theprocessofcurriculumdevelopmentcanbeseenasnarrow(developingaspecificcurricularproduct)orbroad(alongterm,ongoingprocessofcurriculumimprovement,oftenincludingmanyrelatedaspectsofeducationalchange,e.gteachereducation,schooldevelopment,testingandexaminations).Inordertounderstandproblemsofcurriculumdecision-makingandenactment,abroaderdescriptionofcurriculumdevelopmentisoftenmostappropriate:usuallyalongandcyclicprocesswithmanystakeholdersandparticipants;inwhichmotivesandneedsforchangingthecurriculumareformulated;ideasarespecifiedinprogramsandmaterials;andeffortsaremadetorealizetheintendedchangesinpractice.

Moreover,curriculacanberepresentedinvariousforms.Clarificationofthoseformsisespeciallyusefulwhentryingtounderstandtheproblematiceffortstochangethecurriculum.Acommonbroaddistinctionisbetweenthethreelevelsofthe‘intended’,‘implemented’,and‘attained’curriculum.Amorerefinedtypology(vandenAkker,2003)isoutlinedinbox1.

INTENDED Ideal Vision (rationale or basic philosophy underlying a curriculum)

Formal/Written Intentions as specified in curriculum documents and/or materials

IMPLEMENTED Perceived Curriculum as interpreted by its users (especially teachers)

Operational Actual process of teaching and learning (also: curriculum-in-action)

ATTAINED Experiential Learning experiences as perceived by learners

Learned Resulting learning outcomes of learners

Box 1: Typology of curriculum representations

Traditionally,theintendeddomainreferspredominantlytotheinfluenceofcurriculumpolicymakersandcurriculumdevelopers(invariousroles),theimplementedcurriculum

1) PISAistheOECDProgrammeforInternationalStudentAssessment,asurveyeverythreeyearsofthe15-year-olds.TIMSSistheTrendsInMathematicsandSciencesStudy,conductedevery4yearsbytheInternationalAssociationfortheEvaluationofEducationalAchievement(IEA)inprimaryandsecondaryeducation.


relatesespeciallytotheworldofschoolsandteachers,andtheattainedcurriculumhastodowiththestudents.

Besidesthisdifferentiationinrepresentations,curriculumproblemscanbeapproachedfromvariousanalyticalangles.Forexample,Goodlad(1994)distinguishesthefollowingthreedifferentperspectives:• substantive,focusingontheclassicalcurriculumquestionaboutwhatknowledgeisof

mostworthforinclusioninteachingandlearning;• technical-professional,referringtohowtoaddresstasksofcurriculumdevelopment;• socio-political,referringtocurriculumdecision-makingprocesses,wherevaluesand

interestsofdifferentindividualandagenciesareatstake.Somemightarguethatthislististoolimitedasitrefersespeciallytocurriculumissuesfor‘traditional’planningforlearninginschools,anddoesnotincludethemore‘critical’perspectivesthatareamplypresentincurriculumtheoryliterature(e.g.Pinar,Reynolds,Slattery&Taubman,1995).However,fromaprimaryinterestincurriculumimprovement,thethreeperspectivesseemusefulandappropriate.

The vulnerable curriculum spider web

Oneofthemajorchallengesforcurriculumimprovementiscreatingbalanceandconsistencybetweenthevariouscomponentsofacurriculum(i.e.planforlearning).Whatarethosecomponents?TherelativelysimplecurriculumdefinitionbyWalker(2003)includesthreemajorplanningelements:content,purposeandorganizationoflearning.However,curriculumdesignandimplementationproblemshavetaughtusthatitiswisetopayexplicitattentiontoamoreelaboratedlistofcomponents.Elaboratingonvarioustypologies,wehavecometoadheretoaframework(seeBox2)oftencomponentsthataddresstenspecificquestionsabouttheplanningofstudentlearning.

Rationale or Vision Why are they learning?

Aims & Objectves Toward which goals are they learning?

Content What are they learning?

Learning activities How are they learning?

Teacher role How is the teacher facilitating learning?

Materials & Resources With what are they learning?

Grouping With whom are they learning?

Location Where are they learning?

Time When are they learning?

Assessment How to measure how far learning has progressed?

Box 2: Curriculum components


The‘rationale’(referringtooverallprinciplesorcentralmissionoftheplan)servesasmajororientationpoint,andthenineothercomponentsareideallylinkedtothatrationaleandpreferablyalsoconsistentwitheachother.Foreachofthecomponentsmanysub-questionsarepossible.Notonlyonsubstantiveissues(seethenextsection),but,forexample,alsoon‘organizational’aspectsas:• Grouping: - Howarestudentsallocatedtovariouslearningtrajectories? - Arestudentslearningindividually,insmallgroups,orwhole-class?• Location: - Arestudentslearninginclass,inthelibrary,athome,orelsewhere? - Whatarethesocial/physicalcharacteristicsofthelearningenvironment?• Time: - Howmuchtimeisavailableforvarioussubjectmatterdomains? - Howmuchtimecanbespentonspecificlearningtasks?Therelevanceofthesecomponentsvariesacrossthepreviouslymentionedcurriculumlevels(supra,macro,meso,micro,nano)andrepresentations.Afewexamplesmayillustratethis.• Curriculumdocumentsatthemacro-levelwillusuallyfocusonthefirstthree

components(rationale,aims&objectives,content;ofteninratherbroadterms),sometimesaccompaniedbyanoutlineoftimeallocationsforvarioussubjectmatterdomains.

• Whenonetakestheoperationalcurriculuminschoolsandclassroomsinmind,alltencomponentshavetobecoherentlyaddressedtoexpectsuccessfulimplementationandcontinuation.

• Thecomponentsoflearningactivities,teacherrole,andmaterials&resourcesareatthecoreofthemicro-curriculumintheclassroom.

• Thecomponentofassessmentdeservesseparateattentionatalllevelsandrepresentationssincecarefulalignmentbetweenassessmentandtherestofthecurriculumappearstobecriticalforsuccessfulcurriculumchange.

Ourpreferentialvisualizationofthetencomponentsistoarrangethemasaspiderweb(Figure1),notonlyillustratingitsmanyinterconnections,butalsounderliningitsvulnerability.Thus,althoughtheemphasisofcurriculumdesignonspecificcomponentsmayvaryovertime,eventuallysomekindofalignmenthastooccurtomaintaincoherence.AstrikingexampleisthetrendtowardintegrationofICTinthecurriculum,withusuallyinitialattentiontochangesinmaterialsandresources.Manyimplementationstudieshaveexemplifiedtheneedforamorecomprehensiveapproachandsystematicattentiontotheothercomponentsbeforeonecanexpectrobustchanges.


Thespiderwebalsoillustratesafamiliarexpression:everychainisasstrongasitsweakestlink.Thatseemsanotherveryappropriatemetaphorforacurriculum,pointingtothecomplexityofeffortstoimprovethecurriculuminabalanced,consistentandsustainablemanner.

Figure 1: Curricular spider web

Perspectives on substantive choices

Aclassicapproachtotheeternalcurriculumquestionofwhattoincludeinthecurriculum(orevenmoredifficultaswellasurgent:whattoexcludefromit)istosearchforabalancebetweenthreemajorsourcesororientationsforselectionandprioritysetting:• Knowledge:whatistheacademicandculturalheritagethatseemsessentialfor

learningandfuturedevelopment?• Society:whichproblemsandissuesseemrelevantforinclusionfromtheperspectiveof

societaltrendsandneeds?• Learner:whichelementsseemofvitalimportanceforlearningfromthepersonaland

educationalneedsandinterestsofthelearnersthemselves?

Asse

ssm

ent

Aims & Objectives

Content

Learning activitiesTe

ache

r rol

e

Materials &

Resources

Location

Time

Rationale

Grouping


Answerstothesequestionsusuallyconstitutetherationaleofacurriculum.Inevitably,choiceshavetobemade,usuallyinvolvingcompromisesbetweenthevariousorientations(andtheirrespectiveproponentsandpressuregroups).Oftentimes,effortsfailtoarriveatgenerallyacceptable,clearandpracticalsolutions.Theresultofaddingupallkindsofwishesisthatcurriculatendtogetoverloadedandfragmented.Implementationofsuchincoherentcurriculaeventuallytendstoleadtostudentfrustrations,failure,anddropout.Howtocreateabettercurriculumbalance?Easyanswersarenotavailable,butafewalternativesseemtohavesomepromise.First,inviewofthemultitudeof(academic)knowledgeclaims,itsometimeshelpstoreducethebignumberofseparatesubjectdomainstoamorelimitednumberofbroaderlearningareas,combinedwithsharperprioritiesinaimsforlearning(focusingonbasicconceptsandskills).Second,referringtotheavalancheofsocietalclaims,moreinteractionbetweenlearninginsideandoutsidetheschoolmayreducetheburden.However,themosteffectiveresponseisprobablytobemoreselectiveinreactingtoallsortsofsocietalproblems.AsCuban(1992)phraseditclearly:schoolsshouldnotfeelobligedtoscratchthebackofsocietyeverytimesocietyhasanitch.Andthird,aboutthelearners’perspective:worldwide,manyinterestingeffortsareongoingtomakelearningmorechallengingandintrinsicallymotivatingbymovingfromtraditional,teacher-andtextbook-dominatedinstructiontowardsmoremeaningful,activity-basedandautonomouslearningapproaches.

Development strategies

Tosketchcurriculumdevelopmentasaproblematicdomainisactuallyanunderstatement.Fromasocio-politicalstance,itseemsoftenmoreappropriatetodescribeitasawarzone,fullofconflictsandbattlefieldsbetweenstakeholderswithdifferentvaluesandinterests.Problemsmanifestthemselvesinthe(sometimesspectacularandpersistent)gapsbetweentheintendedcurriculum(asexpressedinpolicyrhetoric),theimplementedcurriculum(reallifeinschoolandclassroompractices),andtheattainedcurriculum(asmanifestedinlearnerexperiencesandoutcomes).Atypicalconsequenceofthosetensionsisthatvariousfrustratedgroupsofparticipantsblameeachotherforthefailureofreformorimprovementactivities.Althoughsuchblaminggamesoftenseemratherunproductive,therearesomeseriouscriticalremarkstobemadeonmanycurriculumdevelopmentapproachesworldwide.Firstofall,manycurriculumreformeffortscanbecharacterizedbyoverlybiginnovationambitions(especiallyofpoliticians)withinunrealisticallyshorttimelinesandwithverylimitedinvestmentinpeople,especiallyteachers.Second,oftentimesthereisalackofcoherencebetweentheintendedcurriculumchangeswithothersystemcomponents(especiallyteachereducationandassessment/examinationprograms).And


lastbutnotleast,timelyandauthenticinvolvementofallrelevantstakeholdersisoftenneglected.Fromastrategicpointofview,theliteraturehasofferedusmany(technical-professional)modelsandstrategiesforcurriculumdevelopment.ThreeprominentapproachesareTyler’srational-linearapproach,Walker’sdeliberativeapproach,andEisner’sartisticapproach.Asitdoesnotfitwithinthepurposeofthischaptertoexplainthosemodelsinparticular,thereaderisreferredtoeducativetextsasfromMarshandWillis(2003).Obviously,thecontextandnatureofthecurriculumdevelopmenttaskathandwilldeterminetoalargeextentwhatkindofstrategyisindicated.Itisnoteworthythatwearebeginningtoseemoreblendedapproachesthatintegratevarioustrendsandcharacteristicsofrecentdesignanddevelopmentapproachesinthefieldofeducationandtraining(foranoverviewandaseriesofexamples:seevandenAkker,Branch,Gustafson,Nieveen&Plomp,1999).Somekeycharacteristics:• Pragmatism:Recognitionthatthereisnotasingleperspective,overarchingrationaleor

higherauthoritythatcanresolvealldilemmasforcurriculumchoicestobemade.Thepracticalcontextanditsusersareintheforefrontofcurriculumdesignandenactment.

• Prototyping:Evolutionaryprototypingofcurricularproductsandtheirsubsequentrepresentationsinpracticeisviewedasmoreproductivethanquasi-rationalandlineardevelopmentapproaches.Gradual,iterativeapproximationofcurriculardreamsintorealitiesmaypreventparalysisandfrustrations.Formativeevaluationoftentative,subsequentcurriculumversionsisessentialtosuchcurriculumimprovementapproaches.

• Communication:Acommunicative-relationalstyleisdesirableinordertoarriveattheinevitablecompromisesbetweenstakeholderswithvariousrolesandinterestsandtocreateexternalconsistencybetweenallpartiesinvolved.

• Professional development:Inordertoimprovechancesonsuccessfulimplementation,thereisatrendtowardsmoreintegrationofcurriculumchangeandprofessionallearninganddevelopmentofallindividualsandorganizationsinvolved.

Designordevelopment(al)researchisaresearchapproachthatincorporatessomeofthesecharacteristics,anditbecomesevenmorepromisingbyaddingtheelementofknowledgegrowthtoit(vandenAkker,1999).Suchresearchcanstrengthentheknowledgebaseintheformofdesignprinciplesthatofferheuristicadvicetocurriculumdevelopmentteams,when(morethanincommondevelopmentpractices)deliberateattentionispaidtotheoreticalembeddingofdesignissuesandempiricalevidenceisofferedaboutthepracticalityandeffectivenessofthecurricularinterventionsinrealusersettings.However,thereareseveralpersistentdilemmasforcurriculumdevelopmentthatcannoteasilyberesolved,letalonethroughgenericstrategies.Forexample:howtocombineaspirationsforlarge-scalecurriculumchangeandsystemaccountabilitywiththeneedfor


localvariationsandownership?Thetensionbetweentheseconflictingwishescanbesomewhatreducedwhenoneavoidsthealltoocommon‘onesizefitsall’approach.Moreadaptiveandflexiblestrategieswillavoiddetailedelaborationandover-specificationofcentralcurriculumframeworks.In-stead,theyoffersubstantialoptionsandflexibilitytoschools,teachers,andlearners.Althoughstrugglesaboutprioritiesinaimsandcontentwillremaininevitable,theprincipleof‘lessismore’shouldbepursued.However,whatisincorporatedinacorecurriculumshouldbeclearlyreflectedinexaminationandassessmentapproaches.The‘enactment’perspective(teachersandlearnerstogethercreatetheirowncurriculumrealities)isincreasinglyreplacingthe‘fidelity’perspectiveonimplementation(teachersfaithfullyfollowcurricularprescriptionsfromexternalsources).Thistrendputsevenmoreemphasisonteachersaskeypeopleincurriculumchange.Bothindividualaswellasteamlearningisessential(Fullan,2001).Teachersneedtogetoutoftheircustomaryisolation.Collaborativedesignandpilotingofcurricularalternativescanbeveryproductive,especiallywhenexperiencesareexchangedandreflecteduponinastructuredcurriculumdiscourse.Interactionwithexternalfacilitatorscancontributetocarefulexplorationsofthe‘zoneofproximaldevelopment’ofteachersandtheirschools.Cross-fertilizationbetweencurriculum,teacher,andschooldevelopmentisaconditiosinequanonforeffectiveandsustainablecurriculumimprovement.Theincreasinglypopularmissionstatementsofschoolstobecomeattractiveandinspiringenvironmentsforstudentsandteacherscanonlyberealizedwhensuchintegratedscenariosarepractised.

The potential of curriculum design research

Variousmotivesforinitiatingandconductingcurriculumdesignresearchcanbementioned.Abasicmotivestemsfromtheexperiencethatmanyresearchapproaches(e.g.experiments,surveys,correlationalanalyses),withtheirfocusondescriptiveknowledge,hardlyprovideprescriptionswithusefulsolutionsforavarietyofdesignanddevelopmentproblemsineducation.Probablythegreatestchallengeforprofessionaldesignersishowtocopewiththemanifolduncertaintiesintheircomplextasksinverydynamiccontexts.Iftheydoseeksupportfromresearchtoreducethoseuncertainties,severalfrustrationsoftenarise:answersaretoonarrowtobemeaningful,toosuperficialtobeinstrumental,tooartificialtoberelevant,and,ontopofthat,theyusuallycometoolatetobeofanyuse.Curriculumdesignersdoappreciatemoreadequateinformationtocreateasolidgroundfortheirchoicesandmoretimelyfeedbacktoimprovetheirproducts.Moreover,theprofessionalcommunityofdevelopersasawholewouldbehelpedbyagrowingbodyofknowledgeoftheoreticallyunderpinnedandempiricallytesteddesignprinciplesandmethods.


Anotherreasonforcurriculumdesignresearchstemsfromthehighlyambitiousandcomplexnatureofmanycurriculumreformpoliciesineducationworldwide.Thesereformendeavorsusuallyaffectmanysystemcomponents,areoftenmulti-layered,includingbothlarge-scalepoliciesandsmall-scalerealization,andareverycomprehensiveintermsoffactorsincludedandpeopleinvolved.Thoseradical‘revolutions’,ifpromisingatall,cannotberealizedonthedrawingtable.Thescopeofdiverseneedsisoftenverywide,theproblemstobeaddressedareusuallyill-specified,theeffectivenessofproposedinterventionsismostlyunknownbeforehand,andtheeventualsuccessishighlydependentonimplementationprocessesinabroadvarietyofcontexts.Therefore,suchcurriculumreformeffortswouldprofitfrommoreevolutionary(interactive,cyclic,spiral)approaches,withintegratedresearchactivitiestofeedtheprocess(bothforwardandbackward).Suchanapproachwouldprovidemoreopportunitiesfor‘successiveapproximation’oftheidealsandformorestrategiclearningingeneral.Inconclusion:curriculumdesignresearchseemsawiseandproductiveapproachforcurriculumdevelopment.

Features of curriculum design research

Curriculumdesignresearchisofteninitiatedforcomplex,innovativetasksforwhichonlyveryfewvalidatedprinciplesareavailabletostructureandsupportthedesignanddevelopmentactivities.Sinceinthosesituationstheimageandimpactoftheinterventiontobedevelopedisoftenstillunclear,theresearchfocusesonrealizinglimitedbutpromisingexamplesofthoseinterventions.Theaimisnottoelaborateandimplementcompleteinterventions,buttocometo(successive)prototypesthatincreasinglymeettheinnovativeaspirationsandrequirements.Theprocessisoftencyclicorspiral:analysis,design,evaluationandrevisionactivitiesareiterateduntilasatisfyingbalancebetweenidealsandrealizationhasbeenachieved.

Towhatextentdothesedesignresearchactivitiesdifferfromwhatistypicalfordesignanddevelopmentapproachesinprofessionalpractices?Whataretheimplicationsoftheaccountabilityofresearcherstothe‘scientificforum’?Attheriskofexaggeratingthedifferences,letusoutlinesomeofthem,basedonwhatisknownaboutroutinizedstandard-patternsincurriculumdevelopmentpractices.Ofcourse,alotofactivitiesaremoreorlesscommonforbothapproaches,sothefocuswillbeonthoseadditionalelementsthataremoreprominentindesignresearchthanincommondesignanddevelopmentpractices.

(1)PreliminaryinvestigationAmoreintensiveandsystematicpreliminaryinvestigationofcurriculumtasks,problems,andcontextismade,includingsearchingformoreaccurateandexplicitconnectionsofthat


analysiswithstate-of-the-artknowledgefromliterature.Sometypicalactivitiesinclude:literaturereview;consultationofexperts;analysisofavailablepromisingexamplesforrelatedpurposes;casestudiesofcurrentpracticestospecifyandbetterunderstandneedsandproblemsinintendedusercontexts.

(2) TheoreticalembeddingMoresystematiceffortsaremadetoapplystate-of-the-artknowledgeinarticulatingthetheoreticalrationaleforcurriculumdesignchoices.Moreover,explicitfeedbacktoassertionsinthedesignrationaleaboutessentialcharacteristicsoftheintervention(substantivedesignprinciples)ismadeafterempiricaltestingofitsquality.Thistheoreticalarticulationcanincreasethe‘transparency’and‘plausibility’oftherationale.Becauseoftheirspecificfocus,thesetheoreticalnotionsareusuallyreferredtoas‘mini’-or‘local’theories,althoughsometimesconnectionscanalsobemadeto‘middle-range’theorieswithasomewhatbroaderscope.

(3) EmpiricaltestingClearempiricalevidenceisdeliveredaboutthepracticalityandeffectivenessofthecurriculumfortheintendedtargetgroupinrealusersettings.Inviewofthewidevariationofpossibleinterventionsandcontexts,abroadrangeof(direct/indirect;intermediate/ultimate)indicatorsfor‘success’shouldbeconsidered.

(4) Documentation,analysisandreflectiononprocessandoutcomesMuchattentionispaidtosystematicdocumentation,analysisandreflectionontheentiredesign,development,evaluationandimplementationprocessandonitsoutcomesinordertocontributetotheexpansionandspecificationofthemethodologyofcurriculumdesignanddevelopment.

Morethanmostotherresearchapproaches,designresearchaimsatmakingbothpracticalandscientificcontributions.Inthesearchforinnovative‘solutions’forcurriculumproblems,interactionwithpractitioners(invariousprofessionalroles:teachers,policymakers,developers,andthelike)isessential.Theultimateaimisnottotestwhethertheory,whenappliedtopractice,isagoodpredictorofevents.Theinterrelationbetweentheoryandpracticeismorecomplexanddynamic:isitpossibletocreateapracticalandeffectivecurriculumforanexistingproblemorintendedchangeintherealworld?Theinnovativechallengeisusuallyquitesubstantial,otherwisetheresearchwouldnotbeinitiatedatall.Interactionwithpractitionersisneededtograduallyclarifyboththeproblematstakeandthecharacteristicsofitspotentialsolution.Aniterativeprocessof‘successiveapproximation’or‘evolutionaryprototyping’ofthe‘ideal’interventionisdesirable.Directapplicationoftheoryisnotsufficienttosolvethosecomplicatedproblems.Onemightstate


thatamore‘constructivist’developmentapproachispreferable:researchersandpractitionerscooperativelyconstructworkableinterventionsandarticulateprinciplesthatunderpintheeffectsofthoseinterventions.Anotherreasonforcooperationisthatwithoutinvolvementofpractitionersitisimpossibletogainclearinsightinpotentialcurriculumimplementationproblemsandtogeneratemeasurestoreducethoseproblems.Newinterventions,howeverimaginativetheirdesign,requirecontinuousanticipationatimplementationissues.Notonlyfor‘social’reasons(tobuildcommitmentandownershipofusers)butalsofor‘technical’benefits:toimprovetheirfitnessforsurvivalinreallifecontexts.Therefore,rigoroustestingofpracticalityisaconditiosinequanonindesignresearch.

Emphasis on formative evaluation

Ashasbecomeclearintheprevioussections,formativeevaluationholdsaprominentplaceincurriculumdesignresearch.Themainreasonforthiscentralroleisthatformativeevaluationprovidestheinformationthatfeedsthecycliclearningprocessofcurriculumdevelopersduringthesubsequentloopsofadesignanddevelopmenttrajectory.Itismostusefulwhenfullyintegratedinacycleofanalysis,design,evaluation,revision,etcetera,andwhencontributingtoimprovementofthecurriculum.Thebasiccontributionofformativeevaluationistoqualityimprovementofthecurriculumunderdevelopment.Quality,however,isanabstractconceptthatrequiresspecification.Duringdevelopmentprocesses,theemphasisincriteriaforqualityusuallyshiftsfromrelevance,toconsistency,topracticality,toeffectiveness2.Relevancereferstotheextentthattheintendedcurriculumisperceivedtobearelevantimprovementtopractice,asseenfromthevariedperspectivesofpolicymakers,practitionersandresearchers.Consistencyreferstotheextentthatthedesignofthecurriculumisbasedonstate-of-the-artknowledgeandthatthevariouscomponentsoftheinterventionareconsistentlylinkedtoeachother(cf.thecurricularspiderweb).Practicalityreferstotheextentthatusers(andotherexperts)considertheinterventionasclear,usableandcost-effectivein‘normal’conditions.Effectivenessreferstotheextentthattheexperiencesandoutcomeswiththeinterventionareconsistentwiththeintendedaims.

Themethodsandtechniquesforevaluationwillusuallybeattunedtothatshiftincriteria.Forexample,validitycanadequatelybeevaluatedthroughexpertappraisal,practicalityviamicro-evaluationsandtry-outs,andeffectivenessinfieldtests.Inlaterstagesofformativeevaluation,methodsofdatacollectionwillusuallybelessintensivebutwithanincreasingnumberofrespondents(e.g.achievementtestformanystudentscomparedtoin-depthinterviewwithafewexperts).

2) Seeforthesecriteriaalsochapters1and5.


Formativeevaluationwithindevelopmentresearchshouldnotonlyconcentrateonlocatingshortcomingsoftheinterventioninitscurrent(draft)version,butespeciallygeneratesuggestionsonhowtoimprovethoseweakpoints.Richnessofinformation,notablysalienceandmeaningfulnessofsuggestionsinhowtomakeaninterventionstronger,isthereforemoreproductivethanstandardizationofmethodstocollectandanalyzedata.Also,efficiencyofproceduresiscrucial.Thelowerthecostsintimeandenergyfordatacollection,processing,analysisandcommunicationwillbe,thebiggerthechancesonactualuseandimpactonthedevelopmentprocess.Forexample,samplesofrespondentsandsituationsfordatacollectionwillusuallyberelativelysmallandpurposivecomparedtosamplingproceduresforotherresearchpurposes.Theaddedvalueofgetting‘productive’informationfrommoresourcestendstodecrease,becausetheopportunitiesfor‘rich’datacollectionmethods(suchasinterviewsandobservations)arelimitedwithbignumbers.Toavoidanoverdoseofuncertaintyindatainterpretation,oftentriangulation(ofmethods,instruments,sources,andsites)isapplied.Theseargumentsespeciallyholdtrueforearlystagesofformativeevaluation,whentheinterventionisstillpoorlycrystallized.

Generalization of curriculum design research findings

Thepracticallymostrelevantoutcomeofcurriculumdesignresearchisitscontributiontowardsoptimizationofthecurricularproductanditsactualuse,leadingtobetterinstructionalprocessesandlearningresults.However,amajorcontributiontoknowledgetobegainedfromdesignresearchisintheformof(bothsubstantiveandmethodological)‘designprinciples’tosupportdevelopersintheirtask.Thoseprinciplesareusuallyheuristicstatementsofaformatsuchas:“IfyouwanttodesigncurriculumX[forthepurpose/functionYincontextZ],thenyouarebestadvisedtogivethatcurriculumthecharacteristicsA,B,andC[substantiveemphasis],andtodothatviaproceduresK,L,andM[proceduralemphasis],becauseoftheoreticalandempiricalargumentsP,Q,andR.”Obviouslythoseprinciplescannotguaranteesuccess,buttheyareintendedtoselectandapplythemostappropriate(substantiveandprocedural)knowledgeforspecificdesignanddevelopmenttasks.Itisnotuncommonindesignresearchthatsuchknowledge,especiallythesubstantiveknowledgeaboutessentialcurriculumcharacteristics,canpartlybeextractedfromaresultingprototypeitself.Thatisoneofthereasonsthatmakeitsoprofitabletosearchforandcarefullyanalyzealreadyavailablecurriculatogenerateideasfornewdesigntasks.However,thevalueofthatknowledgewillstronglyincreasewhenjustifiedbytheoreticalarguments,well-articulatedinprovidingdirections,andconvincinglybacked-upwithempiricalevidenceabouttheimpactofthoseprinciples.Moreover,thoseheuristicprincipleswillbeadditionallypowerfuliftheyhavebeenvalidatedinsuccessfuldesignofmoreinterventionsinmorecontexts.Chancesforsuchknowledgegrowthwillincrease


whendesignresearchisconductedintheframeworkofresearchprograms,becauseprojectscanthenbuildupononeanother.Sincedatacollectionindesignresearchisoftenlimitedtosmall(andpurposive)samples,effortstogeneralizefindingscannotbebasedonstatisticaltechniques,focusingongeneralizationsfromsampletopopulation.Insteadonehastoinvestin‘analytical’formsofgeneralization(cf.Yin,2003):readers/usersneedtobesupportedtomaketheirownattemptstoexplorethepotentialtransferoftheresearchfindingstotheoreticalpropositionsinrelationtotheirowncontext.Reportsondesignresearchcanfacilitatethattaskofanalogyreasoningbyacleartheoreticalarticulationofthedesignprinciplesappliedandbyacarefuldescriptionofboththeevaluationproceduresaswellastheimplementationcontext.Especiallya‘thick’descriptionoftheprocess-in-contextmayincreasethe‘ecological’validityofthefindings,sothatotherscanestimateinwhatrespectsandtowhatextenttransferfromthereportedsituationtotheirownispossible.Anotheroptionthatmaystimulateexplorationofpossibilitiesfor(virtual)generalizationistoorganizeinteractivemeetingswithexpertsfromrelatedcontextstodiscusstheplausibilityoftheresearchfindingsandrecommendationsforrelatedtasksandcontexts.Lastbutnotleast,designresearchmayofferdraftsofvariousrelevantcurriculumversions(withprovenconsistencyandpracticality)thatcanbecomparedinmorequantitative,large-scalestudies.

References

Cuban,L.(1992).Curriculumstabilityandchange.InP.Jackson(Ed.),Handbook of research on curriculum(pp.216-247).NewYork:Macmillan.

Goodlad,J.(1994).Curriculumasafieldofstudy.InT.Husén,&T.Postlethwaite(Eds.),The international encyclopedia of education(pp.1262-1276).Oxford:PergamonPress.

Fullan,M.(2001).The new meaning of educational change.NewYork:TeachersCollegePress.Marsh,C.,&Willis,P.(2003).Curriculum: Alternative approaches, ongoing issues(third

edition).UpperSaddleRiver,NJ:Merrill/PrenticeHall.Pinar,W.,Reynolds,W.,Slattery,P.,&Taubman,P.(1995).Understanding curriculum. An

introduction to the study of historical and contemporary curriculum discourses.NewYork:PeterLang.

Taba,H.(1962).Curriculum development: Theory and practice.NewYork:Harcourt,Brace&World.

vandenAkker,J.(1999).Principlesandmethodsofdevelopmentresearch.InJ.vandenAkkeretal.(Eds.),Design approaches and tools in education and training(pp.1-14).Dordrecht:KluwerAcademicPublishers.


vandenAkker,J.(2003).Curriculumperspectives:Anintroduction.InJ.vandenAkker,W.Kuiper&U.Hameyer(Eds.),Curriculum landscapes and trends(pp.1-10).Dordrecht:KluwerAcademicPublishers.

vandenAkker,J.,Branch,R.,Gustafson,K.,Nieveen,N.&Plomp,T.(Eds.)(1999).Design approaches and tools in education and training.Dordrecht:KluwerAcademicPublishers.

vandenAkker,J.,Gravemeijer,K.,McKenney,S.,&Nieveen,N.(Eds.)(2006).Educational design research.London:Routledge.

Walker,D.(2003).Fundamentals of curriculum: Passion and professionalism.Mahwah,NJ:LawrenceErlbaumAssociates.

Yin,R.K.(2003).Case Study Research: Design and Methods.London:Sage.


3. TheIntegrativeLearningDesign Framework:AnIllustratedExample fromtheDomainofInstructional Technology Brenda Bannan

Articulatingacleardefinitionandprocessofdesignresearchisacurrentandprominenttopicamongeducationalresearchers(Kelly,Lesh&Baek,2008;vandenAkker,Gravemeijer,McKennyandNieveen,2006).Designresearchstudiesinvolvecomplexinteractionsandfeedbackcyclesthatcansignificantlyblurtherolesofresearchers,teachers,curriculumdevelopers,instructionaldesignersandassessmentexperts(Kelly,Lesh,Baek&Bannan-Ritland2008).Aseducationalresearchersstruggletoclarifythisresearchmethod,theycontinuetoraisesignificantquestionssuchashowisdesignresearchdifferentfromtheprocessofdesign?Whatareappropriatemethodsandprocessesthatcanbeusedindesignresearch?Howdowesystematicallycreate,testanddisseminatedesignorteachinginterventionsthatwillhavemaximumimpactonpracticecapitalizingondesignresearch?Howdowegenerateboththeoreticalandpracticalknowledgerelatedtocomplexeducationalsettings?Kelly(2006)andothers(Fishman,Marx,Blumenfeld,Krajcik&Soloway,2004;Zaritsky,Kelly,Flowers,Rogers&O’Neill,2003;Rogers2003;Collins,1999;Design-basedResearchCollective,2003)advocatethattheseemergingmethodscallforthearticulationofnewprocessesandcriteriaincludingfactorssuchastheusefulnessandusabilityofknowledge,itsshareability,andmarketability,howwellitdisseminatesandtheextenttowhichitpositivelyimpactspractice.Sabelli(personalcommunication,May16,2002)citesaneedfororganizationalstructureandprotocolforthediffusionofresearchintopracticeandstatesthateducationalresearchsituationsareextremelycomplexsystemsthatcanbenefitfromintegratedsystemresearchstrategies.Thereisaneedforcomprehensivemodelstoguidedesignresearchaddressingtheprocessofdesigning,developingandassessingtheimpactofaneducationalinnovation.Inthischapter,Ipresentanintegrationofexistingdesignandresearchprocessesofferingaguidingframeworkthatgoesbeyondtheindividualdomainsofsocialscience,behavioralscienceandcommunicationtheoryandattemptstointegratethesystematicprocessesoftherelatedfieldsofinstructionaldesign,softwareengineering,productdesign,hencethenameIntegrativeLearningDesignFramework(ILDF).Buildingontheintegrationofprocessesfrommultiplefieldssuchasinstructionaldesign,objectorientedsoftwaredevelopment,productdevelopmentanddiffusionofinnovationsandeducationalresearch,theILDFpresenta“meta-methodological”viewthatattemptstointegratethebestofdesign,researchanddiffusionofeducationalinnovations.Thisframeworkconsistingoffourphases(seeFigure1)challengesresearcherstoprovideimprovedarticulationofdesignresearchprocessesbyphaseandtoconsidertheentirescopeofresearchfrominitialconceptualizationtodiffusionandadoption.


Figure 1: Questions and Methods for Design Research by ILDF Phase

Web-enabled Proto-D

iffusion

ConsequensesD

iffusion Adoption

Adaptation

Publish Results

Evaluate Results

Implem

ent.Theory/System

Refinem.

Formative

TestingD

etailed D

esignArticulated Prototype

Research/System

D

esign

AudienceCharacteriza-

tion

TheoryD

evelop.Survey

LiteratureN

eedsAnalysis

Informed Exploration

Enactment

Evaluation: Local Impact

Evaluation: Broader Impact

Questions:

- What are identified gaps/problem

s in theory, practice and/or the m

arketplace?- W

hat information can be gleaned from

existing data or research?- H

ow can w

e characterize the problem or

leaner need?- W

hat are the systemic social, cultural, and

organizational influences or constraints on design?- W

hat are characteristics of the audience?

Methods:

- Benchmarking

- Performance/needs analysis

- Interviews

- Survey of Experts- Focus G

roups- O

bservation/Role Modeling

- Case Studies

Questions:

- What are the learning targets for

innovation?- W

hat design principles or strategies may

be applicable?- H

ow to identify and operationalize

cognitive and performance processes

in design?- To w

hat extent does the design embody

the theoretical model?

Methods:

- Task Analysis- Contextual Analysis- D

esigner Logs- Expert Review- Audience Review

Questions:

- Is the enacted design usable, valid and

relevant?- Is the design instance accessible and efficient in delivering instruction or supporting learning?- W

hat is the local impact or effectiveness

of the design instance?- H

ow effective is the design solution in

achieving learning targets at its highest fideliy in full context?

Methods:

- Usability Testing

- Expert Review- O

bservation or Video records- Interview

s- Form

ative Evaluation- Pre-post Com

parative Studies- Q

uasi-experimental studies

Questions:

-What factors influence diffusion, adoption

and adaption of innovation?- W

hat are the pragmatic dem

ands of the learning environm

ent that influences adoption of design?- W

hat policies and cultures shape participants use of innovation?

Methods:

- Analysis of computer log files

- Multi-site Interview

s, Surveys and O

bservations- D

ata mining

- Correlational studies- Q

uasi-experimental studies

ILD

Guiding

Questions

for Research

ApplicableResearchM

ethods

Adaptation


ThefourphasesofInformedExploration,Enactment,LocalEvaluationandBroadEvaluationpresentedintheILDFencompassaprocessmodelforconductingdesignresearchbasedonseveralyearsofattemptstoincorporateprogressivelymorerigorous,research-basedcycleswithinatechnology-basedinstructionaldesigneffort.Thistypeofeffortisdifferentthantraditionalinstructionaldesignastheiterativecyclesareessentiallymicro-cyclesofresearch(morecomprehensivequalitativeandquantitativeresearcheffortsthanformativeevaluationcycles)conductedtolearnmorethanhowtoimprovethetechnologysystem,althoughthestudiesmayalsoresultinthatoutcome.ThereferencetolearningwithintheILDFistoplaceemphasisonthelearningthatcanresultinthecontextandactivityofdesign.Forexample,asresearchersorinstructionaldesignerswemaygenerateinformationabouttheteachingandlearningprocess,participants,context,andculturethatisoftennotattendedto,discardedandcapturedinarigorousmannerforotherstolearnfromandcapitalizeon.Whetherourdesignactivityinvolvesclassroom-basedinterventions,technologyorsomecombinationofboth,theinterconnecteddesignresearchcyclescangenerateknowledgeaboutdesignprinciplesbutalsoproviderichinformationonaspectsoflearning,cognition,expertandnoviceperspectives,aswellasstakeholderpositionstodirectdesignanddesigndecision-making.Thecoreissueathandisthattherich,complex,designprocessmayoffermultipleopportunitiestogenerateresearch-basedknowledgehowever,muchofitislostandnotdocumentedinthecreativedesignprocess.Ourchallenge,asdesignresearchers,istotrytosystematicallygather,analyze,reportandcodifythisinformationinarigorousmannerthatstrivestowardsometypeoflogical,argumentativegrammarworthyofstringentresearchprocesses(Kelly,2006).

Connectedcyclesofresearchcyclesanddesignprocessesresultinimproveddecision-makingbasedondata-drivenresultsfordesign,developmentandresearchpurposes.Thoughclearlyinterventionistandprimarilyformativeinnature,theILDFprocessstandsapartfromtraditionalinstructionaldesignandresearchefforts.Throughoutthemultiplephasesandcyclesofintegratedresearchanddesignprocessesvaluableknowledgeinthecontextofuseisgenerated.Weneedtominewhatislearnedaboutimportantfactorsrelatedtolearning,context,culture,andtechnologywithinthedesignprocess(notseparatefromitinacontrolledsettingasevidencedintraditionalresearch).Ifdesignresearcherscanarticulateanintegrativeresearchanddesignprocess,itmayhavethepotentialtosignificantlyimproveourunderstandingofteaching,learningandtrainingin-situ.Themultiplemacroandmicro-cyclesofdatacollection,analysisandmostimportantly,results-drivendesigndecision-makingiswhatsetsdesignresearchapartfromtraditionalformativeevaluationininstructionaldesignwhichisoftenconductedinaverylimitedmannerorasinglecycleofdata-gatheringandanalysis.


Tessmer(1993)referstoformativeevaluationasa“judgmentofthestrengthsandweakenessesofinstructioninitsdevelopingstages,forpurposesofrevisingtheinstructiontoimproveitseffectivenessandappeal(p.11)”.Althoughmultiplemethodsmaybeusedincludingexpertreview,one-to-oneevaluation,smallgroupandfieldtesting,formativeevaluationcyclesintraditionalinstructionalsystemsdesignmaynotalwaysemployresearchmethodsthatarespecifictoparticularphasesofanintegrativeandconnecteddesignresearchcycle.Formativeevaluation,despiteitsmostrigorousandcomprehensiveapplicationdoesnotprogressivelygenerateknowledgeaboutcognition,contextandcultureofusebutprovidesalimitedfocusonaparticulartechnologysystemofinstructionandjudgesitseffectiveness,appealandefficiency.Incontrast,designresearchcyclesarebasedonathorough,systematicprocessintegratedmultipledesignandresearchprocessestoprogressivelyimproveunderstandingaboutlearners,learning,context,orcultureaswellasiterativelyimproveanintervention.Therefore,formativeevaluationmethodsaresubsumedasoneselectedmethodinwhatcouldbedescribedasa“meta-methodological”orinvolvingmultipleresearchmethodsacrossthedesignresearchprocess.Whatiscriticalindesignresearchisthetheoreticalyieldoftheefforttobeviewedasimportantastheimprovementoftheintervention(seePlompchapter1).

Designresearchcyclesaredynamicandintegratemultipleexploratory,constructiveand/orempiricalresearchmethodsaswellasmultipledesign/developmenttechniques(seeFigure1).Exploratoryresearchmethodsstructureandidentifynewproblemssuchasfeasibilitytesting,benchmarkingandqualitativeresearchapproaches.Constructiveresearchdevelopssolutionstoproblemsandmayincludetestingofaconstructortheoryagainstapredefinedcriteriaandmay,forexample,includeformativeevaluationtestingofaninstructionaltechnologysystem.Incontrast,empiricalresearchteststhefeasibilityofasolutionusingempiricalordirectorindirectobservationorevidenceinthetraditionofthescientificmethod.Designresearchmayemployallthreeformsofresearchmethodsaswellasincorporateformativeevaluationmethodsatdifferentphasesintheprocess.However,traditional,formativeevaluationperspectiveswhileofferingvaluableiterativeprocesses,donotinisolation,addressthecomplexityinherentineducationalpractice.Mosteducationalresearchprojectsadvocateonlyonecycleofqualitativeorquantitativeempiricaltestingatafixedpointintimeforagiveninstructionalinterventionforthesolepurposeofgeneratingknowledge.Incontrast,designresearchattemptstoprogressivelyanddynamicallygenerate(exploratoryresearch),improve(constructiveresearch)andlearnabout(empiricalresearch)aparticularphenomenonfrominterconnectedresearchanddesigncycles.

Inresponsetothischallenge,theILDFmodelattemptstoprovideacomprehensiveyetdynamicandflexibleguidingframeworkthatpositionsmultiple,microandmacrodesign


researchcyclesasprimarilysocially-constructed,contextualizedprocessofproducingthemosteducationallyeffectiveproductthathasthebestchancetobeusedintheclassroomwhilealsogeneratingknowledgeaboutteachingandlearningwithintheactivityofdesign.Themodelorframeworkattemptstomovepastisolated,individualeffortsofeducationalresearchbyclearlyarticulatingalogically-orderedstructuralframethatconsidersthefullspectrumofresearchmethodologyinadvancingtowardsystemicimpactineducationandmaybeappliedinavarietyofcontexts.Collins(1990;1993)advocatesforasimilarovert,systematicmethodologyforconductingdesignexperimentsandstates:

“When designing a learning environment, whether computer based or not, there are a multitude of design decisions that must be made. Many of these

design decisions are made unconsciously without any articulated view of the issues being addressed or the tradeoffs involved. It would be better if these

design decisions were consciously considered, rather than unconsciously made (1993, p.1).”

TheILDFprocesspresentsonesteptowardasystematicframeworkforthearticulationanddocumentationofcommonphasesandcomplementarystagesbasedonmultipledesignandresearchprocessespromotingmoreconsciousdesignresearch(Collins,1990;1999).Althoughtherearethousandsofdecisionsmadeinadesignresearchcontext,themajorconjectures,learningtargets,taskanalysis,designprinciplesandevaluationorresearchdecision-makingresultingfromexploratory,constructiveand/orempiricalresearchcyclesmaybeuncoveredbyexaminingarichcasestudyaspresentedhereentitledtheLiteracyAccessOnline(LAO)project.TheILDFispresentedhereasastartingpointforresearcherstoconsideraswiththegoalofelicitingquestions,suggestions,limitationsandcriteriathatmayneedtobeconsideredasresearchersstrugglewiththeimplicationsofthisemergingformofeducationalresearch.Inthischapter,IbrieflydescribetheprogressionoftheLAOdesign-basedresearchstudythatencompassedfouryearsofeffortandillustratestheapplicationoftheILDF.TheLAOcasestudyexampleisdescribedaccordingtobroadphasesincluding1)theinformedexplorationphase;2)theenactmentphase;3)thelocalimpactphase;and4)thebroadimpactphaseaswellasthemultiplepotentialappliedandempiricalresearchprocessesthatalignwitheachphase(seeFigure1).ItishopedthattheLAOexamplewillprovideenoughdetailtopotentiallyimproveunderstandingofconductingcyclesofdesignresearchrelatedtoatechnology-basededucationalintervention.


LiteracyAccessOnline - An integrative learning design study

TheLiteracyAccessOnline(LAO)project1providesanexampleofanintegrativelearningdesignstudybasedontheILDFspecificallyillustratingtheintersectionandsystematicexpressionofmultipledesignandresearchmethods.LiteracyAccessOnlineisanefforttoutilizeWeb-basedtechnologytoprovidesupportforteachers,tutors,andparents(literacyfacilitators)inaddressingliteracygoalsforallchildrenwithaparticularfocusonthosewithdisabilities.Afterfouryearsofdesignresearchanddevelopment,LAO(http://literacyaccessonline.com)nowprovidesatechnology-basedlearningenvironmentthatpromotestheuseofspecificliteracystrategiesfortheimprovementoftutoringandreadingperformanceasthechildandliteracyfacilitatorcollaborativelyengageintheprocessofreadingonline.

TheInformedExplorationPhaseTheexploratoryresearchobjectivesoftheLAOintegrativelearningdesignstudyweretwo-fold;1) toinvestigatethenatureandeffectivenessofaconsistenttechnology-based,

collaborativeliteracyenvironmentaswellas;2) togenerateknowledgeabouthowliteracyfacilitatorsandchildrenunderstandand

employreadingsupportstrategies.Theseobjectiveswereoriginallyconceivedasresearch/evaluationquestionsandevolvedfromanextended,progressiveinvestigationintotheprovisionofliteracysupportforfacilitatorsandchildren.This“meta-methodological”designresearchprocessconsistingofmultipleresearchmethods(e.g.survey,focusgroups,interviews,expertreviews,etc)resultedinclearlyarticulatedlearningtargets,taskanalysesoflearningobjectives,theoreticalmodelembeddedinatechnologysystemdesignandcongruentresearch/evaluationquestionsthatdrovemorerigorousqualitativetestingoftheinterventionwhoseresultscontributedtotheoryofliteracysupportforchildrenwithdisabilitiesfurtherelaboratedinsectionsbelow.Tobeginexploratoryresearchcycles,initialexplorationsintotargetaudienceandstakeholderperceptions,relatedproductsandliteratureanddocumentationofthecomplexnatureofsupportingliteracyrevealedmanyplausiblepathsfordesignresearch.TheinterdisciplinaryresearchteaminvolvedintheLAOprojectwerechargedwithdeterminingtheresearchdirectionandconsistedofeducationalresearchers,teachers,graduatestudents,contentexpertsinliteracy,specialeducationandassistivetechnologyaswellasparentsinvolvedinanadvocacygroupforchildrenwithdisabilities.Thebroaddesignresearch

1) TheLiteracyAccessOnline(LAO)projectissupportedbytheOfficeofSpecialEducationProgramsintheDepartmentofEducationSteppingstonesofTechnologyInnovationforStudentswithDisabilitiesGrantCFDA84.327A


focusevolvedfromtheteam’sperceivedlackofsupportforchildrenwhowerestrugglingwiththeliteracyprocess,basedondirectobservationsofthisprobleminbothclassroomandhomeenvironmentswhichthenmanifesteditselfintoseveralindividualbutconnectedresearchstudies.Forexample,weconductedmultipleinterviewswithparentswithchildrenwithdisabilitieswhowerestrugglingwiththereadingprocess.Wealsoinvitedseveralparentstoparticipateonourdesignresearchteam.Theteams’analysisoftheinterviewdataaswellasthedesignresearchteamdiscussionsrevealedourinitialapproach.Wehadinitiallydecidedtodesignatutorial-basedinterventiononlyforthechild’sbenefit,however,acommentinateammeetingdramaticallychangedourdesigndirection.InlinewithCollins’notionofconsciousconsiderationsofdesignasdemonstratingcoreunderlyingdesigndecision-making,oneparentmember/stakeholderontheteamstatedthatsheprimarilyreadinconjunctionwith(notto)hersonandwantedtodosoonlinebutwithadditionalsupportofhigherlevelreadingstrategies.Basedonthatinputandfollow-upmicro-cyclesofinterviewdatacollectionandanalysistoconfirmationtheviabilityofthisdesignapproachwithparents,weconductedaseriesofinterviewsandsurveystodeterminethefeasibilityofthisdesigndirection.Theresultsofourinvestigationanddiscussionsevolvedintoanonlinecollaborativeperformancesupportsystemtosupportbothliteracyfacilitatorsandtheirchildrenintheliteracyprocessasthedetermineddesigndirection.Thiswasadesigndecisionbasedondataanalyzedfrommultipleinterviewsandsurveyprocedures.AlignedwithConfreyandLachance’s(2000)notionofdrawingkeyinferencesfromdissatisfactionwithcurrenteducationalpracticesanddirectexperienceswithchildren,initialtheoreticalconjecturesweredevelopedbasedontheanalyzeddatathatadvocatedforreading,writingandassistivetechnologysupportforchildrenwithorwithoutdisabilitiestoincreasetheirengagementandperformanceinliteracy.

Whiletheseinitialtheoreticalconjecturesprovidedacentralpremiseandbroaddirectionfordesignresearch,moreinformationwasneededtorefinetheseconjecturesresultinginacomprehensiveneedsanalysisandliteraturereviewthatprovidedafirmandcomplementarytheoreticalfoundationfortheintendeddesign.Extensiveexplorationintoappropriateliteracystrategies,tutorialprogramsandprocesses,surveysofexperts,teachersandparentsaswellasqualitativeobservationofchildrenandfacilitatorsengagedinaliteracyexperienceallinformedthisphaseoftheresearch.Thisprovidednotonlywell-defineddesigndirectionsbutalsoaddedtotheresearchliteratureregardingchildrenwithdisabilitiesandtheirparentsunderstandingofassistivetechnologyandliteracylearning(seeJeffs,Behrman&Bannan-Ritland,2006).Manypotentialdesignresearchdirectionswereconsideredbasedontheinitialconjectures,however,datadrawnfromconductedinterviews,directexperiencewithpotentialresearchparticipantsandliteraturereviewconvergedandpointedtheteaminaparticulardirection.


Aprominentthemethatemergedacrossinitialinterviews,surveysandobservationswithexperts,parents,teachersandchildrenrevealedthatliteracyfacilitatorshadacrucialroleinprovidingsupportforchildrenstrugglingtogainliteracyskillsandthequestionremainedhowtobestsupportthisrole.Thesefindingsandrelatedliteratureprovidedinsightforinformedtheoryandimprovedconjecturesbasedontheaforementionedqualitativeinterviewsandliteraturereviews.Resultsfromdatacollectionandliteraturereviewmethodsintheinformedexplorationphaseindicatedthat:1) childrencan,butoftendonotuseeffectivemetacognitivereadingstrategies;2) explicitlyteachingthesestrategiescangreatlyenhancechildren’scomprehensionof

text;3) teachers(aswellasotherliteracyfacilitators)needtobetrainedinhowtoprovide

cognitivestructurefortheirstudentssothatchildrencanlearntoguidetheirowngenerativeprocessesinreading;and

4) one-to-onetutoringisoneofthemosteffectiveformsofinstructionforimprovingreadingachievementbutincreasedsuccessoftendependsupontheskillofthetutororfacilitatorandtheestablishmentofconsistentrolesandexpectations(Wittrock,1998;Wasik,1998).

Thisexplorationintotheliteratureandperspectivesofthoseinvolvedintheseissuesgreatlyrefinedourinitialtheoreticalconjecturesandresultedinadramaticchangeofourintendeddesigndirectionforthisresearchfromadidactic,tutorial,child-focusedinterventiontoacollaborative,story-basedreadingexperienceprovidingembeddedmetacognitivestrategysupportforboththeliteracyfacilitatorandthechild’suse.Therationaleforthisresearchdirectionwasdocumentedinacomprehensiveneedsanalysisthatdetailedthedatacollection,conclusionsandrelatedliteraturereview.

ThenextstageofourdesignresearchinvolvedtheanalysisanddescriptionoftherangeoflearnersandfacilitatorsthatwouldpotentiallyusetheLAOsystem.Directexperiencewith4th-8thgradechildrenwithorwithoutdisabilities,teachers,tutors,andparentsprovideddatathatcharacterizedouraudience.Thesedescriptionsweredepictedasrolemodels(Constantine&Lockwood,1999)orpersonas(Cooper,1999)thatcomprisedabstractcompositeprofilesofaudiencecharacteristicsgleanedfromactualinterviewsandobservationsandprovidedafocalpointfordesign.RolemodelsorpersonasaresimilartoGraueandWalsh’s(1998)qualitativevignettesthatstrivetocapturethesubstanceofasetting,personoreventtocommunicateacentralthemeofqualitativedata,basedonmultipledirectobservationsandareemployedhereasalsoafocalpointfordesign.Exploringthenatureoftheidentifiededucationalproblem,relatedproductsandliteratureaswellascreatingandrefiningtheoreticalconjecturesanddescriptionsoftheaudienceprovidedaninformedperspectiveforgroundeddesignofalearningenvironmentbasedon


articulatedtheory.Theseactivitiesresultedinspecificresearchartifactsincludinganeedsanalysisthatcontainedanextensiveliteraturereview,anarticulatedandcongruentdesignandresearchdirectionanddetailedaudienceanalysisbasedonqualitativeandquantitativedata.ThesedocumentswerehousedonaprojectWebsitethatprovidedacommunicationmechanismbetweenteammembersaswellasanarchiveofshareabledesignresearchprocesses,productsandevidentiarydata.

TheEnactmentPhaseTheembodimentoftheresultsofourinformedexplorationandtheoriesaboutprovidingliteracysupportforchildrenandliteracyfacilitatorsinausablelearningenvironmentwerecollaborativelyconstructedacrossseveralstagesandconstructiveresearchcyclesthatdevelopsolutionstoproblemsculminatinginaWeb-basedprototype.TheinitialdesignoftheLAOlearningenvironmentresulteddirectlyfromthedesignimplicationsarticulatedinthepreviousphaseofexploratoryresearch,analysesandreview.Theseimplicationsweretranslatedintoanarticulatedprototypeinitiallydevelopedbybuildinganabstract,paper-basedmodelofthesystemforresearcherandteacherinputaccordingtoproceduresadaptedfromusage-centereddesignprocessespreviouslymentionedasrolemodels(Constantine&Lockwood,1999).Rolemodelsareatechniquetocharacterizeprimaryandsecondarytargetaudiencesforthepurposesofdesign.Forexample,wecreatedrolemodelsandpersonasforchildrenwithlearningdisabilities(suchasattentiondeficitdisorder)basedonourdirectexperiencewithachildwhowasstrugglinginthereadingprocessandhismotherwhodidnothaveanyknowledgeofadvancedreadingsupportstrategies.Thesetechniquesarebasedonreal-worldexperienceswithrepresentativesofthetargetaudiencesyourinterventionorsystemisbeingdesignedforbutevolveintoaarchetypalcompositeoftheattributesofmanyindividuals.Therefore,rolemodelsandpersona’sbecomeaqualitativeprofiletocontinuallytargetdesigneffortstomaintaintheaudience(s)oruser(s)perspectives.

Abstractorlow-fidelitymodeling/prototypingoftheinstantiatedorenacteddesignprovidedopportunitiesforinputandco-constructionofLAOwithseveralaudiencememberspriortothemoretime-intensivecomputer-basedproductionofthelearningenvironment.WeutilizedConstantineandLockwood’s(seeforuse.com)proceduresofusage-centereddesignthatencompassedlow-fidelityrepresentationandorganizationofallthefeaturesofthedatabase-drivenWebsite.ForLAO,wedeliberatelyultimatelydesignedaWebdatabasesystemthatwouldpermitperformancesupportfortheparent-childdyadinprovidingmeta-cognitivepromptsforbothparticipantsbasedonresearch-basedreadingstrategiesthroughoutacollaborativeandgenerativeprocessofengagingwithtext.


Inthecontextofaconstructiveresearchapproachthatattemptstovalidateaparticularconstruct(e.g.theory,model,softwareorframework)againstidentifiedcriteriaorbenchmarks,theteamconductedseveraliterativecyclesofdata-gatheringandanalysisofexpertreviewsandtargetaudiencereviews.Theseprogressive,micro-cyclesofdatacollectionandanalysisresultedindata-drivencyclicalrevisionsofthearticulatedprototypewhichwerereflectedindetaileddesigndocumentationincludingtheproductionofflowcharts,technicalspecificationsandstoryboards.Thedesignresearchprocessofemployingmicro-cyclesofconstructiveresearchdatagatheringandanalysiselicitedfeedbackateachcycleanddesignrevisionsagreeduponbytheteamwhichresultedintheinitialcreationandthenprogressiveimprovementofaWeb-basedprototypevalidatedbydatacollectedinaconstructiveresearchapproach.Asateam,weconstructedspecificcriteriarelatedtousabilityofsystemandobservationsandvideoanalysisofactualuseofthesystembyliteracyfacilitatorsandchildren.ThespecificmethodsofdatacollectionemployedatthisstageincludeddesignerlogspostedontheprojectWebsite,expertpanelreviewsofthedesignanddocumentedreviewsofthedesignbycontentexperts,audiencemembersandtheresearchteam.

TheLocalImpactPhaseOnceaphysicalWeb-basedprototypewasinplace,theincorporationofformativeevaluationandqualitativemethodsinanempiricalmannercouldcommenceandbegantocharacterizetherich,highlyiterativenatureofthelocalimpactphaseasitprogressivelyinformed,revisedandrefinedourtheoreticalconstructsaswellastheWeb-basedinstructionaldesignapproachandredesignefforts.Thecomplexinteractionsbetweenfacilitatorsandchildrenthatcanoccurinmultiplesettingsformedtheseriesofmicro-cyclesinLAOexaminingthesespecificconstructsthatgroundedrelatedresearchquestions:1)parent-childdyadsinaninformalsettingwithextensiveinvolvementbyresearchers;2)parent-childdyadsinastructuredworkshopexperiencesupportedbyresearchersand;3)pre-service-teacherdyadsinafieldtrialprogressingtowardmorecloselymodelingauthenticconditionsexperiencedwiththeprototype.Whenafullyfunctioningprototypewasnotyetavailable,studiesattemptedtocloselymimicthetasksthatwouldbeembeddedinLAO.Thedatagatheringacrossthesethreestudiesincorporatedobservations,interviews,childandparentjournalentries,videotapeduseofsystemandpre-andpost-onlinesurveys(seeJeffs,Behrmann&Bannan-Ritland,2006).Thismulti-tiered,multi-methodevaluationschemegeneratedusefulknowledgeandsubsequentresultsfromeachstageofinquirywerethencycledintochangesofourtheoreticalconjectures,researchdesignaswellassystemdesign.Thisprocessrevealedinsightsintothecoredesignprinciples(vandenAkker,etal.,2006)thatmaysupportthecollaborativelearningandimplementationofmetacognitiveprocessesbyliteracyfacilitatorsandchildreninatechnology-basedenvironment.


OurcoredesignprinciplesthatevolvedandwererefinedincludedthefollowingthatwhenengagedinacollaborativeliteracyprocessthatprovideshighlevelreadingstrategymetacognitivesupportinaWeb-basedcontext:1) parentliteracyfacilitatorscoulddevelopgreaterawarenessandskillinimplementing

readingactivitiesandidentifysupportsfortheirchildinastructuredsetting;2) childrenshowedimprovementinliteracyskillsusingtechnology-basedsupportwhen

participatinginaguidedworkshopenvironment;and3) pre-serviceteachersfeltthatthestrategiesandactivitiesembeddedintheLAO

environmentfacilitatedchildren’scomprehension,motivationandinterestwhenworkingwiththeminthisenvironment(seeJeffs,et.al.2006).

MorerigorousevaluationsareplannedforthefuturetosystematicallyincreasingnumberofparticipantsandvaryingcontextsfortheuseofLAOinschool,homeandtutoringenvironments.Thesestudiesinvolvedetailedtrackingofcomputer-basedactivitiesofthedyadsinschoolandhomesettings,assessmentoffacilitatorandchilduseofmetacognitivestrategiespriortousingLAOandpre-andpostcomprehensionmeasuresafterseveralweeksofusingthesystem.

Inaddition,aseriesofexpertreviews,usabilitytesting,one-to-to-one,smallgroup,andfieldtestingwereimplementedinprogressivelymoreauthenticsettings.Themoreintensivestudiesfocusedprimarilyonqualitativestudiesthatcharacterizedthetargetaudienceinteractionwiththeenactedtheoreticalmodelinrelationtothelearningtargets.Specifically,theteamwasinterestedinhowfacilitatorsandlearnersperceivedandinteractedwithWeb-basedsupportinthecollaborativeliteracyprocess(whichincludedbothreadingandwritingtasks).ApilotstudywasinitiallyconductedthatsimulatedsometaskswithinLAOandprovidedfeedbackontheemergingsitewithfivedyadsofmostlyparentfacilitatorsandonesiblingfacilitator.Methodsincludedcollectingdatathroughsemi-structuredinterviewsandobservationsofparent-childinteractionwiththeprototypeandcomplementaryassistivetechnologies(e.g.text-to-speech,etc.)thatpromotedinreadingandwritingactivities.

ThepreliminarystudyrevealedthatthechildrenweremotivatedtocompletereadingandwritingactivitiesontheWebandthatfacilitatorsdevelopedawarenessforimplementingreadingactivitiesinacollaborativeprocessbutdesiredadditionalsupportforchildren’sdisabilities.WhiletheWeb-basedactivitiesandsupportsforthereadingprocesswereusefulforprovidingmoreauthenticandself-initiatedreadingandwritingactivities,theresearchalsorevealedthatinteractionbetweenparentandchilddyadsduringtheseactivitiesoftencreatedtensionsthatwerenotpresentwhenchildrenwereworkingwithnon-familymembers.RevisionstothetheoreticalmodelandenacteddesignofLAObased


onthiscycleofevaluationincludedamongothers,behavioralpromptsdirectedtowardtheparent-childdyadtopotentiallyreleasetension(suchaspromptstotakeabreak,positivereinforcementtechniques,etc.)whenengagedincollaborativereadingandwritingtasksandadditionalreadingstrategysupportsandactivities.

Tofurtherinvestigatetheenactedtheoreticalmodel,afollow-upsmallgroupqualitativestudywasconductedwitheightparent/childdyadsthatrepresentedavarietyofskilllevelsanddisabilities(Jeffs,2000).Thespecificgoalsofthiscycleofresearchwastoidentifythecharacteristicsofparent/childdyadsworkingtogetherspecificallyinliteracyskilldevelopment,depicttheinteractionsofthedyadandinvestigatetheimpactofvariousformsoftechnology(Internet,EPSSandanyassistivetechnology)onattitudesoftheparticipants.Participantsincludedparentsandchildrenwithvariousdisabilitiesingrades4ththrough6thwhowerereadingatleasttwogradesbelowgradelevelandhadatendencytoavoidreadingandwritingtaskspriortoparticipationinthestudy.Thestudyrevealedthatparentsrecognizedtheimportanceofimmediatefeedbackandassistivetechnologyfeaturesintheprovidedtools.Otherresultsrevealedthatwiththesupportoftheirparents,childrencanselectappropriatetechnologiesandwithintegrateduseoftheInternetandassistivetechnologies,children’swritingsamplesimprovedinbothquantityandquality.SuggestedrevisionsfortheLAOprototypebasedontheseresultsincludedbuilt-inassistivetechnologyfeatures(insteadofmerelyreferencestooutsideresources)suchastext-to-speechcapabilitiesandreadingselectionsreflectingvaryingabilitiesandareasofinterest–featuresthatweresubsequentlyincorporatedintotheLAOdesign.

Ineachofthesecyclesofproblem-state,datacollection,analysisandsubsequentdesignmoveorformativeevaluationprocess,thetheoreticalmodelenactedwithintheLAOprototypeexpandedtoincorporatenewandrevisedelementsbasedontargeteddatacollectionandresearchresults.Atthispoint,traditionalresearchanddesignprocessessomewhatdivergeinthattheanalyzedresultsarenotanendinandofthemselves,butareusedfordata-drivendecisionmakingorproblemsolvingtobuilduponorrevisetheoreticalassumptionsandimprovedesign.Often,basedontestingresults,wewouldneedtothrowoutpreviousprototypefeaturesandtotallyredesign,reviseoraddnewfeatures.Theteam’sinformeddesignjudgmentandcollaborativesocialnegotiationwaskeytothisdecision-making.

Thelocalimpactphaseisatime-intensivephasewithmultiplecyclesthatstrivestoyieldausableandinternallyvalidintervention.Testingtheinterventioninprogressivelymorerealisticsettingsprovidesvaluableinformationtoinformtheoreticalassumptionsrelatedtothedesignbutalsotobegintoisolatevariablesthatmightbefurtherempiricallytested.IntheLAOresearchconductedtodate,theintegrationofreadingstrategyscaffoldsand


assistivetechnologysupportsinthecollaborativeliteracyprocessbetweenfacilitatorsandchildrenwitharangeofdisabilitieswasidentifiedasonefactor,ofmany,thatseemtoholdpromiseforimprovingliteracyskills.Conductingadditionalresearchtofurtherinvestigatethecollaborativeprocesspromotedbythetechnologicalenvironmentaswellasisolatingtheeffectsofthemultiplereadingsupportsandassistivetechnologiesaffordedbytheprototyperemainsanimportantobjectiveinthisresearch.

AlthoughthefundingcycleforLAOhasceased,inordertoprogressfromlocaleffectstomoreexternallygeneralizableeffects,additionalcyclesoftestingareneededtoisolateandtestparticularvariablesusingmultiplesites,diverseparticipantsandsettingsprogressivelylimitingtheresearcher-participantinteraction.Basedonavailablefunding,fieldtestsortrialsareplannedforLAOtocollectsignificantamountsofquantitativeandqualitativedatafromseveralsitesandover50participantdyadsusingselectedmeasurements,onlinesurveysandinterviewsincludingparentsandchildreninhomeschoolenvironments,pre-serviceteachersandin-serviceteachersthatcouldrepresentotherliteracyfacilitatorsinseveralgeographicallocationsinteractingwithchildrenwitharangeofdisabilities.Thisdatawouldprovideadditionalevidencefortheeffectivenessofenactedtheoreticalassumptionsforthecollaborativereadingandliteracyprocessaswellasprovideevidencefortheeffectivenessoftheprototypeatitshighestfidelityinfullcontextoftheintendeduse.

TheBroadEvaluationPhaseThelastphaseofthisdesign-basedresearcheffortinvolvesdisseminatingLAOintothebroadeducationalsystem.AlthoughtheLAOresearchhasnotyetfullyprogressedthroughthisstage,initialexplorationsinthisareahaveyieldedsomeuniqueinsightsintothedisseminationprocess.However,thereadershouldnotethatthedisseminationprocesscanencompassanentireresearcheffortinitself.Forexample,Fishman(2006)hasappliedadesignresearchframeworkrelatedtothesustainabilityoftechnology-basedcurriculuminterventionswithinanentireschooldistrictorsystem.LAO,asaWeb-basedlearningenvironment,affordstheopportunitytopublishcurrentworkingprototypesonlineforopenuseandinputthathasresultedinanearlyanduniquediffusionandadoptionprocessbegunpriortothecompletionofafullyfunctioningsystem.Whilestillindevelopment,wehavetrackedover100potentialadoptersthathavediscoveredandexploredtheLAOsite.Theprofilinganddata-basecapabilitiesofthesitepermittrackingandanalysisofthisinformationthathasprovideddetailedinformationonpotentialadoptersofthesystemprovidingsignificantinsightandimpactonsourcesforourlaterdiffusionefforts.Weplantoincorporatemoresophisticatedcomputer-baseddatacollectionandanalysistechniquessuchasdatamining(Tsantis&Castellani,2001)thatmayyieldevenmoreinsightsintoearlyadopters’behaviors,profilesanduseofthisnewtool.We


havejustbeguntopublishourresultsofthedesignbasedresearchconductedrelatedtoLAOintraditionalacademicjournalsandnon-traditionalWebpublishingthatprovideavenuesforadditionalformsofreviewandevaluation.TheresultsofourinitialstudieshavepromptednewresearchdirectionssuchasexploringtheinteractionofanonlinecommunityforparentsofchildrenwithdisabilitiesincorporatedintheLAOenvironment.Giventheiterativenatureofthistypeofresearch,itishighlylikelythatdeterminingtheconsequencesoftheLAOdesignresearcheffortwillyieldnewtheoreticalandappliedquestionsthatwillprompttheentireprocessonceagain.

Theoretical yield of literacy access online design research study

GiventhedesignresearchprocessbasedontheIntegrativeDesignLearningFrameworkdescribedabove,whatdidwelearn?ThecharacteristicsofaninterventionorasvandenAkker,et.al.(2006)describethe“designprinciples”areanimportantyieldofdesignresearch.IntheLAOproject,thesedesignprinciplesincludedprovidingmetacognitivereadingstrategysupportwhileaparent,teacherortutorisengagedinthecollaborativereadingprocesswiththechilddeliveredthroughacomprehensiveWeb-basedperformancesupportsystem.

Designresearchisoftenemployedtobegintogeneratetheory(Design-basedResearchCollective,2003).WithLAO,therewerenoliteraturesources,theoreticalprinciplesorresearchstudiesdirectlyapplicabletoaWeb-supportedcollaborativereadingprocesssotheteamintegratedinsightsfromtutoring,readingstrategiesandreal-timeperformancesupport.Zaritskyet.al.(2003)speaktogoing“…beyondsimpledevelopmentofaninterventionandbeyondstandardcognitiveanalysesallowingtheoryandmodelingthataccountsforthecontent,thecognitionandtheenactmentbyrealpeopleinrealandrichcontextswithreallimitsonresources(p.11).TheLAOdesignresearchteamwentbeyondtraditionaldevelopmentwithintensivecyclesofinterviews,surveys,observationalstudiesaswellasdeepinvestigationoftheone-on-onetutoringandreadingstrategiesliteraturetobuildanewtheoreticalmodelofreal-timemetacognitivereadingstrategyandassistivetechnologysupportforboththeliteracyfacilitatorandthechildwithdisabilities.Muchoftheseinsightswereanintegrationofdataanalyses,directexperiencewithtargetaudiencemembersandagroundedliteratureinreadingprocesses,tutoringandcollaborativeperformancesupport.Thedesignresearchprocesswasconductedsystematicallyto:1) uncovertheinitialconjecturesabouthowlearningmightoccurinthistypeofsetting;2) statedlearningtargets,taskanalyses(inthiscasebasedonActivityTheory);3) thedesignedinterventionwhichembodiesthecoredesignprinciples(metacognitive

readingstrategysupportinacollaborativeperformancesupportcontext);


4) localimpactorevaluationquestionsthatdrovethemoreintensiveresearchcycles(seeFigure2).

Thisprogressiondemonstratesanalignmentorcongruencyfrominitialconjecturesthroughlocalimpactorevaluationquestionsthatevolvedduringthedesignresearchstudy.ThespecifictheoreticalinsightsthatweretestedandrevealedbasedonthisprocessareincludedinFigure2.ThemultiplephasesoftheIDLFprocessuncoveredmanyinformalandformaltheoreticalinsightsbasedonmacroandmicrodatacollectionandanalysiscyclesconductedwithintheprocessofdesignthatcanbetypicallyoverlookedinthetraditionalinstructionaldesignprocess.Forexample,extendingbeyondatraditionallearneranalyses,weconductedmultiplecyclesofsurveys,interviews,andobservationsoftargetaudiencememberinteractionthatrevealedtheoreticalinsightsthatgobeyondjustthedesignoftheintervention.Ourstudiesrevealedthatparentshavelittleformalknowledgeanduseofgoodreadingstrategieswhenengagedwiththeirchildinthereadingprocess.ThisinsightparlayedintothedesignprinciplesofLAObutalsostandapartfromitasafindingthatmaycontributetotheliteratureinthereadingfield.Byformalizingandextendingthemethodsoftraditionalinstructionaldesigntopromoterichcyclesofdatacollectionthatthencaninformourknowledgeofparticularaudiences,learningcontextsandprocesses–separatebutconnectedtothedesignofaparticularintervention,wecanbegintoprogresstowardgeneratingknowledgeandusefultheoreticalinsightsthataretypicallyoverlookedindesign.Thisbecomesaninformation-lossprocessoflearningaboutlearners,contexts,andprocesseswithintheactofdesignthatdesignresearchcanrecapture,whichreferstothenotionthatinthecontextofbothisolateddesignandresearchefforts,wedonottakeadvantageofformalizingmuchofourlearninginanexploratory,confirmatoryorempiricalmanner(Bannan-Ritland&Baek,2008).


Figure 2: Initial Progressive Formulation from Conjectures to Local Evaluation Questions in LAO

ConjecturesLearning Targets

Task Analysis of Learning Task,

(based on Activity Theory)

Designed

InterventionLocal Evaluation

Questions

Provinding a consistent environm

ent and reading support strategies for literacy facilitators

and children would collaboratively

engage the dyad in higher level literacy processes

1) Literacy facilitators w

ill acknowledge the im

portance of and dem

onstrate their ability to im

plement reading strategies w

hen provided technology-based support

in a collaborative reading session w

ith a child

2) Children with of w

ithout disabilities w

ill demonstrate their

abilities to access information,

activities and assisitive technology support related to reading as w

ell as interact w

ith literacy facilitators in a collaborative reading session

3) The facilitator-child dyad will

be able to explore and select appropriate assistive technology intergrated w

ith Internet-based supports that can facilitate

performance in reading and w

riting

4) Children, regardless of disability w

ill be able to capitalize on technology-based supports and a collaborative process to im

prove their literacy skills

Subject: facilitator-child dyad

Object: LAO

system

Tools: literacy strategies, assisitive technologies

Division of Labor

Comm

unity: dyads in school, hom

e and tutoring contexts

Theoretical Model Em

bedded in Electronic Perform

ance Support System

(EPSS)

How

do facilitators and learners perceive and interact w

ith W

eb-based support in the collaborative literacy process?

What are the specific

characteristics of parent/child dyads w

orking toward literacy

skill development?

What im

pact does various forms

of technology support have on the attitudes of facilitators and children w

hen engaged in the collaborative literacy process?

How

should the interaction betw

een literacy facilitators and children w

hen engaged in the collaborative literacy process

be depicted?


LimitationswillcertainlyalsoexistfortheILDFframework,astheknowledgegeneratedisonlyasgoodastherigoroftheresearchmethodsemployed.Integratingbasicqualitativeandquantitativeresearchcyclestoinformdesignatparticularpointsandgeneratebothdesignprinciplesbutalsoknowledgeaboutlearners,learningandlearningcontextsistheultimategoal.Limitationsmayexistintime,qualityofinformationuncoveredindatacyclesthatmayimpactdesign,smallNtoprovidemostlyqualitativeinsightsinitiallyandthefailureinherentinthegenerationoftheoryinthediscoveryresearchprocess.However,itisthroughapplicationindifferentdesignresearchcontextsthatmoreformalizedprocesseswillbegintobeunveiled.TheIDLFandLAOexampleareonecaseofafewcurrentlyfordesignresearcherstouncoverthelogicandwarrantsofthisnewformofresearch(Kelly,2006).Therearemanychallengesthatremainbutcapitalizingonthedesignprocesstogenerateresearch-baseddata-driveninsightsisaworthygoal,indeed.

Conclusion

ThischapterhaspresentedabriefexampleandintroductiontotheILDFframeworkthatcomprisesameta-methodologicalviewofthedesignresearchprocessinanattempttoelucidatecommonphasesandstagesinthisspecificresearchmethodology.Theframeworkispresentedtobegintoestablishcommonterminologyandprocessesthatcanpromoteconsciousdesignresearch.Mostimportantly,theILDFframeworkisanattempttoprovidearoadmapforfuturedesignresearcherstoinvestigate,articulate,documentandinformeducationalpractice.

* IamgreatlyindebtedtoDr.AnthonyE.Kellywhoseinsightsandfeedbackonthischapterwereinvaluableinextendingmythinkinginthisarea.MyappreciationalsogoestoDr.TjeerdPlomp,Dr.NienkeNieveenandDr.JanvandenAkker,esteemedcolleaguesandreviewersofthismanuscriptfortheirsuggestionsforrevision.

References

Akker,J.vanden,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational design research.London:Routledge.

Bannan-Ritland,B.&Baek,J.(2008).Investigatingtheactofdesignindesignresearch:Theroadtaken.InA.E.Kelly,R.A.Lesh,&J.Baek(Eds.),Handbook of design research methods in education: Innovations in science, technology, mathematics and engineering.Mahway,NJ:Taylor&Francis.

Cooper,A.(1999).The inmates are running the asylum: Why high-tech products drive us crazy and how to restore the sanity.Indianapolis,IN:SAMS.


Collins,A.(1990).Toward a design science of education. Technical Report Issue No. 1 Center for Technology in Education.NewYork,NY.ResourcesinEducation(ERIC)ED326179.

Collins,A.(1993).Design issues for learning environments.TechnicalReportNo.27CenterforTechnologyinEducation,NewYork,NY.ResourcesinEducation(ERIC)EDED357733.

Collins,A.(1999).Thechanginginfrastructureofeducationresearch.InE.C.Lagemann&L.S.Shulman(Eds.),Issues in education research: Problems and possibilities (pp.289-298).SanFrancisco:Jossey-Bass.

Confrey,J.&Lachance,A.(2000).Transformativeteachingexperimentsthroughconjecture-drivenresearchdesign.InA.E.Kelly&R.ALesh(Eds.),Handbook of research design in mathematics and science education (pp.231-265).Mahway,NJ:LawrenceErlbaumAssociates.

Constantine,L.L.&Lockwood,L.A.(1999)Software for use: A practical guide to the models and methods of usage-centered design.Addison,WesleyLongman,Inc.

TheDesign-BasedResearchCollective(2003).Design-basedresearch:Anemergingparadigmforeducationalinquiry.Educational Researcher 32(1) 21-24.

Fishman,B.,Marx,R.,Blumenfeld,P.,Krajcik,J.S.&Soloway,E.,(2004).Creatingaframeworkforresearchonsystemictechnologyinnovations.Journal of the Learning Sciences, 13(1),43-76.

Graue,M.E.&Walsh,D.(1998).Studying children in context.ThousandOaks,CA:SageJeffs,T.(2000).Characteristics, interactions, and attitudes of parent/child dyads and their use

of assistive technology in literacy experience on the internet.Unpublisheddissertation,GeorgeMasonUniversity,Fairfax,VA.

Jeffs,T.,Behrmann,M.&Bannan-Ritland,B.(2006).Asssistivetechnologyandliteracylearning:Reflectionsofparentsandchildren.Journal of Special Education Technology 21(1), 37-44.

Kelly,A.E.,Lesh,R.A.&Baek,J.Y.(Eds).(2008).Handbook of Design Research Methods in Education Innovations in Science, Technology, Engineering, and Mathematics Learning and Teaching.NewYork:LawrenceErlbaumAssociates.

Kelly,A.E.(2006).Qualitycriteriafordesignresearch.In:J.J.H.vandenAkker,K.Gravemeijer,S.McKenney,&N.Nieveen(Eds.).Educational design Research.London:Routledge.

Kelly,A.E.,Baek,J.,Lesh,R.,&Bannan-Ritland,B.(2008).Enablinginnovationsineducationandsystematizingtheirimpact.InA.E.Kelly,R.Lesh,andJ.Baek(Eds.),Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching.NewYork:Routledge.

LiteracyAccessOnlinePhaseFourProjectWebSite(2000).RetrievedAugust30,2002,fromGeorgeMasonUniversity,ImmersionProgramWebSite:http://chd.gse.gmu.edu/immersion/lao/deliver/needs.htm

Rogers,E.M.(2003).Diffusion of innovations.NewYork:TheFreePress.Sabelli,N.(personalcommunication,May15,2002).


Tessmer,M.(1993).Planning and conducting formative evaluations.London:KoganPage.Tsantis,L.&Castellani,J.(2001).Enhancinglearningenvironmentsthroughsolutionbased

dataminingtools:Forecastingforself-perpetuatingsystemicreform. Journal of Special Education Technology, 16(4).

Wasik,B.A.(1998).Usingvolunteersasreadingtutors:Guidelinesforsuccessfulpractices.Reading Teacher,51(7),562-70.

Wittrock,M.C.(1998).Cognitionandsubjectmatterlearning.InN.M.LambertandB.L.McCombs(Eds.),How Students Learn(pp.143-152).Washington,DC:AmericanPsychologicalAssociation.

Zaritksy,R.,Kelly,A.E.,Flowers,W.,Rogers,E.,&O’Neill,P.(2003).Clinicaldesignsciences:Aviewfromsisterdesignefforts.Educational Researcher, 32(1), 32-34.


4. WhenisDesignResearchAppropriate Anthony E. Kelly

Introduction

Designresearchhasbeendescribedindetailinmanypublications,mostrecentlybytheDutch(VandenAkker,Gravemeijer,McKenney&Nieveen,2006,withe.g.,Kelly,2006),andtheKelly,LeshandBaek(2008)collectionofpapers(e.g.,Kelly,Lesh,Baek&Bannan-Ritland,2008;Middleton,Gorard,Taylor&Bannon-Ritland,2008).Plomp(chapter1ofthisbook)alsoprovidesanoverview.Forthatreason,Iwillnotreiteratethedescriptionhere.Rather,Iwillassumethatthereaderisfamiliarwiththesesourcesandthespecialissuesofjournals(e.g.,Barab&Squire,2004;Kelly,2003,2004)thathaveappeared.

Instead,Iwishtoplacedesignresearchwithintheframeofalargercontextforresearchoninterventions.Inherseminalpiece,Bannan-Ritland(2003)describedaportfolioofresearchactivitiesusingthefollowingcategories:

• InformedExploration• Enactment• Evaluation:LocalImpact - Quasi-experimentaldesigns - Randomizedtrials - HierarchicalLinearModeling• Evaluation:BroaderImpact - Implementationinnewcontexts(DesignandResearch) - ImplementationatScale - ScalingupDesignandResearch - Web-enabledprotodiffusion - DiffusionofInnovations(Rogers)• Adoption,adaptation,acceptance,rejection

Ofcourse,thislargerframeworkcallsformanydifferentresearchmethods.Inhispaper(Plomp,chapter1),brieflycapturesthefunctionsofresearchmethods:

• survey:todescribe,tocompare,toevaluate• case studies: todescribe,tocompare,toexplain• experiments: toexplain,tocompare• action research:todesign/developasolutiontoapracticalproblem


• ethnography:todescribe,toexplain• correlational research:todescribe,tocompare• evaluation research: todeterminetheeffectivenessofaprogram

HethenprovidesexamplesrelatedtotheChinesecontext:

1. to describe:e.g.whatistheachievementofChinesegrade8pupilsinmathematics;whatbarrierstostudentsexperienceinthelearningofmathematicalmodelling

2. to compare:e.g.whatarethedifferencesandsimilaritiesbetweentheChineseandtheNetherlandscurriculumforprimaryeducation;whatistheachievementinmathematicsofChinesegrade8pupilsascomparedtothatincertainothercountries

3. to evaluate:e.g.howwelldoesaprogramfunctionintermsofcompetencesofgraduates;whatarethestrengthsandweaknessesofacertainapproach;etc

4. to explain or to predict:e.g.whatarethecausesofpoorperformanceinmathematics(i.e.insearchofa‘theory’predictingaphenomenonwhencertainconditionsorcharacteristicsaremet)

5. to design and develop:e.g.whatarethecharacteristicsofaneffectiveteachingandlearningstrategyaimedatacquiringcertainlearningoutcomes;howcanweimprovethemotivationoflearners.

BothBannan-RitlandandPlompprovideabroadercontextforresearch.Withinthislargerframework,wemayask,therefore:Whenisdesignresearchappropriate?Wemayapproachananswerbyasking,first,whenisdesignresearchinappropriate?

When is design research inappropriate?

Areviewofthemanypublishedexamplesofdesignresearch(e.g.,Kelly,Lesh&Baek,2008)demonstratetheheavyinvestmentoftimeandresourcesnecessarytomakeprogressinthefaceofsometimesdauntingcircumstances.Designresearchrequiresinvestmentofsubstantialresourcesatmanylevels:schooldistrictadministrators,teachers,students,andthedesignresearchteam(whichmayincludeeducationresearchers,softwaredevelopers,curriculumspecialists,andsoforth).Thus,designresearchisinappropriateiftheeducationalproblemisfairlysimple.

Iftheproblemhasaknownorstandardsolution,andthereisgeneralagreementonwhentoapplythesolution,andthesolutionhasbeenregularlysuccessfullyappliedinvarioussettings,designresearchisprobablyapooruseofresources.


Evenformorechroniclearningproblemssuchasreading,ifthereareadequatetrainingprograms,andclearmeasuresofsuccessorprogress(e.g.,useofphonicstoteachdecodingskills),designresearchisprobablynotindicated.If,however,newresearchsuggestsapowerfulinnovation,designresearchmaybeareasonablechoice(seebelow,andMcCandliss,Kalchman&Bryant,2003)

Generally,designresearchisprobablynotrecommendedforclosedproblems(e.g.,improvingmathematicscalculationfluency),wherethe:

• Initialstate(s)areknown(e.g.,twonumbersaretobemultiplied;achessboardisreadytoplay).

• Goalstate(s)areknown(e.g.,aproductoftwonumbersistoproduced;checkmateorstalemateinchess).

• Operatorstomovefrominitialstatestogoalstatesareknownandcanbeapplied.(e.g.,theproceduresofmultiplication;therulesofchess).

When is design research appropriate?

Designresearchisrecommendedwhentheproblemfacinglearningorteachingissubstantialanddauntinghow-to-doguidelinesavailableforaddressingtheproblemareunavailable.Further,asolutiontotheproblemwouldleadtosignificantadvancesinlearningoratleastasignificantreductioninmalfunctionintheeducationalsystem.

Thereshouldbelittleagreementonhowtoproceedtosolvetheproblem,andliteraturereviewstogetherwithanexaminationofothersolutionsappliedelsewhere(i.e.,benchmarking)shouldhaveprovenunsatisfactory.Designresearchisfurthersuggestedifpriortrainingorinterventionshaveconsistentlyprovenunsuccessful.Designresearchisoftenindicatedforcriticaleducationalgoals,evenwhenthereisnotacleardefinitionofsuccess,ordesigningadequateindicatorsofsuccessispartoftheoverallproblem.

Inotherwords,designresearchismostappropriateforopen,ormoreappropriately,wickedproblems.TheconceptofawickedproblemwasdescribedbyRittelandWebber(1977)todescribeproblemsthatsharethefeaturesofopenproblems,butthatalsoengageelementsthatmaketheirsolutionfrustratingorpotentiallyunattainable.


Followingfromthedescriptionofclosedproblems,above,inopenproblems,someormoreofthefollowingapply:

• Initialstate(s)areunknownorareunclear.• Goalstate(s)areunknownorareunclear.• Operatorstomovefrominitialstatestogoalstatesareunknownorhowtoapplythe

operatorsisunclear.

Forwickedproblems(e.g.,Camillus,2008;Horn&Weber,2007;Richey,2007),thecharacterofopenproblemspertain.Plus,therearetypicallyinadequateresources,unclear“stoppingrules”(conditionsthatindicateasolutionisathandortheprojectshouldbeabandoned),uniqueandcomplexcontexts,andinter-connectedsystemicfactorsthatimpingeonprogress.Mostfrustrating,theseotherfactorsmaythemselvesbesymptomsofproblemsofassociatedwickedproblems.Forexample,attemptingtoteachnumeracyinasocietywithhighpovertyandHIVrates.

Therefore,oneofthebroadgoalsofdesignresearchistodynamicallyclarifytheinitialandgoalstatesandtheoperators,andtoilluminatethenatureoftheproblem–i.e.,to“tame”awickedproblembybetterspecifyingitscharacterandmakingitopentointervention.Ineducationalsettings,designresearchisrecommendedwhenoneormoreofthefollowingconditionsoperatetomaketheproblemmorewickedandopenthansimpleandclosed,forexample:

• Whenthecontent knowledgetobelearnedis new or being discoveredeven by the experts.

• Whenhowtoteachthecontentisunclear:pedagogical content knowledge is poor.• Whentheinstructional materialsarepoorornotavailable.• Whentheteachers’ knowledgeandskillsareunsatisfactory.• Whentheeducational researchers’ knowledgeofthecontentandinstructionalstrategies

orinstructionalmaterialsarepoor.• When complex societal, policy or political factors may negatively affect progress.

AnumberofexamplesofmaybefoundinKelly,LeshandBaek(2008).Someotherexamplesfrommathematics,science,andreadingarebrieflypresentedinthenextsection.

Examples from mathematics, science and reading

Thissectionpresentsbrieflyanumberofexamplesofwhenapplyingdesignresearchistheappropriateresearchapproach.


1. IntroducingExistingScienceorMathematicsatEarlierGradeLevelsForexample,someeducationauthoritieshaveadvocatedtheteachingofalgebrainearliergrades(asearlyasthe8thgradeintheUS),seeFoundationsforSuccess:ReportoftheNationalMathematicsAdvisoryPanel(http://www.ed.gov/about/bdscomm/list/mathpanel/index.html).Afewpolicymakershaveevenadvocatedstartingalgebrainstructionintheearlyelementary.

Howshouldoneproceedtointroduceideasofalgebraicreasoningintheearlyelementarygrades?Isthisrecommendationadvisable?Thisissueclearlymeetsthecriteriasetout,above.SomeofthecomplexitiesassociatedwithansweringthisquestioncanbegaugedbyreadingsomeoftherecentworkonthistopicbyCarraherandcolleagues(e.g.,Carraher&Schliemann,2007;Carraher,Schliemann,&Schwartz,2007;Peled,&Carraher,2007;Schliemann,Carraher&Brizuela,2007).

2. Learningneworemergingsciencecontent(e.g.,genetics)Researchinmicrobiologyisinrevolutionwithstunningfindingsappearingonfrontpagesofnewspapers,almostdaily,worldwide.Howcanhistoricalscienceeducationbeupdatedtopreparehighschoolteachersandstudentstomeetthischallengeandopportunity?Moreover,howcanhighschoolspreparestudentstobesuccessfulinemergingintegratedbiologyprogramssuchastheoneatPrincetonUniversity(http://www.princeton.edu/integratedscience/)?RutgersUniversityhasexploredthischallengethroughitsmicrobiologyprogram(http://avery.rutgers.edu/WSSP/Begin/index.html).Areviewofitsvariedsolutionstothischallengeexemplifiesthisrichcontextfordesignresearch.

3. UncoveringthePotentialContributionsofNeuroscienceforMathematicsLearningTheauthorhasjoinedotherwriters(e.g.,Varma,McCandliss&Schwartz,2008)inoutliningthecaseforcultivatingtheintersectionofneuroscienceandmathematicslearning.(e.g.,Kelly,2002,2008).

Whyisthereagrowinginterestinneuro-mathematicseducation?Anumberoffactorshavecoincidedtosupportasurgeininterestinbrain-basedmathematicseducationresearch(seeOECD,2007foracomprehensivereviewofbrain-relatedresearchineducation):• Confidenceduetorecentgainsinunderstandingthebrainbasesforprocessesof

decodinginreading.• Emergentfindingsintheneuralbasesformathematicalthought.• Decadesofbehavioralandcognitivesciencefindingsonlearningmathematicsand

relatedhigher-orderprocessesfromwhichtodraw.


• Adesiretodisambiguateandconstrainresearchhypothesesatthebehavioral,cognitiveandsociallevelsofanalysis.

• Adesiretosharpenandgrounddiagnosisandremediationofmathematicallearningdifficultieswithimprovedassessments.

• Adesiretoconstructnewmixed-methodsresearchmethodologiesforthesocialsciences.

• Adesiretoscientificallydebunklearningandteaching“neuromythologies”.• Asenseofurgencytobringscientificdiscourse,evidenceandreasoningtotheslateof

ethicalissuesthatareemergingthatpertainbothtolearningandteaching.• Agoaltoimprovemethodsofteachingofmathematics.• Agoaltoimproveeducationalmaterials,includingthosethatusecomputerhardware

andsoftware.• Morecomprehensiveandtestablemodelsoflearningemergingfromcognitivescience

(e.g.,Bruer,1997).• Adesiretounderstandandpromotesignificantmathematicalcreativity.• Tochallengeneuroscientiststocontinuetopushtheboundariesofimaging

technologies,andtoco-formulateclinicallearningtasks.Thepointtobedrawnhereisthatthecoincidenceofthesefactors,alone,doesnotdictateteachingorlearningstrategiesorevenprovideprinciples,materials,curricula,interventions,orassessmentapproachestosupporteitherlearningorteaching.How,then,shouldresearchersproceedtobringthelaboratoryfindingsofcognitiveneuroscienceintotheclassroominviableways?Again,theproblemmeetstheaboverequirementsforusingdesignresearch.

4. CyberinfrastructureCyberinfrastructureencompassestheuseofdistributedinternetresourcessuchascomputingsystems,data,informationresources,networking,digitallyenabled-sensors,instruments,virtualorganizations,andobservatories(NSF,2007).Itallowstolinkgroupsofscientiststoattackmulti-levelcomplexproblems.Theseproblemswillhaveassociatedchallengesforlearning,teaching,andassessment.Importantquestionsarehoweducationshouldcapitalizeoncyberinfrastructureresources.Whatitmeanstostudysciencecontentwithinacyberinfrastructureframework,andwhatthecurricular,instructionaldesign,assessment,teacherprofessionaldevelopment,andpolicyquestionsthatareraised,andhowtheymustbeansweredtofullyexploitthehigh-technologyinvestmentinscienceatthislevel.Asimportant,whatarethemethodologicalchallengesinstudyinglearningwithinacyberinfrastructureproject?Forexample,howareclaimsofcausalityhandledinacomplexnetworkedandnestedlearningenvironment,andwhatevidencewouldmakesuchclaimscredible(e.g.,Kelly&Yin,2007)?Thisisaclearexample,spanningmanysciencedisciplines,forwhichdesignresearchisanappropriateinvestment.


TheAppendixtothischapterdiscussesinmoredetailthemeaningandpossibilitiesofcyberinfrastructureore-scienceingeneralandforeducation.

5. ReadingandInquiryScienceThereader’sattentionisdrawntotwoexamplesfromBrendaBannan-Ritland,currentlyatGeorgeMasonUniversity.Heranalysisofhowdesignresearchworkswithinherintegrativelearningdesignframework(Bannan-Ritland,2003;Bannan-Ritland&Baek,2008;seealsopaperinthisvolume)providesexamplesinnarrower,ifnolessimportant,applications.

Forexample,designresearchisappropriatewhendevelopingcreativeorinnovativeeducationalproducts,blueprintsordesignsthataredirectedatchroniceducationalproblems.Inanumberofpapers(seeLiteracyAccessOnline,Bannan-Ritland&Baek,2008;http://immersion.gmu.edu/lao/spring2003/projectResources.htm),andBannan-Ritland’schapter(thisvolume)describestheprocessesundertakentoaddressachronicprobleminmostcountries,howtoteachreadingtostrugglingreaders.

FollowingherworkonLiteracyAccessOnline,Bannan-Ritlandextendedherworkonreadingdesignintothelearningofinquiryscienceatthe4thgrade.Basedonthisexperience,Bannan-Ritlandsignificantlyaddedtothebroadeninguseofdesignresearchprinciplesbymethodologicallyincorporatingteachersasdesignersintheoveralldesignresearchparadigm.Thisexcitingnewdirection,calledteacherdesignresearch(whichdovetailswithworkbyZawojewskietal.,2008),isdescribedinBannan-Ritland(2008).Theareaofapplicationinthereportisearthsciencesintheearlyelementaryschool.

The growing need for design assessment research

Arecentreviewofcontributionstodesignresearchshowanincreasingawarenessoftheneedfortacklingtheproblemofhowtoassesslearninginemergingareasoflearning,particularlywhenthereisanemphasisoninnovationininstructionalpractices(Kelly,Baek,Lesh&Bannan-Ritland,2008).Theynote:

In design research as currently practiced, assessment is not directed at some summative sense of learning, though a summative measure of student

learning would be central to later attempts at confirmatory studies, i.e. to show local impact (Bannan-Ritland, 2003). . . . Design research also differs from forma-

tive assessment with regard to the student’s knowledge end state and how feedback loops are enacted. Formative assessment is the gathering of data

relative to some predetermined fixed point, providing feedback that informs the students and teacher of their current knowledge state in relation to some


end state (see Black & Williams, 1998). In design research, assessment may be used formatively in order to dynamically determine progress toward mastery of disciplinary knowledge (e.g., Cobb & Gravemeijer, [Kelly, Lesh & Baek, 2008])

or to guide the design of a prototype and to inform its iterative re-design as necessary or both. In fact, sensitivity to assessment practices themselves may

inform changes to the act of assessment itself (e.g., Lobato, [Kelly, Lesh & Baek, 2008]; Lesh et al., [Kelly, Lesh & Baek, 2008]). Ultimately, design researchers are

challenging the assumptions about learning, teaching, and knowing that underlie available assessment techniques, not only in terms of the psychome-tric assumptions (like item response theory), but also the function of assess-

ment itself within and across the stages of design research (see Sloane & Kelly, [Kelly, Lesh & Baek, 2008]).

Inotherwords,whenasuitablecontextfordesignresearchisidentified,totheextentthattheapplicationisnovel(e.g.,teachingalgebraicconceptsintheearlyelementarygrades,readingcomprehension)ortheknowledgeunfolding(e.g.,genomics,cyberinfrastructure),therewillbearequirementandaresponsibilityforresearchersnotonlytoiterativelyinvestigatetheimpactoflearningprototypes,butalsotoaddressdirectlythequestionofhowthisimpactwillbemeasured.Thepointhereisnotthatassessmentisnecessary,ratherthatthetargetsforassessmentmayarisedynamicallyinthecourseofdesignresearchandmeasuresmaynotbeavailableapriori.Asaresult,manyofthequestionsaboutthevalidityandreliabilityofmeasureshavetobeactivelyreconsidered.Inpractice,toooften,prototypesareredesignedwithoutspecifyingtheevidencebase(viaassessmentdesign)fortheredesign.Inmanycases,designresearchersappeartorelyonjudgmentorsubjectivefactors.Addingtotheunfoldingneedfornewmethodsindesignassessmentwillbeamajorchallengeandopportunityforscholarsinthenextdecade(e.g.,Kelly,2005a,2005b).

Whatistheevidencetosupportclaimsofeffectivenessduringiterations,andlater,astheinnovationissubjecttomorerigoroustests?

Design research in general practice

Thegoalofthischapterwastocharacterizedesignresearchatabroadlevel,andtoprovidesomeexamplesofwherethesignificantresourcesassociatedwithdesignresearchmightbespent.Iwillfinishwithageneraloutlineofhowdesignresearchcyclesunfoldwithinalargerframeworkofresearch(Bannan-Ritland,2003;Plomp,thisvolume).Usingcognitivescience,cognitivepsychologyandothersocialsciencemethodssuchassurveys,casestudies,clinicalinterviews,ethnography:


• Identifyorcharacterizetheinitialstates.Clarifytheinitialknowledgeandgoalknowledgestates(ofstudents,teachers,researchers,experts)usingtheinterventions.

• Identifyorcharacterizethegoalstates.Designformativeassessmentstomonitorprogresstowardthegoalstate.

• Identifyorcharacterizetheoperators.Dynamicallyusingthecognitiveandotheranalyses,iterativelydesignandspecifytheoperators(interventions,supports,environments)tosupportlearning.See,inparticular,theworkofBannan-Ritland(2008)andZawojewskietal.,(2008).

• Informre-designcyclesoriterationsusingdatagatheredfromunfolding,andparallelworkindesignassessment.

• Worktowarddevelopingamatureprototypethatcanbesubjecttoamoredefinitivetest(e.g.,randomizedclinicaltrial),seeBannan-Ritland’s(2003)localimpactphase.

One final note: Prototyping and theory building

Byperturbingthesystemusingtheinterventionsinthisiterativeresearchprocess,designresearchtranscendseachofthelocalmethodsused.Inotherwords,designresearchinvolvesnotonlytheuseofdifferentmethods(e.g.,surveys,casestudies,clinicalinterviews),butcombinesthefruitsofeachmethod,overtime,tospecifytheoryandmodelsrelatedtolearning,teachingandassessingthetargetknowledge(seeCobb&Gravemeijer,2008).Thus,designresearchgoes beyond simple development of an interventionandgoes beyond standard cognitive analysesandallowstheoryandmodelingthataccountsforthecontent,thecognition,andtheenactmentbyrealpeopleinrealandrichcontextswithreallimitsonresources(seeZaritskyetal.,2003).

Thequestionofthe“theoreticalyield”ofdesignresearchisnotasimpleone.Notethatthischapterwasframedintermsofcomplex,openandwickedproblems.Forsuchproblems,thereexistsnosimpletheoreticalmodel(atleastnoneisperceivedatthetime).Forthatreason,if“theory”issomethingthatisassumedtobeinformedbyhypothesistestingofasomewhatdefinitivequestion,thendesignresearch(inearlystages)willlikelynotposeoreasilyanswersimplehypotheses,andthusnothavesimpletheoreticalyield.Schwartz,ChangandMartin(2008;inKelly,Lesh&Baek)viewsthedesignresearchcyclesaspreparatoryfortheoreticalyieldfromlaterrandomizedclinicaltrialsorotherlaboratorytests.Iftheobservationisborneoutthatmuchofeducationalinterventionoccursincomplexsystems,thenthetheoreticalyieldwillnotbeassociatedwithonetheory,butmany(perhapsinterdependent)subtheories.Ifso,thentheyieldmaybediffuseandobfuscatedbytheinfluenceofmanyfactorsthatarenotcontrolledindesignresearchsettings.Someresearchershaveattemptedtoframedesignresearchwithinanoverarchingtheory(say,“variation”theory,Holmqvist,Gustavsson,&Wernberg,2008).Thepay-offofthisapproachwillinformusgreatlyabouttheroleoftheoryindesignresearch.


Somewritersusetheword“theory”moregenerallytoencompass“designprinciples,”anditmaybethecasethatsuchprinciplescanindeedbeidentified(seeKali,2008).Suchrecommendationsfordesignpracticeareusefulheuristics.Iftheseheuristicsshowevidenceofdurableapplicabilityacrossmanyprojectsandcontexts,itislikelythatsomenecessary(asopposedtocontingent)principlesarebeingevoked(seeKelly,2004),whichwouldopentheseheuristicstotheoreticalanalysis.

References

Akker,J.J.H.vanden,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds.)(2008).Educational design Research.London:Routledge.

Bannan-Ritland,B.(2003).Theroleofdesigninresearch:Theintegrativelearningdesignframework.Educational Researcher 32(1) 21-24.

Bannan-Ritland,B.(2008).Teacherdesignresearch:Anemergingparadigmforteachers’professionaldevelopment.InA.E.Kelly,R.Lesh,andJ.Baek(Eds.),Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching.NewYork:Routledge.

Bannan-Ritland,B.,&Baek,J.Y.(2008).Investigatingtheactofdesignindesignresearch.InA.E.Kelly,R.Lesh,andJ.Baek(Eds.),Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching.NewYork:Routledge.

Barab,S.&Squire,K.(2004).Design-basedresearch:puttingastakeintheground.Journal of the Learning Sciences, 13(1),1-14.

Camillus,JohnC.,“StrategyasaWickedProblem,”HarvardBusinessReview,May2008.Carraher,D.W.&Schliemann,A.D.(2007).Earlyalgebraandalgebraicreasoning.InF.Lester

(ed.)Second handbook of research on mathematics teaching and learning: A project of the National Council of Teachers of Mathematics.VolII.Charlotte,NC:InformationAgePublishing,pp.669-705.

Carraher,D.W.,Schliemann,A.D.&Schwartz,J.(2007).Earlyalgebraisnotthesameasalgebraearly.InJ.Kaput.D.Carraher,&M.Blanton(Eds.),Algebra in the early grades. Mahwah,NJ,Erlbaum,pp.235-272(nowTaylor&Francis).

Chin,G.,Jr.,&Lansing,C.S.(2004).Capturingandsupportingcontextsforscientificdatasharingviathebiologicalsciencescollaboratory,Proceedings of the 2004 ACM conference on computer supported cooperative work,409-418,NewYork:ACMPress.

Cobb,P.&Gravemeijer,K.(2008).Experimentingtosupportandunderstandlearningprocesses.InA.E.Kelly,R.Lesh,andJ.Baek(Eds.),Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching. NewYork:Routledge.


Cogburn,D.L.(2003).HCIintheso-calleddevelopingworld:what’sinitforeveryone,Interactions,10(2),80-87,NewYork:ACMPress.

Holmqvist,M.,Gustavsson,L.,&Wernberg,A.(2008).Variationtheory:Anorganizingprincipletoguidedesignresearchineducation.InA.E.

Kelly,R.Lesh,andJ.Baek(Eds.),Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching. NewYork:Routledge.

Horn,RobertE.,Knowledge Mapping for Complex Social Messes,apresentationtothe“FoundationsintheKnowledgeEconomy”conferenceattheDavidandLucilePackardFoundation,July16,2001.

Kali,Y.(2008).TheDesignPrinciplesDatabaseasmeansforpromotingdesign-basedresearch.InA.E.Kelly,R.Lesh,andJ.Baek(Eds.),Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching. NewYork:Routledge.

Kelly,A.E.(2003).Theroleofdesigninresearch.Educational Researcher, 32(1),3-4.Themeissueguesteditor.

Kelly,A.E.(2004).Designresearchineducation:Yes,butisitmethodological?Journal of the Learning Sciences, 13(1),115-128.

Kelly,A.E.(2005a).“How do modeling perspectives inform program assessment?”Presentationatthe27thannualmeetingoftheNorthAmericanChapteroftheInternationalGroupforthePsychologyofMathematicsEducation.Roanoke,VA.

Kelly,A.E.(2005b).“Theintersectionofdesignandassessmentformathematicslearning.”Presentation,NationalScienceFoundationWorkshop,ParkCity,UT.

Kelly,A.E.(2006).Qualitycriteriafordesignresearch.In:J.J.H.vandenAkker,K.Gravemeijer,S.McKenney,&N.Nieveen(Eds.).Educational design Research.London:Routledge.

Kelly,A.E.(2008).Brainresearchandeducation:Potentialimplicationsforpedagogy.InEducation, sciences cognitives et neurosciences.Paris:PressesUniversitariesdeFrance.

Kelly,A.E.,Baek,J.,Lesh,R.,&Bannan-Ritland,B.(2008).Enablinginnovationsineducationandsystematizingtheirimpact.InA.E.Kelly,R.Lesh,andJ.Baek(Eds.),Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching. NewYork:Routledge.

Kelly,A.E.&Yin,R.(2007).Strengtheningstructuredabstractsforeducationalresearch:Theneedforclaim-basedstructuredabstracts.Educational Researcher.133-138.

McCandliss,B.D.,Kalchman,M.,&Bryant,P.(2003).Designexperimentsandlaboratoryapproachestolearning:Stepstowardcollaborativeexchange.Educational Researcher, 32(1),14-16.

Middleton,J.,Gorard,S.,Taylor,C.,&Bannan-Ritland,B.(2008).The“compleat”designexperiment:Fromsouptonuts.InA.E.Kelly,R.Lesh,andJ.Baek(Eds.),Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching.NewYork:Routledge.


NationalScienceFoundation(2007).Cyberinfrastructure vision for 21st Century discovery.OfficeofCyberinfrastructure.March.

OECD(2007).Understanding the brain: The birth of a learning science.Paris:OECD.Olson,G.M.,Teasley,S.,Bietz,M.J.,&Cogburn,D.L.(2002).Collaboratoriestosupport

distributedscience:theexampleofinternationalHIV/AIDSresearch,Proceedings of the 2002 annual research conference of the South African institute of computer scientists and information technologists on enablement through technology,44–51.

Peled,I.&Carraher,D.W.(2007).Signednumbersandalgebraicthinking.InJ.Kaput.D.Carraher,&M.Blanton(Eds.),Algebra in the early grades.Mahwah,NJ,Erlbaum,pp.303-327(nowTaylor&Francis).[608k]

Richey,Tom;Wicked Problems: Structuring Social Messes with Morphological Analysis,SwedishMorphologicalSociety,lastrevised7November2007

Schliemann,A.D.,Carraher,D.W.,&Brizuela,B.(2007).Bringing out the algebraic character of arithmetic: From children’s ideas to classroom practice.LawrenceErlbaumAssociates(nowTaylor&Francis).

Schwartz,D.L.,Chang,J.,&Martin,L.(2008).Instrumentationandinnovationindesignexperiments:Takingtheturntowardsefficiency.InA.E.Kelly,R.Lesh,andBaek(Eds.),Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching. NewYork:Routledge.

Thomas,J.J.,&Cook,,K.A.(Eds.),(2005).Illuminating the path: The research and development agenda for visual analytics. LosAlamitos,CA:IEEEComputerSociety.

Varma,S.,McCandliss,B.D.,Schwartz,D.L.(2008).Scientificandpragmaticchallengesforbridgingeducationandneuroscience.Educational Researcher, 37(3),140-152.

Wulf,W.(1989,March).Thenationalcollaboratory.InTowards a national collaboratory. UnpublishedreportofaNationalScienceFoundationinvitationalworkshop,RockefellerUniversity,NewYork.

Zaritksy,R.,Kelly,A.E.,Flowers,W.,Rogers,E.,&O’Neill,P.(2003).Clinicaldesignsciences:Aviewfromsisterdesignefforts.Educational Researcher, 32(1),32-34.

Zawojewski,J.,Chamberlin,M.,Hjalmarson,M.A.,&Lewis,C.(2008).Developingdesignstudiesinmathematicseducationprofessionaldevelopment:Studyingteachers’interpretivesystems.InA.E.Kelly,R.Lesh,andJ.Baek(Eds.),Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching. NewYork:Routledge.


Appendix

FortheUSNationalScienceFoundation(NSF,2007),theopportunitiesinthecomplementaryareasthatmakeupcyberinfrastructure:computingsystems,data,informationresources,networking,digitallyenabled-sensors,instruments,virtualorganizations,andobservatories,alongwithaninteroperablesuiteofsoftwareservicesandtoolsprovidechallengesalongthreelines:(a)data,dataanalysis,andvisualization;(b)virtualorganizationsfordistributedcommunities;and(c)learningandworkforcedevelopment.

AmajorparallelactivityincyberinfrastructureisunderwayinEurope,whichislabeled“e-science”.e-sciencedescribessimilaractivitiestotheUScyberinfrastructure.NotunlikeearlyvisionsofUScyberinfrastructure,theUKlaunchingdocument,(http://www.nesc.ac.uk/documents/OSI/report.pdf),didnotexplicitlylisteducationasoneofthekeyareasofconcerninsettingupacyberinfrastructure.Itfocused,rather,onnetworks,middleware,digitallibraries,andcomputationalresources.AsintheUS,thisimbalanceisbeingrecognized.InEurope,itbeingaddressedbythecreationofICEAGE:“Theinternationalcollaborationtoextendandadvancegrideducation”(http://www.iceage-eu.org/v2/partners.cfm).ICEAGE,whileinternational,isprimarilyaEuropeaneffort,withbranchesinEdinburgh,Scotland,UniversityofCatania,Sicily,SPACI(SouthernPartnershipforAdvancedComputationalInfrastructures),anItalianuniversity-basedeffort(http://www.spaci.it/),CERN,nearGeneva(http://public.web.cern.ch/Public/Welcome.html),theRoyalInstituteofTechnologyinSweden(http://www.kth.se/?l=en_UK),andTheComputerandAutomationResearchInstitute,HungarianAcademyofSciences(http://www.sztaki.hu/institute).

Cyberinfrastructuredescribestheuseofdistributedinternetresourcestolinkgroupsofscientiststoattackmulti-levelcomplexproblems.Theseproblemswillhaveassociatedchallengesforlearning,teaching,andassessment.Forexample,adesignresearchproblemwouldbehowtodescribeandcreditastudent’slearninginacyberinfrastructureresearchcollaboratoryingeosciences:Scientifically,acrucialconcernindetectingearthquakesistomeasureminutechangesinelevation.Traditionalradar,whichusesradiowavesasthemeansofdetectingdistancesfromthesource,areoflimitedvalueinprecisemeasurementsduetothelengthoftheradiowaves.TheuseofLiDAR(LightDetectionandRanging)technologyallowstheuseofwavelengthsintheultraviolet,visible,ornearinfraredrange(fromabout10micrometerstotheUV(ca.250nm).Theseshorterwavelengthsallowdetectionofsmokeandotherdiffuseparticulates,whichhasledtotheuseofLiDARinmeteorology.Forearthquakeprediction,LiDARcanbeusedtolocatefaults,andtomeasureuplift.Faultsdescribethelineoffractureanddemarcationbetweenplates(McKnight&Hess,2000).


Upliftistypicallyduetotectonicplateactivity(Kearney&Vine,1990),technically“orogenicuplift”orduetotheremoval(duetoerosion)ofheavymaterial,technically“isostaticuplift.”ThesignificantadvantageofLiDARoverradaristhatLiDARcangeneratedigitalelevationmodels(DEMs)oftheshapetheearth’ssurfaceatresolutionsnotpreviouslypossible.Complexifyingtheproblem,anearthquakeissometimesassociatedwithvolcanicactivity.Forexample,the“PacificRimofFire”isassociatedwithcollidingtectonicplates.Insuchcases,LiDARmaybeusednotonlytomakeprecisemeasurementsofelevation,butalsotocharacterizethedensityandeventhechemicalmakeupofthegasesandashemittedbyavolcano.LiDARdataonMountSt.Helen’svolcanomaybefoundathttp://wagda.lib.washington.edu/data/type/elevation/lidar/st_helens/.

LearningaboutgeomorphologyusingLiDARiscomplex,andsomepubliclyavailablewebsiteshaveattemptedtoprovideinstruction(e.g.,http://lidar.cr.usgs.gov/andhttp://gisdata.usgs.net/website/lidar/viewer.php).ThemostcomprehensiveactivityhasbeenconductedbytheGEONnetwork(http://www.geongrid.org/).Thisnetworkispartofacyberinfrastructureresearchcollaborator.TutorialsontheuseofLiDARwithinandoutsideofgeoscience(e.g.,coastalerosion,flooding,rivercourses,forestmappingandmining)maybefoundhere;http://home.iitk.ac.in/~blohani/LiDAR_Tutorial/Airborne_AltimetricLidar_Tutorial.htm.

WecannowseejustafractionoftheassociatedscientificconceptsthatarepertinentinunderstandingtheuseofLiDARinunderstandinggeoscience:e.g.,radartechnologyvsLiDARtechnology,thescienceofplatetectonics,digitalelevationmodels,readingandunderstandingcomputervisualizations,modelingcomplexinter-relatedscientificprocesses,reasoningaboutimplicationsforhumanactivity,includingurbangrowth,andsoforth.Whichofthese(orotherrelatedconcepts)aremostpertinentforscientistsinacyberinfrastructureresearchcollaboratorywillbeanempiricalquestion.Howtoidentifythecentralconstructspertinenttoahigh-schoolscienceeducationwillprovideasignificantmeasurementchallenge,includinghowtodesignauthenticassessmentstomeasureunderstandingoftheseconcepts.Identifyingandmappingoutthecontentandcognitivedemandsofsuchmeasurementcouldbeamajorfocusofthedesignresearchwork.Ofparticularinterestwillbehowtoestablishcontent,construct,predictive,concurrentandotherformsofvalidityforthesemeasures.

Factorsconvergingtosupportthedevelopmentofcyberinfrastructure.

1. ExistingcomputingdatagridsintheUSandoverseas a. TheTeraGridproject(http://www.teragrid.org/about/)combinesthepowerofNCSA,

SDSC,ArgonneNationalLaboratory,CACR,PSC,ORNL,TACC,andvariousuniversity


partnersintegratedbytheGridInfrastructureGroupattheUniversityofChicago.Europeane-sciencelinksfacilitiesontheContinentwiththoseintheUK.SimilaractivitiesoccurinJapan.IndustrypartnersincludeIBM,Intel,Hewlett-PackardandOracle.

2. Theavailabilityofmassivedatastoragecapacityandspeed a. TheTeraGridcurrentlyoffersover100teraflopsofcomputingpower;andover3

petabytesofrotatingstorage3. Thedevelopmentofmiddlewareandsoftwaretogatherandanalyzestoreddata a. TheTeraGridsupportsdataanalysisandvisualizationproductioninterconnectedat

10-30gigabits/second.4. Theemergenceoflargeteamsofscientistsdedicatedtosolvingsharedscience problems(actingthroughscience“collaboratories”and“gateways”) a. Acollaboratory(Wulf,1989)is“morethananelaboratecollectionofinformationand

communicationstechnologies;itisanewnetworkedorganizationalformthatalsoincludessocialprocesses;collaborationtechniques;formalandinformalcommunication;andagreementonnorms,principles,values,andrules”(Cogburn,2003,p.86).Collaboratoriesexistinmanyareasofscience,includingbiology,chemistry,geoscienceandastronomy(e.g.,Chin&Lansing,2004;Olson,Teasley,Bietz,&Cogburn,2002).

b. Sciencegatewaysareweb-basedportalsorinterfacesforthestructuresanddataofthecyberinfrastructureinmanyscienceareas(foralistingof24gateways,seehttp://www.teragrid.org/programs/sci_gateways/).

5. Developmentsinscientificvisualization. Scientificvisualizationdrawsonhumanspatialandvisualprocessinginordertomodel

andanalyzecomputationallyintensethegraphicdisplayofcomplexdata(foracomprehensivereview,seeThomas&Cook,2005).Existingmethodsandmodelsforscientificvisualizationaresignificantlychallengedbycyberinfrastructure(e.g.,http://www.teragrid.org/userinfo/data/vis/vis_gallery.php;Chinetal.,2006).

6. Funding. Theestablishmentandfundingofnationalandinternationaleffortstocoordinateand

developtheinfrastructuretobetterservescienceand,morerecently,education(e.g.,theOfficeofCyberinfrastructure–NSF;CERN,Dutch(VL-e)andUKinitiatives).ThepromiseofcyberinfrastructureforeducationisthatthevastinvestmentbyUSagencies(upwardsof$250Moverthenext5years,alone)willprovidetest-bedsforexploration.


5. FormativeEvaluationinEducational DesignResearch Nienke Nieveen

Introduction

Inthischapterandinlinewiththegeneralintroductionofthisbook,wedefineeducationaldesignresearchas:the systematic study of analyzing, designing and evaluating educational interventions in order to solve complex educational problems for which no ready-made solutions are available and to gain insight in key design principles.Designresearchprojectsstriveaftertwotypesofmainresults.Thefirstaimcompriseshigh-qualityinterventions(suchasprograms,productsandprocesses)designedtosolvecomplexeducationalproblems.Thistypeofoutputputsforwardthepracticalrelevanceofdesignresearch.Itisforthatreasonthatdesignresearchisalsolabeledasbeinguse-inspired,appliedorientedand/orsociallyresponsibleresearch(vandenAkker,1999;Reeves,2000).Thesecondmainoutputofdesignresearchistheaccompanyingsetofwell-articulateddesignprinciples(Linn,Davis&Bell,2004;vandenAkker,1999)thatprovideinsightinthe:• purpose/functionoftheintervention;• keycharacteristicsoftheintervention(substantiveemphasis);• guidelinesfordesigningtheintervention(proceduralemphasis);• itsimplementationconditions;• theoreticalandempiricalarguments(proof)forthecharacteristicsandprocedural

guidelines.Thesecomprehensivedesignprinciplesserveseveralpurposesforavarietyoftargetgroups.Fromaresearchperspective,theseprinciplesshowthecontributionofdesignresearchtotheexistingknowledgebasewithinformationonhowtheinterventionworksinpractice,theeffectsofusingtheinterventionandexplanationoftheworkingmechanisms.Foreducationaldesigners,theseprinciplescarryrichinformationonhowtodesignsimilarinterventionsforsimilarsettings.Fromtheperspectiveoffutureusers,theprinciplesprovideinformationneededforselectingandapplyinginterventionsinthespecifictargetsituationandprovideinsightsintherequiredimplementationconditions.Finally,forpolicymakers,theseprinciplesassistinmakingresearch-baseddecisionsforsolvingcomplexeducationalproblems.

Inordertoreachbothtypesofoutput(highqualityinterventionsanddesignprinciples),designresearcherscarefullycombinedesignandresearchactivitiesresultinginaniterativedevelopmentapproach.Inthiscontributionwewillfirstexplorethisiterativenatureofdesignresearch(herelabeledwiththetermprototypingapproach)andthenelaborateon


therolethatformativeevaluationplaysindesignresearchprojectsinordertooptimizeinterventionsanddesignprinciples.Thecontributionwillendwithsomeremarksontheroleofdesignresearchersconcerningformativeevaluationactivities.

Prototyping approach

Designresearchisbynaturehighlyiterative(Design-basedresearchcollective,2003;vandenAkker,1999).Eachiterationhelpstoimproveprototypesofbothendresultsofdesignresearchefforts:1.theeducationalinterventionunderdevelopment;and2.itsaccompanyingtentativedesignprinciples.Thissectiondiscussesbrieflythenotionofaniterativeorprototypingapproach.

Aprototypeisapreliminaryversionofthewholeorapartofaninterventionbeforefullcommitmentismadetoconstructandimplementthefinalproduct.Prototypesmaybeusedintwoways(cf.Smith,1991).Ontheonehand,aprototypemaybecontinuallyrefined(basedonformativeevaluationresultsandreflectionsofdevelopersontheprototype)andevolvetowardsafinaldeliverable.Thisrefiningapproachcanbereferredtowiththetermevolutionaryprototyping.Ontheotherhand,developerscandesignthrow-awayprototypes,suchasscenariosorpaper-basedmock-ups(Nieveen,1999).Ascenarioisanarrativedescriptionoftypicalandcriticalsituationsthatprospectiveusersparticipatein.Scenariosmaybeusedtomakethetentativedesignspecificationsmoreconcrete.Thismakesiteasiertocommunicatethepotentialsofasystemwiththetargetgroup.Apaper-basedmock-upcomprisesapileofpapersrepresentingallscreenswhichmayappearduringtheuseoftheintervention.Thiskindofprototypeisoftenusedinsoftwaredevelopmentprojects.Usersmay‘walkthrough’thescreenstogetanideaoftheintentionsofthesoftwareapplication.Paper-basedprototypesfocustheattentionoftheusermoreoncontentandoverallstructurethanonappearance.Afterbeingevaluated,athrow-awayprototypewillbediscardedanditsevaluationresultsaretakenintoaccountinthenextprototype.Thisprocesswillcontinueuntilalluncertaintiesarecoveredandthefinalproductorinterventioncanbedelivered.

Especiallyindesignresearchprojectsthataimatinnovativeandcomplexproducts,withfewexperiencesordesignprinciplesfromwhichtodraw,suchaprototypingapproach(eitherevolutionaryorthrow-away)isrecommendable.Somedesignresearchprojectscombinebothkindsofprototypes,forinstancebyfirstdesigningandevaluatingthrow-awayprototypesandthenshiftingtoanevolutionaryapproach(Nieveen,1999).Tomaketheprototypingapproachwiththrow-awayand/orevolutionaryprototypesfeasible,thenotionof‘thinkbig,butstartsmall’ishelpful.Byfirstdevelopingasmallpartoftheproposedintervention,onekeepsthedevelopmentprocessmanageableandonecan


learnfromfailuresandapplysuccesseswhendesigningthesubsequentparts.Inordertokeepanoverviewontheentiredevelopmentprocessitisoftenfunctionaltodecomposetheinterventionintoseveralcomponentsthatcouldbebuiltseparately.Educationalinterventionscanbedecomposedintoatleasttwokeyaspectswhichwillrequiremajorattentionduringthedesignprocess(cf.Nieveen,1999;Nieveen&vandenAkker,1999):• theconceptual framework oftheintervention,referringtoallnotionsthatare

underlyingtheintervention.Incaseofaneducationalintervention,itrefersforinstancetotheconceptualizationofall10curriculumcomponentspresentedinthecurricularspiderweb(vandenAkker,2003);

• thepresentation-modeoftheintervention,referringtotheformatthatassuresthattheinterventionisusableforitsusers.Toassistusersfindingthecontentoftheirpreference,allinterventions(paper-basedandcomputer-based)needasounduser-interfaceincludingconsistentlayoutandtransparantnavigation.

Thevariousconceptualandpresentationelementsmaybeindifferentstagesofdevelopmentineachprototype.However,towardsthefinaldeliverable,allelementsneedtobeconsistentwithoneanother.Forexample,inamathprojectaimedatsolvingproblemsoflow-achievingstudentswithmeasuringquantities,interventionsweredevelopedtohelpthesestudentstoacquiretherequiredmathematicalproblemsolvingskills.Thedesignresearchteamputmuchemphasisonthedevelopmentofinnovativelearningandteachingactivities(twocomponentsofthecurricularspiderweb/theconceptualframework)gearedtotheproblemsofthesestudents.Subsequently,thefirstversionofthelessonmaterialswasdesignedaccordingtothisspecificpedagogy.Duringtheformativeevaluationofthisfirstprototype,thedesignresearchteamwasespeciallyinterestedinthequalityofthenewlearningandteachingactivities(beingpartoftheconceptualframework)andlessinthelayout(beingpartofthepresentation-mode).However,towardstheendofthestudy,thelayoutofthematerialsgotspecificattentioninordertoimprovetheoverallpracticalityofthematerials.

Formative evaluation

Inaprototypingapproachempiricaldataareneededtogaininsightintothequalityofthetentativeinterventionanddesignprinciples.Forthatreason,formativeevaluationisacrucialfeatureofeachprototypingapproachandthusofeachdesignresearchproject.Itprovidesinsightinthepotentialsoftheinterventionanditskeycharacteristics.Resultsoftheformativeevaluationgivegroundforboth1.improvingtheprototypeoftheinterventiontowardsahigh-qualityfinaldeliverableand2.sharpeningtheunderlyingtentativedesignprinciplestowardsanelaboratedsetofdesignprinciples.Inthisway,eachprototypingcyclecontributestosuccessiveapproximationofbothoutputsofadesignresearchproject.IntheGenericDesignResearchModelofWademan(2005),seeFigure1,thisisnicelyillustratedintheprototypingandassessmentphase.


Figure 1: Generic Design Research Model (Wademan, 2005)

Inthissection,theconceptofformativeevaluationwillbefurtherelaboratedandplacedinadesignresearchcontext.Asfarasthetermevaluationisconcerned,theJointCommitteeonStandardsforEducationalEvaluation(1994)usesthefollowingdefinition:“Evaluationisthesystematicassessmentoftheworthormeritofsomeobject.”Meritreferstotheobject’sinherent,intrinsicvalue,whileitsworthisdefinedasitscontextuallydetermined,place-boundvalue(Lincoln&Guba,1979).Scriven(1967)wasthefirstauthorwhomadethedistinctionbetweenformativeandsummativeevaluation.Formativeandsummativeevaluationsservedifferentfunctions.Thefunctionofformative evaluationis‘toimprove’.Itfocusesonuncoveringshortcomingsofanobjectduringitsdevelopmentprocesswiththepurposetogeneratesuggestionsforimprovingit.Thefunctionofsummative evaluationis‘toproof’.Asummativeevaluationiscarriedouttogainevidencefortheeffectivenessoftheinterventionandfindargumentsthatsupportthedecisiontocontinueorterminatetheproject.Summativeevaluationsarebeingcarriedoutwithoutthedirectintentiontoreveal

Practitioner and User Participation

Researchers

Other Sources

Collaboratives

Practitioners

ConsultExperts &

Practitioners

FocusedLiterature

Review

AnalyzePromisingExamples

AnalyzePracticalContext

Tentative Product

Approaches

Tentative Design

Principles

Reflection

FormativeEvaluation

PracticalProduct/Results

Contributionto

Theory

Refinement of Problem, Solution and Method

Refinement of Design Theory

Preliminary investigation of Problem, Context,

& Approaches

Problem in Context

Phases

Prototyping & Assessment of Preliminary Products & Theories

Identification of Tentative Products & Design Principes

Problem Identification Tentative Products& Theories

Problem Resolution & Advancing Theory

Successive Approximation of Theory

Successive Approximation of Product

Redesign & Refinementof Products & Theories


pointsofimprovement.However,itisnotalwayspossibletodrawasharplinebetweenformativeandsummativeevaluation.Theresultsofsummativeevaluationsareusuallytakenintoaccountwhiledevelopingasecondreleaseoftheproduct.

Basedoncomparingandsynthesizingdefinitionsofvariousscholarsinthefieldofformativeevaluation(cf.Brinkerhoff,Brethouwer,Hluchyj&Nowakowski,1983;Flagg,1990;Scriven,1967,Tessmer,1993)wedefineformativeevaluationinthecontextofdesignresearchas:a systematically performed activity (including research design, data collection, data analysis, reporting) aiming at quality improvement of a prototypical intervention and its accompanying design principles.Asstatedbefore,adesignresearchprojectusuallyneedsseveraliterationsbeforeanoptimalsolutionforthecomplexproblemcanbeenreached.Eachdesignresearchcycleoriterationconcentratesonspecificresearchquestionsandneedsanappropriateresearchdesign.Theremainderofthissectionwillelaborateonissuesrelatedtotheresearchdesignofformativeevaluationactivities.

FormulatingresearchquestionsThemainresearchquestionofaformativeevaluationisbuiltaroundthekindofvaluejudgmentthatisexpectedfromevaluatingtheprototypeandtwokeyattributesoftheprototypicalintervention:1.thestageofdevelopmentoftheprototype;and2.themainelementoftheprototypethattheevaluationwillfocuson.

Firstofall,itisnecessarytomakeclearthetypeofvaluejudgmentthattheevaluationneedstoresultin.Inthisrespect,wedistinguishfourqualitycriteriathatareapplicabletoawidearrayofeducationalinterventions(seeTable1).Attheendofadesignresearchproject,theinterventionshouldsufficeallofthesecriteria.However,usuallyeachiterationconcentratesononeortwoofthesecriteria.


Criterion

Relevance (also referred to as content validity)

There is a need for the intervention and its design is based on state-of-the-art (scientific) knowledge.

Consistency (also referred to as construct validity)

The intervention is ‘logically’ designed.

Practicality ExpectedThe intervention is expected to be usable in the settings for which it has been designed and developed. ActualThe intervention is usable in the settings for which it has been designed and developed.

Effectiveness ExpectedUsing the intervention is expected to result in desired outcomes. ActualUsing the intervention results in desired outcomes.

Table 1: Criteria for high quality interventions

Itisimportanttopointheretothedistinctionbetweenexpectedandactualpracticalityandeffectiveness.Onlywhenthetargetusershavehadpracticalexperiencewithusingtheinterventiononewillbeabletogetdataontheactual practicalityoftheprototype.Similarly,onlywhentargetusershavehadtheopportunitytousetheinterventioninthetargetsetting,theevaluatorwillgetdataontheactual effectiveness.Inallotherinstances,suchasagroupdiscussionsbasedonthematerials,theresearcherwillonlygetdataontheexpected practicality and/or effectiveness.

Moreover,whenpreparingaformativeevaluationitisimportanttodescribetheboundariesoftheprototypethatwillbeevaluated.Inadesignresearchprojecta(throw-awayorevolutionary)prototypeisusuallyinoneofthefollowingdevelopmentstages:• Designspecifications:Afirstandgeneraldescriptionoftheinterventioninwhich

attentionispaidtoitssubstantiveparts.Thissketchhasbeenbasedonpreliminaryresearchactivities(includingproblemandcontextanalysisandliteraturereview).

• Globalintervention:Someorallcomponentsoftheinterventionaregivensomedetail.Thiscouldbetermedasahorizontalprototype.Itgivesanideaofhowtheinterventionwilleventuallyappear,howeveritcannotyetbeusedinpractice.Forexample,inthecaseofthedevelopmentofanewcurriculumatthisstagetheinterventioncouldtaketheformofatableofcontentswithabriefdescriptionofsub-componentsormodules.

• Partoftheinterventionindetail:Atthisstage,apartorcomponentoftheinterventionhasbeenelaboratedtoaconcretelevelforusebythetargetgroup.Thiscouldbecalledaverticalprototype.Onecanimaginevarioussub-stageswitheachofthemaddressingonlyaspecificpartofthetotalinterventionforuseinpractice.


• Completeintervention:Thetotalinterventionissufficientlydetailedthatitcouldbeusedintheintendeduser-setting.

Anotherissuethatneedstobeclearbeforestartingaformativeevaluationisthemainelementsoftheprototypethattheevaluationwillfocuson.Thesecanbealloroneelementsrelatedtotheconceptualframeworkandpresentation-modeoftheintervention.

Thethreecharacteristics(qualitycriteria,elementsandthestageofdevelopment)giveinputforthemainresearchquestions.Thesyntaxoftheseresearchquestionsis:‘Whatisthe[qualitycriteriuma,b,cand/ord]of[elementoftheconceptualframeworkorpresentationmode]oftheinterventionthatisin[developmentstagew,x,y,z].Instancesofthesequestionsare:• Whatistherelevancy[qualitycriterium]ofthecontent[conceptualelement]ofaquick

referencemanualforusingChinesecharactersthatisinaglobalstage[developmentstage]?

• Whatistheinternalconsistency[qualitycriterium]oftheattainmenttargets[conceptualelement]forscienceinuppersecondaryeducationofwhichthreeoutofsevendomainsareelaboratedindetail[developmentstage]?

• Whatisthepracticality[qualitycriterium]ofthelayout[elementofpresentationmode]oftheMathtextbookmodulesthatisinacompletedstage[developmentstage]?

SelectingappropriatemethodsDesignresearchersneedtoselectthoseformativeevaluationmethodsthatfittheresearchquestions.Buildingonearlierresearch(Nieveen,1997,1999),Table2providesanoverviewoftherelationshipbetweentheresearchquestions(withonthevericalaxisthequalitycriteriaconcerningtheelementsofaninterventionandonthehorizontalaxisthedevelopmentstages,seeprevioussection)andsuitableformativeevaluationmethods(indicatedinthecells).Herewedistinguishthefollowingmethods.PleasereferforanextensiveoverviewforinstancetoTessmer(1993)orBrinkerhoff,etal.(1983):• Screening:membersofthedesignresearchteamcheckthedesignwithsomechecklists

onimportantcharacteristicsofcomponentsoftheprototypicalintervention.• Expertappraisal:agroupofexperts(forinstance,subjectmatterexperts,instructional

designexperts,teachersreviewthematerials)reactsonaprototypeofanintervention,usuallyonthebasisofaguidelinewithcentralquestionsofthedesignresearchteam.Usuallythisisdonebyinterviewingtherespondents.

• Walkthrough:thedesignresearcherandoneorafewrepresentativesofthetargetgrouptogethergothroughthesetupoftheintervention.Usuallythisiscarriedoutinafacetofacesetting.


• Micro-evaluation:asmallgroupoftargetusers(e.g.learnersorteachers)usespartsoftheinterventionoutsideitsnormalusersetting.Here,themainactivitiesoftheevaluatorareobservingandinterviewingtherespondents.

• Try-out:alimitednumberoftheusergroup(e.g.teachersandlearners)usesthematerialsinthedaytodayusersetting.Iftheevaluationfocusesonpracticalityoftheintervention,thefollowingevaluationactivitiesarecommon:observation,interviewing,requestinglogbooks,administeringquestionnaires;iftheevaluationhasitsfocalpointontheeffectivenessoftheintervention,evaluatorsmaydecidetorequestlearningreportsand/orgiveatest.

Summativeevaluationmethods,suchas(quasi-)experiments,surveysandaccompanyingcase-studies,followtheseformativeevaluationactivitiesassoonastheinterventionhasbecomefullygrownandhasbeenimplementedineducationalpractice(seeforinstanceRossi,Freeman&Lipsey,1999).

Table 2: Table for selecting formative evaluation methods

Wheninterventionsbecomemoredetailed,thefocusoftheformativeevaluationwillgraduallyshiftwithrespecttotheaforementionedqualitycriteria.Inanearlystage,themainfocuswillbeontherelevancyandconsistencyofaprototype.Assoonasaglobalinterventionhasbeendesigned,designresearchersalsowouldliketoassesstheexpectedpracticalityoftheintervention.Whentheinterventionisevenmoreelaborated,thenthefocuswillshifttowardstheactualpracticalityandeffectiveness.Intable2,itisindicatedingreythatwiththisshiftinfocusalsoother,moresuitable,evaluationmethodswillcomeintoplay.Moreover,eachdevelopmentstagemayconsistofseveralcyclesofanalysis,designandformativeevaluationbeforetheprototypewillgrowintoanextdevelopmentstage.

Table 6: Table for selecting formative evaluation methods

Design stage

Quality criterion

Designspecifications

Global design Partly detailedintervention

Completeintervention

Implementedintervention

Relevance - Screening- Expert appraisal

- Screening- Expert appraisal



Consistency - Screening- Expert appraisal




Practicality expected - Screening- Expert appraisal


- Expert appraisal- Walkthrough

- Expert appraisal- Walkthrough

actual - Micro-evaluation - Micro-evaluation- Try-out

Survey, (Quasi)experiment, Case-study

Effective-ness

expected - Screening- Focus group

- Screening- Focus group

- Expert appraisal - Expert appraisal

actual - Micro-evaluation - Micro-evaluation- Try-out

Survey, (Quasi)experiment, Case-study


Sampling-selectingrespondentsTobeabletoanswertheresearchquestionswiththechosenevaluationmethods,therequiredtypeandnumberofrespondentsneedtobediscussed.Thetypeandsamplesizedependontheresearchquestions.Withrespecttothetypeofrespondents,oneneedstoselectthoserespondentsthatcanhelpansweringtheresearchquestions.Forinstance,incasedesignresearcherswanttogaininsightintherelevancyofthedesignfromasubjectmatterperspectivetheywillselectanumberofexpertsinthatspecificdomaintodoanexpertappraisal.Incaseinsightsareneededintheactualpracticalityofalearningpackageforlearnersbyperformingamicro-evaluation,studentsneedtobesampledwhowillhavetoworkwiththeintervention.Moreover,themainpurposeoftheevaluationalsoinfluencesthesamplesize.Incaseofaformativeevaluationduringearlystagesoftheproject,themainpurposeistolocateshortcomingsintheinterventionandtogeneratesuggestionsforimprovement(seealsodefinitionofformativeevaluation),thenumberofrespondentsislesscritical:aremarkofonlyonerespondentcouldbehighlyvaluablebecauseofitssalience.Smallsamplesofrespondentsareusuallysufficientiftheyarecarefullyselected.Samplesareusuallydeliberatelychosen(alsoreferredtoaspurposivesamplingwheresubjectsareselectedbecauseofsomecharacteristic),insuchawaythatthecommentsandreactionswillbeasinformation-richaspossible.Thismeansthatforinstancefororganizingamicro-evaluationinordertogaininsightsintothepracticalityofaprototypeofsomelearnermaterials,nexttohigh-achievingstudentsalsoagroupoflow-achievingandagroupofavaragestudentsneedtobeselected.Triangulationisimportanthereinordertoenhancethereliabilityandinternalvalidityofthefindings(cf.Miles&Huberman,1994).Onecouldtriangulatebyusingdifferenttypeofpersons,differenttimes,differentplaces.Theeffectivenessoftriangulationrestsonthepremisethattheweaknessesineachsingledatasourcewillbecompensatedbythecounterbalancingstrengthofanother.Incaseofasummativeevaluation,whenthemainpurposeofanevaluationisto prooftheactualpracticalityandeffectiveness,(quasi-)experimentalresearchdesignswithexperimentalandcontrolsettingsarerequiredwithlargesamplesizes.FormoreinformationonsamplingseeforinstanceCreswell(2008),Denscomb(2007)andMills,Gay,AirasianandAirasian(2008).

Wheninvitingrespondentsforaformativeevaluationitisnecessarytoilluminatetheirrole.Theycouldfulfilltheroleoflearner,criticand/orrevisor(Weston,McAlpine&Bordonaro,1995).Respondentswithalearnerrolearenotspecificallyexpertinthesubjectmatterwhichiscoveredbythematerials.Onecouldthinkofstudentswholearnanewsubject;butalsoteacherswhohavenottaughtinacertainmannerbefore.Inmanycasesexpertsrepresentthiscategoryaswell.Forinstance,educationaltechnologyexpertsdonotalwayshaveexpertiseinthesubjectmatterdomainoftheeducationalintervention.Theywilltaketheroleofalearnerfirst,beforetheywillgivecommentsonmattersrelatedto


educationaltechnology(inwhichtheyareexperts).Criticsarerespondentswhoareaskedtocommentonthematerialsfromtheperspectiveoftheirexpertise.Thisgroupconsists,forinstance,ofsubjectmatterexpertsandteacherswhoareinvitedtomakestatementsaboutthedifficultyorreadabilityoflearnermaterials.Revisorswillnotonlygivecommentsonthematerials(likecriticsdo),buttheywillalsoprovidesuggestionsforimprovements.Forinstance,asubjectmatterexpertmayindicatewhattypeof‘state-of-the-artknowledge’ismissinginthelearnermaterialsandwherethisknowledgecouldbefound.Itisimportanttonotethatindividualsmayplayseveralrolessimultaneouslyduringtheformativeevaluation.Thenextsectionwillelaborateontheroleoftheresearchersduringaformativeevaluation.

Researchers’ role during formative evaluation

Sinceadesignresearchprojectcomesintoplaywhenaneedarisestosolveacomplexeducationalproblemforwhichnoready-madesolutionsareavailable,oftentimesamulti-disciplinedteamisbroughttogethertoworkonit.Suchteamsusuallycompriseofexpertsindomainsthatweredistinguishedwhendecomposingtheintervention(e.g.fromaconceptualpointofview:subjectmatterexperts,pedagogicalexperts,instructionaldesigners;fromapresentation-modepointofview:user-interfacedesigners)aswellasmembersofthetargetgroup.Monk,Wright,HaberandDavenport(1993,p.5)stressthat“Itrequiresaccesstopeopletypicalofthosewhowillactuallyusethesystem,nottheirrepresentativesormanagement.”Involvingfutureusersinadesignresearchteamhasseveraladvantages(cf.Moonen,1996;Shneiderman,1992):moreaccurateinformationaboutcomplexityoftheproblemathand,moreintensivediscussionsabouttherequirementsoftheintervention,increaseofusercommitmentandownershipofthefinaldeliverable,increaseofinsightsintotherequirementsofthecontextinwhichtheinterventionwillbeused,andstimulationoftheprofessionaldevelopmentofallparticipants.

Oneofthekeyresponsibilitiesofthedesignresearchteamistoworkontheformativeevaluationoftheprototypes.Forreasonsofscientificrigor,itisoftenrecommendedtolookforexternalevaluators.However,certainlyintheearlystagesofadesignresearchprojectitseemslegitimateorevenadvisablethatdesignresearchersthemselvescarryouttheformativeevaluationoftheprototype.Engaginginformativeevaluationactivitiestendtoleadtoimportantlearningexperiencesofthedesignresearchers.Theywillexperiencethemselvestheproblemsthatoccurandhearoutoffirsthandthesuggestionsforimprovementthatrespondentscomeupwithduringtheiruseofaprototype(forexample,byobservingorinterviewingteachersorstudents).Thisusuallyhasstrongerandmore


directimpactontheirthinkinganddesignactivities,comparedtocaseswhereexternalevaluatorsreporttheresultstothedevelopers.Ofcourse,designresearchersneedtobeawareofseveralpitfallswhentheyareinvolvedintheformativeevaluationoftheinterventiontheyarealsodesigning(cf.McKenney,Nieveen&vandenAkker,2006).Theymayeasilybecometoo‘attached’totheirprototypewhichcouldleadtoalessobjectiveviewtowardproblemsandcommentsfromtherespondents.Inthisrespect,Scriven(1991)warnsofa(co-)authorshipbias.Moreover,respondentscouldbebiasedduringtheevaluation,aswell.Forinstance,iftheyknowhowmucheffortthedesignresearchteamhasputintotheprototype,theymayhesitatetobefullycriticalofit.Toovercomethesebiases,itseemsessentialtoincludeformativeevaluationearlyoninthedevelopmentprocessandtoapplytriangulationofdatasources,methods(observation,interview,questionnaires,etc.),evaluators(differentevaluators)andtheories(differentconceptualframeworks).

Closing remark

Thischapterfocusesontheiterativenatureofeducationaldesignresearch.Eachiterationorcycleofanalysis,designandformativeevaluationgivesthedesignresearchteamfirmergroundandargumentsfortheinterventiontheteamisworkingoninordertosolveacomplexeducationalproblem.Theempiricaldatatheteamcollectsduringaformativeevaluationwillnotonlyprovidesuggestionsforimprovingtheintervention,butwillalsoassistinsharpeningtheaccompanyingdesignprinciples.Proceedingthroughseveraloftheseiterationswillendinafinalstageofthescientificcycleinwhichclaimsofcausalitycanbestudiedinsummativeevaluationsettings(cf.Nieveen,McKenney&vandenAkker,2006).Inthiscontributionweconcentratedontheresearchdesignforeachformativeevaluationperformedwithinsuchaniterativeorprototypingapproach.Weelaboratedontheresearchquestions,selectionofappropriatemethodsandrespondents.Weareawarethatthereismuchmoretosayaboutformativeevaluationingeneral,andintegratedindesignresearchprojectsinparticular.Forinstance,wecouldhavepaidattentiontoevaluationinstruments,datacollection,dataanalysisandreporting.Severalhelpfulbooksandarticlesareavailabletoassistinsystematicallyconductingformativeevaluationineducation(cf.Brinkerhoff,etal.,1983;Flagg,1990;Tessmer,1993).Althoughthesesourceswerenotwrittenwiththespecificneedsandwishesofdesignresearchersinmind,theycanprovideampleinspiration.


References

Brinkerhoff,R.O.,Brethouwer,D.M.,Hluchyj,T.,&Nowakowski,J.R.(1983).Program evaluation: A practitioner’s guide for trainers and eduacators. Boston:Kluwer-Nijhoff.

Creswell,J.W.(2008).Educational research: planning, conducting and evaluating quantitative and qualitative research.UpperSaddleRiver,NJ:Pearson.

Denscomb,M.(2007). The good research guide for small-scale social research projects. Maidenhead(UK):OpenUniversityPress.

Design-basedresearchcollective.(2003).Design-basedresearch:Anemergingparadigmforeducationalinquiry.Educational Researcher, 32(1),5-8.

Flagg,B.N.(1990).Formative evaluation for educational technologies.Hillsdale,NJ:LawrenceErlbaumAssociates.

JointCommitteeonStandardsforEducationalEvaluation(1994).The program evaluation standards. ThousandOaks,CA:SagePublications.

Lincoln,Y.&Guba,E.G.(1979).Thedistinctionbetweenmeritandworthinevaluation.Paper presented at the Annual Meeting of the Evaluation Network(5th,Cincinnati,OH,September24-26,1979)retrievedOctober20,2008,http://www.eric.ed.gov/ERICDocs/data/ericdocs2sql/content_storage_01/0000019b/80/32/b5/c8.pdf

Linn,M.,Davis,E.A.,&Bell,P.(2004).Internet environments for science education.Hillsdale:LawrenceErlbaum.

McKenney,S.Nieveen,N.&Akker,J.vanden(2006).Designresearchfromacurriculumperspective.InJ.vandenAkker,K.Gravemeijer,S.McKenney,&N.Nieveen(Eds.),Educational design research(pp.67-90).London:Routledge.

Miles,M.B.,&Huberman,A.M.(1994).Qualitative data analysis: An expanded source book.London:Sage.

Mills,G.,Gay,L.R.,Airasian,P.&Airasian,P.W.(2008).Educational research: competencies for analysis and applications.UpperSaddleRiver,NJ:PrenticeHall.

Monk,A.,Wright,P.,Haber,J.,&Davenport,L.(1993).Improving your human-computerinterface: A practical technique.NewYork:PrenticeHall.

Moonen,J.(1996).Prototypingasadesignmethod.InTj.Plomp,&D.P.Ely(Eds.),International encyclopedia of educational technology(pp.186-190).Oxford:Pergamon.

Nieveen,N.M.(1997).Computer support for curriculum developers: A study on the potential of computer support in the domain of formative curriculum development,Doctoraldissertation.Enschede:UniversityofTwente.

Nieveen,N.(1999).Prototypingtoreachproductquality.InJ.vandenAkker,R.Branch,K.Gustafson,N.Nieveen,andT.Plomp(Eds.),Design approaches and tools in education and training(pp.125–36).Dordrecht:KluwerAcademicPublishers.

Nieveen,N.M.,&Akker,J.J.H.vanden(1999).Exploringthepotentialofacomputertoolforinstructionaldevelopers.Educational technology research & development, 47(3),77-98.


Nieveen,N.,McKenney,S.,&Akker,J.vanden(2006).Educationaldesignresearch:Thevalueofvariety.InJ.vandenAkker,K.Gravemeijer,S.McKenney,andN.Nieveen(Eds.),Educational design research(pp.151-158).London:Routledge.

Reeves,T.(2000).Enhancingtheworthofinstructionaltechnologyresearchthrough“designexperiments”andotherdevelopmentalstrategies.RetrievedOct.20,2006fromhttp://it.coe.uga.edu/~treeves/AERA2000Reeves.pdf

Rossi,P.H.,Freeman,H.E.&Lipsey,M.W.(1999).Evaluation: A systematic approach.ThousandOaks,CA:Sage.

Scriven,M.(1967).Themethodologyofevaluation.InR.W.Tyler,R.M.Gagné,&M.Scriven(Eds.),Perspectives of curriculum evaluation.AERAMonographseriesoncurriculumevaluation.nr.1.Chicago,MI:RandMcNally.

Scriven,M.(1991).Beyondformativeandsummativeevaluation.InM.W.McLaughlin,&D.C.Phillips(Eds.),Evaluation and education: At quarter century(pp.19-64).Chicago:UniversityofChicago.

Shneiderman,B.(1992).Designing the user interface: Strategies for effective human-computer interaction.Reading,MA:Addison-Wesley.

Smith,M.F.(1991).Software prototyping: Adoption, practice and management.London:McGraw-Hill.

Tessmer,M.(1993).Planning and conducting formative evaluations: Improving the quality of education and training.London:KoganPage.

VandenAkker,J.(1999).Principlesandmethodsofdevelopmentresearch.InJ.vandenAkker,R.Branch,K.Gustafson,N.Nieveen,andT.Plomp(Eds.),Design approaches and tools in education and training (pp.1–15).Dordrecht:KluwerAcademicPublishers.

VandenAkker,J.(2003).Curriculumperspectives:Anintroduction.InJ.vandenAkker, W.Kuiper&U.Hameyer(Eds.),Curriculum landscapes and trends(pp.1-10).Dordrecht:

KluwerAcademicPublishers.Wademan,M.(2005).Utilizing development research to guide people capability maturity

model adoption considerations. Unpublisheddoctoraldissertation. Syracuse:SyracuseUniversity.Weston,C.,McAlpine,L.,&Bordonaro,T.(1995).Amodelforunderstandingformative

evaluationininstructionaldesign.Educational training research & technology, 43(3),29-48.


6. ReferencesandSourcesonEducational DesignResearch Tjeerd Plomp and Nienke Nieveen

Introduction

Thisbibliographyhasbeencompiledtosupportresearchersandgraduatestudentsingettingaccesstokeypublicationsondesignresearch.Wedonotclaimthattheselectionofsourcesincludedinthischapteriscompleteandexhaustive–it is coloured by our background and bias as well as our knowledge and familiarity with publications.Importantcriteriaforustoincludetitlesinthisbibliographyare(i)provenusefulnessofsourcesforourownwork,and(ii)representingimportantperspectivesandgroupsthatare(orhavebeen)activelyworkinginthisdomain.Inthefirstsectionwepresentjustanoverviewofrelevantsourcesavailable.Thisisfollowedbyasectioninwhichwepresentthestructureandcontentoftheexcellentwebsite‘Design-basedResearchEPSS’(http://projects.coe.uga.edu/dbr/index.htm)–createdbyInstructionalTechnologyPh.D.studentsatTheUniversityofGeorgia(lastupdateNovember2006).Giventhequalityandcompletenessofthiswebsite(atleasttillNovember2006),wedecidedtointroduceitinaseparatesectionandintheothersectionsofthischapterwewillrefertopartsofthiswebsite,butalsointroduceanumberofothersources.Inthefollowingtwosectionswepointthereadertoselectedjournalarticlesandbookchaptersontheconceptandmethodologyofdesignresearchandondesignresearchindomainssuchascurriculum,instructionaltechnology,andthelearningofreadingandwriting,mathematicsandscience.Inthefinalsection,welisttheURLsofanumberofdoctoralthesesthathavebeendefendedinTheNetherlandsutilizingdesignresearchasaresearchapproach.Asstated,ourselectioniscolouredbyourbiasandexperience,butallthesepublicationsrefertoawiderangeofwritingsondesignresearchandwetrustthattheythereforeserveasausefulintroductiontothereader.

Overview of sources

Thissectionpresentstitlesandreferencestovariousspecialissuesofjournalsandbooksthathavebeenpublishedaboutdesign(-based)research.Besidesanumberofwebsiteswillbelisted.


Specialissuesofjournals

• EducationalPsychologist,39(4),2004 Specialissue‘Design-basedresearchmethodsforstudyinglearningincontext’,edited

byW.Sandoval&P.Bell,including: - Sandoval,W.A.,&Bell,P.L.(2004).Design-BasedResearchMethodsForStudying

LearningInContext:Introduction.Educational Psychologist, 39(4),199-201. - Hoadley,C.(2004).Methodologicalalignmentindesign-basedresearch.Educational

Psychologist, 39(4),203-212. - Sandoval,W.A.(2004).Developinglearningtheorybyrefiningconjecturesembodied

ineducationaldesigns.Educational Psychologist, 39(4),213-223. - Tabak,I.(2004).ReconstructingContext:NegotiatingtheTensionbetween

ExogenousandEndogenousEducationalDesign.Educational Psychologist, 39(4),225-233.

- Joseph,D.(2004).ThePracticeofDesign-BasedResearch:UncoveringtheInterplayBetweenDesign,Research,andtheReal-WorldContext.Educational Psychologist, 39(4),235-242.

- Bell,P.L.(2004).Onthetheoreticalbreadthofdesign-basedresearchineducation.Educational Psychologist, 39(4),243-253.

- AlsoincludescommentarybyAngelaO’Donnell.

• EducationalResearcher32(1),January/February2003 SpecialissuepreparedbyA.E.Kelly,including: - Kelly,A.E.(2003).ThemeIssue:TheRoleofDesigninEducationalResearch.

Educational Researcher, 32,3-4. - TheDesign-BasedResearchCollectiveDesign-BasedResearch(2003).AnEmerging

ParadigmforEducationalInquiry.Educational Researcher, 32,5-8. - Cobb,P.,Confrey,J.,diSessa,A.,Lehrer,R.&Schauble,L.(2003).DesignExperimentsin

EducationalResearch.Educational Researcher, 32,9-13. - McCandliss,B.D.,Kalchman,M.&Bryant,P.(2003).DesignExperimentsand

LaboratoryApproachestoLearning:StepsTowardCollaborativeExchange.Educational Researcher, 32,14-16.

- Lobato,J.(2003).HowDesignExperimentsCanInformaRethinkingofTransferandViceVersa. Educational Researcher, 32,17-20.

- Bannan-Ritland,B.(2003).TheRoleofDesigninResearch:TheIntegrativeLearningDesignFrameworkEducational Researcher, 32,21-24.

- Shavelson,R.J.,Phillips,D.C.,Towne,L.,&Feuer,M.J.(2003).OntheScienceofEducationDesignStudies.Educational Researcher, 32,25-28.


- Sloane,F.C&Gorard,S.(2003).ExploringModelingAspectsofDesignExperiments.Educational Researcher, 32,29-31.

- Zaritsky,R.,Kelly,A.E.,Flowers,W.,Rogers,E.,&O’Neill,P.(2003).ClinicalDesignSciences:AViewFromSisterDesignEfforts.Educational Researcher, 32,32-34.

• EducationalTechnology,45(1),2005 SpecialissuepreparedbyC.Dede,including: - Dede,C.(2005).Whydesign-basedresearchisbothimportantanddifficult.

Educational Technology, 45(1),5-8. - Squire,K.D.(2005).Resuscitatingresearchineducationaltechnology:Usinggame-

basedlearningresearchasalensforlookingatdesign-basedresearch.Educational Technology, 45(1),8-14.

- Barab,S.A.,Arici,A.,&Jackson,C.(2005).Eatyourvegetablesanddoyourhomework:Adesign-basedinvestigationofenjoymentandmeaninginlearning.Educational Technology, 45(1),15–21.

- Nelson,B.,Ketelhut,D.J.,Clarke,J.,Bowman,C.,&Dede,C.(2005).Design-basedresearchstrategiesfordevelopingascientificinquirycurriculuminamultiuservirtualenvironment.Educational Technology, 45(1),21–28.

- Kafai,Y.B.(2005).Theclassroomas“livinglaboratory”:Design-basedresearchforunderstanding,comparing,andevaluatinglearningsciencethroughdesign.Educational Technology, 45(1),28–34.

- Hay,K.E.,Kim,B.,&Roy,T.C.(2005).Design-basedresearch:Morethanformativeassessment?AnaccountoftheVirtualSolarSystemProject.Educational Technology, 45(1),34-41.

- Hoadley,C.(2005).Design-basedresearchmethodsandtheorybuilding:AcasestudyofresearchwithSpeakEasy.Educational Technology, 45(1),42-47.

- Reeves,T.C.(2005).Design-basedresearchineducationaltechnology:Progressmade,challengesremain.Educational Technology, 45(1),48-52

• JournaloftheLearningSciences,13(1),2004 Specialissue,including: - Barab,S.,&Squire,K.(2004).Design-BasedResearch:PuttingaStakeintheGround.

Journal of the Learning Sciences, 13(1),1-14. - Collins,A.,Joseph,D.,&Bielaczyc,K.(2004).DesignResearch:Theoreticaland

MethodologicalIssues.Journal of the Learning Sciences, 13(1),15-42. - Fishman,B.,Marx,R.W.,Blumenfeld,P.,Krajcik,J.,&Soloway,E.(2004).Creatinga

FrameworkforResearchonSystemicTechnologyInnovations.Journal of the Learning Sciences, 13(1),43-76.


- diSessa,A.A.,&Cobb,P.(2004).OntologicalInnovationandtheRoleofTheoryinDesignExperiments.Journal of the Learning Sciences, 13(1),77-103

- Dede,C.(2004).IfDesign-BasedResearchistheAnswer,WhatistheQuestion?ACommentaryonCollins,Joseph,andBielaczyc;diSessaandCobb;andFishman,Marx,Blumenthal,Krajcik,andSolowayintheJLSSpecialIssueonDesign-BasedResearch.Journal of the Learning Sciences, 13(1),105-114.

- Kelly,A.(2004).DesignResearchinEducation:Yes,butisitMethodological?Journal of the Learning Sciences, 13(1),115-128.

Books• VandenAkker,J.,Gravemeijer,K,McKenney,S.&Nieveen,N.(Eds).(2006).Educational

design research.London:Routledge.ISBN10:0-415-39635-2(pbk)(163pages) Availableathttp://www.taylorandfrancis.co.uk/shopping_cart/products/product_

detail.asp?sku=&ppid=118302&isbn=9780415396356 ThisbookcomprisesthepaperspresentedataseminarorganizedbytheNetherlands

OrganizationforScientificResearch,inparticularbytheProgramCouncilforEducationalResearch.Theseminar,conductedinDecember2003,hasbeenameetingplaceofdesignresearchersfromtheUSAandTheNetherlands.Thebookreflectsthevariousanglesfromwhichresearchersinthedomainsofcurriculum,instructionaltechnologyand(mathematicsandscience)educationaddresstheneedtodevelopresearchbasedsolutions(interventions)toproblemsforwhichnoguidelinestosolutionsareavailable.Thebookillustratesthatauthorswithvariousbackgroundshaveclearlyacommongroundwhenreflectingondesignresearchasaresearchapproach.

Thebookhasfourparts: Part1.Whatandwhy 1. IntroducingEducationalDesignResearch-JanvandenAkker,KoenoGravemeijer,

SusanMcKenney,NienkeNieveen 2. TowardProductiveDesignStudies-DeckerWalker Part2.Examplesfromthefield 3. DesignresearchfromtheLearningDesignPerspective-KoenoGravemeijer,Paul

Cobb 4. DesignResearchfromtheTechnologyPerspective-ThomasReeves 5. DesignResearchfromaCurriculumPerspective-SusanMcKenney,NienkeNieveen,

JanvandenAkker


Part3.Quality 6. AssessingtheQualityofDesignResearchProposals:SomePhilosophicalPerspectives

-D.C.Phillips 7. BalancingInnovationandRisk:AssessingDesignResearchProposals-DanielC.

Edelson 8. QualityCriteriaforDesignResearch:EvidenceandCommitments-AnthonyE.Kelly Part4.Movingahead 9. FromDesignResearchtoLarge-ScaleImpact:EngineeringResearchinEducation-

HughBurkhardt 10. EducationalDesignResearch:TheValueofVariety-NienkeNieveen,SusanMcKenny,

JanvandenAkker

• Kelly,A.E.,Lesh,R.A.&Baek,J.Y.(Eds).(2008).Handbook of Design Research Methods in Education Innovations in Science, Technology, Engineering, and Mathematics Learning and Teaching.NewYork:LawrenceErlbaumAssociates.ISBN:978-0-8058-6059-7(pbk)(560pages)

Availableathttp://www.routledgeeducation.com/books/Handbook-of-Design-Research-Methods-in-Education-isbn9780805860597

Theannouncementofthebookstatesthatthehandbookpresentsthelatestthinkingandcurrentexamplesofdesignresearchineducation.Design-basedresearchinvolvesintroducinginnovationsintoreal-worldpractices(asopposedtoconstrainedlaboratorycontexts)andexaminingtheimpactofthosedesignsonthelearningprocess.Designedprototypeapplications(e.g.,instructionalmethods,softwareormaterials)andtheresearchfindingsarethencycledbackintothenextiterationofthedesigninnovationinordertobuildevidenceoftheparticulartheoriesbeingresearched,andtopositivelyimpactpracticeandthediffusionoftheinnovation.

TheHandbook of Design Research Methods in Educationismeanttofillaneedinhowtoconductdesignresearchbythosedoingsorightnow.Thechaptersrepresentabroadarrayofinterpretationsandexamplesofhowtoday’sdesignresearchersconceptualizethisemergentmethodologyacrossareasasdiverseaseducationalleadership,diffusionofinnovations,complexitytheory,andcurriculumresearch.

Thehandbookhaseightsections: - Designresearchanditsargumentativegrammar - Modelingstudentlearningduringdesignresearch - Modelingteacherlearningusingdesignresearch - Modelingstakeholderscommitmentsusingdesignresearch - Reflectingondesignresearchattheprojectlevel


- Reflectingondesignresearchattheprogramlevel - Extendingdesignresearchmethodologically - Trackingthediffusionofdesignresearch.

• Reinking,D.&Bradley,B.A.(2008).On Formative and Design Experiments: Approaches to Language and Literacy Research.NewYork&London:TeachersCollege,ColumbiaUniversity.ISBN:978-0-8077-4841-1(pbk)(134pages)

Thisbookletprovidesaniceintroductionintoformativeanddesignexperiments,atermsynonymousforwhatwecalldesignresearchandothersdesign-basedresearch.Itprovidesathorough,butpracticalandusefuloverviewofdesignresearchaddressingthefollowingquestions:

- Whatareformativeanddesignexperiments(Ch1)? - Whatarethemethodsofformativeanddesignexperiments(Ch2)? - Whataresomegoodexamplesofformativeanddesignexperiments(Ch3)? - Isthereaformativeordesignexperimentinyourfuture(Ch4)?

• Richey,R.&Klein,J.D.(2007).Design and development research: methods, strategies, and issues.London:Routledge.ISBN080585732X,9780805857320(180pages)

Thisvolumediscussesmethodsandstrategiesappropriateforconductingdesignanddevelopmentresearch.Richwithexamplesandexplanations,thebookdescribesactualstrategiesthatresearchershaveusedtoconducttwomajortypesofdesignanddevelopmentresearch:1)productandtoolresearchand2)modelresearch.Commonchallengesconfrontedbyresearchersinthefieldwhenplanningandconductingastudyareexploredandproceduralexplanationsaresupportedbyawidevarietyofexamplestakenfromcurrentliterature.

Websites• http://projects.coe.uga.edu/dbr/index.htm(lastupdateNovember2006): titled‘Design-basedResearchEPSS’–createdbyInstructionalTechnologyPh.D.students

atTheUniversityofGeorgiaundersupervisionofTomReeves(comprehensivetilllastupdateofNovember2006).Thiswebsiteissummarizedinthenextsection.

• http://cider.athabascau.ca/CIDERSIGs/DesignBasedSIG/dbrreferences(lastupdateearly2005):

ThisbibliographyisdrawnupbyTerryAndersonoftheUnivesityofAthabasca(Edmonton,Alberta,Canada).Andersoncallsitasnapshotofmostcurrent(early2005)literaturerelatedtodiscussion,explorationandexamplesofdesign-basedresearch.ThereferencesarepresentedwithURLs(ifavailable)alongwithabstractsandoccasionallyquotationsorannotationsbyAnderson.HasmuchoverlapwiththeUniversityofGeorgiawebsite.


• http://www.designbasedresearch.org/index.html(lastupdatenotclear,butnoreferenceslaterthan2004)

ThisisthewebsiteoftheDesign-BasedResearchCollective,asmallgroupofresearcherswhoengageindesign-basedresearch,oftenintechnologyenhancedlearningenvironments.Itcontainsreferencesofanumberofpublications,aswellasanumberoflinkstorelevantrelatedwebsites.OtherpublicationsApartfromthesourcesmentionedabove,manyarticlesandbookchaptershavebeenpublisheddealingwithconceptualand/ormethodologicalaspectsofdesignresearch,orreportingaboutdesignresearchprojects.Manyofthesereferences(plusabstracts)canbefoundonthewebsitesmentionedinthissection,butwehaveselectedanumberwhicharesummarizedinthefinalsectionofthischapter.

UGA Website ‘Design-based Research EPSS’, November 2006

TheURLoftheUGAwebsiteElectronicPerformanceSupportSystem(EPSS)is:http://projects.coe.uga.edu/dbr/index.htm.Thewebsite(November2006),createdbyInstructional Technology Ph.D. students at The University of Georgia(UGA),supervisedbyTomReeves1,hasthreeparts:1. PEERTutorial2. Webliography3. ExpertInterviews

1.PEERTutorialThisusefultutorialiscomposedoffourprimarysections:(i) tutorialsurvey,(ii) explanation,(iii)enactment,and(iv)reflection.Parts(ii)and(iii)areusefulandinformativetobecomefamiliarwithdesignresearchandhowtogetstarted.(ii)Explanation:Thepurposeoftheexplanationpartofthetutorialistoprovidetheuserwithfundamentalknowledgeandinsightaboutdesignresearchcomposedoffivesections:• WhatisDesign-basedResearch(DBR)?• HowdoesDBRdifferfromotherapproaches?• HowdidDBRgetstarted?• WhatarethebenefitsofDBR?

1) TheauthorsofthiswebsiteusethetermDesignBasedResearch(DBR),whilstweusethroughoutthisbookletthe term‘DesignResearch’:bothtermsshouldbeseenassynonymous.


• Whataresomecriticalperspective?Eachsectionisconciseandprovidescoreinformationwithamplereferencestoliterature.(iii)Enactment:Thispartconsistsofthreesections:- HowdoIgetstartedwithDesign-BasedResearch(DBR)?- SomeexamplesofDBR- WhatarethechallengesofdoingDBR?EachsectiondiscussesanumberofrelevanttopicsforthosewhowanttoconductDesignResearch(DR)orDesign-BasedResearch.Asstated,thisisausefultutorial.Butkeepinmindthatauthorsmaydifferinemphasis,approachand/oruseoftermsandconcepts.Butwhenyouareanopen-minded,criticalreaderyouwillfindmanyusefulideasandsuggestionsinthistutorial.

2.WebliographyThepurposeofthispartofthewebsiteis–accordingtoitscreators–toprovidevarioustypesofresourcesthatmayprovideinterestedresearchersabeginningpointforinvestigatingandpursuingthetopicofdesign-basedresearch.Warning:Asthewebsitehasbeenpreparedin2006,itmaybepossiblethatsomeoftheURLsreferredtoarenolongeraccessibleoractive.Thewebliographyhasthefollowingsections:(i) GlossaryAlimitednumberofkeyconceptsaredefined,themostimportantbeingdesign-basedresearch.(ii)PrintedresourcesOnebookislisted(VandenAkker,etal.,2006)andquiteanumberofjournalarticles(andtheirabstracts),dividedintomethodologicalarticlesandresearchexamples.Thisisausefullistofpublicationsup‘tillNovember2006,andespeciallythosepublishedinNorthAmerica.(iii)OnlineresourcesThissectionhasanumberofsub-sections:anumberofusefulwebsites,twoonlinejournals(toolimitedtobereallyuseful),andtheURLsofanumberofusefularticles.(iv)OrganizationsTwoorganizationsarementioned,viz.Design-basedResearchCollective(DBRC)andtheDesign-basedResearchSIGoftheCanadianInstituteofDistanceEducationResearch,butthewebsitesofbothorganizationsseemnottobeuptodate.Nevertheless,thewebsiteofDBRC(http://www.designbasedresearch.org/index.html)givesusefulreferencestotwospecialissuesofjournals:


• Kelly,E.A.(Ed.).(2003).Theroleofdesignineducationalresearch[specialissue].Educational Researcher, 32 (1).• Sandoval,W.&Bell,P.(Eds.).(2004).Design-basedresearchmethodsforstudying

learningincontext[SpecialIssue].Educational Psychologist, 39(4)2.

(v)Expertsindesign-basedresearchContainsshortbiographicalnotesandapictureofanumberofexpertsindesignresearch(amongstthemallexceptthefirstauthorofthisbooklet).(vi)FrequentlyAskedQuestions(FAQ):Threequestionsareaddressed,vizwhatisdesign-basedresearch(DBR),howtobeginwithDBR,andhowdoesDBRdifferfromotherresearchmethodologies.

3.ExpertinterviewsThispartofthewebsitecontainsanumberofvideotapedinterviewswithanumberofexpertsindesignresearch.

Selected journal articles and book chapters on concept and methodology

Therearesomanypublicationsoneducationaldesignresearchthatitisimpossibletodrawupacomprehensivebibliography.Anumberofpublicationshavealreadybeenlisted(withabstracts)ontheUGAwebsite(see6.2).Howeverwewanttopointthereadertoanumberofarticlesandchapterthathavehelpedustogetinvolvedindesignresearchandtounderstandthemainissuesinourfield.Giventhisrationaleforselectingthesetitles,thereaderwillfindthatsomeofthetitlesarealsoreferredtoontheUGAwebsite.

Akker,J.vanden(1999).Principlesandmethodsofdevelopmentresearch3.InJ.vandenAkkeretal.(Eds.),Design approaches and tools in education and training(pp.1-14).Dordrecht:KluwerAcademicPublishers. Abstract:Thischapterdiscussestheroleofresearchinrelationtoeducationaldesign

anddevelopmentactivities.Thefirstpartofthechapterfocusesontherationaleandbasicprinciplesofdevelopmentresearchbyoutliningmotivesforconductingformativeresearch,analyzingdefinitionsandaimsofvarioustypesofdevelopmentresearch,anddiscussingseveralofitskeycharacteristics.Thesecondpartofthechapterdealswith

2) Thewebsiteofthisspecialissueofferstheopportunitytopurchasearticlesforb22.00plusVAT.3) Theconceptdevelopmentresearch,usedinsometitles,issynonymoustodesignresearch.


methodsofdevelopmentresearch,exploringsomeofitstypicalproblemsanddilemmas,anddiscussingseveralchallengesforfurtheractionandreflection.

Akker,J.vanden&Plomp,Tj.(1993).Developmentresearchincurriculum:propositionsandexperiences.PaperpresentedattheannualconferenceoftheAmericanEducationalResearchAssociation,April1993,Atlanta(GA,USA). Abstract:Reasontoincludethispaperisthatitisthefirstpaperfromthegroupatthe

UniversityofTwenteonwhattheycalledatthattimedevelopmentresearch.Basedontheassertionthatbothcurriculumdevelopmentandcurriculumresearchhavemuchrelevancetogainfromacloseliaison,theauthorssuggestthatboundariesbetweenthetwoshouldfade,whichcanbedoneinanewresearchstrategycalleddevelopmentresearch.Thepaperpresentsthepurpose,aconceptualframeworkandsomecharacteristicsofdevelopmentresearchincurriculum

See:www.leerplanevaluatie.slo.nl/taakhulp/lezen

Bannan-Ritland,B.(2003).Theroleofdesigninresearch:Theintegrativelearningdesignframework. Educational Researcher, 32(1),21-24. Abstract(fromUGAwebsite):Inthisarticle,ageneralmodelisproposedfordesign

researchineducationthatgrowsoutoftheauthor’sresearchandworkinrelateddesignfields.Themodelemphasizesthestagesensitivityof(a)researchquestions,(b)dataandmethods,and(c)theneedforresearcherstodesignartifacts,processes,andanalysesatearlierstagesintheirresearchthatcanthenbeprofitablyused(perhapsbydifferentresearchers)inlaterstages.

Barab,S.A.,andSquire,K.D.(2004).Design-BasedResearch:PuttingaStakeintheGround.Journal of the Learning Sciences, 13(1),1-14. Abstract(fromUGAwebsite):Thearticlehighlightsandproblematizessomechallenges

thatarefacedincarryingoutdesign-basedresearch.Itstatesthattheemergingfieldoflearningsciencesisonethatisinterdisciplinary,drawingonmultipletheoreticalperspectivesandresearchparadigmssoastobuildunderstandingsofthenatureandconditionsoflearning,cognitionanddevelopment.Afundamentalassumptionofmanylearningscientistsisthatcognitionisnotathinglocatedwithintheindividualthinkerbutisaprocessthatisdistributedacrosstheknower,theenvironmentinwhichknowingoccursandtheactivityinwhichthelearnerparticipates.Inotherwords,learning,cognition,knowingandcontextareirreduciblyco-constitutedandcannotbetreatedasisolatedentitiesorprocesses.


Brown,A.L.(1992).Designexperiments:Theoreticalandmethodologicalchallengesincreatingcomplexinterventionsinclassroomsettings.Journal of the Learning Sciences, 2,141-178. Abstract(fromUGAwebsite):Thisistheseminalarticleondesignresearch.Discusses

theoreticalandmethodologicalchallengesincreatingcomplexinterventionsinclassroomsettings.Movementfromtheclassicalpsychologicalpositionofconcentratingonatheoreticalstudyofthelearningprocessesofindividualstudentstoaconcentrationonconceptualchangeinteachersandstudents;Classroomrestructuring;Designexperiments;Experiencesonlearningtheory.

Design-BasedResearchCollective.(2003).Design-basedresearch:Anemergingparadigmforeducationalinquiry.Educational Researcher, 32(1),5-8. Abstract(fromUGAwebsite):Theauthorsarguethatdesign-basedresearch,which

blendsempiricaleducationalresearchwiththetheory-drivendesignoflearningenvironments,isanimportantmethodologyforunderstandinghow,when,andwhyeducationalinnovationsworkinpractice.Designbasedresearchers’innovationsembodyspecifictheoreticalclaimsaboutteachingandlearning,andhelpusunderstandtherelationshipsamongeducationaltheory,designedartifact,andpractice.Designiscentralineffortstofosterlearning,createusableknowledge,andadvancetheoriesoflearningandteachingincomplexsettings.Designbasedresearchalsomaycontributetothegrowthofhumancapacityforsubsequenteducationalreform.

Kelly,A.E.(2006).Qualitycriteriafordesignresearch.In:J.vandenAkker,K.Gravemeijer,S.McKenney,&N.Nieveen(Eds.).Educational design Research.London:Routledge. Abstract:thischapterdiscussesforeachofthreedifferentusesfordesignresearchin

educationanumberofcharacteristicsandexemplaryexamples.Itintroducesthenotionofthecommissivespaceofdesignresearch,meaningthat(amongstothercharacteristics)designresearchdoesnotstriveforcontext-freeclaimsbutseescontextsascentraltoitsconceptualdomain,thatdesignresearchisexperimentalbutnotanexperiment,andthatdesignresearcherschoosetoworkinthe“contextofdiscovery”,ratherthaninthe“contextofverification”utilizingrandomizedtrials.

Reeves,T.(2000).Enhancingtheworthofinstructionaltechnologyresearchthrough“designexperiments”andotherdevelopmentalstrategiesPaperpresentedattheAnnualMeetingoftheAmericanEducationalResearchAssociation,April2000,NewOrleans(LA,USA).RetrievedOct.20,2006fromhttp://it.coe.uga.edu/~treeves/AERA2000Reeves.pdf


Abstract:Theauthorarguesthatingeneralresearchintheareaofinstructionaltechnologyispoor,notprovidingpractitionerswithsufficientguidance.Hediscussesvarioustypesofinstructionaltechnologyresearchgoalsandmethodsandsuggeststhat‘use-inspiredbasicresearch’isneededinthedomainofinstructionaltechnologyreferringtoapproacheslikedevelopmentresearchanddesignexperiments.Hepresentsaframeworkandcharacteristicsfordevelopmentresearchintheareaofinstructionaltechnology.

Selected journal articles and book chapters on design research in domains

Overthelastfewyears,increasinglyexamplesofdesignresearchhavebeenpublished.Thissectioncontainsjustafewexemplaryreferencestoarticlesandchaptersinbooksofdesignresearchinvariousdomains,ofwhichafewaretakenfromtheUGAwebsiteWehaveaddedinthenextsectionreferencestosomePhDdissertationsreportingondesignresearchconductedatDutchuniversitiesthatcaneasilybeaccessedthroughtheWorldWideWeb.

DomainofmathematicseducationAseminalchapteris:Gravemeijer,K.&Cobb,P.(2006).Designresearchfromthelearningdesignperpective,invandenAkker,K.Gravemeijer,S.McKenney,&N.Nieveen(Eds.)EducationalDesignresearch:Thedesign,developmentandevaluationofprograms,processesandproducts.London:Routledge,17-51. Abstract:thischapterpresentsanapproachtodesignresearchthathasbeenusedand

refinedinaseriesofdesignresearchprojectsinwhichthetwoauthorscollaboratedoveraten-yearperiod.Theirapproachisfallingwithinthebroadercategoryofdesignresearchthataimsatcreatinginnovativelearningecologiesinordertodeveloplocalinstructiontheoriesontheonehand,andtostudytheformsoflearningthatthoselearningecologiesareintendedtosupportontheotherhandinthedomainofmathematicseducation(includingstatisticseducation).

Theapproachtodesignresearchhasitsrootsinthehistoryofthetwoauthorswhichistheworkonrealisticmathematicseducation(RME)thatiscarriedoutintheNetherlands(firstauthor)andthatofsocio-constructivistanalysisofinstruction(secondauthor).

Somereferences4fordesignresearchcasesinthedomainofmathematicseducationconductedintheUSAare:

4) WewanttoexpressourthankstoPaulCobbforprovidingthesereferences.


Bowers,J.S.,Cobb,P.,&McClain,K.(1999).Theevolutionofmathematicalpractices:Acasestudy.Cognition and Instruction, 17,25-64.Cobb,P.(1999).Individualandcollectivemathematicallearning:Thecaseofstatisticaldataanalysis.Mathematical Thinking and Learning, 1,5-44.Cobb,P.,McClain,K.,&Gravemeijer,K.(2003).Learningaboutstatisticalcovariation.Cognition and Instruction, 21,1-78.Confrey,J.,&Smith,E.(1995).Splitting,covariation,andtheirroleinthedevelopmentofexponentialfunctions.Journal for Research in Mathematics Education, 26,66-86.Lehrer,R.,&Schauble,L.(2004).Modelingnaturalvariationthroughdistribution.American Educational Research Journal, 41,635-679.Lobato,J.(2003).Howdesignexperimentscaninformarethinkingoftransferandviceversa.Educational Researcher, 32(1),17-20.Simon,M.A.(1995).Reconstructingmathematicspedagogyfromaconstructivistperspective.Journal for Research in Mathematics Education, 26,114-145.Stephan,M.,Bowers,J.,&Cobb,P.(Eds.).(2003).Supporting students’ development of measuring conceptions: Analyzing students’ learning in social context. Journal for Research in Mathematics Education Monograph No. 12. Reston,VA:NationalCouncilofTeachersofMathematics.

Anillustrativeexampleofdesignresearchinthecontextofadevelopingcountryis:Vos,P.,Devesse,T.G.,andPinto,A.A.R.(2007).DesigningMathematicsLessonsInMozambique:StartingFromAuthenticResources.African Journal of Research in SMT Education, 11(2),pp.51-66 Abstract:Thisarticledescribesresearchonthedesignofstudent-centredinstructionin

Mozambique.Thestartingpointwastheuseofreal-liferesources,suchastraditionalartcraftobjectsandauthenticnewspaperclippings.Theresearchwasbasedonaninstructionaldesignmodel,whichattemptstoaligntheorywithpracticeandwhichisgearedtowardsimprovingpractice.Intwoparallelstudies,oneongeometryandoneonstatistics,student-centredinstructionwasfacilitatedthroughtheuseofworksheetswithopen-endedquestionstailoredforgroupwork.Inacyclicprocess,theprototypematerialsandtheassociatedinstructionalmethodwereformativelyevaluated.Theevaluationsshowedthatthedesignswereusefuleveninclassroomspackedwithmorethansixtystudents.

DomainofscienceeducationHoadley,C.M.,&Linn,M.C.(2000)Teachingsciencethroughonline,peerdiscussions:SpeakEasyintheknowledgeintegrationenvironment.International Journal of Science Education,22(8),839-857.


Abstract:Thisarticlediscusseswhetherstudentscanlearnsciencefromcarefullydesignedonlinepeerdiscussions.Contraststwoformatsofcontributedcomments--historicaldebateandnarrativetext--andassessestheimpactofanasynchronousdiscussiononstudentunderstandingofthenatureoflight.Italsoreportsthatstudentsgainintegratedunderstandingofthenatureofcolorfrombothdiscussionformats.

Kafai,Y.B.,&Ching,C.C.(2001).Affordancesofcollaborativesoftwaredesignplanningforelementarystudents’sciencetalk.Journal of the Learning Sciences, 10(3),323-363. Abstract:Thisarticleinvestigateswhethersciencepermeatesthedesignenvironment

andisthuscontextedwithintheotheractivitiesofcollaborativemanagementandtechnology.Focusesonwhichcontextsgaverisetosciencetalk.Studiesaclassroomwith(n=33)studentsdividedintoseventeams

Knippels,M.C.P.J.,Waarlo,A.J.,andBoersma,K.Th.(2005).Designcriteriaforlearningandteachinggenetics.Journal of Biological Education,39(3),108-112. Abstract:Whilelearningandteachingdifficultiesingeneticshavebeenabundantly

exploredanddescribed,therehasbeenlessfocusonthedevelopmentandfield-testingofstrategiestoaddressthem.Toinformthedesignofsuchastrategyareviewstudy,focusgroupinterviewswithteachers,acasestudyofatraditionalseriesofgeneticslessons,studentinterviews,andcontentanalysisofschoolgeneticsteachingwerecarriedout.SpecificdifficultiesreportedintheliteraturewerecomparabletothoseperceivedbyDutchteachersandfoundinthecasestudyandthestudentinterviews.Theproblemsassociatedwiththeabstractandcomplexnatureofgeneticswerestudiedinmoredetail.Theseparationofinheritance,reproductionandmeiosisinthecurriculumaccountsfortheabstractnatureofgenetics,whilethedifferentlevelsofbiologicalorganisationcontributetoitscomplexnature.Finally,fourdesigncriteriaaredefinedforalearningandteachingstrategytoaddresstheseproblems:linkingthelevelsoforganism,cellandmolecule;explicitlyconnectingmeiosisandinheritance;distinguishingthesomaticandgermcelllineinthecontextofthelifecycle;andanactiveexplorationoftherelationsbetweenthelevelsoforganisationbythestudents.

Key words:Biologyeducation;Genetics;Learningandteachingdifficulties;Designcriteria

Lijnse,P.L.(1995).“DevelopmentalResearch”asawaytoanempiricallybased“DidacticalStructure”ofScience.Science Education, 29(2),189-199. Abstract:Theauthorarguesthatdevelopmentalresearch(inthisbookcalled´design

research’)isneededinwhichsmall-scalecurriculumdevelopmentiscyclicallycoupledtoin-depthclassroomresearchofteaching-learningprocesses.Suchresearchshouldresultinworkedoutexamplesofsuccessfulwaysofteaching,accordingtonew


conceptualcurriculumstructures.Designingsuch`didactical´structuresconstitutesalongertermresearchprogram,whichasksforinternationalexchangeandcooperatition.

Domainofreading-writingAbbott,S.P.,Reed,E.,Abbott,R.D.,&Berninger,V.W.(1997).Year-longbalancedreading/writingtutorial:Adesignexperimentusedfordynamicassessment.Learning Disability Quarterly,20(3),249-263. Abstract:Sixteenchildrenwithseverereadingproblemsinfirstgradereceivedayear-

longindividualtutorialintervention.Growthcurveanalysesfoundsignificantgainsonmeasuresoforthographicandphonologicalcoding,wordidentification,wordattackskills,readingcomprehension,letterautomaticity,andspellingandmarginallysignificantgainsinwritingcomposition.

DeCorte,E.,Verschaffel,L.,&vandeVen,A.(2001).Improvingtextcomprehensionstrategiesinupperprimaryschoolchildren:Adesignexperiment.The British Journal of Educational Psychology,71,531-559. Abstract:Withrespecttotheacquisitionofcompetenceinreading,newstandardsfor

primaryeducationstressmorethanbeforetheimportanceoflearningandteachingcognitiveandmetacognitivestrategiesthatfacilitatetextcomprehension.Therefore,thereisaneedtodesignaresearch-basedinstructionalapproachtostrategicreadingcomprehension.Thedesignexperimentaimedatdeveloping,implementingandevaluatingaresearch-based,butalsopracticallyapplicablelearningenvironmentforenhancingskilledstrategyuseinupperprimaryschoolchildrenwhenreadingatext.Thisdesignexperimentshowsthatitispossibletofosterpupils’useandtransferofstrategicreadingcomprehensionskillsinregularclassroomsbyimmersingtheminapowerfullearningenvironment.Butthisinterventiondoesnotautomaticallyresultinimprovementofperformanceonastandardizedreadingcomprehensiontest.

Neuman,S.B.(1999).Booksmakeadifference:Astudyofaccesstoliteracy.Reading Research Quarterly,34(3),286-311. Abstract:Thisarticleexaminestheimpactofaninterventiontargetingeconomically

disadvantagedchildrenthatfloodedover330child-carecenterswithhigh-qualitychildren’sbooksandprovided10hoursoftrainingtochild-carestaff.Itexaminestheproject’simpactandgivessupportforthephysicalproximityofbooksandthepsychologicalsupporttochild-carestaffonchildren’searly-literacydevelopment.

DomainofinstructionaltechnologyBannan-Ritland,B.(2003).Theroleofdesigninresearch:Theintegrativelearningdesignframework.Educational Researcher,32(1),21-24.


Abstract:(fromUGAwebsite):Inthisarticle,ageneralmodelisproposedfordesignresearchineducationthatgrowsoutoftheauthor’sresearchandworkinrelateddesignfields.Themodelemphasizesthestagesensitivityof(a)researchquestions,(b)dataandmethods,and(c)theneedforresearcherstodesignartifacts,processes,andanalysesatearlierstagesintheirresearchthatcanthenbeprofitablyused(perhapsbydifferentresearchers)inlaterstages.

Herrington,J.,&Oliver,R.(1997).Multimedia,magicandthewaystudentsrespondtoasituatedlearningenvironment.Australian Journal of Educational Technology,13(2),127-143.Availableat:http://www.ascilite.org.au/ajet/ajet13/herrington.html Abstract:Thisarticlepresentsadesignofaninteractivemultimedialearning

environmententitledInvestigatingassessmentstrategiesinmathematicsclassrooms,whichrepresentstheoperationalizedcharacteristicsofsituatedlearning.Theauthorsalsosuggestthecriticalguidelinesforthedesignofthemultimediasoftwaretoenableittosupportasituatedlearningenvironment.Theythenreportastudythatinvestigatespatternsofbehaviorofstudentsimmersedinthismultimediasituatedlearningenvironment.Thefindingssuggestthattheuseofthesituatedlearningmodelissuccessfulinprovidingguidelinesforthedevelopmentofaninteractivemultimediaprogram.Theyalsorevealthatininstanceswherelearnersareempoweredandareenabledtoassumehigherdegreesofresponsibilityfortheiractivityandconductinalearningsetting,theresearchersneedtobecognizantofthevariousdesignfactorswhichcanimpedeorenhancelearning.Inmultimediaenvironments,theseincludesuchelementsasthemotivationalaspectsoftheenvironment,theinterfacedesign,andthenavigationelementsemployed.Inconclusion,theauthorssuggestthatitisalsoimportanttopracticeresearchwhichexplorestheimpactofthemoretangibleaspectsofmultimediadesignsuchasthoseexploredinthisstudy.

Herrington,J.,&Oliver,R.(2000).Aninstructionaldesignframeworkforauthenticlearningenvironments.Educational Technology Research and Development,48(3),23-48.Availableat:http://edserver2.uow.edu.au/~janh/Assessment/Authentic%20Assessment_files/ETR%26D.pdf Abstract:Theinstructionaltechnologycommunityisinthemidstofaphilosophical

shiftfromabehaviouristtoaconstructivistframework,amovethatmaybegintoaddressthegrowingriftbetweenformalschoollearningandreal-lifelearning.Onetheoryoflearningthathasthecapacitytopromoteauthenticlearningisthatofsituatedlearning.

Thepurposeofthisthreepartstudywasfirstly,toidentifycriticalcharacteristicsofasituatedlearningenvironmentfromtheextensiveliteraturebaseonthesubject;secondly,tooperationalisethecriticalcharacteristicsofasituatedlearningenvironmentbydesigningamultimediaprogramwhichincorporatedtheidentifiedcharacteristics;


andthirdly,toinvestigatestudents’perceptionsoftheirexperiencesusinganmultimediapackagebasedonasituatedlearningframework.

Thelearningenvironmentcomprisedamultimediaprogramforpreserviceteachersonassessmentinmathematics,togetherwithrecommendedimplementationconditionsintheclassroom.Eightstudentswereobservedandinterviewedtoexploretheirperceptionsofthesituatedlearningenvironment.Findingssuggestthattheuseofthesituatedlearningframeworkappearedtoprovideeffectiveinstructionaldesignguidelinesforthedesignofanenvironmentfortheacquisitionofadvancedknowledge.

Reeves,T.(2006).Designresearchfromatechnologyperspective.In:J.vandenAkker,K.Gravemeijer,S.McKenney,&N.Nieveen(Eds.).Educational design Research.London:Routledge. Abstract:Theeffectivenessofthefieldknownaseducationaltechnologyin

fundamentallyenhancingteachingandlearninghasincreasinglybeencalledintoquestion,ashastheefficacyofeducationalresearchingeneral.Doubtsabouteducationaltechnologyresearchprimarilystemfromdecadesofanarguablyflawedresearchagendathathasbeenbothpseudoscientificandsociallyirresposible.Itisproposedthatprogressinimprovingteachingandlearningthroughtechnologymaybeaccomplishedusingdesignresearchasanalternativemodelofinquiry.Designresearchprotocolsrequireintensiveandlong-termcollaborationinvolvingresearchersandpractitioners.Itintegratesthedevelopmentofsolutionstopracticalproblemsinlearningenvironmentswiththeidentificationofreusabledesignprinciples.Examplesofdesignresearchendeavorsineducationaltechnologyaredescribedhere.Thechapterendswithacallfortheeducationaltechnologyresearchcommunitytoadoptdesignresearchmethodsmorewidely.

Reeves,T.C.,Herrington,J.,&Oliver,R.(2004).Adevelopmentresearchagendaforonlinecollaborativelearning.Educational Technology Research and Development, 52(4),53-65. Abstract:Althoughimportant,traditionalbasic-to-appliedresearchmethodshave

providedaninsufficientbasisforadvancingthedesignandimplementationofinnovativecollaborativelearningenvironments.Itisproposedthatmoreprogressmaybeaccomplishedthroughdevelopmentresearchordesignresearch.Developmentresearchprotocolsrequireintensiveandlong-termcollaborationamongresearchersandpractitioners.Inthisarticle,weproposeguidelinesforimplementingdevelopmentresearchmodelsmorewidely,andconcludewithaprescriptionforanonlinecollaborativelearningresearchagendaforthenextfivetotenyears.

Reinking,D.,&Watkins,J.(2000).Aformativeexperimentinvestigatingtheuseofmultimediabookreviewstoincreaseelementarystudents’independentreading.Reading Research Quarterly, 35(3),384-419.


Abstract:Thisstudyinvestigateshowacomputer-basedinstructionalintervention(creatingmultimediareviewsofbooks)mightincreasefourthandfifthgraders’independentreading.Thestudyfindsthatthesuccessoftheinterventionwasrelatedtothemediatingeffectsofusingtechnology,changesintheinteractionsamongstudentsandteachers,andstudents’engagementinrelationtotheirreadingability.Italsonotesseveralotherfactors.

DomainofcurriculumMcKenney,S.&vandenAkker,J.(2005).Computer-basedsupportforcurriculumdesigners:Acaseofdevelopmentalresearch.Educational Technology Research & Development, 53(2)41-66. Abstract:Inthisarticle,weexplorethepotentialofthecomputertosupportcurriculum

materialsdevelopmentwithinthecontextofsecondarylevelscienceandmathematicseducationinsouthernAfrica.Duringthefour-yearcourseofthestudy,acomputerprogramwasdevelopednamedCASCADE-SEA,whichstandsforComputerAssistedCurriculumAnalysis,DesignandEvaluationforScience(andmathematics)EducationinAfrica.Bycarefullydocumentingtheiterativeprocessofanalysis,prototypedesign,evaluation,andrevision,wesoughtinsightintothecharacteristicsofavalidandpracticalcomputer-basedtoolthatpossessesthepotentialtoaffecttheperformanceofitsusers.TheresultsofthisstudyincludetheCASCADE-SEAprogramitself,whichassistsusersinproducingbetterqualitymaterialsthantheyotherwisemight,whiletheyalsolearnfromthedevelopmentprocess.Further,thisresearchhascontributedtothearticulationofdesignprinciplesandrelateddevelopmentalresearchmethods.Thisarticlehighlightstheresearchanddevelopmentthattookplace,andonlybrieflyaddressesthetoolitself.

McKenney,S.,Nieveen,N,&vanderAkker,J.(2002).Computersupportforcurriculumdevelopers:CASCADE.Educational Technology Research and Development,50(4),25-35. Abstract:Thisarticleexaminesresearchonacomputer-basedtool,CASCADE(Computer

AssistedCurriculumAnalysis,DesignandEvaluation),thatwasdevelopedattheUniversityofTwente(Netherlands)toassistincurriculumdevelopment.Thearticlediscusseselectronicperformancesupportsystemsandtheneedforincreasedattentiontoimplementationandimpactstudies.

Nieveen,N.M.(1999).Prototypingtoreachproductquality.In:J.J.H.vandenAkker,R.Branch,K.Gustafson,N.M.Nieveen,&Tj.Plomp(Eds.),Design approaches and tools in education and training(pp.125-136).Dordrecht:Kluwer. Abstract: Thischapterprovidesaframeworkforproductqualityconsistingofthe

followingthreecriteria:validity,practicalityandeffectiveness,andprovidesinsightinto


theapplicabilityoftheframeworkinvariousdomainsofeducationalproductdevelopment.Inordertoreachproductquality,theprototypingapproachisseenandunderstoodasasuitableapproach.Thischapterdiscussesthreesignificantcharacteristicsofaprototypingapproach:extensiveuseofprototypes,highdegreeofiterationandtheroleofformativeevaluation,andtheparamountimportanceofuserinvolvement.Thechapterillustratesthewaytheprototypingapproachhasbeeninstrumentalindevelopingacomputersupportsystemforinstructionaldevelopers.Duringtheprototypingprocess,theframeworkassistedindecidingthefocusofeachprototypeandenhancedthetransparencyoftheentireprocess.

Nieveen,N.M.&Akker,J.J.H.,vanden(1999).Exploringthepotentialofacomputertoolforinstructionaldevelopers. Educational Technology Research & Development,47(3),77-98. Abstract:Informationandcommunicationtechnologytoolscurrentlypermeatealmost

everyprofessionaldomain.Thosegearedtowardthefieldofinstructionaldevelopmenthaveemergedinrecentyears.Thisarticleexploresthepotentialforlinkingthedomainsofcomputersupportandinstructionaldevelopment.ThisarticlereportsonthedesignandevaluationofCASCADE(ComputerAssistedCurriculumAnalysis,DesignandEvaluation),acomputersystemthatsupportsinstructionaldevelopersduringformativeevaluationefforts.Fiveprototypesofthesystemwerecreatedandevaluatedonthebasisoftheirvalidity(reflectionofstate-of-the-artknowledgeandinternalconsistency);practicality(abilitytomeettheneeds,wishesandcontextualconstraintsofthetargetgroup);andeffectiveness(improvedusertaskperformance).TheresultsofthisstudysuggestthattheuseofCASCADEcould:(a)improvetheconsistencyofformativeevaluationplansandactivities;(b)motivatedevelopersbyelevatingtheirconfidenceinusingformativeevaluationactivities;(c)savetime;and(d)helptoprovidejustificationsfordecisionsmade.

Some PhD theses utilizing design research as a research approach

Overtheyears,variousPhDthesehavebeenwritteninwhichdesignresearchhasbeenappliedasthemainresearchapproach.InthissectionwejustmentionafewthathavebeendefendedinTheNetherlandsattheUniversityofTwenteandtheUniversityofUtrecht.Undoubtly,manymoredissertationscanbefoundviasearchmachinesontheWWWeb,butwejustwanttopointthereadertoafewexamplesofdesignresearchwearefamiliarwith.

DomainofcurriculumdevelopmentNieveen,N.(1997).Computer support for curriculum developers.Doctoralthesis.Enschede(TheNetherlands)UniversityofTwente,.


Availablefrom:http://projects.edte.utwente.nl/cascade/original/McKenney,S.(2001).Computer-based support for science education materials developers in Africa: exploring potentials. Doctoralthesis.Enschede(TheNetherlands):UniversityofTwente.Availablefrom:http://projects.edte.utwente.nl/cascade/seastudy/

Kouwenhoven,W.(2003).Designing for competence in Mozambique: towards a competence-based curriculum for the Faculty of Education of the Eduardo Mondlane University. Doctoralthesis.Enschede(TheNetherlands):UniversityofTwente.Availablefrom:http://doc.utwente.nl/41442/1/thesis_Kouwenhoven.pdfNote:thisisanexampleofdesignresearchinwhichtheresearcherwasnotactivelyinvolvedinallphasesofthedesignprocess.

DomainofprofessionaldevelopmentofteachersTeclaiTecle,Andemariam(2006). The potential of a professional development scenario for supporting biology teachers in Eritrea.Doctoralthesis.Enschede(TheNetherlands):UniversityofTwente.Availablefrom:http://purl.org/utwente/55985

DomainofmathematicseducationArmanto,Dian(2002).Teaching multiplication and division realistically in Indonesian primary schools : a prototype of local instructional theory.Doctoralthesis.Enschede(TheNetherlands)UniversityofTwente.Availablefrom:http://purl.org/utwente/58710

Bakker,A.(2004).Design research in statistics education : on symbolizing and computer tools.Doctoralthesis.Utrecht(TheNetherlands:UniversityofUtrecht.Availablefrom:http://igitur-archive.library.uu.nl/dissertations/2004-0513-153943/inhoud.htm

Fauzan,Ahmad(2002).Applying realistic mathematics education (RME) in teaching geometry in Indonesian primary schools. Doctoralthesis.Enschede(TheNetherlands)UniversityofTwente.Availablefrom:http://purl.org/utwente/58707

DomainofscienceeducationKnippels,M.C.P.J.(2002).Copingwiththeabstractandcomplexnatureofgeneticsinbiologyeducation:Theyo-yolearningandteachingstrategy.Doctoralthesis.Utrecht(TheNetherlands:UniversityofUtrecht.Availablefrom:http://igitur-archive.library.uu.nl/dissertations/2002-0930-094820/inhoud.htm


Mafumiko,FideliceSimbagungileMbaruku(2006).Micro-scale experimentation as a catalyst for improving the chemistry curriculum in Tanzania.Doctoralthesis.Enschede(TheNetherlands)UniversityofTwente.Availablefrom:http://purl.org/utwente/55448Ottevanger,W.(2001).MaterialsdevelopmentasacatalystforsciencecurriculumimplementationinNamibia.Doctoralthesis.Enschede:UniversityofTwente.

Tilya,FrankNicodem(2003).Teacher support for the use of MBL in activity-based physics teaching in Tanzania. Doctoralthesis.Enschede(TheNetherlands)UniversityofTwente.Availablefrom:http://purl.org/utwente/41462

Westbroek,H.B.(2005).Characteristics of meaningful chemistry education - The case of water quality.Doctoralthesis.Utrecht(TheNetherlands:UniversityofUtrecht.Availablefrom:http://igitur-archive.library.uu.nl/dissertations/2005-0922-200121/index.htm


AuthorbiographiesJanvandenAkkerJanvandenAkkerisdirectorgeneralofNetherlandsInstituteforCurriculumDevelopment[SLO].Besides,heispart-timeprofessor(chaironCurriculumDesign)attheUniversityofTwente.Inhiswideteaching,research,supervision,andconsultancyexperiences(bothintheNetherlandsandabroad)hetendstoapproachcurriculumdesignchallengesfromabroadereducationalinnovationperspective.Overtheyearshispreferencefordesignresearchhasincreasedbecauseofitsstrongcombinationofpracticalrelevanceandknowledgegrowth.Somebooksoverthelastdecade(editedwithsomecolleagues)thatrepresenthisorientation:Design Approaches and Tools in Education and Training (1999), Curriculum Landscapes and Trends (2003),and Educational Design Research (2006).

BrendaBannanBrendaBannanisanassociateprofessorintheinstructionaltechnologyprogramatGeorgeMasonUniversityinFairfax,Virginia,USA.Herresearchinterestsprimarilyinvolvetheintegrationofdesignandresearchprocessesrelatedtoeducationaltechnologydesignanddevelopment.ShehasauthoredseveralarticlesandchaptersondesignresearchintheHandbookforDesignResearchMethodsinEducation(Eds.Kelly,Lesh&Baek,2008)aswellastheEducational Researcher,vol32,2003.Dr.BannanwasawardedanNSFCAREERgrantawardin2003toconductafive-yearcycleofdesignresearchandwasinvitedasavisitingscholaratStanfordUniversityfortheacademicyearof2005-2006.

AnthonyE.KellyAnthonyE.KellyisaprofessorofeducationalpsychologyatGeorgeMasonUniversityinVirginia,USA.Hisresearchinterestsextendtoresearchmethodologydesign,andresearchattheintersectionofcognitiveneuroscienceandeducation.HecurrentlyhastwoNationalScienceFoundationgrants,oneonmodelinglearningincyberinfrastructurenetworksinearthsciences(withBrendaBannan-Ritland),andoneonlearninginmegacities.HisNSFgrantondesignresearchmethodswithRichardLesh,producedtheHandbookofDesignResearchMethods(Kelly,Lesh&Baek,2008).Kellyeditedthespecialissueondesignresearch,EducationalResearcher,vol32,2003.HeistheeditorofthespecialissueontheUSNationalMathematicsAdvisoryPanelReport,Educational Researcher,forthcoming.HeservedasaprogrammanagerattheNationalScienceFoundationfrom1997-2000,and2006.

NienkeNieveen(authorandeditor)NienkeNieveenworksattheresearchdepartmentofNetherlandsInstituteforCurriculumDevelopment[SLO].HerworkcentersaroundcoordinatingandsupportingtheInstitute’sdesignresearchactivitiesbytakingseriesofconcertedactionstoassistcurriculum


developersininterweavingdesignandformativeevaluationactivities.Indoingso,SLOaimsatknowledgegrowthandprovenqualityforallofitsproducts,i.e.curriculumframeworksandteachingandlearningmaterials.Herdissertation,in1997,wasbasedonafouryeardesignresearchprojectand,alsoafterthisperiod,shecontinuedworkingondesignresearchprojectsattheUniversityofTwente.Shehasauthoredseveralarticlesandchaptersondesignresearchandco-editedthebookEducational design research.

TjeerdPlomp(authorandeditor)TjeerdPlompisemeritusprofessorofcurriculumoftheUniversityofTwenteinEnschede,TheNetherlands.Hehasbeeninchargeofteachingeducationaldesignmethodologyinthe(atthattime)FacultyofEducationalScienceandTechnology.HewaschairofIEA,theInternationalAssociationfortheEvaluationofEducationalAchievement,from1989-1999.IntheIEAheservedaschairforthe‘ComputersinEducation’study(Comped),theThirdInternationalMathematicsandScienceStudy(TIMSS)andtheIEASecondInternationalTechnologyinEducationStudy(SITES).Hisresearchinterestsareeducationaldesignanddesignresearch,internationalcomparativeresearch,andinformationtechnologyinthecurriculumandteachereducation.Hehasbeenrecentlyinvolvedasadvisorinvariousresearchprojectsandprogramsutilizingdesignresearch,bothinTheNetherlandsandinternationally.


SLO • Netherlands institute for curriculum developmentSLO

PO box 20417500 CA EnschedeThe Netherlands

T +31(0)53 484 08 40F +31(0)53 430 76 92E [email protected]

www.slo.nl

SLO is the Netherlands institute for curriculum development. We are bridging the contexts of policy, research, and practice. Our expertise focuses on the development of curricular goals and content for various educational levels, from national policy to classroom practices. We closely collaborate with many different stakeholders from policy circles, schools (boards, principals, teachers), research, civic organizations, and the society at large. This allows us to design and validate relevant curriculum frameworks, to elaborate exemplary materials and to evaluate these in school practices. Our products and services support both policy makers and schools and teachers in making substantive curricular decisions and in elaborating these into relevant, inspiring and effective education.

ISBN: 978 90 329 2329 7

An In

trod

uctIo

n to

Edu

cAtIon

Al dEsIg

n rEsEArch

Editors: tjeerd Plomp &

nienke n

ieveen

Editors:Tjeerd Plomp & Nienke Nieveen

introduction 20to 20 education 20design research

Documents