Overview of Research Methodologies for Social

Software Research

AarhusUniversity,DenmarkJanuary21,2009TerryAnderson,Ph.D.

CanadaResearchChairinDistanceEducaAon

ThecontextofSocialSoFwareImplementaAon

•  DisrupAve(Christensen,2008)simpler,notwantedbymainstreamcustomers

•  RapidgainsinfuncAonality•  Cheaper•  AdapAve

StudyingTechnologicalBasedModelsofChange

•  Technologicaldeterminism(Marx,Zuboff,Kurzweil)

•  AffordancesandSocialConstrucAvist(Bjiker,StewartandWilliams,Dalsgaard

•  DomesAficaAon/AppropriaAon–(Silverston,Ling)

Research Paradigms

Paradigm1QuanAtaAveResearch

•  “thosewhoareseekingthestrictwayoftruthshouldnottroublethemselvesaboutanyobjectconcerningwhichtheycannothaveacertaintyequaltoarithmeAcorgeometricaldemonstraAon”–  (ReneDescartes)

•  Inordinatesupportandfaithinrandomizedcontrolledstudies

QuanAtaAve1–CMCContentAnalysis

•  Anderson,Garrison,Rourke1997‐2003–  hcp://communiAesofinquiry.com‐9papersreviewingresults

focusingonreliable,quanAtaAveanalysis–  IdenAfiedwaystomeasureteaching,socialandcogniAve

‘presence’–  MostreliablemethodsarebeyondcurrentAmeconstraintsof

busyteachers–  QuesAonsofvalidity–  ServesasbasicresearchasgroundingforAImethodsofthe

future–  ServesasqualitaAveheurisAcforteachersandcoursedesigners

QuanAtaAve–Meta‐Analysis

•  UngerleiderandBurns(2003)•  SystemaAcreviewofeffecAvenessandefficiencyofICT

•  The type of interventions studied were extraordinary diverse –only criteria was a comparison group

•  “Only 10 of the 25 studies included in the in-depth review were not seriously flawed, a sobering statistic given the constraints that went into selecting them for the review.”

IsDEBecerthanClassroomInstrucAon?Project1:2000–2004

•  QuesAon:HowdoesdistanceeducaAoncomparetoclassroominstrucAon?(inclusivedates1985‐2002)

•  Totalnumberofeffectsizes:k=232

•  Measures:Achievement,AmtudesandRetenAon(oppositeofdrop‐out)

•  DividedintoAsynchronousandSynchronousDE

11

Primaryfindings•  DEandCIareessenAallyequal(g+≈0.0tolowaverageeffect)onallmeasures

•  EffectsizedistribuAonsareheterogeneous;someDE>>CI,someDE<<CI

•  Generallypoormethodologicalquality•  PedagogicalstudyfeaturesaccountformorevariaAonthanmediastudyfeatures(Clark,1994)

•  InteracAveDEanimportantvariable*

12

*Lou, Y., Bernard, R.M., & Abrami, P.C. (2006). Media and pedagogy in undergraduate distance education: A theory-based meta-analysis of empirical literature. Educational Technology Research & Development, 54(2), 141-176.

Summaryofresults:Achievement

13

Achievement Outcomes

*Significantly heterogeneous average effect

Summaryofresults:Amtudes

14

Attitude Outcomes

*Significantly heterogeneous average effect

Summaryofresults:RetenAon

15

Retention Outcomes

*Significantly heterogeneous effect sizes

Equivalency:ArealltypesofInteracAonnecessary?

Anderson, 2003 IRRODL

Anderson’sEquivalencyTheorem(2003)

Moore(1989)disAncAonsare:  ThreetypesofinteracAon

o  student‐studentinteracAono  student‐teacherinteracAono  Student‐contentinteracAon

Anderson(2003)hypothesesstate:

  Highlevelsofoneoutof3interacAonswillproducesaAsfyingeducaAonalexperience

  IncreasingsaAsfacAonthroughteacherandlearnerinteracAoninteracAonmaynotbeasAmeorcost‐effecAveasstudent‐contentinteracAvelearningsequences

17

DothethreetypesofinteracAondiffer?Moore’sdisAncAons

18

Achievement and Attitude Outcomes

Achievement Attitudes Interaction Categories k g+adj. k g+adj. Student-Student 10 0.342 6 0.358 Student-Teacher 44 0.254 30 0.052 Student-Content 20 0.339 8 0.136 Total 74 0.291 44 0.090 Between-class 2.437 6.892*

Moore’s distinctions seem to apply for achievement (equal importance), but not for attitudes (however, samples are low for SS and SC)

DoesstrengtheninginteracAonimproveachievementandamtudes?Anderson’shypotheses

19

Anderson’s first hypothesis about achievement appears to be supported

Anderson’s second hypothesis about satisfaction (attitude) appears to be supported, but only to an extent (i.e., only 5 studies in High Category)

Achievement and Attitude Outcomes

Achievement Attitudes Interaction Strength k g+adj. SE k g+adj. SE Low Strength 30 0.163 0.043 21 0.071 0.042 Med Strength 29 0.418 0.044 18 0.170 0.043 High Strength 15 0.305 0.062 5 -0.173 0.091 Total 74 0.291 0.027 44 0.090 0.029 (Q) Between-class 17.582* 12.060*

 Bernard,Abrami,Borokhovski,Wade,Tamin,&Surkes,(inpress).ExaminingThreeFormsofInteracAoninDistanceEducaAon:AMeta‐AnalysisofBetween‐DEStudies.ReviewofResearchinEduca2on

BecausesocialsoFwareuseissoemergentneedfor

•  Demographicstudies– Whoisusingwhatfordoingwhat?

– HowaretheyusingmulA‐usetools?

•  VisualizaAonofacAvityandrelaAonships•  Networkanalysis•  ConnecAonbetweencontextandtypeofuse•  Datamining

QuanAtaAveSummary

•  CanbeusefulespeciallywhenfinetuningwellestablishedpracAce

•  Providesincrementalgainsinknowledge,notrevoluAonaryones

•  Theneedto“control”contextoFenmakesresultsofliclevaluetopracAcingprofessionals

•  InAmesofrapidchangetooearlyquanAtaAvetesAngmaymaskbeneficialposiAvecapacity

•  Willweeverbeabletoaffordblindreviewed,randomassignmentstudies?

Paradigm2QualitaAveParadigm

•  ManydifferentvarieAes•  GenerallyanswerthequesAon‘why’ratherthen‘what’or‘howmuch’?

•  PresentsspecialchallengesindistributedcontextsduetodistancebetweenparAcipantsandresearchers

•  CurrentlymostcommontypeofDEresearch(RourkeandSzabo,2002)

QualitaAvestudyofSocialSoFware

•  CriAcallyimportant:–  InearlystagesofadopAon– Totrackeffectsofusercompetenceandefficacy– Ascontextsarepersonalized–  astoolsareappropriatedbyusersforenArelydifferenttasksthanthoseintendedbydevelopers

QualitaAveExample

– Dearnley(2003)Student support in open learning: Sustaining the Process

– PracAcingNurses,weeklyF2Ftutorialsessions

– Phenomenologicalstudyusinggroundedtheorydiscourse

Corecategorytoemergewas“Findingtheprofessionalvoice”

QualitaAveexample2

•  Mann,S.(2003)Apersonalinquiryintoanexperienceofadultlearningon‐line.Instruc8onalScience31

•  Conclusions:–  TheneedtofacilitatethepresentaAonoflearnerandteacher

idenAAesinsuchawaythattakesaccountofthelossofthenormalchannel

–  TheneedtomakeexplicitthedevelopmentofoperaAngnormsandconvenAons

–  reducedcommunicaAvemediathereisthepotenAalforgreatermisunderstanding

–  Theneedtoconsiderwaysinwhichthedevelopinglearningcommunitycanbeopentotheotherofuncertainty,ambiguityanddifference

3rdParadigmCriAcalResearch

•  Askswhogainsinpower?•  DavidNoble’scriAqueof‘digitaldiplomaMills’mostprominentCanadianexample

•  AreprofitsgeneratedfromusergeneratedcontentexploitaAve?

•  ConfronAngthe“netchangeseverything”mantraofmanysocialsoFwareproponents.

•  WhoisbeingexcludedfromsocialsoFware

IstheextracAonofinformaAonfromthemasses

exploitaAveorempowering?

•  WhydoesFacebookownallthecontentthatwesupply?

•  Doesthepowerofthenetfurthermarginalizethenonconnected?

•  Whobenefitsfromvoluntarydisclosure?

•  WhydidtheOneLaptopPerChildfail?

QuanAtaAvevs.QualitaAveParadigmWarsRekindled

•  Currentresearch“moreresemblesthependulumswingscharacterisAcofartorfashion,ratherthantheprogressiveimprovementscharacterisAcofscienceandtechnology”(p.16).•  Slavin(2002)inEducaAonalResearcher

•  SoluAontoembrace“evidencebasedlearning”•  Projectedtoincreasefrom5%to75%ofUSGov.fundingby2007for“researchthataddressescausalquesAonsandusesrandomassignments….”Slavin,2002p.15

DoEitherQualitaAveorQuanAtaAveMethodsMeetRealNeedsofPracAcing

DistanceEducators?

ButwhattypeofresearchhasmosteffectonpracAce?

– Kennedy (1999) - teachers rate relevance and value of results from each of major paradigms.

– No consistent results – teachers are not a homogeneous group of consumers but they do find research of value

– “The studies that teachers found to be most persuasive, most relevant, and most influential to their thinking were all studies that addressed the relationship between teaching and learning.”

ButwhattypeofresearchhasmosteffectonPracAce?

– “The findings from this study cast doubt on virtually every argument for the superiority of any particular research genre, whether the criterion for superiority is persuasiveness, relevance, or ability to influence practitioners’ thinking.” Kennedy, (1999)

4thParadigmDesign‐BasedResearch

•  Relatedtoengineeringandarchitecturalresearch

•  Focusesonthedesign,construcAon,implementaAonandadopAonofalearninginiAaAveinanauthenAccontext

•  Relatedto‘DevelopmentResearch’

•  Closesteducatorshavetoa“homegrown”researchmethodology

Design‐BasedResearchStudies

– iterative, – process focused, – interventionist, – collaborative, – multileveled, – utility oriented, – theory driven and generative

•  (Shavelson et al, 2003)

Design‐basedresearch•  Methodologydevelopedbyeducatorsfor

educators•  DevelopedfromAmericanpragmaAsm–Dewey

(Anderson,2005)•  RecentThemeIssues:

–  TheJournaloftheInstruc1onalSciences,(13,1,2004),–  Educa1onalResearcher(32,1,2003)and–  Educa8onalPsychologist(39,4,2004)–  Seebibliographyat

hKp://cider.athabascau.ca/CIDERSIGs/DesignBasedSIG/

•  Myar8cleatwww.cjlt.ca/abstracts.html

IntegraAveLearningDesign(Bannan‐Ritland,2003)

•  “design‐basedresearchenablesthecreaAonandstudyoflearningcondiAonsthatarepresumedproducAvebutarenotwellunderstoodinpracAce,andthegeneraAonoffindingsoFenoverlookedorobscuredwhenfocusingexclusivelyonthesummaAveeffectsofanintervenAon”Wang&Hannafin,2003

Wang and Hannafin, 2003

•  IteraAvebecause•  ‘InnovaAonisnotrestrictedtothepriordesignofanartefact,butconAnuesasartefactsareimplementedandused”

•  ImplementaAonsare“ineviAtablyunfinished”(StewartandWilliams(2005”

DesignBasedresearchandtheScienceofComplexity

•  ComplexitytheorystudiestheemergenceoforderinmulAfaceted,changingandpreviouslyunorderedcontexts

•  ThisemergingorderbecomesthefocusofiterateintervenAonsandevaluaAons

•  Orderemergesatthe“edgeofchaos”inresponsetorapidchange,andfailureofpreviousorganizaAonmodels

D‐BResearchexamples

Design-Based Research Strategies for Studying Situated Learning in a Multi-user Virtual Environment Chris Dede, 2004

•  Needtostudyusability,scalabilityandinnovaAonadopAonwithinbureaucraAcsystems

•  AllowknowledgetoolstoevolveinnaturalcontextthroughsupporAvenourishmentofstaff

Groundedtheory

•  Theneedfortheoryandmodelbuilding•  “ThereisnothingsopracAcalasagoodtheory”

•  Weneedmodelsandtheoriesnowtoguidetherapiddeploymentthatistakingplace

•  Bebraveandexpansiveinyourconclusions

ReasonsforCriAcalMassinNetworksofPracAce

Landquvist &Teigland, 2007

PersonalAcademicuseofsocialsoFware

•  Docreateablogtodocumentyourresearchjourney

•  Followblogsofthoseyouadmire

•  Selfarchiveandonlygiveyourcopyrightawayunderunusualcircumstances

•  Supportopenaccessjournalsiewww.irrodl.org

FinalHints

•  SubscribetoIRRODLwww.irrodl.org•  CustomizeGoogleScholartoscanyourlibraryconnecAonswww.googlescholar.com

•  SubscribetomyblogtheVirtualCanuckatterrya.edublogs.org

Conclusion

•  EducaAonresearchisgrosslyunder‐resourcedtomeetthemagnitudeofopportunityanddemand

•  ParadigmwarsareunproducAve•  Design‐basedresearchoffersapromisingnewresearchdesignmodel

•  ThesemanAcwebofferspromiseofgreatlyenhancedcollaboraAonandresearchdisseminaAon