DesignDesign-Based -Based Research for Research for Advancing Educational TechnologyAdvancing Educational Technology

EDIT 9990EDIT 9990

Goals Critique the state-

of-the-art of educational research.

Describe applications of “design-based research.”

Encourage new thinking about why and how we do research.

Improving the quality of teaching and learning through educational research is critical to our survival.

Global Warming

Dutch Floating Homes

The lack of scientific literacy is appalling in even the most developed countries.

Bad News

Most educational research has little impact on practitioners and yields few discernable benefits.

Oh...no!

The Failure of Educational Research

– Vast resources going into education research are wasted.

– They [educational researchers] employ weak research methods, write turgid prose, and issue contradictory findings.

The Failure of Educational Research

– Too much useless work is done under the banner of qualitative research.

– Qualitative research…. [yields] ….little that can be generalized beyond the classrooms in which it is conducted.

College of EducationThe University of Georgia

Ranked 27th of 187 education colleges in the USA

240 faculty members in 9 departments

5,000 students in 33,000+ student university

Research Productivity 1997-2001Refereed Journal Articles (in-cites.com)

U. of Wisconsin - 202 U. of Georgia - 201 U. of Michigan - 164 Indiana U. - 161 U. of Maryland - 146

Georgia vs. Wisconsin

$7,824

44,073

51%

49th

$8,604

42,232

78%

7th

Per pupil

Salary

HS Grad.

Ranking

It is time we put the PUBLIC back in publication!

Bush Administration Position “There’s been no

improvement in education over the last 30 years, despite a 90 percent increase in real public spending per pupil.”

Promotes randomized controlled trials as used in medical research.

Accountability: Guaranteeing Results

Flexibility: Local Control for Local Challenges

Research-Based Reforms: Proven Methods with Proven Results

Parental Options: Choices for Parents, Hope for Kids

Four Reform Four Reform PrinciplesPrinciples

The What Works Clearinghouse (WWC) has been established by the U.S. Department of Education’s Institute of Education Sciences to provide educators, policymakers, researchers, and the public with a central and trusted source of scientific evidence of what works in education.

What Works What Works ClearinghouseClearinghouse

Slavin’s 5 Questions for Valid Educational Research Is there a control group? Are the control and

experimental groups assigned randomly?

If a matched study, are the groups extremely similar?

Is the sample size large enough?

Are the results statistically significant? Robert Slavin

“What Works” Position “Once we have dozens or

hundreds of randomized or carefully matched experiments going on each year on all aspects of educational practice, we will begin to make steady, irreversible progress.”

NCLB funds “scientifically based research.” Robert Slavin

“It Won’t Work” Position Double blind experiments

impossible in education Implementation variance

reduces treatment differences

Causal agents are under-specified in education

Goals, beliefs, and intentions of students and teacher affect treatments David R. Olson

Medical and health knowledge is rarely applied sufficiently.

Another “It Won’t Work” Position The What Works Clearinghouse

(WWC) standards “ignore the critical realities about social, organizational, and policy environments in which educational programs and interventions reside.”

Advocates “decision-oriented” evaluation research over “conclusion-oriented” academic research.

Recommends extended-term mixed-method (ETMM) designs as a viable alternative.

Madhabi Chatterji

American Evaluation Association

The priority given to randomized controlled trials “manifests fundamental misunderstandings about 1) the types of studies capable of determining causality, 2) the methods capable of achieving scientific rigor, and 3) the types of studies that support policy and program decisions. We would like to help avoid the political, ethical, and financial disaster that could well attend implementation of the proposed priority.”

Randomized controlled trials are the only way

we’ll ever be able to prove “what works” in education!

Randomized controlled trials promotes

pseudoscience and will limit effective change!

Educational researchers have failed to make a clear appeal to the public for their support.

People learn when…..

Ellen Lageman argues that educational researchers, in a misguided effort to be “scientific,” have turned away from the pragmatic vision of John Dewey.

She attacks the excessive emphasis on quantitative measurement.

Kieran Egan argues that progressive ideas from Herbert Spencer, John Dewey, and Jean Piaget are responsible for the “general ineffectiveness” of our schools.

He also assails the notion that education can be improved through research as traditionally conceived.

Thomas Kuhn The Structure of Scientific Revolutions

"I'm not sure that there can now be such a thing as really productive educational research. It is not clear that one yet has the conceptual research categories, research tools, and properly selected problems that will lead to increased understanding of the educational process. There is a general assumption that if you've got a big problem, the way to solve it is by the application of science. All you have to do is call on the right people and put enough money in and in a matter of a few years, you will have it. But it doesn't work that way, and it never will."

Complexity of Interactions We cannot store up

generalizations and constructs for ultimate assembly into a network.

When we give proper weight to local conditions, any generalization is a working hypothesis, not a conclusion.

Lee Cronbach

Learning Styles“Research into learning styles can, in the main, be characterised as small-scale, non-cumulative, uncritical and inward-looking. It has been carried out largely by cognitive and educational psychologists, and by researchers in business schools and has not benefited from much interdisciplinary research.”

Dichotomies convergers versus divergers verbalisers versus imagers holists versus serialists deep versus surface learning activists versus reflectors pragmatists versus theorists adaptors versus innovators assimilators versus explorers field dependent versus field

independent globalists versus analysts assimilators versus

accommodators imaginative versus analytic

learners non-committers versus plungers common-sense versus dynamic

learners concrete versus abstract learners random versus sequential learners

initiators versus reasoners intuitionists versus analysts extroverts versus introverts sensing versus intuition thinking versus feeling judging versus perceiving left brainers versus right brainers meaning-directed versus undirected theorists versus humanitarians activists versus theorists pragmatists versus reflectors organisers versus innovators lefts/analytics/inductives/successive

processors versus rights/globals/deductives/ simultaneous processors executive, hierarchic, conservative

versus legislative, anarchic, liberal.

If Sisyphus were a scholar, his field would be educational research.

- David Laberee

Educational Technology Research

Pseudoscience Results

Insert cone of experience example

Pseudoscience Results

Insert cone of experience example

Educational technology researchers are not doing much better than other educational researchers.

NCLB Requirements "every student is

technologically literate by the time the student finishes the eighth grade," and

"that technology will be fully integrated into the curricula and instruction of the schools by December 31, 2006."

Abundant technology has not led to extensive use of computers for “tradition-altering classroom instruction.”

The small percentage of computer-using instructors only use it to maintain existing classroom practices.

Teachers have legitimate concerns. Is it simple enough for

me to learn quickly? It it versatile? Will it motivate students? Is it aligned with skills I’m

expected to teach. Is it reliable? It it breaks, who will

help? Will it weaken my

classroom authority?

Ed. Tech Research Reality Isolated researchers conduct

individual studies rarely linked to a research agenda or concerned with any relationship to practice.

Studies are presented at conferences attended by other researchers and published in journals few people read.

Occasional literature reviews and meta-analyses are published.

Ed. Tech Research Reality Many educational technology

studies claim to have predictive goals (testing theories) and use quasi-experimental designs with quantitative measures.

Research reviewers usually must reject 75 percent or more of the published studies to find the few worthy of further review or inclusion in meta-analyses.

Ed. Tech Research Reality Dillon & Gabbard’s 1998 literature

review of “Hypermedia as an Educational Technology” highlights problems with IT research.

Major conclusion: “Clearly, the benefits gained from the use of hypermedia technology in learning scenarios appear to be very limited and not in keeping with the generally euphoric reaction to this technology in the professional arena.”

Ed. Tech Research Reality Fabos & Young 1999 literature review

of “Telecommunications in the Classroom: Rhetoric Versus Reality” is another bad sign.

Major conclusion: “…many of the expected benefits of telecommunications [enhancing writing, multicultural awareness, and economic possibilities] are inconclusive, optimistic, and even contradictory.”

Bernard et al. (2004) Meta-analysis: “How Does Distance Education Compare to Classroom Instruction?”

a very small but positive mean effect size for interactive distance education over traditional classroom instruction on student achievement

small negative effect for retention rate

DE Research from 1985-2002

1,010 potential “studies” retrieved

232 studies met all criteria

599 independent effect sizes

47,341 students (achievement)

Results: Overall Effects 325 independent outcomes (total

achievement)

Hedges’ g = +0.0122, p < .001

Range of findings from –2.17 to +2.66

177 outcomes with low methodology removed

Hedges’ g = +0.017, p > .05

Significantly heterogeneous

Distribution of Effect

Sizes

-3

-2

-1

0

1

2

3

1 21 41 61 81 101 121 141 161 181 201 221 241 261 281 301 321

Distribution of Effect Sizes for Achievement Outcomes

Magnit

ude o

f Eff

ect

Siz

e

Hedges’g

Effect Sizes Ordered by Magnitude

325 independent outcomes (achievement)Hedges’ g = +0.0122, p < .001

Sir John Daniel - UNESCO… the futile tradition of comparing test performances of students using new learning technologies with those who study in more conventional ways…is a pointless endeavor because any teaching and learning system, old or new, is a complex reality. Comparing the impact of changes to small parts of the system is unlikely to reveal much effect and indeed, “no significant difference” is the usual result of such research.

Chewing Gum More Effective than Interactive Multimedia CD-ROM

Dr. Ken Allen at NYU wanted to compare CD-ROM with lectures

Wrigley’s wanted to fund chewing gum study

Combined study Gum chewers = B-

Abstainers = C+ CD-ROM no better

Media comparison studies are akin to comparing copper bracelets with voodoo dolls as medical cures.

Experimental approaches to educational technology research won’t work in the way that they do in medical research!

MedicalCures

GeneticsResearch

Basic Applied

“Pasteur’s Quadrant” approach to research is needed (Stokes, 1997).

Considerations of use?

No Yes

Quest for fundamental understanding?

No

Yes

Bohr

Research is inspired by:

Edison

Pasteur

Good News There are new strategies for conducting “design-based research” that can improve our research so that it can become a socially responsible enterprise.

Thankgoodness!

Educational researchers often fail to distinguish between research goals and methods.

Six Ed. Tech research goals:

TheoreticalPredictiveInterpretivistPostmodernDesign/DevelopmentAction/Evaluation

Theoretical Goals

Focus on explaining phenomena through logical analysis and synthesis of principles and results from other studies

EXAMPLE: Gagne’s theory of the conditions of learning

Predictive Goals Focus on determining

how education works by testing hypotheses related to theories of learning, teaching, performance, etc.

EXAMPLE: cooperative learning and control studies by Hooper, Temiyakarn, and Williams Simon Hooper

Interpretivist Goals Focus on determining

how education works by describing and interpreting phenomena related to learning, teaching, performance, etc.

EXAMPLE: Delia Neuman’s observations of disabled children using commercial software Delia Neuman

Postmodern Goals

Focus on examining the assumptions underlying educational programs with the goal of revealing hidden agendas and empowering disenfranchised minorities

EXAMPLE: Ann DeVaney’s analysis of IT in relation to race, gender, and power

Design/Development Goals Focus on dual

objectives of developing creative approaches to solving problems and constructing reusable design principles

EXAMPLE: Sasha Barab’s “Quest Atlantis” project and “Learning Engagement Theory”

Action/Evaluation Goals

Focus on describing, improving, or estimating the effectiveness and worth of a particular program

EXAMPLE: Hill and Reeves four-year evaluation of ubiquitous computing initiative.

Methods should not be selected until goals & research questions are clear:

Quantitative

Qualitative

Critical Theory

Historical

Literature Review

Mixed-methods

So what does design-based research look like in the real world?

Design-Based Research Collective

Goals of designing learning environments and theories are intertwined

Development and research occur in continuous cycles

Research on designs leads to sharable theories relevant to practitioners

Research must account for how designs function in authentic settings

Development of accounts relies on methods that connect actions to outcomes

Design-Based Research Strategies Define a pedagogical outcome and create

learning environments that address it. Emphasize content and

pedagogy rather than technology.

Give special attention to supporting human interactions.

Modify learning environments until outcome is reached.

Chris Dede – Harvard University

River City Curriculum & Situated Learning Theory

Yasmin Kafai - UCLA

van den Akker, Nieveen, McKenney – University of Twente

Design-Based Research Example “Authentic Learning in Interactive

Multimedia Environments.” Ph.D. dissertation by Jan Herrington at

Edith Cowan University in Australia. Supervised by Professor

Ron Oliver. Winner of AECT

Young Researcherof the Year in 1999.

Outcome Practitioners Desired

New teachers will use a wider variety of assessment methods in their student teaching experience and eventual practice.

Learning Environment Design

Identified the critical characteristics of a situated learning model.

Developed an interactive multimedia learning environment based on those characteristics.

Situated Learning Model Herrington 1997

Provide an authentic context reflecting the way the knowledge will be used in real-life

Provide authentic activities Provide access to expert

performances and the modeling of processes

Provide multiple roles and perspectives

Support collaborative construction of knowledge

Situated Learning Model Herrington 1997

Promote reflection to enable abstractions to be formed

Promote articulation to enable tacit knowledge to be made explicit

Provide coaching and scaffolding at critical times

Provide for integrated assessment of learning within the tasks.

Mixed Methods Design Videotaped

preservice teachers using program

Interviewed teachers and their supervisors in schools during student teaching practicum

Logged usage data

Findings Problem Solved:

– Novice teachers acquired advanced knowledge while engaging in higher order thinking

– New knowledge and skills applied in practicum

Design Principles:– Situated learning model is

a successful design model for interactive learning

Authentic Learning Team

Reeves, Herrington, Oliver

What challenges do we face in adopting design-based research approaches?

Design-Based Research Challenges

–Sampling bias

–Response bias

–Researcher bias

–Overwhelming data

–Confounded variables

–Dissemination

–Scaling up Ann Brown

Design-Based Research CollectiveWe suggest that the value of design-based research should be measured by its ability to improve educational practice.

Because we are a design profession (not a discipline), educational technologists should pursue design-based research that integrates the desire to solve problems with the search for knowledge.

“The status of research deemed educational would have to be judged, first in terms of its disciplined quality and secondly in terms of its impact. Poor discipline is no discipline. And excellent research without impact is not educational.” - Charles W. Desforges (2000)