katewinterevaluation.com education research 101 a beginner’s guide for s stem principal...

KATEWINTEREVALUATION.com

Education Research 101

A Beginner’s Guide for S STEM Principal Investigators


Welcome and Overview• Structure of today’s webinar• About 30 minutes of content and time for Q&A• Quick overview of educational research • Brief synopsis of the Common Guidelines • Pulse checks (interaction points) to gauge engagement

• About me


Objectives• Increase awareness of the “Common Guidelines for

Education Research and Development,” including:• The 3 broad categories of research• The 6 general types of research, with their:• Purpose• (Very) Abbreviated theoretical and empirical justifications• Abbreviated guidelines for evidence produced• Guidelines for external review and feedback


What is Education Research? • Research that uses various methodologies to address

educational processes and outcomes• Research that provides the evidence behind “evidence-

based” practices/strategies/interventions in education• Increasingly commonly by following established guidelines for rigor

and significance set forth by IES and NSF, including the What Works Clearinghouse (WWC) standards


Interaction #1

What was your background with the Common Guidelines for Education Research and Development

PRIOR to this webinar?


Disclosure• In *many* places, this presentation draws text verbatim

from the Common Guidelines. When you have time, I highly encourage you to find, bookmark, and review the full document from IES and NSF.

http://ies.ed.gov/pdf/CommonGuidelines.pdf


Assumptions behind the Common Guidelines•While the 6 research types generally follow a sequence of

development:• Knowledge development is not linear and can be informed from

various places along the continuum• Investigations can skip “phases” (MOOCs make large-scale testing

possible without prior small-scale testing)• Single studies may include aspects of more than one research type


Foundational Research & Early-Stage or Exploratory Research• Contribute to Core Knowledge in education, including basic

understanding of teaching and learning (i.e., cognition); components and processes involved in learning and instruction; the operation of education systems; and, models of systems and processes• Include:• Research Type #1: Foundational Research• Research Type #2: Early-Stage or Exploratory Research


Design and Development Research • Develops solutions to achieve a goal related to education or

learning, such as improving student engagement or mastery of a set of skills• Represents Research Type #3


Impact Studies• Contribute to evidence of impact, generating reliable

estimates of the ability of a fully-developed intervention or strategy to achieve its intended outcomes• Includes:• Research Type #4: Efficacy Research• Research Type #5: Effectiveness Research• Research Type #6: Scale-Up Research


Interaction #2

Which statements are TRUE?


Research Type #1: Foundational Research

• Provides the fundamental knowledge that may contribute to improved learning and other relevant education outcomes• Seeks to test, develop, or refine theories of teaching or

learning and may develop innovations in methodologies and/or technologies that will influence and inform research and development in different contexts


Foundational Research (cont.)• Purpose: To advance the frontiers of education and learning;

develop and refine theory and methodology; and provide fundamental knowledge about teaching and/or learning• Justifications: Should address important research problems

or questions related to education and learning; should outline the theoretical and empirical bases for the project and why it is necessary for core knowledge generation


Foundational Research (cont.)• Evidence to be produced: • Advances in theory, methodology, and/or understanding of

important constructs in education and learning • Findings that could serve as the basis for future studies

May examine phenomenon without establishing explicit links to education outcomes


Research Type #2: Early-Stage or Exploratory Research• Examines relationships among important constructs in

education and learning to establish logical connections that may form the basis for future interventions or strategies to improve education outcomes• Connections are usually correlational rather than causal


Early-Stage or Exploratory Research (cont.)• Purpose: To investigate approaches to education problems to

establish the basis for design and development of new interventions or strategies, and/or to provide evidence for whether an established intervention or strategy is ready to be tested in an efficacy study• Justifications: Should address a practical education problem

and provide a compelling case for how the study will inform the development, improvement, or evaluation of education programs, policies, or practices


Early-Stage or Exploratory Research (cont.)• Evidence to be produced: • Empirical evidence regarding associations between malleable

factors and student learning outcomes• A well-specified framework behind these associations• Justification for proceeding to a Design and Development project,

an Efficacy study, or if additional foundational or early-stage or exploratory research is needed first


Interaction #3

Who has engaged in a study or project that would be considered foundational research or early-stage or

exploratory research?


Research Type #3: Design and Development Research • Draws on existing theory and evidence to design and

iteratively develop interventions or strategies (solutions)•May include pilot tests of fully developed interventions to

determine if they achieve their intended outcomes under various conditions• Results could lead to additional work to better understand

the foundational theory behind the results or could indicate that the intervention or strategy is sufficiently promising to warrant more advanced testing


Design and Development Research (cont.)• Purpose: To develop new or improved interventions or

strategies to achieve well-specified learning goals or objectives, including making refinements on the basis of small-scale testing. Typically entails four components• Justifications: Explain why the proposed project has the

potential to improve learning or education outcomes or increase efficiencies in the education system or institutional setting beyond what current practice provides


Design and Development Research (cont.)• Evidence to Be Produced: • Fully developed version of the proposed design/research• A well-specified theory of action• Descriptions of the major iterations and the resulting evidence to

support or question key assumptions about the theory of action• Description and empirical evidence of the adjustments that

resulted from design testing• Measures with evidence of technical quality for assessing the

intervention and data demonstrating the project’s success• Pilot data on the intervention’s promise for generating the

intended beneficial learner outcomes


Interaction #3

Who has engaged in a study or project that would be considered design and development research?


Research Type #4: Efficacy Research

• Allows for testing of a strategy or intervention under “ideal” circumstances•May limit the investigation to a single population


Efficacy Research (cont.)• Purpose: To determine whether an intervention or strategy

can improve outcomes under “ideal” conditions• Justifications: Must compelling explain why testing the

impact under “ideal” circumstances is warranted, with evidence from prior studies/implementation that supports the value of the intervention and the need for additional testing


Efficacy Research (cont.)• Evidence to Be Produced (same for all Impact Studies): • Final report as outlined by the What Works Clearinghouse (WWC)• Reliable estimates of the intervention’s average impact, with

estimates by sample subgroups as appropriate• Implementation details for the intervention and counterfactual

condition, with connections to outcomes as appropriate• Implications of the findings for theory of action:• Replication details for projects demonstrating favorable impacts• Otherwise, possible reasons why no favorable impact was demonstrated


Research Type #5: Effectiveness Research• Examines the effectiveness of a strategy or intervention

under circumstances that would typically occur


Effectiveness Research (cont.)• Purpose: To estimate the impacts of an intervention or

strategy when implemented under conditions of routine practice• Justifications: Must compelling explain why testing the

impact under routine circumstances is warranted, with evidence from prior studies/implementation that supports the value of the intervention and the need for additional testing


Research Type #6: Scale-Up Research• Examines effectiveness in a wide range of populations,

contexts, and circumstances, without substantial developer involvement in implementation or evaluation (i.e., under “normal” conditions)


Scale-Up Research (cont.)• Purpose: To estimate the impacts of an intervention or

strategy when implemented under conditions of routine practice AND across a broad spectrum of populations and settings• Justifications: Must compelling explain why testing the

impact under routine circumstances and with a broad sample is warranted, with evidence from prior studies/implementation that supports the value of the intervention and the need for additional testing


Guidelines for External Review and Feedback• All research types should be subject to a series of external,

critical reviews of design, implementation, analysis and reporting• External reviews should be independent and rigorous• Appropriate review activities include:• Peer review of proposed projects• Ongoing monitoring and review by grant-making personnel• External review panels or advisory boards• Third-party evaluator• Peer review of publications and/or conference presentations


Interaction #4

Who has engaged in a study or project that would be considered an impact study?


Recap• Three categories:

1. Foundational research and early-stage or exploratory research2. Design and development research3. Impact studies:• Efficacy research• Effectiveness research• Scale-up research

• OK to mix different approaches in a single project, if it makes sense to do so


Questions or Comments?

Raise your hand to be un-muted, orType your question in the question box

Please let us know if you would be interested in additional webinars on the specific research types, including aspects related to design plans and WWC standards

katewinterevaluation.com education research 101 a beginner’s guide for s stem principal...

Documents