American Educational Research AssociationAnnual Meeting

AERA San Diego, CA - April 14, 2006

Denise Huang

Theory and Methods in OST Research

2 / 28

Experimental Qualitative Evaluation

Purpose New knowledge, Truth To uncover and understand what lies behind any phenomenon

Mission achievement, product delivery

Outcome Generalizable conclusions Emergence of theory Specific decisions

Value Explanatory and predictive power

Rich descriptive dataInsiders’ point of view

Determining worth and social utility

Impetus Curiosity and ignorance Understanding Needs and goals

Conceptual Basis Cause and effect relationships Experiences as a whole, in context, interactive

Means ends processes

Key event Hypothesis testing Observations of multiple, shifting axioms

Assessing attainment of an objective

Classic paradigms Experimental methodCorrelational method

Grounded theory approach System approachObjective approach

Discipline Control & manipulation of variance

Iterations Program planning and management

Criteria Internal and external validity Fit, understanding, generality, and control

Isomorphism and credibility

Functional type Pure and appliedTrue experimental Quasi-experimental

Grounded theoryEthnography,Phenomenon approachCase studies, etc.

Formative – summativeProcess – product

3 / 28

What do all these mean? The gold standard for research design and analysis involve

experimental randomization.

Satisfactory research findings required generalization and replication.

The afterschool population has distinct characteristics

Ethical issues

Self-selective group

Difficulty in obtaining comparison group (consent forms)

Accuracy of data and availability of longitudinal records

High transience

Qualitative strategies are vital tools in finding out the intricate rich data about a program and help to explain, elaborate, and triangulate quantitative findings…

Matching study questions with design and appropriate methodology

4 / 28

Most afterschool studies are evaluation research

The systematic application of social research procedures for assessing the conceptualization, design, implementation, and utility of social intervention programs (Strauss & Corbin, 1990).

• Focus on finding and explaining educational /social effects or outcomes from the intervention, and

• Devising instructional strategies that will improve the outcomes.

Formative and summative evaluations

Process and impact evaluations

5 / 28

Different types of evaluation research

Process Evaluations:

Formative studies

• Evaluations that assesses the conduct of the program during the initial design and testing stages with the intent to improve the program

Program monitoring

• Systematic examination of program coverage and delivery- (target population, fidelity, efficiency)

• Identifying successful implementation strategies for program diffusion

6 / 28

Impact Evaluations

Assesses the changes in the well-being of individuals that can be attributed to a particular intervention, such as a project, program or policy (Strauss & Corbin, 1990).

Summative evaluation

• Summative evaluation provides information on the product's efficacy ( it's ability to do what it was designed to do)

• By looking at the intervention group, the evaluator can examine the learning materials and learning process together with the outcomes-- hence the name Summative Evaluation.

Impact Evaluation

• Impact evaluation involves constructing a counterfactual

• Random selection and isolation from interventions are seldom practicable and sometimes ethically difficult to defend.

• Quasi-experimental method is often used.

7 / 28

Which theoretical perspective informed your work, and why did you select this theory?

• Goal setting theory

• Social cognitive theory

• The contextual understanding of the social ecological model

8 / 28

The role that theory play in research

Theory is an invention aimed at organizing and explaining specific aspects of the environment (Pedhazur & Schmelkin, 1991)

• Theory provides the researcher with a “selective point of view-an orientation

• Helps to determine what variable is relevant and which are not relevant

Being a way of seeing, a theory is also a way of not seeing.

9 / 28

How did this theory shape your research methods?

• Mixed methods

• Process/outcome

• SEM

• HLM

10 / 28

Examples: Formative, Process, & Summative

The ASL/KidzLit Formative Study

ASL is a reading and writing project that was based on the theoretical foundation of the literature-based instructions (social cognitive theory)

• How do the LA’s BEST teachers evaluate the ASL project in terms of training, materials, and support?

• To what extent have staff capabilities been enhanced after the ASL training?

• If the staff members teach during the regular school day and in the after-school program, to what extent have they carried the ASL strategies into their regular school day teaching?

• What changes in student attitudes might be associated with ASL? Are there any preliminary indicators that participation in ASL is affecting students?

11 / 28

Teacher survey I-formative evaluation

For the effectiveness of the ASL training, the evaluation based on ASL’s four core components:

• Care

• Talk

• Read

• Connect

To evaluate the ASL project as a whole, teachers’ perspectives were examined:

• ASL materials

• ASL training

• Support for the ASL project

For an affective/motivational outcome the following indicators were examined:

• Students’ enjoyment

• Teacher attitude

12 / 28

The student survey

Based on the social cognitive theory that learning is a social phenomenon and social collaboration is essential in the classroom climate, we investigate on students’:

• Enjoyment of their reading class;

• Perception of the social support in the classroom;

• Perception of their relation with their teachers.

Based on the motivational theory, we look at the students’:

• Attitudes towards reading and writing; and

• Reading and writing self-efficacy.

And students’ perception of the classroom climate:

• The autonomy orientation of their classroom climate; and

• The emphasis on effort allocation.

13 / 28

Qualitative studyExploring the intellectual, social, and organizational capitals at LA’s

BEST

Relationships

Expectations &Social Norms

Trust

Staff & Student Relationships

Staff & Student Relationships

Student Engagement

Student Engagement

Organizational CapitalOrganizational

CapitalOrganizational

Capital

Social CapitalSocial Capital

Social Capital

Intellectual CapitalIntellectual Capital

Intellectual Capital

Internal Networks External Networks Innovation and Staff Development

Competence

Attitudes

Problem-solving& Agility

14 / 28

Quantitative outcome evaluations

Keeping kids in school study

• 4 cohorts of students (all LA’s BEST participants and a stratified random sample of non-participants)

• 3 participation level ( 1,2,3+ years)

• 6-9th grade in 1998-1999 followed through to 2002-2003

• Chi-square analysis

• Cox survival analysis

15 / 28

Comparison of dropout rates for LA’s BEST vs. LAUSD non-participants (3+ Years).

16 / 28

The Afterschool Hours-SEM and HLM

Examining the Relationship between Afterschool Staff-Based Social Capital and Student Engagement in LA’s BEST

• What are LA’s BEST staff perceptions of collective staff efficacy, teamwork, and communication, and the quality of their relationships with students?

• What are student perceptions of their relationships with LA’s BEST staff? To what extent do they value education and have high aspirations for their futures? What are their reported levels of student engagement in LA’s BEST and the day school?  

• How are staff-student relationships, teamwork and communication, and collective staff efficacy at the site level, related to student perceptions of their relationships with staff?  

• What is the association between student perceptions of their relationship with staff, their value of education, future aspirations, and engagement in the afterschool program and day school?

17 / 28

Hierarchical Linear Modeling

• Examine the relationship between the social capital predictors measured by the staff surveys (i.e., staff-student relationships, collective staff efficacy, and communication and teamwork) and student perceptions of social capital (i.e. staff-student relationship) as measured by the student surveys.

• The demographic variables of gender, grade level, and languages spoken were also included in each model to control for individual student differences.

• A total of 2,270 students and 395 staff from 50 school sites were included in the HLM analysis.

18 / 28

Structural Equation Modeling

Staff/Student Relationships

LAB Engagement

School Engagement

Value of Education

0.82

0.23

0.34

0.74

-0.30

0.41

Future Aspirations

0.87

19 / 28

DOJ study-using propensity matching & cost-benefit analysis

Quasi-experimental design

Longitudinal sampling of academic and juvenile crime data

Advanced multilevel propensity scores methods to establish study samples

Hierarchical growth modeling and survival analysis (multilevel discrete-time hazard)

Students were followed from 1994-1995-2002-2003 school years

Cost-benefit analysis

20 / 28

How did your method(s) contribute to our knowledge of OST, and what were the limitations of the method?

• Better comparison sample—propensity matching

• The importance of “dosage”

• Importance of goal setting

• Importance of employing a continuous improvement model

21 / 28

Why results are inconsistent?

• Linking results to program content

__Design___________________________________________

• Linking results to implementation procedures(program fidelity)

__Treatment________________________________________

• Linking results to program attendance

__Dosage___________________________________________

22 / 28

What does that theory mean for what you would do with children and youth during the out-of-school time and inform program design?

Importance of having

• A theory of change

• A logic model

• A continuous improvement strategy

23 / 28

Why create a logic model?

• They are powerful tools for designing, planning, implementing, and evaluating OST programs

• They set up a plan of how a program is expected to work

• Provide a map of how to achieve goals (cause and effect-program process and outcomes)

• Stimulate clear thinking, preparation, and organization

24 / 28

Example: Theoretical-based logic model designed for the Afterschool Partnership study

Program SupportManagement

Staff /ResourcesPeriodical Evaluation for Continuous

Improvement

Periodic Assessment of

StudentPerformance

Linkage to School Day

Linkage to School Day Research Based

Practice & Strategies

Research Based Practice & Strategies

Content Practice Quality

After School Center Quality

Content Alignedto Standards

Content Alignedto Standards

Set Goals

Align activitiesto goals

Motivation

Engagement Opportunities to Practice

Process

Structure

Content

Process

25 / 28

Example: Outcome based logic model

Linkage with School

ContentAcademic

Design

ContentAcademic

Design

ProcessAfterschool

Program Implementation

ProcessAfterschool

Program Implementation

Attitudes & Skills

Attitudes & Skills

EngagementEngagement

Linkage with Standard

Research-based

Internal Program Structure

External Communication

& Support

Student/Adult Interaction

Parent Satisfaction

Structure Program Structure

Structure Program Structure

Indicators Design & Process Immediate Outcome Expected Outcome Long-term Outcome

Professional Development

AchievementAchievement

26 / 28

Using data for continuous improvement

Plan of ActionPeriodic

Assessment

Analysis of Results

Specification of MonitoringContinuous

Adjustment

Data Based decision Making Process

Theory and Research-

based Programming

Student Engagement

Student Retention

Student Improvement

27 / 28

Looking Forward

Gap in knowledge as we are working to develop the 21st century skills:

• Technology divides

• The contextual environmental

• Social capital and human networking system

dhuang@cse.ucla.edu