Education 795 Class Notes

Non-Experimental DesignsANCOVA

Note set 5

Today’s Agenda

Announcements (ours and yours)

Q/A

Non-experimental design

Categorical predictors

Statistical control

ANCOVA

Interactions

Nonexperimental Designs

What distinguishes designs are a. manipulation of IV’s

b. randomization

Experiments have both a and b

Quasi-experiments have a but not b

Nonexperimental designs have neither a nor b

Prediction vs. Explanatory

Predictive research is aimed at predicting outcomes from a collection of variable

Explanatory research is aimed at testing hypotheses formulated to explain phenomena of interest

Major difference is the role of theory. Explanatory research usually lacks a theoretical framework and it is a search to uncover independent variables

Logic of Comparison

Experimental Designscomparison made among groups that, because of randomization, are equal across all things except for the “treatment”

Quasi-experimental Designscomparison made among groups that have been exposed to different “treatments” but are groups that are often not like to begin with

Logic of Comparison

Nonexperimental designsOften grouped based on the dependent variable… e.g. persisters/nonpersisters, users of technology, nonusers of technology… We then try to uncover the variables that have “caused” the observed differences in groups. So we have group problems:

they are not alike across the independent variablesthey may have been exposed to the same treatment

Sampling and design are key. In nonexperimental research we need to be very careful of the results and implications we put forth based on our samples and designs

Control vs. Comparison Groups

The major threat to validity in nonexperimental research comes from uncontrolled unmeasured covariates… confounding variables

Categorical Independent Variables

Used when we classify people into groups and are interested in group differencesUnless a categorical variable is a “treatement”, there is no causation implied… in other words a difference between males and females on an outcome becomes a question, not an answer.What is it about males and females that make them differ on the phenomena being studied?

How We Treat Categorical IV’s

For Continuous OutcomesDichotomous independent variables

t-tests

Polychotomous independent variablesANOVA or multiple regression

For nominal outcomescrosstabs, Chi-Square

Coding Categorical IV’s

We are familiar with the dichotomous independent variable sex: female/maleWe have also been using race: students of color/whiteReminder: the groups must be mutually exclusiveWe still stick to Dummy Coding in this class. (Behind the scenes, SPSS is doing effects coding)

Dummy Coding

Consists of 1s and 0s with 1 signifying membership in a category and 0 signifying nonmembership

Let’s extend our dichotomous IV to n levels. You will have n columns representing the n groups but will only use n-1 groups in a regression model.

Example With Data

RELIGION J C M O1 Jewish 1 0 0 02 Christian 0 1 0 03 Muslim 0 0 1 04 Other 0 0 0 15 Muslim 0 0 1 06 Jewish 1 0 0 0

Four columns represent four mutually exclusive groups. In this example, the first and sixth subjects are Jewish

Using Categorical IV’s

In our example, we only need three of the columns to correctly specify a contrast between two groupsThe group you leave out of the regression, will become your “reference category”A reference category is technically represented by the constantContrasts will be between the three included groups and the group you left out

Our Example

RegressionDependent Variable: Promote Racial UnderstandingIndependent Variable: Religion

If I include C, M and O as variables and leave out J. Then I will have three variables representing three contrastsC-the difference between Christians and JewishM-the difference between Muslims and JewishO-the difference between Others and JewishIf we want a different set of contrasts we choose a different group as the referent group to leave out

Statistical Control

Forms of controlmanipulation, elimination/inclusion, statistical, randomization

Manipulation—the control the researcher has over manipulation of a treatmentElimination/Inclusion—either we eliminate by holding them constant (only study females) or we include them so they can be estimatedStatistical—include them as covariates, control for them but not interested in themRandomization—with random assignment, we control for observed and unobserved covariates!!

Statistical Control inOur Context

Rather than holding IVs constant through experimental control, influence is held constant by statistical techniques (by removing influence of confounding factors)

Application to non-experimental designs makes causal interpretations difficultMeasurement concerns are important for the IVs

ANCOVA

The age old question: “What is the difference between ANCOVA and Regression?”

Those trained in experimental research are usually taught to apply and “speak” in ANCOVA termsThose trained in quasi-experimental, correlational research are usually taught to apply and “speak” in Regression terms

ANCOVA

They are parallel analytical techniques. One usually employs ANCOVA in cases where the “treatment” is manipulated and of a causal nature.The field of higher education primarily utilizes Multiple Regression.

Doing ANCOVA through a multiple regression program not only enables one to see clearly what is taking place but also affords the control necessary to carry out the analysis required. (Pedhazur & Pedhazur, 1991, p. 568).

Theoretical: Interactions

Without interactions between predictors in the model, we assume a constant effect for all levels of each independent predictor

Interactions allow the effects of variables depend on the level of OTHER independent variables.

Example: the effect of race for promoting racial understanding differs for genders implies an interaction between sex and race

Interactions

Ordinal—The regression lines do not intersect within the range of another independent variable (the rank order of the effect does not change)

Disordinal—The regression lines intersect within the range of another independent variable (the rank order of the effect changes, flips)

Graphical Interactions or Lack Of

ANCOVA Example

Reminder: Including variables as statistical controls reduces the error variance thus increases the sensitivity of the analysis.

Our example: dependent—promote racial understanding

independents—attend cultural awareness workshop

controls—sex, race

interaction—race*cultural awareness workshop

Assumptions

All the normal regression assumptions about normality, homogeneity, independence

Assume effect of attending a cultural awareness workshop is constant across males and females

Allow the effect of attending a cultural awareness workshop to vary across whites/students of color by adding the interaction

Results

Note: 21% of students attended a workshop, attending at approximately equal rates across the race groups

Results

Intepretation

The interaction is not significant so we can say:

the effect of attending a cultural awareness workshop is constant across whites vs. students of color

all three effects, sex, race, workshop are significant

females, students of color and those attending workshops are more likely to believe promoting racial understanding is important

Graphing Interactions

Effect of Attending a Cultural Awareness Workshop for Males

0

0.51

1.5

2

2.53

3.5

No Attend Attend

Workshop

Prom

ote R

acial

Un

ders

tand

ing White

SOC

Last But Not Least

Adusted MeansWe do this by plugging values

White, Male, Attend=1.99

White, Male, No Attend=1.41

SOC, Male, Attend=2.86

SOC, Male, No Attend=2.12

For Next Week

Read Pedhazur Ch 3 32-39

Read Pedhazur Ch 4 p66-70

Read Pedhazur Ch 22 p590-606