lally school of management & technology michael j. kalsher psychometrics mgmt 6971 1 mgmt 6971...

Lally School of Management &

Technology

Michael J. Kalsher

PSYCHOMETRICSMGMT 6971

1MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher

Week 1: Introduction and Research Design

MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher

Course Overview

• Review of research design/methodology and statistical concepts

• Review of SPSS (data entry; setting up variables; graphing; syntax; etc.)

• Statistical analysis techniques– Covariance, correlation, simple regression, multiple regression– t-tests, ANOVA / ANCOVA / MANOVA– Non-parametric statistics– Factor analysis, Multilevel Linear Models, Structural Equation

Models

• Grading requirements– Exams, Labs, Problem Sets, Data Collection/Analysis Project

2


Research Methods & Design: Establishing Control over your variables

• Historical foundations of scientific research in the behavioral and social sciences.

• The importance of research design– Ruling out alternative explanations.– Establishing control of IVs.

• Research Design vs. Statistical Analysis

3


Methods of Establishing Truth• Tenacity

– “It’s so because it’s so”• Authority

– “Aristotle said it’s so”• Logical Deduction (Rationalism)

– Aristotle said women have fewer teeth than men (Premise)– You are a woman– Therefore, you have fewer teeth than I

• Empiricism– Combines Logical Deduction with observation

(measurement)– “Let’s count your teeth”

4


Scientific Method

• Shared observations– Rules out individual experiences like religious

revelations or esthetic experiences (William James).

• Reproducible Effects– “No miracles”

• Conditional Truths– Premises may be wrong– Necessary Connection may be wrong

5


Types of Relationship (between two concepts)


Spurious Relationships


Spurious RelationshipsIce Cream Sales

Swimming Pool Drownings

Heat Wave

A city's ice cream sales are found to be highest when the rate of drownings in the city’s swimming pools is highest. To allege that ice cream sales cause drowning, or vice-versa, would be to imply a spurious relationship between the two. In reality, a third variable, in this instance a heat wave, more likely caused both.


Sets of Relationships (a theory)


High Experimental Research

Differential Research

Correlational Research

Case-study Research

Low Naturalistic Observation

Exploratory Research

De m

and

Research plan becomes increasingly detailed (e.g., precise hypotheses and analyses) but less flexible.

Research plan may be general, ideas, questions, and procedures relatively unrefined.

A Model of the Research Process: Levels of Constraint

(Model used to illustrate the continuum of demands placed on the adequacy of the information used in research and on the nature of the processing of that information.)

10


Independent variable A variable that is actively manipulated by the researcher to see what its impact

will be on other variables.

Dependent variable A variable that is hypothesized to be affected by the independent-variable

manipulation.

Extraneous variable Any variable (usually unplanned or uncontrolled factors), other than the

independent variable, that might affect the dependent measure in a

study.

A constant Any variable prevented from varying (by holding variables constant, they do

not affect the outcome of the research).

Classes of Research Variables:

Variables defined by their use in research

11


Variable values are represented by numbers, but these numbers may not demonstrate all the characteristics of true numbers.

1. Nominal. A variable made up of discrete, unordered categories. Each category is either present or absent and categories are mutually exclusive and exhaustive (e.g., gender).

2. Ordinal. A variable for which different values indicate a difference in the relative amount of the characteristic being measured.

3. Interval. A variable for which equal intervals between variable values indicate equal differences in amount of the characteristic being measured.

4. Ratio. Ratios between measurements as well as intervals are meaningful because there is a starting point (zero).

Classes of Research Variables:

The Measurement Model

12


Levels of Measurement

Nominal Ordinal Interval Ratio

Diagnostic categories Socioeconomic Test scores; Weight; length;

brand names; political class; ranks personality and reaction time;

or religious affiliation attitude scales # of responses

Identity Identity; magnitude Identity; magnitude Identity; magnitude;

equal intervals equal intervals;

true zero point

None Rank order Add; subtract Add; subtract;

multiply; divide

Nominal Ordered Score Score

Chi Square Mann-Whitney t-test; ANOVA t-test; ANOVA

U-test

Examples

Properties

MathematicalOperations

Type of Data

Typical Statistics

Scales of Measurement: Some Examples

13


The Role of Variance

14

- In an experiment, IV(s) are manipulated to cause variation between experimental and control conditions.

- Experimental design helps control extraneous variation--the variance due to factors other than the manipulated variable(s).

Sources of Variance- Systematic between-subjects variance

Experimental variance due to manipulation of the IV(s) [The Good Stuff]

Extraneous variance due to confounding variables.

Natural variability due to sampling error

- Non-systematic within-groups varianceError variance due to chance factors (individual differences) that affect some participants more than others within a group

[The Not-So-Good Stuff]


Separating Out The Variance

15

SST = Sums of Squares Total

SSM = Sums of Squares Model

SSR = Sums of Squares Error

SST

SSM SSR


Controlling Variance in Experiments

16

In experimentation, each study is designed to:

1. Maximize experimental variance.

2. Control extraneous variance.

3. Minimize error variance.• Good measurement• Manipulated and Statistical control


Controlling Variance in Observational Studies

17

• Choose IV’s with large natural variance• Control for alternate explanations by

measuring confounding variables and statistically removing their variance

• Minimize error variance– Good measurement– Statistical control


Maximizing Experimental Variance: Strong manipulations and Manipulation Checks

18

Experimental Variance (The Good Stuff)

Due to the effects of the IV(s) on the DV(s)

Ensure that experimental manipulations are strong and reliable!

Manipulation CheckProcedures designed to determine whether manipulation

of the IV(s) had the intended effect(s) on the DV(s)


Controlling Extraneous Variance

19

Extraneous variables: Between-group variables--other than the IV(s)--that have effects on whole groups and thus may confound the results.

Goal: To prevent extraneous variables from differentially affecting the groups.

Solution: Take steps to ensure that: (1) the experimental and control groups are equivalent at the beginning of the study; and (2) groups are treated exactly the same--save for the intended manipulation (of the IV).

Methods (for controlling extraneous variance): 1. Random Assignment of subjects to experimental conditions2. Select participants on the basis of one or more potentially confounding

variables (e.g., age, ethnicity, social class, IQ, sex). 3. Build the confounding variables into the study as additional IVs.4. Match participants on confounding variable or use within-subjects design


Test Statistics

20

Essentially, most test statistics are of the following form:

Test statistic = Systematic variance

Unsystematic variance

Test statistics are used to estimate the likelihood that an observed difference is real (not due to chance), and is usually accompanied by a “p” value (e.g., p<.05, p<.01, etc.)


A Very Simple Statistical Model

outcomei = (model) + errori

• model – an equation made up of variables and parameters

• variables – measurements from our research (X)

• parameters – estimates based on our data (b)

outcomei = (bXi) + errori

outcomei = (b1X1i + b2X2i + b3X3i)+ errori

21


Examples of Statistical Models

• One Predictor (e.g. deviance):outcomei = (bXi) + errori

outcomelecturer1 = mean + errorlecturer1

errrorlecturer1 = mean – outcomelecturer1 = 1 – 2.6 = -1.6

• Multiple Predictors (e.g. sum of squared errors):outcomei = (b1X1i + b2X2i…)+ errori

errori = (outcome1 – model1)2 + (outcome2 – model2)2 …

= (-1.6)2 + (-0.6)2 + (0.4)2 + (1.4)2 = 5.20

22


Types of Hypothesis

• Neyman and Pearson proposed organizing scientific statements into testable hypotheses.– H0 – null hypothesis, that no effect will occur

• Adding a narrative component to a video game will not affect gameplay experience

– H1 – alternative (or experimental) hypothesis, that the effect you are testing for will occur

• Playing a game with a narrative component will improve your gameplay experience

• Data cannot prove alternative hypotheses, only reject null ones 23


Null Hypothesis Significance Testing (NHST)

• NHST combines Fisher’s work with Neyman and Pearson’s– Initially assume null hypothesis is true– Choose a statistical model that represents an

alternative hypothesis– Calculate p-value of the null hypothesis producing

this model– If p < .05 (generally), model fits and alternative

hypothesis is supported

• We’re never certain, we just have evidence24


One- and Two-tailed Tests

• One-tailed: directional results (effect is present or not)

• Two-tailed: directional results (effect increases, decreases, or no effect)

25


Types of Mistakes

26

Statistical decision

Reject Ho

Don’t reject Ho

True state of null hypothesis

Ho true Ho false

Type I error Correct

Correct Type II error


Inflated Error Rates

• A measure of how well Type I errors have been avoided

• In most research, the complexity of the question requires more than one test. The rate of error increases with the number of tests done, increasing the Type I error. This is called familywise error.

• Solution? Choose a stricter p-value for each individual test (Bonferroni correction)

required p-value per test = (desired overall p-value)/(number of tests)

27


Statistical Power

• A measure of how well Type II errors have been avoided (i.e. how well a test is able to find an effect)

• = 1 – type II error rate• Power should be 0.8 or higher, so Type

II error rate should not exceed .20.

28


Confidence Intervals & Statistical Significance

• p-value of H0 decreases with the amount of overlap between two confidence intervals

• Moderate overlap (defined as ½ the average Margin Of Error) indicates p = .05.

• MOE = ½ the length of the confidence interval:

• So moderate overlap is:

(

29


Sample Size & Statistical Significance

• Because MOE is a result of sample size (via the confidence interval), small differences can be significant in large samples, and large differences might not be significant in small samples.– This is because larger samples have more

power to detect effects when they exist.

30


Effect Sizes: The Correlation coefficient

31

The statistical test only tells us whether it is safe to conclude that the means come from different populations. It doesn’t tell us anything about how strong these differences are. So, we need a standard metric to gauge the strength of the effects.

The correlation coefficient (r) is one metric for gauging effect size.

• Ranges from 0 – 1 (no effect to perfect effect)• Rough cutoffs (nonlinear, that is twice the r value

doesn’t necessarily mean twice the effect)– 0.10 – small effect (explains 1% of the variance)– 0.30 – medium effect (explains 9% of the variance)– 0.50 – large effect (explains 25% of the variance)


Effect Sizes: The coefficient of determination

32

The statistical test only tells us whether it is safe to conclude that the means come from different populations. It doesn’t tell us anything about how strong these differences are. So, we need a standard metric to gauge the strength of the effects.

r2 (r-Square), or the “Coefficient of Determination”, is one metric for gauging effect size.

Rules of Thumb regarding effects sizes:

Small effect: 1-3% of the total variance

Medium effect: 10% of the total variance

Large effect: 25% of the variance

r2 =SSM

SST


– Uses the same unit for all data (standard deviation units)

– Provides information about the signal-to-noise ratio – how large is the effect in comparison to other effects on the same data?

– = (the difference of the means) divided by the standard deviation

– Effect cutoffs (but remember this is only rough):• 0.2 – small• 0.5 – medium• 0.8 – large

33

Effect Sizes: Cohen’s d


Meta-Analysis

• An average of the effect size of multiple studies that all address the same question– Weighted to favor more precise studies over less

precise ones

• Useful for getting the most accurate information about the population as a whole

• Not easily done in SPSS

34


Reporting Statistical Models

• APA recommends exact p-values for all reported results; best to include an effect size, too– Effect “x” was not statistically significant in condition y, p

= .24, d = .21

• Report a mean and the upper and lower boundaries of the confidence interval as M = 30, 95% CI [20,40]– If all confidence intervals you are reporting are 95%, it’s

acceptable to say so and then later say something like:In this condition, effect x increased, M = 30 [20,40].

35


Reliability Getting the same result when a measurement device is applied to the same quantity repeatedly.

Validity The extent to which a measurement tool (test, device) measures what it purports

to measure.

Control Behavior can be influenced by many factors, some known and others unknown to the researcher. Control refers to the systematic methods employed by a researcher to reduce threats to the validity of the study posed by extraneous influences on the behavior of both the participants and the observer.

Importance Does the research question we are trying to answer warrant the expenditure of resources (i.e., time, money,

effort) that will be required to complete the study).

Essential Elements of Research: Reliability, Validity, Control and Importance

36


Types of Reliability

Test-retest ReliabilityConsistency of measurement over time

Internal Consistency Inter-item correlation

Interrater Reliability Level of agreement between independent observers of behavior(s). Assessed via correlation or the procedure at right.

AgreementAgreement + Disagreement

x 100

37


Evaluating Measures: Effective Range

Effective Range: Scales sensitive enough to detect differences among one group of subjects may be insensitive to detect differences among another.

Scale Attenuation (or range restriction). A problem associated with scales not ranging high enough, low enough, or both.

Leads to “ceiling” effects and “floor” effects that distort data by not measuring the full range of a variable.

38


Types of Validity

Face validity. The (non-empirical) degree to which a test appears to be a sensible measure.

Content validity. The extent to which a test adequately samples the domain of information, knowledge, or skill that it purports to measure.

Criterion validity. Now (concurrent) and Later (predictive). Involves determining the relationship (correlation) between the predictor (IV) and the criterion (DV).

Construct validity. The degree to which the theory or theories behind the research study provide(s) the best explanation for the results observed.

39


Internal ValidityExtent to which causal/independent variable(s) and no other extraneous factors caused the change being measured.

External Validity (generalizability)Degree to which the results and conclusions of your study would hold for other persons, in other places, and at other times.

Internal vs. External Validity

40


Selection

History

Maturation

Repeated Testing

Instrumentation

Regression to the mean

Subject mortality

Selection-interactions

Experimenter bias

Threats to Internal Validity:Factors that reduce our ability to draw valid conclusions

41


The role of ControlBehavior is influenced by many factors termed—confounding variables—that tend to distort the results of a study, thereby making it impossible for the researcher to draw meaningful conclusions. Some of these may be unknown to the researcher.

Control refers to the systematic methods (e.g., research designs) employed to reduce threats to the validity of the study posed by extraneous influences on both the participants and the observer (researcher).

Reducing Threats to Internal Validity

42


Group/Selection threatOccurs when nonrandom procedures are used to assign subjects to conditions or when random assignment fails to balance out differences among subjects across the different conditions of the experiment.

Example:A researcher is interested in determining the factors most likely to elicit aggressive behavior in male college students. He exposes subjects in the experimental group to stimuli thought to provoke aggression and subjects in the control group to stimuli thought to reduce aggression and then measures aggressive behaviors of the students. How would the selection threat operate in this instance?

43


History threat

Events that happen to participants during the research which affect results but are not linked to the independent variable.

Example:

The reported effects of a program designed to improve medical residents’ prescription writing practices by the medical school may have been confounded by a self-directed continuing education series on medication errors provided to the residents by a pharmaceutical firm's medical education liaison.

44


Maturation threat

Can operate when naturally occurring biological or psychological changes occur within subjects and these changes may account in part or in total for effects discerned in the study.

Example:A reported decrease in emergency room visits in a long-term study of pediatric patients with asthma may be due to subjects outgrowing childhood asthma rather than to any treatment regimen introduced to treat the asthma.

45


Repeated testing threatMay occur when changes in test scores occur not because of the intervention but rather because of repeated testing. This is of particular concern when researchers administer identical pretests and posttests.

Example:

A reported improvement in medical resident prescribing behaviors and order-writing practices in the study previously described may have been due to repeated administration of the same short quiz. That is, the residents simply learned to provide the right answers rather than truly achieving improved prescribing habits.

46


Instrumentation threat

When study results are due to changes in instrument calibration or observer changes rather than to a true treatment effect, the instrumentation threat is in operation.

Example:

In Kalsher’s Experimental Methods and Statistics course, he evaluates students progress in understanding principles of research design at week 3 of the semester. A graduate T.A. evaluates the students at the conclusion of the course. If the evaluators are dissimilar enough in their approach, perhaps because of lack of training, this difference may contribute to measurement error in trying to determine how much learning occurred over the semester.

47


Statistical Regression threatThe regression threat can occur when subjects have been selected on the basis of extreme scores, because extreme (low and high) scores in a distribution tend to move closer to the mean (i.e., regress) in repeated testing.

Example:

if a group of subjects is recruited on the basis of extremely high stress scores and an educational intervention is then implemented, any improvement seen could be due partly, if not entirely, to regression to the mean rather than to the coping techniques presented in the educational program.

48


Experimental Mortality threatExperimental mortality—also known as attrition, withdrawals, or dropouts—is problematic when there is a differential loss of subjects from comparison groups subsequent to randomization, resulting in unequal groups at the end of a study.

Example:

Suppose a researcher conducts a study to compare the effects of a corticosteroid nasal spray with a saline nasal spray in alleviating symptoms of allergic rhinitis (irritation and inflammation of the nasal passages). If subjects with the most severe symptoms preferentially drop out of the active treatment group, the treatment may appear more effective than it really is.

49


Selection Interaction threatsA family of threats to internal validity produced when a selection threat combines with one or more of the other threats to internal validity. When a selection threat is already present, other threats can affect some experimental groups, but not others.

Example:If one group is dominated by members of one fraternity (selection threat), and that fraternity has a party the night before the experiment (history threat), the results may be altered for that group.

50


People, Places, and Times

Demand Characteristics

Hawthorne Effects

Order Effects (or carryover effects)

Threats to External Validity:Ways you might be wrong in making generalizations

51


Example: You learn that the grant you submitted to assess average drinking rates among college students in the U.S. has been funded. In late November, you post an announcement about the study on campus to get subjects for the study. 100 students sign up for the study. Of these, 78 are members of campus fraternities; the other 22 are members of the school’s football team.

People threat:Are the results due to the unusual

type of people in the study?

52


Example: Suppose that you conduct an “educational” study in a college town with lots of high-achieving educationally-oriented kids.

Places threat:Did the study work because of the unusual place you did the study in?

53


Example: Suppose that you conducted a smoking cessation study the week after the U.S. Surgeon General issued the well publicized results of the latest smoking and cancer studies. In this instance, you might get different results than if you had conducted the study the week before.

Time threat:Was the study conducted at a peculiar time?

54


Demand CharacteristicsParticipants are often provided with cues to the anticipated results of a study.

Example:

When asked a series of questions about depression, participants may become wise to the hypothesis that certain treatments may work better in treating mental illness than others. When participants become wise to anticipated results (termed a placebo effect), they may begin to exhibit performance that they believe is expected of them.

Making sure that subjects are not aware of anticipated outcomes (termed a blind study) reduces the possibility of this threat.

55


Hawthorne Effects

Similar to a placebo, research has found that the mere presence of others watching a person’s performance causes a change in their performance. If this change is significant, can we be reasonably sure that it will also occur when no one is watching?

Addressing this issue can be tricky but employing a control group to measure the Hawthorne effect of those not receiving any treatment can be very helpful. In this sense, the control group is also being observed and will exhibit similar changes in their behavior as the experimental group therefore negating the Hawthorne effect.

56


Order Effects (carryover effects)

Order effects refer to the order in which treatment is administered and can be a major threat to external validity if multiple treatments are used.

Example: If subjects are given medication for two months, therapy for another two months, and no treatment for another two months, it would be possible, and even likely, that the level of depression would be least after the final no treatment phase. Does this mean that no treatment is better than the other two treatments? It likely means that the benefits of the first two treatments have carried over to the last phase, artificially elevating the no treatment success rates.

57

MGMT 6971 PSYCHOMETRICS © 2014, Michael Kalsher 58

The Role of Experimental Design

In most social and behavioral research studies, we attempt to obtain at least one score from each participant (usually more!). Any obtained score is comprised of a number of components:

1. A ‘true score’ for the thing we hope we are measuring.

2. A ‘score for other things’ that we measure inadvertently.

3. Systematic (non-random) bias (usually ok as long as it affects all participants equally).

4. Random (non-systematic) error (which should cancel out over large numbers of observations).

We want our obtained score to consist of as much ‘true score’, and as little of the other factors, as possible.


Research Study Control

Control removes sources of error in inferences– Reduces the chance of wrong conclusions– Increases the power of statistics to find

relationships in the presence of random error (“noise”)

Types of Control– Direct Manipulation– Randomization– Statistical Control

59


Types of Control: Direct Manipulation

Sources of error held constant by research design or sampling decisions– Example: a researcher investigating the effects of

seeing justified violence in video games on children knows that young children cannot interpret the motives of characters accurately. She decides to limit her study to older children only, to eliminate random responses or unresponsiveness of younger children.

60


Types of Control: Randomization

Unknown sources of error are equalized across all research conditions by randomly assigning subjects or by randomly choosing experimental materials.– Example: Many different factors are known to affect

the amount of use of Internet social networking sites. A researcher wants to test two different site designs. He randomly assigns subjects to work with each of the two designs. This equalizes the amount of confounding error from unknown factors in both groups.

61


Known confounding variables are measured, and mathematical procedures are used to remove their effect.– Example: A political communication researcher

interested in studying emotional appeals versus rational appeals in political commercials suspects that the effects vary with the age of the viewer. She measures age, and uses it as an independent predictor (with multivariate statistics) to isolate, describe, and remove its effect.

62

Types of Control: Statistical Control


Contrasting Methods of Control

Type of Control

Strength Weakness

Direct Manipulation

• Removes effect completely • Must know source of effect• Reduces generalizability

Randomization • Don’t have to know source of effect• Equalizes effect so there is no systematic confound

• Reduces statistical power by adding to unsystematic error variance

Statistical control

• Estimates effect of confounding variables• Expands theoretical model

• Must know source of effect• Requires more complex statistics

63


Basic Types of Research

• Observational Methods

• Quasi-Experimental Designs

• True Experimental Designs


Observational Methods

65

No direct manipulation of variables by the researcher. Behavior is merely recorded--but systematically and objectively so that the observations are potentially replicable.

Advantages• Reveals how people normally behave.• Experimentation without prior careful observation can lead to a

distorted or incomplete picture.

Disadvantages• Generally more time-consuming.• Doesn’t allow identification of cause and effect.


Quasi-Experimental Design

66

In a quasi-experimental study, the experimenter does not have complete control over manipulation of the independent variable or how participants are assigned to the different conditions of the study.

Advantages• Natural setting• Higher face validity (from practitioner viewpoint)

Disadvantages• Not possible to isolate cause and effect as conclusively as with a

“true” experiment.


Types of Quasi-Experimental Designs


One Group Post-Test Design

68

Treatment Measurement

Change in participants’ behavior may or may not be due to the intervention.

Prone to time effects, and lacks a baseline against which to measure the strength of the intervention.

Time


One Group Pre-test Post-test Design

69

Measurement Treatment Measurement

Comparison of pre- and post-intervention scores allows assessment of the magnitude of the treatment’s effects.

Prone to time effects, and it is not possible to determine whether performance would have changed without the intervention.

Time


Interrupted Time-Series Design

70

Measurement

Measurement

Measurement

Treatment

Measurement

Measurement

Measurement

Time

Don’t have full control over manipulations of the IV. No way of ruling out other factors. Potential changes in measurement.


Static Group Comparison Design

71

Treatment (experimental group)

Measurement

MeasurementNo Treatment

Group A:

Group B:(control group)

Participants are not assigned to the conditions randomly.

Observed differences may be due to other factors. Strength of conclusions depends on the extent to which we can identify and eliminate alternative explanations.

Time


Experimental Research:

Between-Groups and Within-Groups Designs


Between-Groups Designs

Separate groups of participant are used for each condition of the experiment.

Within-Groups (Repeated Measures) Designs Each participant is exposed to each condition of the experiment (requires less participants than between groups design).

73


Between-Groups Designs

74

Advantages• Simplicity• Less chance of practice and fatigue effects• Useful when it is not possible for an individual to

participate in all of the experimental conditions

Disadvantages• Can be expensive in terms of time, effort, and number of

participants• Less sensitive to experimental manipulations


Examples of Between-Groups Designs

75


Post-test Only / Control Group Design

76

Treatment (experimental group)

Measurement

MeasurementNo Treatment

Group A:

Group B:(control group)

Randomallocation:

If randomization fails to produce equivalence, there is no way of knowing that it has failed. Experimenter cannot be certain that the two groups were comparable before the treatment.

Time


Pre-test / Post-test Control Group Design

77


No Treatment

Group A:

Group B:

Random

allocation:

Measurement

Measurement

Measurement

Pre-testing allows experimenter to determine equivalence of the groups prior to the intervention. However, pre-testing may affect participants’ subsequent performance.

Time


Solomon Four-Group Design

78


No Treatment

Group A:

Group B:

Ran

dom

allo

catio

n:

Measurement

Measurement

Measurement

Measurement

Measurement

Treatment

No Treatment

Group C:

Group D:

Time


Within-Groups Designs: Repeated Measures

79

• Economy

• Sensitivity

Advantages

Disadvantages

• Carry-over effects from one condition to another

• The need for conditions to be reversible


Repeated-Measures Design

80


No Treatment

Random Allocation

Measurement

Measurement

Measurement

No Treatment

Treatment

Potential for carryover effects can be avoided by randomizing the order of presentation of the different conditions or counterbalancing the order in which participants experience them.

Time


Latin Squares Design

81

One group of participants

Another group of participants

Yet another group of participants

A B C

B C A

C A B

order of conditions or trials:

Three Conditions or Trials

Order of presentation of conditions in a within-subjects design can be counterbalanced so that each possible order of conditions occurs just once. Problem not completely eliminated because A precedes B twice, but B precedes A only once. Same with C and A.


Balanced Latin Squares Design

82

One group of participants

Another group of participants

Yet another group of participants

A B C D

B D A C

D C B A

order of conditions or trials:

And yet another group of participants C A D B

Four Conditions or Trials

Note: This approach works only for experiments with an even number of conditions. For additional help with more complex multi-factorial designs, see: http://www.jic.bbsrc.ac.uk


Factorial Designs

83

• include multiple independent variables

• allow for analysis of interactions between variables

• facilitate increased generalizability


Important Concepts

84

Alternative hypothesis Dispersion Null hypothesis Score-level variable Standard Deviation

Between-groups design Effect Size Observational study Skew Standard Error

Categorical variable Experimental research One-tailed test Standard Deviation Systematic variation

Central tendency Face validity Ordinal variable Standard Error Two-tailed test

Confidence intervals Frequency distribution Outcome variable Systematic variation Type I error

Confounding variable Independent variable Platykurtic Two-tailed test Type II error

Construct validity Kurtosis Power Type I error Unsystematic variation

Content validity Leptokurtic Practice effects Type II error Validity

Continuous variable Level of Measurement Predictor variable Unsystematic variation Variance

Correlational research Mean Quasi-exp. research Validity Within-groups design

Counterbalancing Measurement error Randomization Variance z-scores

Criterion validity Median Range Within-groups design

Degrees of Freedom Mode Reliability z-scores

Dependent variable Nominal variable Repeated measures Score-level variable

Discrete variable Normal Distribution Sampling distribution Skew

lally school of management & technology michael j. kalsher psychometrics mgmt 6971 1 mgmt 6971...

Documents

kalsher psychometrics

research design slide

michael kalsher methods

michael kalsher week

research process

importance of research

theory slide

citys ice cream sales