tuesday ets air symposium data analysis

8/7/2019 Tuesday ETS AIR Symposium Data Analysis

1/44

An Overview of Data Analysis

Jonathan SteinbergSenior Research Data Analyst, Data Analysis Research

Bruce A. Kaplan

Director, Data Analysis Research


2/44

Copyright 2008 by Educational Tes ting Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of

Educational Testing Servic e (ETS). The AIR logo is a trademark of the Associati on for Institutional Res earch. 9116

Outline

Key Considerations in Data Analysis

Components of a Data Analysis Plan

Distinguishing Data Types Distinguishing Different Types of Analyses

Overview of Different Statistical Software

Case Study: ACESTM


3/44



Key Considerations in

Data Analysis

Identify the purpose of the analysis or project

Understand the sample(s) under study

Understand the instruments being used to

collect data

Be cognizant of data layouts and formats

Establish a unique identifier if matching ormerging is necessary

Plan your work and work your plan!


4/44



Components of a

Data Analysis Plan

Statement of research questions

Methods used to answer research questions

Timeline Budget

File restructuring procedures (syntax creation, adding new

variables as needed)

Algorithms for scoring, equating, etc. Data cleaning procedures (e.g. removing outliers)

Quality control procedures at every step in the project


5/44



Examples of Analyses

Frequency Distributions and Cross -Tabulations

Descriptive Statistics (Means, Std. Deviations, Correlations)

T-tests and Analysis of Variance (ANOVA)

Regression

Principal Components/Factor Analysis (Data Reduction)

Cluster and Discriminant Analyses (Segmentation)

Latent Class Analysis (Classification)

Hierarchical Linear Modeling (HLM)

Differential Item Functioning (DIF)


6/44



More Advanced the Analysis,

Greater the Amount of Preparation

Most analyses can be executed straight from a

working data file Some analyses may require transformations of

the raw data, subsets, or specific input data to

comply with statistical software


7/44



Data Types & Representation

Variables may require special coding for

different data representation Numeric

String

Date & time

Monetary


8/44



ASCII Text Files

Usually rectangular in structure One record per observation

Each data variable in same position on each record

Each record may have multiple instances of data Arrays

Repeating blocks (sets of variables)

File may have multiple records per observation Number of records per observation can be variable

Most government data files come in this format at aminimum

Every software package can handle this file type


9/44



CSV Files

(Comma-Separated Values)

Individual data elements separated by commas

Usually rectangular structure One record (line) per observation

Fixed number of elements on each record

Problems if data elements contain delimiter or blank

spaces (i.e. text strings) Missing data must be represented by nulls


10/44



System Files

(SAS

, SPSS

)

Data stored in binary (machine) format

Issues of portability across platforms

Structured as rectangular tables

SAS files can be indexed for direct access

Self-contained documentation

Data variable labels & formats

Data value labels

Most analysis packages provide facility for reading (but not writing!)system files from other packages (SPSS more than SAS)

Using default data formats can yield system files that are much largerthan source ASCII files


11/44



Relational Databases

Accessible using Structured Query Language

(SQL)

Optimal for filtering & sorting

One or more rectangular tables contain data of

interest

This format is often at odds with statistical analysis

needs


12/44



Relational Database

Example

Index Student School Gender Ethnicity

1 29101 291 F 3

2 29102 291 M 2

3 29201 292 F 6

Student

SAT

Index Student TestDate Measure Score

1 29101 10/13/2006 Verbal 650

2 29101 10/13/2006 Math 640

3 29102 5/21/1995 Verbal 370

4 29102 5/21/1995 Math 400

5 29201 2/3/2000 Verbal 520

6 29201 2/3/2000 Math 550

Index Student TestDate Test Measure Score

1 29101 5/6/2006 General Verbal 760

2 29101 5/6/2006 General Quantitative 540

3 29102 5/3/2006 General Verbal 340


5 29102 5/3/2006 General Analytical 0

6 29102 Subject Subject1 540


8 29201 5/6/2006 General Verbal 530



GRE


13/44



Hierarchy of

Statistical Software

Excel

General Stats

Psychometric

Advanced Modeling

SAS, SPSS

IRT: PARSCALE, BILOG

Factor Analysis: LISREL, EQS

Nested Models: HLM


14/44



Overview

SAS and SPSS are most commonly used and tend to focus onthe classic statistical routines: Descriptive statistics and non-parametric (distribution-free) tests

ANOVA / Regression Factor analysis

However, many psychometric procedures (e.g. IRT) and newerstatistical models are not as well supported by these programs Very specialized programs are used

Designed to do a specific task or validate a theory

Specialized programs may have issues Interface not very user-friendly

Additional data types or files required

Expense


15/44



What is Excel?

Data are organized by worksheets, rows and columns

Worksheet limits are 256 columns and 65,536 total cells

C

ells contain data or formulas with relative or absolute references toother cells

Direct manipulation of data and flexibility to move data around (e.g.sorting, replacing, merging)

Opens many file types

Quite useful in prepping files for use in SPSS, SAS or other programs Conditional formatting

Also features macro capabilities, replicating user actions, allowingsimple automation of regular tasks


16/44



Data Presentation

Options in Excel

Tables and graphs can be exported to a wide variety of software packages

Can tweak and perfect example graph or table and then replicate by

replacing only the data being used

Main advantage is ability to combine data from multiple sources not just

what is found in the data file

Two-for-One deal - table creation usually puts data into a format that leads

to easy graph creation

User has control over virtually all aspects of a graph - size, colors, fonts, titles,

legends, labels, etc. Can combine graphs with tables and use cell layout to produce more complex

presentations

Final graphs can be of publication quality


17/44



Data Presentation

Options - Example

0%

10%

20%

0%

0%

0%

60%

0%

80%

90%

100%

00 and

belo

10 to

800

810 to

900

910 to

1000

1010 to

1100

1110 to

1200

1210 to

1 00

1 10 to

1

00

1

10 to

1 00

1 10 to

1600

C

mbi

(V

rb

l + M

t

)

r

i trib ti f Fr m G b C mbi r

A(

.90 to

.2!

)

A-(

.60 to

.89)

" +(

.

0 to

.!

9)

" (2.90 to

.29)

"

-

(2.60 to 2.89)

C+

(2.

0 to 2.!

9)

C(1.90 to 2.29)

C-

(1.60 to 1.89)D

(0.60 to 1.!

9)

#

("

elo$

0.!

9)


18/44



What is SAS?

A general purpose statistical package with a basicprogramming capability utilizing scores of statistical andmathematical functions in numerous modules

Can readily access data from a wide variety of sources,perform data management, and present findings in avariety of report and graph formats

Provides powerful tools for both specialized andenterprise-wide analytical needs


19/44



SAS - Strengths

Versatile data input and output formats

SAS provides both SQL and DATA steps tomanipulate data: SQL provides a way of carrying out relational algebra

on tables and views SAS data sets can be indexed for direct access or

processed sequentially, without reading all records

into memory, which is sometimes much more efficient


20/44



SAS - Weaknesses

Steep learning curve; volume of functions, options anddocumentation can be overwhelming for the novice

Inconsistent syntax across different procedures ormodules

Not a good choice for applications that interact withexternal systems such as hardware devices orsoftware programs because of its inconvenient

interface

Difficult interaction with other programming languages

Expensive


21/44



What is SPSS?

A commercially produced statistical software packagethat is widely used in the fields of Education andPsychology

Program functionality is broken into over a dozendifferent modules which are sold individually

Most commonly used are Base, Regression Models, andAdvanced Models

Other modules can be installed to run more complexanalyses

SPSS data files include both the data and also variableinformation (variable and value labels, formats andmissing values)


22/44



SPSS - Strengths

Easily opens data from other programs such as Excel and SAS

Variable view screen allows for quick overview of file contents andallows for easy modifications of names, formats, labels, and variable

order Having all data information in a single file allows sharing files on a

project to be very easy

Point-and-click menus do not require memorizing syntax for majorityof procedures

Many procedures can be expanded beyond the menu options in syntax

Split-file command allows all output to be replicated for variousgroups through a single command

Journal file tracks all commands used for life of program, with goodresources to find code accidentally deleted


23/44



SPSS - Weaknesses

Ease of doing data manipulation can sometimes lead tomistakes as the program does not preclude inappropriatemodifications to the data

Matching feature requires exact match Duplicate records generate warnings but can be marked in file

Error logs are hard to interpret at times

Incompleteness of menus means some options are onlyavailable via syntax

While the majority of output is saved as pivot tables allowinggreat flexibility in modifying tables

Output tables and graphs generally not done as well as Exceland are harder to manipulate


24/44



LISREL

Ideal for discrete data types

Test data, Likert scale item data

Data can be imported in various types

ASCII, Access, Excel, SAS, SPSS, etc.

Variable names have length restrictions

Data files then stored as system files for later use

Basic statistics (e.g. means and correlations) are generated in an

underlying program called PRELIS LISREL itself is used to confirm the structural validity of a

measurement model for any assessment

Requires syntax and input matrices


25/44



EQS

Ideal for continuous data types (test subscales)

Data can be imported in various types ASCII,

Access, Excel, SAS, SPSS, etc. but variable names

have length restrictions

Data files then stored as system files for later use

EQS itself is used to confirm the structural validity ofa measurement model for any assessment

Some model syntax can be built through the menus


26/44



HLM

Hierarchical Linear Modeling (HLM) is becoming a more

popular type of analysis, namely in cohort trend modeling

Also allows you to look at variance component estimatesand regression models given a nested sample ofrespondents

Students within countries within global regions on personality

variables

More tedious to set up analysis with fewer available filetypes

Also requires more upfront work as multiple data files are

needed


27/44



What Program

Should I Use?

Microsoft Excel is the most basic and accessible spreadsheet programavailable today

It is most ideal for general data exploration, histograms, scatter plots, etc.

Appearance of tables can be customized to meet APA standards Allows for easy transition to other programs to complete analyses and

write reports

However, its heritage is not as a statistical analysis program

Certain statistical programs are designed for specific analytic tasks

Balance the results and what will being presented Choose wisely in the interests of efficiency and accuracy of results

Some output is good forlookingat the data through basic exploration andto generate basic tables, but not to present the data


28/44



Summary

Be very clear about the analysis objectives

Be very familiar with all aspects of what defines your data

Develop and stay true to your data analysis plans and

research questions

Be cognizant of which statistical software programs can best

answer your research questions and present your results

Be thorough in your analyses, express openness to additionalinvestigations, yet be mindful of limitations given the data and

the programs you are using


29/44

Overview of the Admitted Class

Evaluation Service (AC

ES)Bruce A. Kaplan

Director, Data Analysis Research


30/44



What is ACES?

ACES is a validity study service offered by theCollege Board that is operated and maintained

by ETS.


31/44



Types of ACES Studies

There are two types of ACES studies:

Admission

Evaluate the validity of tests in predicting how well

students will do in an institution of higher education

(used for admitting students into an institution)

Placement Evaluate the validity of tests in placing students into

or out of a class


32/44



ACES Admission -

Types of Data

Institutions have many variables to choose from

when requesting their models The required variables are:

First-Year Grade Point Average (FGPA), supplied by the

institution

A measure of high school success (either from the SATQuestionnaire or supplied by the institution)

A choice from a list of SAT Test scores


33/44



ACES Admission -

Types of Data (continued)

The optional variables are:

Course data (name of course, grade received, and

number of credits awarded) for each student,supplied by the institution

A choice from a list of SAT Subject Test scores.

Additional predictors, either supplied by the

institution or chosen from a list of ACES predictors Additional subgroups to be analyzed, either

supplied by the institution or chosen from a list of

ACES subgroups


34/44



Data Merging, Matching, -

and Cleaning

ACES data poses many challenges

Merging

Institutions have choices of format type, and sending two files orone

ACES merges all this information in a more or less standardized

format

Matching

Institution data is matched against a file of college seniors to pull

SAT scores, SAT subject scores and SAT QuestionnaireResponses

Cleaning

Data is checked for outliers, out of range, and implausible values


35/44



ACES Admission -

Types of statistics used

Basic Statistics

Means and Standard Deviations for Total Group andby Gender

Correlations for various predictor variables with:

First-Year Grade Point Average (FGPA) for Total group andsubgroups

Adjusted First-Year Grade Point Average (AGPA)

Adjusted for course difficulty

Both are corrected for Restriction of Range


36/44



ACES Admission -

Types of statistics used (continued)

Various Regressions with up to six classes of

predictor variables

SAT (SAT-r) variables only

SAT-r and high school measure (either HSGPA or HS rank)

SAT-r, HS measure and school requested variables

SAT-r and SAT Subject Tests (SAT-s)

SAT-r, SAT-s, and HS measure

SAT-r, SAT-s, HS measure and school requested variables


37/44



ACES Admission -

Types of statistics used (continued)

At Risk Students are students whose Actual FGPA

fell 1.5 or more standard deviations below their

Predicted Grade Point Average (PGPA) PGPA for a student is computed from regression

model with the most data available for that student

Return to institutions the data files used so they can

conduct further research. This includes any data sentto ACES along with the matched SAT variables, and

calculated values like PGPA and AGPA


38/44



ACES Admission -

Restriction of Range

What is Restriction of Range?

We want to measure the correlation between FGPAand SAT scores for the applicant pool for a given

institution

We only have the attending student pool for a giveninstitution

What does that do to our correlations?

Demonstration of Restriction of Range: http://cnx.org/content/m11196/latest/

Our solution: use the SAT senior cohort to helpestimate the correlations


39/44



ACES Admission -

Course Adjustment

Are all FGPAs at an institution really equal?

Adjusted FGPA (AGPA) using institution-supplied coursegrades

For the institution, calculate each students predicted FGPA using alinear regression with SAT CR, SAT M, SAT W (if used), and HSvariable as independent variables and FGPA as a dependentvariable

Calculate an average residual per course

C

alculate an adjustment factor for each student, using the averageresiduals for the courses they took, weighted by the number ofcredits for that course

Apply the adjustment factor to the FGPA to obtain theAGPA


40/44



ACES Placement -

Statistics Used

Distributions of Predictor variables (Accuplacer

scores, CLEP scores, SAT Scores, or school

supplied variables) by Different levels ofCourse

Grade

Logistic regression calculated for two dependent

variables

Pass = B or higher, else Fail

Pass = C or higher, else Fail


41/44



ACES Placement -

Statistics Used (continued)

Models provided for single predictors, and multiple

predictors

For each Model

A measure of association between the dependent and

predictor variables is computed

Percent correctly placed

Cut Scores Associated with Predicted Probability of

Success


42/44



You can submit a study

for your Institution

Examples of sample reports:

Admissions Report

http://www.collegeboard.com/prod_downloads/highered/apr/aces/sample_admiss.pdf

Placement Report

http://www.collegeboard.com/prod_downloads/highered/apr/aces/sample_place.pdf

ACES site to request a study

http://professionals.collegeboard.com/higher-

ed/validity


43/44



Contact Information

Jonathan Steinberg:[email protected]

Bruce Kaplan: [email protected]

For additional ACES help and information:

[email protected] Or call (609) 921-9000 and ask for help with ACES


44/44

Questions orComments

tuesday ets air symposium data analysis

Documents