SIZING UP STANDARDIZED ASSESSMENTS:

BREAKING DOWN THE ETS PROFICIENCY PROFILE, MAJOR FIELD TESTS, AND THE ROOWRITER

By Nathan Lindsay and Dan Stroud

January 30, 2013

ASSESSMENT AT UMKC

Consists of a multifaceted approach:

Formative and Summative Assessments

Direct and Indirect Assessments

Direct assessments include embedded assessments (in the classroom), and standardized assessments

UMKC STANDARDIZED TESTS OVERVIEW

For many years, undergraduate students at UMKC have participated in the following standardized tests:

The ETS-Proficiency Profile test that measures General Education learning outcomes

Major Field Tests (MFTs) that measure learning outcomes for students in their disciplines.

The WEPT (soon to be the RooWriter) that measures students’ writing ability

For the ETS tests, only this past year did we purchase a reporting license that allows us to closely analyze the data

QUESTION

What are some of the pros and cons of standardized tests?

WHY STANDARDIZED TESTS MIGHT BE USEFUL ALONG SIDE IN-HOUSE ASSESSMENTS

o Standardized tests allow comparisons to both national and peer data (which are requested by external constituents).

o There are benefits in the use of longitudinal data.

o ETS test design includes strong:1) Validity2) Reliability3) Fairness/Equity4) Representation

HOW ETS DEVELOPS TEST QUESTIONS

Below is the process by which tests and test questions are developed for future ETS Proficiency Profile tests and MFTs:

http://www.ets.org/s/understanding_testing/flash/how_ets_creates_test_questions.html

TEST SCORING

Two Types of Test Scores

Norm-referenced (scaled) scores - These scores pull their meaning from comparing the scores between and individual student or group of students with those of either the same individual or group or another at different points of time. The majority of test results are of this nature.

Criterion-referenced (proficiency classified) scores - These pertain to the level of proficiency students obtained within a particular skill set. This type of score stands on its own – with no need to compare to another group, though they can still use specific comparisons as well.

Sampling the Proficiency Profile

Sampling the Proficiency Profile

Sampling the Proficiency Profile

ETS – PROFICIENCY PROFILE (IN 2011-2012, N = 1448 STUDENTS)

Test given to all students after they have completed 70 credit hours. Completion is a requirement of graduation.

Scores were above benchmark peers for all four areas: Reading Critical Thinking Writing Mathematics

Scores on individual items exceeded the national average 51 of 53 times.

Higher scores may be due to the fact that most UMKC students take the ETS-PP right before they graduate (which is not always true at other institutions)

MAJOR FIELD TESTS (MFTS)

UMKC students take the MFTs in the following areas: Biology Business Psychology Political Science Chemistry Computer Science Mathematics Physics

20 students have to take the test to have sufficient comparative data, so results are only available for some of these disciplines.

TWO REPORTS AVAILABLE FOR MAJOR FIELD TESTS

The Custom Comparative Data Report indicates how UMKC’s students' skills and knowledge compare with the skills and knowledge of students at similar institutions.

In the Item Information Report, the items are prioritized first by content, and subsequently by sub-content to make comparisons easier for grouping. Scores compared in this report are referenced to the overall national percentage for each item.

MAJOR FIELD TESTS RESULTS AT UMKC

Biology (N = 87 Students) – UMKC scored above average on 8 of 9 Assessment Indicators.

Business (N = 192 Students) – UMKC scored above average on 4 of 9 Assessment Indicators.

Psychology (N = 81 Students) – UMKC scored above average on 2 of 6 Assessment Indicators.

Political Science (N = 20 Students) – UMKC scored above average on 1 of 3 Assessment Indicators.

DISSEMINATING THE ETS RESULTS

The General Education scores have been reviewed by the General Education Committee, the University Assessment Committee, and the Assessment Academy Team.

These MFT reports have been sent to all relevant departments, which have been encouraged to discuss how they can enhance their curriculum to address the areas of weakness within each test.

A summary of our standardized tests will be shared with the Higher Learning Commission.

We will collect data during the 2012-2013 academic year to determine any changes over time.

DEPARTMENTAL DIALOGUE: CONVERTING THE DATA TO ACTION

Use the data to stimulate conversation in the department about the curriculum and pedagogy surrounding these topics. Address the following questions:1. How do the content areas of the test relate to our degree’s

learning outcomes? Should test items be added?2. Are there any areas of the curriculum that could/should be

emphasized more strongly? [curriculum mapping]3. Do the results have face validity? How do the results

compare to our findings from course-based assessments and our program/degree assessments?

4. How did different student groups/demographics score on the test? Should more sub-groups for analysis be created?

5. What are other ways that we could address/assess the areas that were lower? Should we develop our own test?

6. What trends do the scores show longitudinally?7. How could we increase student motivation for the test?

HOW TO CUSTOMIZE MFT REPORTING GROUPS

Cohorts

- Determine which students to put in what

groups as you design report emphasis

Subgroups

- You can create up to 16 subgroups

- For each subgroup, you can design two questions with up to 8 possible responses.

There must be a minimum of 5 students in each group, cohort, or subgroup. Minimums for reporting

cohorts or subgroups must add up together to 20

as a minimum to report.

DISCUSSIONAre there other ways that your unit/department could develop test questions that would evaluate the learning outcomes for your degree(s)?

Questions to consider: o What’s the point/purpose of the test?

o What are the intended uses of this test?

o What aspects of each unit or department do we wish to measure?

o What scores will be reported?

o Is the test in question statistically reliable, valid, and fair?

STUDENT MOTIVATION FOR STANDARDIZED ASSESSMENTS

Key Issue - High stakes for institutions - Low stakes for students

Incentivizing is two tiered - Get students to take the test - Get students to do their best

Best Practice—Connect to Academic Outcomes - Participation grade for a class - Students who attain a score of _______

or higher receives _______?

DISCUSSION

What incentives could be used to help students to take standardized testing more

seriously?

OTHER WAYS TO ENCOURAGE MEANINGFUL PARTICIPATION

Inform students well in advance

Include the MFT in the syllabus

Send letter or email from Dean/Department Chair

Develop faculty understanding of the purposes of the test

Create interdepartmental competition

Develop an Assessment Day (Long range goal)

WEPT This is a Pass/Fail test, and approximately 5% of

students have failed the test the last few semesters

Each test is rated on 4 areas:  Point/Thesis/Idea – Evaluated on the clarity of purpose

throughout the essay. Development/Support/Evidence – Evaluated on the

relevance and convincingness of the proof offered in support of the thesis.

Organization/Logic/Coherence – Evaluated on the ease of understanding and logical arrangement of the essay.

Usage/Mechanics/Documentation – Evaluated on the ease of reading and grammatical and style correctness

The test is graded on a four-point scale The average scores for area one (i.e., “Point/Thesis/Idea”)

were the highest at 3.10, up from the 3.07 in the previous semester. The remaining three scores average fell near 2.82 (2.82, 2.83, and 2.80, respectively).

THE ROOWRITER To replace the WEPT for incoming freshmen

in Fall 2013 An online diagnostic/formative test with

many benefits for students, faculty, and the institution (see attached handout)

The University Writing and Reading Board is currently working on finalizing all of the logistics for the test

Please refer to the RooWriter website for an overview: http://dev.umkc.edu/roowriter/index.cfm

QUESTIONS

Are there other standardized tests that UMKC should be considering?

What other questions about standardized tests do you still

have?