guide to the use and interpretation of scores for the ... • exadep guide to the use and...

49694-00062 • EXADEP Guide to the Use and Interpretation-English • InDCS4 • Fonts: Avenir, Helvetica, Helvetica Neu, Symbol, Times New Roman • D1 3/2/11 • RI64645 • D1 3/2/11 RI64645 • D1b 4/5/11 RI64645 • D2 4/6/11 RI64645

www.ets.org/exadep

Guide to the Use and

Interpretation of Scores

for the EXADEPTM Test


2

The Examen de Admisión a Estudios de PosgradoTM (EXADEPTM) is administered by Educational Testing Service.

Please send all your questions to the ETS office in Puerto Rico.

ETS Puerto Rico Office Phone: 787-753-6363Global Division TDD: 787-758-4598American International Plaza Fax: 787-250-7426250 Muñoz Rivera Ave. E-mail: [email protected] Floor, Suite 315 Web site: www.ets.org/exadepHato Rey, PR 00918

8th Edition, 2011.Copyright © 2011 by Educational Testing Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS) in the United States of America and other countries. EXADEP and EXAMEN DE ADMISIÓN A ESTUDIOS DE POSGRADO are trademarks of ETS.


3

TABLE OF CONTENTS Page

PREFACE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

DESCRIPTION OF THE TEST. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6EXADEP Measures Specific Reasoning Skills That Are Developed Over a Long Period of Time . . . . 7

GUIDELINES FOR THE USE OF EXADEP SCORES. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9Normally Appropriate and Inappropriate Uses of EXADEP Scores . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Appropriate Uses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10Inappropriate Uses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

CONSIDERATIONS IN SCORE INTERPRETATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11Repeat Test Takers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11Examinees with EXADEP Scores More Than Five Years Old . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11Examinees with Disabilities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

POLICY AND USE OF EXADEP SCORES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12Score Reporting Policies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12Reporting Revised Scores. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12Confidentiality and Authenticity of EXADEP Scores. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12EXADEP Scores and Graduate Admissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12Validity Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

SCORE INTERPRETATION AND STATISTICAL INFORMATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14Verbal, Mathematics and Analytical Reasoning, Written Expression, and English Sections of the Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14Standard Error of Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15Reliability Coefficient. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

STATISTICAL TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16How to Compare the Scores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Mean. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16Percentile Ranks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Table 1: Business . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17Table 2: Science . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18Table 3: Education. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19Table 4: Law . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20Table 5: All Other Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21Table 6: All Candidates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

STEPS THAT ETS TAKES TO ENSURE FAIRNESS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23Involving External Faculty Members in the Development of the Test. . . . . . . . . . . . . . . . . . . . . . . . . 23Differential Item Functioning Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23


4

PREFACE

Our mission at Educational Testing Service® (ETS®) is to advance quality and equity in education by providing fair and valid assessments, research, and related services. The Guide to the Use and Interpretation of Scores for the EXADEP™ supports this mission by helping to ensure that the test is created and scores are used under fair and appropriate ETS guidelines. The purpose of this guide is to help institutions or university members responsible for graduate admissions and fellowship awards to use EXADEP scores as an additional aid in their decision making. Understanding the contents of this guide will help prevent misuse and misinterpretation of the test scores and potential unfairness to applicants.

An absolute commitment to excellence, integrity, and fairness is at the core of everything we do. Our products and services are meant to help educators, educational institutions, businesses, and governments further education for all people worldwide. Additional copies or information about the test or services may be obtained from the ETS Puerto Rico Office by e-mail or visiting our Web site at: www.ets.org/exadep.


5

INTRODUCTION

EXADEP scores can be used by admissions or fellowship panels to supplement undergraduate records and other qualifications for graduate study. The scores provide common measures for comparing the qualifications of applicants and aid in the evaluation of grades and recommendations. Any accredited graduate or professional school or any department or division within a school may require or recommend that its applicants take the test. An institution not accredited by a recognized agency can become a score user if approved by the EXADEP Program.

The weight to be given to EXADEP scores can generally be established by relating what the tests measure to the orientation, curriculum, and aims of a department. Specifically, the content validity of the tests for a graduate department should be determined by reviewing each test carefully and then making subjective decisions as to the weight, if any, the scores on EXADEP tests should receive in relation to other admission factors.

Before taking the EXADEP, students should have completed at least two years of undergraduate study in an institution where the language of instruction is Spanish. Students should indicate on their registration form the institutions to which they want their test scores sent.

The institutions or the faculties within an institution that require or recommend that the candidates take the EXADEP should announce it in their publications and on their Web site, and they should notify the candidates. It is also important to let the candidates know the deadlines for receiving scores. For institutions that are considering using the EXADEP test or for those planning to require EXADEP as a requisite of admission, additional information and convenient forms can be found on the EXADEP Web site (www.ets.org/exadep).


6

DESCRIPTION OF THE TEST

The purpose of the EXADEP test is to measure an examinee’s quantitative, analytical reasoning and verbal abilities in Spanish and in English as a foreign language in order to contribute to the prediction of the examinee’s performance in graduate or professional schools and for fellowship selection.

The test comprises five sections. The first four are in Spanish and the fifth is in English. All questions in the test are multiple-choice questions.

Section 1: Verbal Aptitude (90 minutes) Part A: Antonyms Analogies Sentence Completion Part B: Reading Comprehension

Section 2: Mathematics (40 minutes) Arithmetic Algebra Geometry

Section 3: Analytical Reasoning (40 minutes)

Section 4: Written Expression (30 minutes) Part A: Language Usage Part B: Sentence Correction

Section 5: English (45 minutes) Part A: Sentence Completion Antonyms Part B: Reading Comprehension

The total testing time is four hours and five minutes. There is a ten-minute break between Sections 2 and 3.

• The Verbal section measures the ability to analyze and evaluate written material and synthesize information obtained from it, to analyze relationships among component parts of sentences, and to recognize relationships between words and concepts. In each test edition, the passages are a balance among three different subject-matter areas: humanities, social sciences, and natural sciences.

• The Mathematics section measures basic mathematical skills and understanding of elementary mathematical concepts, as well as the ability to reason quantitatively and to solve problems in a quantitative setting. There is a balance of questions requiring arithmetic, algebra, geometry, and data analysis.

• The Analytical Reasoning section measures the ability to think analytically. It tests the ability to draw inferences and think deductively based on a partially defined scenario and a set of conditions that must hold in any fully developed version of the scenario. It does not test knowledge of any particular subject nor does it require training in formal logic.

• The Written Expression section measures the ability to recognize the use of language essential to a finished piece of writing that would be considered acceptable by most educated readers and writers of Spanish.

• The English section measures certain elements in the spectrum of abilities required to reason effectively in a verbal medium.


7

EXADEP Measures Specific Reasoning Skills That Are Developed Over a Long Period of Time.

EXADEP is designed to measure verbal, quantitative, and analytical reasoning abilities considered important for successful performance in graduate school. The test is designed to be independent of particular courses of study. Instead, it represents knowledge, skills, and abilities that are developed across many courses and that reflect exposure to demanding courses of study over a long period of time. Research on other ETS tests used for similar purposes has shown that short-term study will not alter scores greatly for most people.

It is important that test takers be thoroughly familiar with the test—its contents and procedures—before the actual testing day in order to avoid receiving lower scores than they might otherwise obtain.

The long-term reasoning component measured by EXADEP makes it a test of developed skills. Both words are significant here. The skills are reasoning skills of the type most important to success in graduate study. The developed nature of these skills comes from a lifetime as a reader, as one who thinks through quantitative problems, and as one who is accustomed to communicating and critiquing arguments in writing.

Unlike high school or undergraduate grade reports, EXADEP scores reflect performance on tasks that are common to ALL applicants. The complementarity between EXADEP scores and other elements of the application means that EXADEP scores should always be used in conjunction with those other elements.

Standardized test scores provide information that is highly comparable across examinees and that references a well-documented set of knowledge and skills. Such scores do not, however, provide information about other important characteristics, such as motivation and persistence. Furthermore, they do not typically provide a good indicator for understanding such complexities as changes in level of attainment over time or obstacles overcome to reach a particular level of attainment. Grades and letters of recommendation, in contrast, are less comparable across examinees because they are based on different requirements for different individuals. Grades represent different combinations of courses and are based on grading standards unique to each institution, department, and professor. Letters of recommendation may provide valuable information about an individual, but typically offer little systematic basis for comparing one applicant to another. Despite these weaknesses, grades and letters of recommendation probably provide better information than standardized tests about such traits as persistence, motivation, and ability to overcome obstacles, all of which may be of considerable importance to success in graduate study. This is why it is important to combine grades, letters of recommendation, and test scores to develop a holistic understanding of an applicant’s abilities and of achievement in relation to abilities.

The complementarity of test scores and grades is made clear by the results of validity studies conducted by the EXADEP Program in collaboration with various universities. These studies show that the combination of EXADEP scores and undergraduate GPA predicts first-year grades more effectively than any single piece of information. These various sources of information represent a system of checks and balances in decision making. Fairness is enhanced by using multiple measures because systematic over- or under-prediction will be decreased as additional measures are used. For example, students coming from schools or fields that employ strict grading practices will be less disadvantaged when tests are used along with grades. Conversely, test scores may represent peak performance capacity, while grades and letters of recommendation can offer testimony to an individual’s ability to sustain performance over longer periods of time.


8

GUIDELINES FOR THE USE OF EXADEP SCORES

Introduction

These guidelines have been adopted by the EXADEP Program to provide information about the appropriate use of the EXADEP scores for those who use the scores in graduate admissions and fellowship selection processes and for other approved purposes. They are also intended to protect students from unfair decisions that may result from inappropriate uses of scores. Adherence to the guidelines is important.

EXADEP is designed to assess academic knowledge and skills relevant to graduate study. As measures with known statistical properties and high-quality technical characteristics, the scores from these tests, when used properly, can improve graduate admissions and fellowship selection processes. Any EXADEP test, however, has two primary limitations: (1) it does not and cannot measure all the qualities that are important in predicting success in graduate study or in confirming undergraduate achievement, and (2) it is an inexact measure—that is, only score differences that exceed the standard error of measurement of a given score can serve as reliable indications of real differences in academic knowledge and developed abilities. Although limitations and cautions apply to all admissions measures, the EXADEP Program is obligated to inform users of the appropriate uses of EXADEP scores and to identify and try to rectify instances of misuse. To this end, the following policies and guidelines are available to all EXADEP test takers, institutions, and organizations that receive EXADEP scores.

Policies

In recognition of its obligation to ensure the appropriate use of EXADEP scores, the EXADEP Program has developed policies designed to make score reports available only to approved recipients, to encourage these score users to become knowledgeable about the validity of the tests, to protect the confidentiality of examinees’ scores, and to follow up on cases of possible misuse of scores. The policies are discussed below.

Score Recipients Accredited undergraduate and graduate institutions and non-degree-granting organizations that award graduate fellowships are eligible for consideration as score recipients. Institutions and organizations that do not meet either one of these requirements are, in general, not eligible to be score recipients. The EXADEP Program retains the right to make exceptions to this policy in special circumstances.

Validity The general appropriateness of using EXADEP scores for graduate admissions, fellowship selection, and other approved purposes has been established by research studies carried out by ETS. EXADEP scores may be appropriate for some other purposes, but it is important for the user to validate their use for those purposes. To assist institutions in evaluating proposed uses, these guidelines include information about appropriate and inappropriate uses.

Confidentiality EXADEP scores, whether for an individual or aggregated for an institution, are confidential and can be released only by authorization of the individual or institution or by compulsion of legal process.

Use of Scores in Aggregated Form Information based on EXADEP scores may be useful to prospective students, but use of a precise mean or median should be avoided. Graduate programs and institutions are urged to report EXADEP scores in ranges such as the highest and lowest scores of the middle 50 percent of the admitted students. Presenting information by score ranges emphasizes the diversity of individual scores for any one graduate program or


9

institution and also makes clear the overlap of scores among graduate programs and institutions. Use of EXADEP scores in aggregated form as a measure for ranking or rating graduate programs, institutions, university systems, or states is strongly discouraged except when the scores are used as one indicator among several appropriate indicators of educational quality.

Encouragement of appropriate use and investigation of inappropriate useAll users of EXADEP scores have an obligation to use the scores in accordance with published EXADEP Program policies and guidelines. Institutions have a responsibility to ensure that all individuals using EXADEP scores are aware of the EXADEP Program score-use policies and guidelines and to monitor the use of the scores, correcting instances of inappropriate use when they are identified. The EXADEP Program staff is available to assist institutions in resolving issues of inappropriate score use.

Guidelines

Use Multiple CriteriaRegardless of the decision to be made, multiple sources of information should be used to ensure fairness and balance the limitations of any single measure of knowledge, skills, or abilities. These sources may include undergraduate grade point average, letters of recommendation, personal statement, samples of academic work, and professional experience related to proposed graduate study. EXADEP scores should not be used exclusively. The use of multiple criteria is particularly important when using EXADEP scores to assess the abilities of educationally disadvantaged students, as well as those who are returning to school after an extended absence. Score users are urged to become familiar with factors affecting score interpretation for these groups, as discussed in this publication.

Accept Only Official EXADEP Score ReportsThe only official reports of EXADEP scores are those issued by ETS and sent directly to approved institutions and organizations designated by the test takers. Scores obtained from other sources should not be accepted. If there is a question about the authenticity of a score report, it should be referred to ETS. ETS will verify the accuracy of the scores and whether an official report was issued.

Conduct Validity StudiesDepartments using EXADEP scores for graduate admissions, fellowship awards, and other approved purposes are encouraged to collect validity information by conducting their own studies. The EXADEP Program staff will provide advice on the design of appropriate validation studies without charge.

Maintain Confidentiality of EXADEP ScoresAll individuals who have access to EXADEP scores should be aware of the confidential nature of the scores and agree to maintain their confidentiality. Institutional policies should be developed to ensure that confidentiality is maintained. For example, EXADEP scores should not be placed on documents sent outside the institution.

Consider Verbal Aptitude, Quantitative and Analytical Reasoning, Written Expression, and English Subscores Since the level of skills required for success in graduate school varies by field or department, the subscores provided for each section of the test should be taken into consideration.

Avoid Decisions Based on Small Score DifferencesSmall differences in EXADEP scores (as defined by the standard error of measurement) should not be used to make distinctions among examinees. Standard errors of measurement (SEMs) vary by test and are available in this publication.


10

Do Not Compare Scores from Different TestsEXADEP scores are not directly comparable with scores on other graduate or undergraduate admission tests. Percentile ranks should be compared only if they are based on the same reference population.

Recognize Limitations of Scores Earned on Tests Taken Under Special ConditionsEXADEP tests are offered with special arrangements and test materials to accommodate the needs of students with visual, physical, hearing, and learning disabilities. Depending upon the nature and extent of the disability, the scores may not accurately reflect a student’s educational achievement. For some students, the nature of their disabilities may make it advisable to waive EXADEP score requirements.

Normally Appropriate and Inappropriate Uses of EXADEP Scores

The suitability of EXADEP for a particular use should be explicitly examined before using test scores for that purpose. The following lists of appropriate and inappropriate uses of EXADEP scores are based on the policies and guidelines previously outlined. The lists are meant to be illustrative, not exhaustive, in nature. There may be other appropriate uses of EXADEP scores, particularly for non-admissions purposes, but any use other than those listed below must be discussed in advance with EXADEP Program staff to determine their appropriateness. If a use other than those appropriate uses listed below is contemplated, it will be important for the user to validate the use of scores for that purpose. The EXADEP Program staff will provide advice on the design of such validity studies without charge.

Appropriate UsesProvided all applicable guidelines are adhered to, EXADEP scores are suitable for the following uses:

• selection of applicants for admission to graduate school• selection of graduate fellowship applicants for awards• selection of graduate teaching or research assistants• guidance and counseling for graduate study

Inappropriate UsesThe EXADEP scores are not suitable for the following uses:

• Requirement of a minimum score on the EXADEP for conferral of a degree, credit-by-examination, advancement to candidacy, or any noneducational purpose

• Requirement of scores on the EXADEP for employment decisions, including hiring, salary, promotion, tenure, or retention (except for the awarding of assistantships to graduate students)

Comments, complaints, inquiries, and suggestions about the use of EXADEP scores are welcome. To contact the EXADEP Program office, see the inside front cover.


11

CONSIDERATIONS IN SCORE INTERPRETATION

EXADEP scores should be used to supplement the information provided in a person’s application, such as undergraduate record and letters of recommendation. Officials responsible for admission at each institution must determine the significance of EXADEP scores for each applicant. Particular attention must be paid to the use of EXADEP scores for individuals described below. The experience of institutions or departments should continue to be the best guide to interpretation of EXADEP scores in these instances.

Repeat Test Takers

Individuals are permitted to take EXADEP tests more than once. Score recipients are cautioned not to view an increase in scores necessarily as a reflection of academic gain, especially over a short time period. There are several ways in which graduate departments can judge multiple scores for an individual (e.g., use average of all scores, use most recent score, use highest score). Using the mean score may be the best technique because it is the most objective. Whatever approach is adopted, it should be used consistently with all applicants.

Examinees with EXADEP Scores More Than Five Years Old

The EXADEP Program established a policy of retaining and reporting EXADEP scores earned during the five-year period after the test date. Scores more than five years old are not reported. Candidates who took the test more than five years ago must take it again if they want scores to be sent to institutions.

When institutions decide to consider older scores, they should be aware that the applicant’s competence may have changed in the intervening time. The problem lies in determining how much the applicant’s competence has changed in either direction in the intervening years and how the change affects present potential for graduate work. Institutions may prefer to request that the applicant retake the test.

Examinees with Disabilities

ETS makes special arrangements for individuals who have recently documented visual, physical, hearing, or learning disabilities and are unable to take the tests under standard conditions. The tests are administered in a nonstandard manner chosen to minimize any adverse effect of the examinee’s disability on test performance and to help ensure that, insofar as possible, the final scores should closely represent the examinee’s educational achievement. Depending on the nature and extent of the disability, an examinee’s scores may not fully reflect his or her educational achievement and, because there are so few persons with disabilities taking EXADEP tests and their circumstances vary so widely, it has not been possible to provide special interpretive data for these examinees. Therefore, graduate schools should seriously consider waiving EXADEP requirements for applicants with certain disabilities.


12

POLICY AND USE OF EXADEP SCORES

Score Reporting Policies

EXADEP score reporting policies have been adopted by the EXADEP Program to encourage the appropriate use of EXADEP scores and to protect the right of individuals to control the distribution of their own score reports. Current EXADEP Program policy states that scores are reportable for five years. Score reports for the test are released approximately three to four weeks after the test date to the examinees and to accredited institutions of higher education granting the baccalaureate or higher and approved graduate fellowship-granting sponsors designated by the examinees.

Absentees are reported to institutions. Their names will appear on the institution roster with no scores. Percentile ranks shown on the tables included in this guide and on the score reports are based on the performance of the current reference group. The percentile rank for any score may vary over the years depending on the scores of the group with which the score is compared. Thus, when two or more applicants are being compared, the comparison should be made on the basis of their respective scores; if percentile ranks are considered, they should all be based on the percentile rank tables in the most recent edition of this guide.

Reporting Revised Scores

ETS routinely follows extensive review and quality control procedures to detect and avoid flawed questions and consequent errors in scoring. Nonetheless, if an error is discovered after scores have been reported, the specific circumstances are reviewed carefully, and a decision is made about how best to take corrective action that is fairest to all concerned. Revised scores are sent to the affected students, who may request that ETS send the revised scores to any graduate schools or fellowship sponsors still considering their applications.

Confidentiality and Authenticity of EXADEP Scores

EXADEP scores are confidential and must not be released by an institutional recipient without the explicit permission of the examinee. EXADEP scores are not to be included in academic transcripts. Dissemination of score records should be kept to a minimum, and all staff who have access to them should be explicitly advised of their confidential nature. To ensure the authenticity of scores, the EXADEP Program urges that institutions accept only official reports of EXADEP scores received directly from ETS.

The EXADEP Program recognizes the privacy rights of both institutions and individuals with regard to information supplied by and about them. ETS therefore safeguards from unauthorized disclosure all information stored in its data or research files. Information about an institution (identified by name) will be released only in a manner consistent with a prior EXADEP agreement or with the consent of the institution.

EXADEP Scores and Graduate Admissions

Many factors play a role in an applicant’s admissibility and expectation of success as a graduate student. EXADEP scores are only one element in this total picture and should be considered along with other data. The EXADEP Program believes that EXADEP scores should never be the sole basis for an admissions decision and that it is inadvisable to reject an applicant solely on the basis of EXADEP scores. A cutoff score should not be used without consideration of other admission factors.


13

EXADEP scores permit comparison of one applicant to a graduate school with other applicants for the same program at that institution as well as with everyone else who took the test. Subscores provide further information for consideration. These subscores, which reflect a student’s general strengths and weaknesses in the major areas on which the total score is based, aid in the interpretation of the total score. Often the subscores can suggest areas in which the student may require extra work. A low subscore, however, may be the result of lack of exposure to a particular field. As a result, subscores should always be reviewed in relation to the applicant’s undergraduate history.

For admissions decisions, the weight to be given to EXADEP scores can generally be established by relating what the tests measure to the orientation, curriculum, and aims of the department. Specifically, the content validity of the tests for a graduate department should be determined by reviewing the test carefully and then making subjective decisions as to the weight, if any, the scores on EXADEP should receive in relation to other admission factors.

Validity Studies

One way to determine the weight to give to test scores is to conduct validity studies. Validity is an ongoing process of assembling knowledge supporting interpretations that are made using test scores. A primary way of determining the validity of a test is to examine the correlation between test scores (and perhaps other predictors, such as undergraduate grade point average) and one or more criteria of success in graduate study. It should be noted, however, that where there are small numbers of students, major problems can occur in attempting to carry out adequate validity studies.

In addition to correlation studies, institutions might consider other approaches. One would be to prepare a table of EXADEP scores for those students who do poorly and/or drop out of graduate school. Another approach would be to independently assess the general skills or abilities needed for success in the particular graduate school and compare them to those assessed by the EXADEP.

The EXADEP Program strongly recommends that institutions using EXADEP scores conduct such validity studies, and it will assist institutions that wish to do so. Institutions interested in such assistance should contact the ETS office in Puerto Rico.


14

SCORE INTERPRETATION AND STATISTICAL INFORMATION

Verbal, Mathematics and Analytical Reasoning, Written Expression, and English Sections of the Test

The range of EXADEP scores for each section is from 20 to 80, in one-point increments. If no answers are given for a section, a score of 20 is reported for that section. The total scale score ranges from 200 to 800, and it is the sum of the results of the different sections multiplied by their individual weights. The weight of each section is proportional to the number of questions in the section. The distribution of weights is given in the following table:

Section

Section Score

Weight

1 Verbal Aptitude A 3.52 Mathematics and 3 Analytical Reasoning B 2.5

4 Written Expression C 2.05 English D 2.0TOTAL 3.5A + 2.5B + 2.0C + 2.0D

Note: One score is reported for sections 2 and 3 combined.

There may be candidates who do not answer all questions in a given section of the test. Because the number of answers is incorporated into the calculation of the scores, it is important that test takers answer every question.

The following table indicates the mean, standard deviation, SEMs, and reliability coefficient based on the results of all the candidates who took the test between February 2008 and December 2010.

EXADEP SUMMARY TABLE

(Based on the scale scores of 24,969 candidates who took the test between February 2008 and December 2010)

Section MeanStandard Deviation SEM

ReliabilityCoefficient

1 Verbal Aptitude 45.1 11 4 0.868

2 Mathematics and49.5 10 4 0.832

3 Analytical Reasoning

4 Writing Expression 49.2 12 5 0.798

5 English 47.5 12 5 0.836

TOTAL 475.7 91 22 0.938

SEM = Standard Error of Measurement


15

The EXADEP Summary Table (on page 14) provides data on the standard error of measurement and reliability.

Standard Error of Measurement

As with all educational measurements, the scores obtained by an individual could easily vary from one administration to another, although there may have not been any changes in the candidate’s true skills. The individual score is considered an estimate of the knowledge or skills of the person in the area examined. If a person could take different editions of the test without changes in his or her knowledge and skills, the average of all the scores obtained would be a precise measurement of the knowledge and the skills of the person in the area examined. This hypothetical average is known as the “true score”. The difference between the “true score” and the score obtained at a test administration is called “error of measurement.” The term tries to describe the imprecise measurement of a unique score obtained by the candidate as the measurement of knowledge or skills of the person. It does not mean that an error was committed when developing or evaluating the test.

It is statistically possible to estimate the average of the error of measurement for a large group of candidates. The statistic outcome is expressed in a score that is known as “standard error of measurement” (SEM). If the “true scores” for a group of candidates are distributed according to a normal distribution, the probability is that nearly 68 percent of the candidates will obtain scores within a standard error of measurement above or below their true scores.

The standard error of measurement for the total score of the EXADEP, expressed in scale scores, is 22. Therefore, for the 68 percent of the EXADEP candidates, it can be assumed that their true scores would fall within a range of 22 points above and 22 points below the obtained scores.

The importance of the standard error of measurement is in understanding that, when comparing the performance in the examination of two candidates, the variations in the scores perhaps do not represent a real difference as far as the skills of the respective candidates. The value of the standard error of measurement must be taken into consideration when interpreting the individual results.

Reliability coefficient

The reliability of a test indicates the degree to which individual candidates scores would change the relative position of the scores if the examination had been administered under conditions to a certain extent different (for example, if the candidates were tested by means of different test edition). The reliability is represented by means of a statistical coefficient that is affected by measurement errors. Generally, while smaller the measurement errors in a test, the greater the reliability. The reliability coefficient ranges from 0 to 1. The reliability index of 1 indicates a perfectly reliable test (that is, that there are no measurement errors in the test). The 0 indicates that the test obtains completely inconsistent results.


16

STATISTICAL TABLES

To aid in the interpretation of scale scores, the EXADEP Program describes scores in terms of their standing in appropriate reference groups. Tables 1–6 provide percentile ranks (i.e., the percentage of candidates in a group who obtained scores lower than each possible scale score) for the test. All candidates who took the test between February 2008 and December 2010 were divided into five reference groups according to the graduate program to which they requested the scores to be sent. Tables 1–5 show percentile ranks according to those five groups. Table 6 shows percentile ranks for all candidates. The tables also indicate the means and standard deviations for each group.

• Table 1: Business• Table 2: Science• Table 3: Education • Table 4: Law• Table 5: All Other Programs*• Table 6: All Candidates

How to Compare the Scores

MeanA way to interpret the scores is to compare the results of a candidate with the mean of scores obtained by all candidates that took the test. It can be done for an EXADEP candidate by comparing the results of the candidate with the mean of results in the Table of Summary of the EXADEP that appears on page 14. However, it is even better to compare the scores of the candidate with the reference group in the tables on pages 17–22.

Percentile RanksAnother method to interpreting the EXADEP scores is to compare the score of one candidate with the score of another candidate, as it indicates in percentile rank tables, Table 1 – Table 6.

The candidate’s score can be compared in relation to other candidates by means of the use of the tables. For example, in Table 1, if the scaled score of a candidate is 56 in Section 1 (Verbal Aptitude), by using the Table, then 86 percent of the other candidates obtained lower scores. Also, if a candidate obtained a scaled score of 60 in Sections 2 and 3 (Mathematical and Analytical Reasoning), according to the table 83 percent of the candidates of the group obtained a lower score. If a candidate obtained a scaled score of 68 in Section 4 (Written Expression), then according to the table, 97 percent of the candidates obtained a lower score. If a candidate obtained a scaled score of 61 on Section 5 (English), then the table shows that between 86 and 89 percent of the candidates obtained a lower score.

Also note that not all the total scores appear in the tables, but as multiples of 10. For example, in Table 1, if the total scaled score of a candidate is 563, then (according to the table, 560 is the closest number) we can assume that 84 percent of the candidates obtained a lower score.

*The fifth reference group includes all graduate programs that, from a statistical point of view, did not have enough candidates.


17

Table 1

PERCENTILE RANKS FOR BUSINESS PROGRAM CANDIDATES

Percentage of candidates who obtained scale scores lower than each possible scale score(Based on the results of 24,969 candidates who took the test between February 2008 and December 2010)

SECT

ION

SC

ALE

SCOR

E

VERB

AL A

PTIT

UDE

MAT

HEM

ATIC

S/

ANAL

YTIC

AL

REAS

ONIN

G

WRI

TING

EX

PRES

SION

ENGL

ISH

TOTA

L

SCAL

E SC

ORE

PERC

ENTI

LE R

ANK

OF

THE

TOT

AL

SCAL

E SC

ORE

80 99 99 99 80078 99 99 99 98 78076 99 99 99 98 760 9974 99 98 99 97 740 9972 99 97 99 97 720 9970 99 95 98 95 700 9968 99 94 97 94 680 9866 98 92 95 93 660 9764 96 89 92 91 640 9662 95 86 88 89 620 9460 92 83 84 86 600 9158 89 78 79 83 580 8856 86 74 73 80 560 8454 82 68 66 76 540 7952 77 62 61 73 520 7350 71 54 54 68 500 6548 66 47 49 62 480 5746 57 40 42 57 460 4944 51 32 36 48 440 4142 45 23 30 40 420 3240 38 17 26 31 400 2438 31 12 21 23 380 1736 25 7 17 14 360 1134 20 4 14 8 340 732 15 2 11 5 320 430 12 1 8 2 300 228 8 1 6 1 280 126 6 1 4 1 260 124 5 1 3 1 240 122 3 2 1 22020 1 1 1 200

n 4,316 4,316 4,316 4,316 n 4,316

Mean 43.7 50 47.9 46.8 Mean 467.6

Standard Deviation 11 10 12 11 Standard

Deviation 91


18

Table 2

PERCENTILE RANKS FOR SCIENCE PROGRAM CANDIDATES


SECT

ION

SC

ALE

SCOR

E

VERB

AL A

PTIT

UDE

MAT

HEM

ATIC

S/

ANAL

YTIC

AL

REAS

ONIN

G

WRI

TING

EX

PRES

SION

ENGL

ISH

TOTA

L

SCAL

E SC

ORE

PERC

ENTI

LE R

ANK

OF

THE

TOT

AL

SCAL

E SC

ORE

80 99 99 80078 99 99 99 99 78076 99 99 99 98 760 9974 99 99 99 97 740 9972 99 98 99 96 720 9970 99 98 98 96 700 9968 99 97 97 95 680 9966 97 95 95 93 660 9864 96 93 91 91 640 9762 93 90 87 90 620 9560 90 87 82 88 600 9258 87 83 76 85 580 8856 83 78 71 82 560 8454 79 72 65 78 540 7952 74 67 58 74 520 7250 68 60 52 69 500 6548 62 52 47 63 480 5646 54 44 41 57 460 4844 47 36 35 48 440 3942 41 27 29 41 420 3140 35 21 24 32 400 2438 28 14 19 24 380 1736 22 8 15 16 360 1234 18 4 12 9 340 732 14 2 9 5 320 430 11 1 7 2 300 228 7 1 5 1 280 126 5 1 4 1 260 124 3 1 2 1 240 122 2 1 2 1 22020 1 1 1 1 200

n 5,981 5,981 5,981 5,981 n 5,981

Mean 44.7 48.5 48.5 46.4 Mean 468.2


Deviation 90


19

Table 3

PERCENTILE RANKS FOR EDUCATION PROGRAM CANDIDATES

Percentage of candidates who obtained scale scores lower than each possible scale score (Based on the results of 24,969 candidates who took the test between February 2008 and December 2010)

SECT

ION

SCAL

E SC

ORE

VERB

AL A

PTIT

UDE

MAT

HEM

ATIC

S/

ANAL

YTIC

AL

REAS

ONIN

G

WRI

TING

EX

PRES

SION

ENGL

ISH

TOTA

L SC

ALE

SCOR

E

PERC

ENTI

LE R

ANK

OF T

HE T

OTAL

SC

ALE

SCOR

E

80 99 99 80078 99 99 78076 99 99 99 99 76074 99 99 99 98 740 9972 99 99 99 98 720 9970 99 98 99 97 700 9968 99 98 97 97 680 9966 98 97 95 96 660 9964 97 96 93 95 640 9862 95 95 90 94 620 9760 93 93 87 92 600 9558 90 91 82 90 580 9356 87 88 78 88 560 9054 84 84 72 85 540 8752 79 80 66 83 520 8250 74 75 61 79 500 7648 69 68 54 74 480 6946 63 60 47 69 460 6144 57 50 41 61 440 5342 50 39 35 53 420 4340 44 31 29 44 400 3438 37 22 24 34 380 2536 31 13 19 24 360 1834 25 7 15 15 340 1132 20 3 12 9 320 630 15 2 9 4 300 328 11 1 7 2 280 126 8 1 5 1 260 124 5 1 3 1 240 122 3 2 1 22020 1 1 1 200

n 3,482 3,482 3,482 3,482 n 3,482

Mean 42.3 45.2 46.5 43.4 Mean 441.6


Deviation 86


20

Table 4

PERCENTILE RANKS FOR LAW PROGRAM CANDIDATES


SECT

ION

SCAL

E SC

ORE

VERB

AL A

PTIT

UDE

MAT

HEM

ATIC

S/

ANAL

YTIC

AL

REAS

ONIN

G

WRI

TING

EX

PRES

SION

ENGL

ISH

TOTA

L SC

ALE

SCOR

E

PERC

ENTI

LE R

ANK

OF T

HE T

OTAL

SC

ALE

SCOR

E

80 99 99 99 800 9978 99 99 99 96 780 9976 99 98 99 95 760 9974 99 97 99 94 740 9972 99 96 99 92 720 9970 99 94 97 91 700 9968 98 92 95 88 680 9766 97 88 92 86 660 9664 95 85 88 83 640 9362 92 80 82 81 620 9060 88 75 76 77 600 8658 83 69 68 73 580 8056 78 63 61 69 560 7454 72 56 53 64 540 6552 65 49 45 59 520 5650 56 41 38 53 500 4648 48 33 32 46 480 3646 40 26 26 40 460 2844 33 19 21 32 440 2042 26 13 17 26 420 1440 20 9 13 18 400 938 15 6 10 13 380 636 11 3 8 8 360 434 8 1 6 4 340 232 6 1 4 2 320 130 4 1 3 1 300 128 3 1 3 1 280 126 2 1 2 1 260 124 1 1 1 1 24022 1 1 1 22020 1 1 1 200

n 7,844 7,844 7,844 7,844 n 7,844

Mean 48.2 53.2 52.2 51.2 Mean 509


Deviation 84


21

Table 5

PERCENTILE RANKS FOR ALL OTHER PROGRAMS


SECT

ION

SCAL

E SC

ORE

VERB

AL A

PTIT

UDE

MAT

HEM

ATIC

S/

ANAL

YTIC

AL

REAS

ONIN

G

WRI

TING

EX

PRES

SION

ENGL

ISH

TOTA

L SC

ALE

SCOR

E

PERC

ENTI

LE R

ANK

OF T

HE T

OTAL

SC

ALE

SCOR

E

80 99 99 99 80078 99 99 99 98 780 9976 99 99 99 98 760 9974 99 99 99 97 740 9972 99 99 99 96 720 9970 99 98 98 95 700 9968 98 97 96 94 680 9966 98 96 94 92 660 9864 96 95 91 91 640 9762 94 93 87 89 620 9560 91 90 83 87 600 9358 88 87 78 85 580 9056 84 84 73 82 560 8654 80 79 66 79 540 8152 75 74 60 75 520 7550 70 67 55 71 500 6948 65 60 49 64 480 6146 58 52 44 59 460 5344 52 43 38 52 440 4542 46 33 32 44 420 3740 40 26 27 36 400 2838 33 18 22 27 380 2136 26 11 18 18 360 1534 21 6 14 11 340 1032 17 3 12 6 320 630 13 1 10 3 300 328 9 1 7 1 280 126 6 1 5 1 260 124 4 1 4 1 240 122 3 1 3 1 22020 1 1 1 1 200

n 3,346 3,346 3,346 3,346 n 3,346

Mean 43.6 46.7 47.7 46 Mean 457.4


Deviation 92


22

Table 6

PERCENTILE RANKS FOR ALL CANDIDATES


SECT

ION

SCAL

E SC

ORE

VERB

AL A

PTIT

UDE

MAT

HEM

ATIC

S/

ANAL

YTIC

AL

REAS

ONIN

G

WRI

TING

EX

PRES

SION

ENGL

ISH

TOTA

L SC

ALE

SCOR

E

PERC

ENTI

LE R

ANK

OF T

HE T

OTAL

SC

ALE

SCOR

E

80 99 99 99 800 9978 99 99 99 98 780 9976 99 99 99 97 760 9974 99 98 99 96 740 9972 99 97 99 95 720 9970 99 96 98 94 700 9968 98 95 96 93 680 9866 97 93 94 91 660 9764 96 90 91 89 640 9662 94 87 86 87 620 9360 90 84 81 85 600 9058 87 79 75 81 580 8756 83 75 70 78 560 8254 78 69 62 74 540 7652 73 63 56 70 520 6950 66 56 50 66 500 6148 60 49 44 59 480 5346 52 41 38 54 460 4444 45 33 32 45 440 3642 39 24 27 38 420 2840 33 19 22 30 400 2138 26 13 17 22 380 1536 21 7 14 15 360 1034 17 4 11 9 340 632 13 2 9 5 320 430 10 1 7 2 300 228 7 1 5 1 280 126 5 1 3 1 260 124 3 1 2 1 240 122 2 1 2 1 22020 1 1 1 1 200

n 24,969 24,969 24,969 24,969 n 24,969

Mean 45.1 49.5 49.2 47.5 Mean 475.7


Deviation 91


23

STEPS THAT ETS TAKES TO ENSURE FAIRNESS

ETS has designed several procedures intended to build fairness into its tests: involving external faculty members in the design and oversight of the tests, the fairness review process, and the differential item functioning (DIF) analysis. The purpose of involving faculty members in the design and oversight of the tests is to make sure that the perspectives of a diverse group of people are considered in planning and ongoing operational activities. The purpose of the fairness review process is to ensure that tests reflect the multicultural nature of society and to screen out any material that might be offensive or less accessible to certain groups of test takers, such as those based on age, disability, ethnic group, race, or gender. The purpose of the DIF analysis is to identify any test questions on which members of a particular group of test takers perform differently than would be expected on the basis of their overall ability in the areas covered by the test.

Involving External Faculty Members in the Development of the Test

The EXADEP Program involves undergraduate and graduate faculty members in the design and oversight of the test. The EXADEP Advisory Committee is made up of men and women from different academic disciplines and different Spanish-speaking countries representing a variety of institutions. Members are drawn from different ethnic groups. Drawing on a diverse group of educators who are not ETS employees is one way ETS seeks to ensure the fairness of the test. Every question in an ETS test (and all materials published by ETS) must pass a fairness review. This review is based on a set of written guidelines; each review is conducted by an ETS staff member specifically trained in the application of these guidelines. Any test question that does not pass the fairness review must be revised to comply with the guidelines or be removed from the test. The fairness review does not guarantee that women, minority group members, or individuals with disabilities will perform well on the test, but it does guard against the possibility of distraction caused by language or content that might be found offensive or inaccessible.

Differential Item Functioning Analysis

Differential item functioning occurs when people from different groups and of approximately equal knowledge and skill perform in substantially different ways on a particular test question. Differential item functioning analysis is a statistical technique used as part of the testing process that is designed to identify test questions that are more difficult for members of one group than for members of some other group, controlling for overall ability. It is important to realize that DIF is not synonymous with bias. DIF may occur if a perfectly fair question happens to be measuring a skill that is not well represented in the test as a whole. Each DIF analysis involves a set of comparisons between a group of examinees that is the focus of the study (focal group) and the group with which it is compared (reference group). For example, if the focal group is women, the reference group is men. The DIF analysis is based on a comparison between groups of test takers of the same overall ability as determined by their performance on the test as a whole. A DIF statistic is computed for each test question, indicating the extent to which members of the focal group perform differently from members of the reference group who have similar ability levels. DIF analyses are run before scoring is performed. A test question that appears, on the basis of the DIF analysis, to be functioning in a substantially different way for the focal and reference groups, is reviewed and subject to removal from scoring. The EXADEP Program encourages test takers to report concerns about specific test questions directly to the test center administrator or to the EXADEP Program immediately following the test administration. Subject-matter specialists will review these questions and eliminate them from scoring if potential bias is determined. The test specialists will also respond in writing to the examinees. If a response does not resolve an examinee’s concern, the examinee may pursue the matter further with ETS.





Copyright © 2011 by Educational Testing Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS) in the United States and other countries. EXADEP, EXAMEN DE ADMISIÓN A ESTUDIOS DE POSGRADO are trademarks of ETS. 16463

guide to the use and interpretation of scores for the ... • exadep guide to the use and...

Documents