Draft August, 2015

Guiding tool 2 - General

academic competence and

potential

1

Table of Contents

1. Introduction 2 2. Definition 4

3. Debate standardized testing 5 4. Instruments 7

5. Implementation 11

Annexes (separate documents):

A. Questions on general academic competences in application process B. Examples of Master’s programmes currently using standardized tests

C. Suggestions further reading

2

1. Introduction

This Guiding Tool describes and analyses the categorisations of general academic competence and potential most commonly used in the

admissions process of graduate students to Master’s programmes. It provides an overview of the tests used in admission processes, looks into

the competences addressed there and the ways in which these are

measured.

This guiding tool on general academic competences provides you with relevant background information on general academic competences, food-

for-thought to support you in shaping your own ideas on how relevant and, more precisely, how decisive they are for a successful completion of

your Master’s programme – and therefore to what extent they are a critical or determining factor in your programme’s application process.

This guiding tool provides an overview of available assessment instruments to gain insight into an applicant’s strengths and weaknesses

in these competences. This guiding tool does not aim to convince you that you should use a specific test. If it helps you to clarify what you are

looking for in terms of general academic competencies and how you look for it, it will already serve its purpose.

Appendix A to this guiding tool offers a set of questions for Master coordinators in order to determine whether it could be worthwhile to

include a measurement of general academic competences in the application process. Appendix B to this guiding tool offers you examples of

Master’s programmes that are currently using standardized tests in their admission process to include general academic competence in their

criteria for admission. Appendix C to this guiding tool provides you with an overview of resources should you want to dive deeper into (aspects of)

the world of general academic competences.

2. Definition

The paradigm shift described in the report Changing paradigms in admission to master’s programmes in Europe describes the need to evolve

admission processes from “recognition of diplomas” to “assessment of competencies”. This implies the necessity to measure an applicant’s

competence on a level that is comparable to what can be expected after completing a Bachelor’s level. The shift from diploma recognition towards

the assessment of competency level requires a different framework to judge an applicant’s qualification, but not (necessarily) a different

competency level. And the same goes for the necessity to measure an

3

applicant’s potential with respect to the learning outcomes of the Master’s

programme. In the terminology of the Dublin Descriptors this means

assessing the applicant’s competence at the level of completion of the first cycle:

- have demonstrated knowledge and understanding in a field of study that builds upon and their general secondary education,

and is typically at a level that, whilst supported by advanced textbooks, includes some aspects that will be informed by

knowledge of the forefront of their field of study; - can apply their knowledge and understanding in a manner

that indicates a professional approach to their work or vocation, and have competences typically demonstrated

through devising and sustaining arguments and solving problems within their field of study;

- have the ability to gather and interpret relevant data (usually within their field of study) to inform judgements that include

reflection on relevant social, scientific or ethical issues;

- can communicate information, ideas, problems and solutions to both specialist and non-specialist audiences;

- have developed those learning skills that are necessary for them to continue to undertake further study with a high

degree of autonomy.

But the tools needed should do more, they should also predict the potential

to develop competences in line with completion of the second cycle:

- have demonstrated knowledge and understanding that is

founded upon and extends and/or enhances that typically associated with Bachelor’s level, and that provides a basis or

opportunity for originality in developing and/or applying ideas,

often within a research context; - can apply their knowledge and understanding, and problem

solving abilities in new or unfamiliar environments within broader (or multidisciplinary) contexts related to their field of

study; - have the ability to integrate knowledge and handle

complexity, and formulate judgements with incomplete or limited information, but that include reflecting on social and

ethical responsibilities linked to the application of their knowledge and judgements;

- can communicate their conclusions, and the knowledge and rationale underpinning these, to specialist and non-specialist

audiences clearly and unambiguously;

Tools should relate to Dublin Descriptors:

measure competence at level of

completion first cycle;

predict potential to complete second cycle.

4

- have the learning skills to allow them to continue to study in a

manner that may be largely self-directed or autonomous.

And finally the tools needed should be able to measure general academic

competence and potential in a way that ensures the exclusion of biases that may lead to an undesired limitation of the Master programme’s

attractiveness to an international population of students.

This guiding tool focuses on general academic competence and potential as one of the essential fields to determine whether a potential Master

student should be admitted to a specific Master’s programme. Together with subject-specific knowledge and skills, and with personal

competencies and traits, general academic competence and potential is considered to be a decisive aspect of a student’s chance on successful

completion of a Master’s programme. To phrase it differently: what an applicant knows, who he/she is, and how smart he/she is.

General academic competence Obviously, years of debate can pass discussing which competences and

skills can be labelled as being general academic, and how these can best be assessed. But fortunately years of academic debate have already been

spend on this question. In preparing the feasibility study for the assessment of higher education

learning outcomes, the OECD used the following description of generic skills: “generic skills are considered the general abilities, capacities or

capabilities deployed in all kinds of cognitive activity. These outcomes are seen as broad, enduring skills that, although not domain specific, are

developed by frequent practice and exercise in specific domains. These

skills are those that would be desired of any student regardless of subject matter or discipline, and are teachable and learnable. They are

distinguished from the knowledge or skills of particular performances or domains. Such general abilities are the flexible, fluid capacities developed

by and used in learning new things.”1

When using this description as the background against which to further specify to general

academic competence and skills, the following three main categories can be used to distinguish

general academic from personal competence: analytical skills, verbal reasoning and quantitative reasoning. And obviously, these categories can

be further split into underlying skills and competences.

1 http://www.oecd.org/edu/skills-beyond-school/AHELOFSReportVolume1.pdf, p.115

General Academic Competence:

Verbal reasoning

Quantitative reasoning

Critical thinking

Ability to analyse symbolical

information

5

These three categories are the framework for the most commonly used

standardized tests of potential Master students; Graduate Records

Examinations (GRE) and Graduate Management Admission Test (GMAT). Other tests (IE GAT) also add analysing and handling symbolical

information as a competence to be distinguished.

3. Debate

The debate on both the importance of general academic competence and potential for successful participation in and (timely) graduation from a

Master’s programme, as well as the possibility to measure these, has been

long and intensive. Although the debate still goes on today, several developments took place over the years.

Nowadays, the notion of

academic competence itself is no longer disputed. The

(labelling of) exact components of what makes a

person academic competent may vary, but generally

aspects like (the ability for) verbal and quantitative

reasoning and critical thinking are considered to be crucial

elements. Taking the IE test

(see below) into consideration, we may add analysing and

handling symbolical information as a third category. With reference to the Dublin Descriptors, students who start with a Master’s programme may be

expected to:

be able to apply their knowledge and understanding in a manner that indicates a professional approach to their work or vocation, and

have competences typically demonstrated through devising and sustaining arguments and solving problems within their field of

study; have the ability to gather and interpret relevant data (usually within

their field of study) to inform judgments that include reflection on relevant social, scientific or ethical issues;

be able to communicate information, ideas, problems and solutions

to both specialist and non-specialist audiences. (part of the qualifications to signify completion of the first cycle)

6

Students who successfully complete a Master’s programme may be

expected to:

be able to apply their knowledge and understanding, and problem solving abilities in new or unfamiliar environments within broader

(or multidisciplinary) contexts related to their field of study; have the ability to integrate knowledge and handle complexity, and

formulate judgements with incomplete or limited information, but that include reflecting on social and ethical responsibilities linked to

the application of their knowledge and judgements; be able to communicate their conclusions, and the knowledge and

rationale underpinning these, to specialist and non-specialist audiences clearly and unambiguously.

(part of the qualifications that signify completion of the second cycle)

It takes common sense to understand that without the previously mentioned skills of verbal and quantitative reasoning and critical thinking

the abovementioned capabilities will be impossible to develop and to

prove. Nowadays the discussion focuses much more on influences that stimulate

or hinder the development of academic competence, both before and during a student’s enrolment at university. And especially on the ways to

measure a student’s academic competence and the predictive value of standardized tests.

4. Instruments This chapter provides an overview of available tests for admission

processes taking general academic competence into account. Graduate Record Examinations (GRE), developed and offered by Educational Testing

Service (ETS), and the Graduate Management Admission Test (GMAT), developed and offered by the Graduate Management Admission Council

(GMAC) are the two most-known standardized tests. Both originate in the United States and have – due to criticism on earlier versions of the tests –

fairly recently seen adaptations in the tests.

GMAT always was the preferred test when applying for an MBA, but

nowadays also GRE is accepted by a large number of MBA programmes although GMAT is still accepted more widely. There seems to be a

difference in both tests in how hard prospective students find them (GMAT being considered the harder), and the accents the tests put in the three

components analytical skills, verbal reasoning and quantitative reasoning, also differ.

An alternative test has been developed by IE Business School in Madrid; this claims to be more practical and designed for a more diverse audience

7

than the first two. This test is less well-known than GRE and GMAT,

possibly because the latter two are successfully marketed globally and

offer test facilities world-wide, whilst the IE test can only be taken in Madrid. But IE offers an additional perspective on testing academic

competence (in short, diagrammatical or symbolic reasoning) that may be an important added value for certain Master’s programmes.

Also available is the Miller Analogies Test (MAT), developed by Pearson to offer a somewhat cheaper and shorter alternative to GRE and GMAT. This

test traditionally focuses more on Humanities than other academic fields.

PLEASE NOTE: the information provided below in paragraph 2a-2d is based only on sources from the organisations involved in offering these

tests!

2a. Graduate Record Examinations Graduate Record Examinations (GRE) was developed over 60 years ago,

and is and offered by Educational Testing Service (ETS). In 2011 a revised

version of the test was introduced, the GRE revised General Test. ETS claims thousands of Higher Education Institutions use GRE as part of their

admission process, most of them universities in the US. Some 350 universities in Europe use GRE, sometimes for specific Master’s

programmes. The GRE focuses on verbal reasoning, quantitative reasoning, and critical

thinking and analytical writing skills. Below these are described with an eye on the changes made in the revised version.

Verbal Reasoning

The Verbal Reasoning measure focuses on a test taker’s ability to understand what they read and how they apply their reasoning skills —

with an emphasis on how they analyze and evaluate written material. Plus, antonyms and analogies are no longer part of the test. Vocabulary is

still important, but it is used in the context of a reading passage. This measure demonstrates an applicant’s ability to:

analyze and draw conclusions from discourse; reason from

incomplete data; identify author’s assumptions and/or perspective; understand multiple levels of meaning such as literal, figurative and

author’s intent select important points; distinguish major from minor or relevant

points; summarize text; understand the structure of a text understand the meanings of words, sentences and entire texts;

understand relationships among words and among concepts

Quantitative Reasoning The Quantitative Reasoning measure focuses on a test taker’s ability to

apply basic mathematical skills and concepts (arithmetic, algebra, geometry and data analysis), with an emphasis on data interpretation and

8

real-life scenarios that test takers will encounter in graduate or business

school. This measure demonstrates an applicant’s ability to:

understand quantitative information

interpret and analyze quantitative information solve problems using mathematical models

apply basic mathematical skills and elementary mathematical concepts of arithmetic, algebra, geometry, probability and statistics

Critical Thinking and Analytical Writing

The Analytical Writing measure focuses on a test taker’s critical thinking and analytical writing skills — specifically their ability to articulate complex

ideas clearly and effectively. Test takers are asked to provide focused responses to questions so they can accurately demonstrate their skill in

directly responding to the task presented. This measure demonstrates an applicant’s ability to:

articulate complex ideas clearly and effectively

examine claims and accompanying evidence support ideas with relevant reasons and examples

sustain a well-focused, coherent discussion control the elements of Standard Written English

2b.Graduate Management Admission Test

The Graduate Management Admission Test (GMAT) is also developed over 60 years ago and offered by the Graduate Management Admission Council

(GMAC). A revised version of the test was introduced in 2012. GMAT focuses on analytical writing, integrated reasoning, quantitative and

verbal skills. The analytical writing assessment focuses on the analysis of an argument. Integrated reasoning consists of questions to assess multi-

source reasoning, the interpretation of graphics, two-part analysis and table analysis. Quantitative questions focus on assessing data sufficiency

and problem solving. And verbal questions focus on assessing reading

comprehension, critical reasoning, and sentence correction.

Analytical Writing The Analytical Writing section consists of one writing task—Analysis of an

Argument. The format is one 30-minute written essay on the analysis of an argument. What it measures is the test-taker's ability to formulate an

appropriate and constructive critique of a specific conclusion based on a specific line of thinking. A specific knowledge of the essay topic is not

necessary; only the capacity to write analytically is assessed.

Integrated Reasoning The Integrated Reasoning section gives another data point to differentiate

candidates. This part consists of 12 questions covering four question types: Multi-Source Reasoning, Graphics Interpretation, Two-Part

9

Analysis, and Table Analysis, to be completed in 30 minutes. What it

measures is the test takers ability to analyze and synthesize data in

different formats from multiple sources.

Verbal The Verbal section allows candidates to showcase their verbal abilities.

The format is 41 questions comprised of reading comprehension, critical reasoning, and sentence correction to be completed in 75 minutes. It

measures the test taker's ability to read and comprehend written materials, reason and evaluate arguments, and correct written material to

confirm to standard written English.

Quantitative The Quantitative section highlights data analysis and reasoning skills. This

section consists of 37 questions on Data Sufficiency and Problem Solving, to be completed in 75 minutes. It measures the test taker's ability to

reason quantitatively, solve quantitative problems, and interpret graphic

data. Understand problems involving arithmetic, elementary algebra, and common geometry concepts. Evaluate the amount of information needed

to solve quantitative problems.

2c. IE Admission Test The IE Admission Test is developed by IE Business School Madrid as an

alternative for tests like the GRE and GMAT. This test has been developed as a means of determining a candidate’s ability to make management‐

style decisions under pressure. This exam is different to other standardized tests in a number of ways:

It is aimed at an international audience, with a diversity of educational backgrounds (thus, it does not favor any particular

educational system or area of study); It is a practical exam (as opposed to an academic exam), presenting

participants with more real life situations; No preparation is required (this exam determines participants’

ability to process information and think strategically, rather than repeat previously studied patterns or equations).

The ieGAT consists of three types of tasks: verbal, numerical and

diagrammatical. Questions are scattered randomly throughout the exam, requiring participants to switch rapidly from one line of thinking to

another.

Comprehension and reasoning This section consists of a text box which must be read and understood by

the participant. Each text box is followed by two questions regarding the text. This portion of the test determines participants’ comprehension of

written English in a professional context.

10

Comprehension, reasoning and calculation

This section consists of graphs, and data analysis. Advanced mathematics

is not required. Rather, participants need to think quickly and process large amounts of information and quick calculations.

Logical‐abstract reasoning with symbols

In this section, participants take a predetermined shape through a series of modifications to determine the final product. This section requires

strong visualization skills and the ability to maintain track of multiple

“images” at a single time.

2d. Miller Analogies Test (MAT) Miller Analogies Test is developed by Pearson, focusing on testing an

applicant’s analytical skills. The MAT is a high-level mental ability test requiring the solution of

problems stated as analogies. Performance on the MAT is designed to reflect candidates’ analytical

thinking, an ability that is critical for success in both graduate school and professional life.

Through analogies with content from various academic subjects, MAT scores help graduate schools identify candidates whose

knowledge and abilities go beyond mere memorizing and repeating information.

The analytical skills measured by the MAT are key to success in

graduate school and are highly valued by graduate programs. For more than 50 years, the MAT has proven to be a reliable and

valid test for assessing candidates for graduate school. Predictive validity studies conducted by Pearson and by independent

researchers have shown positive correlations between MAT scores and subsequent success in graduate programs.

5. Implementation Below an overview is provided of steps to be taken after the decision has

been made to implement the assessment of general academic competence and potential into the admission process of your Master’s.

Step 1: See also step 1 knowledge requirements; bring together the core

lecturers of the programme.

Step 2: Identify the academic competences that are most important for successful completion of the master programme and/or entrance into the

job market and that you wish to measure.

11

Consider using the following questions in identifying these:

a. Does your admissions process require proof of an applicant's

writing/mathematical/analytical skills? If so, what kind of proof do you require?

b. Have you formulated criteria against which to determine whether the writing/mathematical/analytical skills are sufficient for

enrolment in your Master's programme? Could you briefly describe these criteria?

Are these criteria communicated to the applicants? c. Did you ever encounter any difficulties in assessing an applicant's

writing/mathematical/analytical skills? If so, could you briefly describe these difficulties?

d. Do you experience difficulties with the writing/mathematical/ analytical skills of students currently enrolled in your Master's

programme? If so, do you offer support to these students?

Step3: Adjust the descriptions of the academic competences into

transparent requirements for applicants.

Step 4: Decide in which way you wish to measure these academic competences (already existing standardized test, newly developed test

etc)

Step 5: Decide how to measure/rate these academic competences

Step 6: Decide what the meaning of the ratings is in your admission process.

One quick and dirty way to handle “General Academic Competencies” for your master’s

programme, could be:

- Sit down with the senior academics of your programme and discuss if your

current indicators for GAC (often the GPA and the reputation of the previous

university) are adequate – also to limit the risk of false negatives.

- Determine if you want to improve admission in terms of GAC.

- If you do, go for the GRE as the most broadly used and validated test with more

than 500 000 test takers per annum.

- If you are concerned about the costs for the students, you can consider to abduct

the GRE fee from the tuition for admitted students.

There may be other and better ways, but this is a good start.