NEAS Conference 2016:

Understanding & Engaging With Computer-Based Testing Scoring Engines Louise FitzGerald

Presentation overview

1. Automated testing

2. PTE Academic - selected items

3. Scoring and scoring engines

4. Questions

Pedagogic disruption

High stakes tests – Study and immigration

Automated testing

PTE Academic

PTE Academic - Item types

Speaking and Writing

Personal introduction • Read aloud • Repeat sentence • Describe image • Re-tell lecture • Answer short question • Summarize written text • Write essay

Reading

• Multiple-choice, choose single answer • Multiple-choice, choose multiple answers • Re-order paragraphs • Reading: Fill in the blanks • Reading & writing: Fill in the blanks

Listening

• Summarize spoken text • Multiple choice, choose multiple

answers • Fill in the blanks • Highlight correct summary • Multiple choice, choose single answer • Select missing word • Highlight incorrect words • Write from dictation

PTE Academic – Scored practice tests

PTE Academic – Scored practice tests Offer

[S] Item 3: Describe image

No. of tasks: 6-7 Preparation time: 25 seconds Speaking time: 40 seconds

Scoring – Describe image

Enabling Skills and Traits

Content 5 Describes all elements of the image and their relationships, possible developments, conclusions or implications 4 Describes all the key elements of the image and their relations, referring to their implications or conclusions 3 Deals with most key elements of the image and refers to their implications or conclusions 2 Deals with only one key element in the image and refers to an implication or conclusion. Shows basic understanding of several core elements of the image 1 Describes some basic elements of the image but does not make clear their interrelations or implications 0 Mentions some disjointed elements of the presentation

Pronunciation 5 Native-like, 4 Advanced, 3 Good, 2 Intermediate, 1 Intrusive, 0 Non-English

Fluency 5 Native-like, 4 Advanced, 3 Good, 2 Intermediate, 1 Limited, 0 Disfluent

Communicative Skills: Speaking

Enabling Skills and Traits

Content 5 Describes all elements of the image and their relationships, possible developments, conclusions or implications 4 Describes all the key elements of the image and their relations, referring to their implications or conclusions 3 Deals with most key elements of the image and refers to their implications or conclusions 2 Deals with only one key element in the image and refers to an implication or conclusion. Shows basic understanding of several core elements of the image 1 Describes some basic elements of the image but does not make clear their interrelations or implications 0 Mentions some disjointed elements of the presentation

Pronunciation 5 Native-like, 4 Advanced, 3 Good, 2 Intermediate, 1 Intrusive, 0 Non-English

Fluency 5 Native-like, 4 Advanced, 3 Good, 2 Intermediate, 1 Limited, 0 Disfluent

[W] Item 6: Summarize written text

No. of tasks: 2-3 Length of text: Up to 300 words Response word limit: 5-75 words Writing time: 10 minutes

Scoring – Summarize written text

Enabling Skills and Traits

Content 2 Provides a good summary of the text. All relevant aspects mentioned 1 Provides a fair summary of the text but misses one or two aspects 0 Omits or misrepresents the main aspects of the text

Formal requirements

1 Is written in one, single, complete sentence 0 Not written in one, single, complete sentence or contains less than 5 or more than 75 words. Summary is written in capital letters

Grammar

2 Has correct grammatical structure 1 Contains grammatical errors with no hindrance to communication 0 Has defective grammatical structure which could hinder communication

Vocabulary 2 Has appropriate choice of words 1 Contains lexical errors with no hindrance to communication 0 Has defective word choice which could hinder communication

Communicative Skills: Reading and Writing

Scoring – Summarize written text

Enabling Skills and Traits

Content 2 Provides a good summary of the text. All relevant aspects mentioned 1 Provides a fair summary of the text but misses one or two aspects 0 Omits or misrepresents the main aspects of the text

Formal requirements

1 Is written in one, single, complete sentence 0 Not written in one, single, complete sentence or contains less than 5 or more than 75 words. Summary is written in capital letters.

Grammar

2 Has correct grammatical structure 1 Contains grammatical errors with no hindrance to communication 0 Has defective grammatical structure which could hinder communication

Vocabulary 2 Has appropriate choice of words 1 Contains lexical errors with no hindrance to communication 0 Has defective word choice which could hinder communication

Communicative Skills: Reading and Writing

Summarize written text

Test takers need to use just one compound or, preferably, complex sentence to summarize the main point and briefly mention the supporting detail.

Test takers need to check they have only written one sentence of between 5 and 75 words, with a capital letter at the beginning and only one full stop at the end.

1. ‘Just-in-time’ production relates goods produced to demand for goods and reduces the amount of wasted time, costs and materials. (19 words)

2. ‘Just-in-time’ manufacturing minimises production time and costs and stored stock is kept low because each stage is finished just before the next one is started. (25 words)

3. ‘Just-in-time’ is a method of manufacturing, which minimises time, costs and stock kept in factories, as each stage of production is completed before the next is started. (27 words)

[W] Item 7: Write essay

No. of tasks: 1-2 Input text : Up to 4 sentences Word limit: 200-300 words Writing time: 20 minutes

[W] Item 7: Write essay

Subskills tested

• Writing for a purpose (to learn, to inform, to persuade)

• supporting an opinion with details, examples and explanations

• organizing sentences and paragraphs in a logical way

• developing complex ideas within a complete essay

• using words and phrases appropriate to the context

• using correct grammar

• using correct spelling

• using correct mechanics

• writing under timed conditions

Scoring

Communication skill:

Writing Enabling skills:

Content

Form

Development, structure and coherence

Grammar

General linguistic range

Vocabulary range

Spelling

Write essay

Students need to analyze the essay prompt so they know how to answer the question. Note key words, limiting words and task words, e.g. discuss.

Practise producing plans for a variety of persuasive essay topics.

Make sure they’ve practiced typing within the time and word limits. In the test, they’ll lose marks if they don’t!

Make sure your students know to leave a few minutes at the end to edit their work. If they know their common errors, they can just concentrate on those. They won’t have time to much more than that.

[L] Item 19: Highlight incorrect words

No. of tasks: 2-3 Length of recording: 15-50 seconds No. of errors: up to 7

Scoring – Highlight incorrect words

Communicative Skills: Listening and Reading

Partial credit scoring * Points deducted for incorrect options chosen 1 Each correct word -1 Each incorrect word 0 Minimum score

Scoring and scoring engines

Score alignment with IELTS

IELTS PTE Academic

4.5 30

5 36

5.5 42

6 50

6.5 58

7 65

7.5 73

8 79

Score alignment with TOEFL iBT

Item Scoring

PTE Academic Score Guide

http://pearsonpte.com/test-takers/results/

Item Scoring Example: Write essay

+ Content (if 0, no further scoring)

+ Form (if 0, no further scoring) + Other traits

Enabling skills scores

+ Vocabulary Vocabulary

+ Spelling Spelling

+ Grammar Grammar

+ Development, structure and coherence Written discourse

+ General linguistic range

= Total item score

Communicative skills score: Writing

Overall score

Scoring and Results

Automated scoring: objective and reliable

Automated Scoring: Writing YouTube - Automated Scoring FAQs: how the machines are trained

Automated Scoring: Writing YouTube – Latent Semantic Analysis

Automated Scoring: Speaking YouTube - Automated Scoring FAQs: how the system is trained

Fascinating Facts: Scoring Engines No.2 Individual,

discrete features of a response can

be analysed. independently.

No.1 The system is not

‘distracted’ by language-

irrelevant factors, such as poor handwriting.

No.3 The scoring engine

for writing is trained to score any traits that humans can

reliably score.

No.4 The scoring engine has been trained for

every new item on the test by human raters before it goes live.

No.5 All standard

conventions for spelling are

accepted, but test takes must use the same convention throughout their

response.

Automated scoring

The automated system acts like a human rater when assessing test takers’ language skills, but does so with the precision, consistency, and objectivity of a machine.

Automated scoring

For more information on Automated Scoring: White paper: http://pearsonpte.com/institutions/scores/ YouTube: https://www.youtube.com/user/PearsonPTE/featured

Official Guide to PTE Academic with CD-ROM

References https://www.ets.org/Media/Tests/TOEFL/pdf/10855_ellguide.pdf

https://www.ets.org/s/toefl/pdf/toefl_ibt_insight_s1v6.pdf https://www.ets.org/Media/Tests/TOEFL/pdf/SampleQuestions.pdf https://www.ets.org/toefl/ibt/scores/get/ https://www.ets.org/Media/Tests/TOEFL/pdf/SampleQuestions.pdf https://www.ets.org/s/toefl/pdf/toefl_ibt_research_insight.pdf http://www.hagitegas.gr/2013/05/%CE%B7-%CF%80%CF%81%CE%BF%CF%83%CE%B5%CF%85%CF%87%CE%AE-%CF%84%CF%89%CE%BD-%CE%B5%CE%BE%CE%B5%CF%84%CE%AC%CF%83%CE%B5%CF%89%CE%BD/ http://www.startupsmart.com.au/advice/growth/why-disruption-is-about-experiences-not-technology/#.Vx72QHMU8v0.linkedin

35

Questions?

Learn more at future webinars and PTEA teacher training events.