Paul Nisbet, CALL ScotlandUniversity of Edinburgh

Abi James, BDA

USING TEXT-TO-SPEECH IN EXAMS - PITFALLS AND

PRACTICAL SOLUTIONS

Reading difficulties in secondary

• “20% of 11 year olds have poor reading comprehension” (data from SAT results)

• “by 14 years of age 33% of pupils have unsatisfactory reading comprehension”

Chris Singleton, Special Children 182 April/May 2008

“While four-fifths of pupils at Key Stage 2 reached national expectations over the last three years, one in five primary pupils did not achieve the expected standard in English.”

Moving English forward: action to raise standards in English, Ofsted 2012

Reading difficulties in secondary

SQA Assessment Arrangements

“Assessment arrangements allow candidates who are disabled and/or who have been identified as having additional support needs appropriate arrangements to access the assessment without compromising its integrity.”

Introduction to Assessment Arrangements for Schools and Colleges

SQA January 2010, revised June 2013

Assessment Arrangements 2013

• Requests for Assessment Arrangements made for –Scotland: 17,263 candidates

(11.3% of candidates)–Rest of UK: 145,430 candidates (6.9%)

• Requests made for 61,680 entries (exams)(8.3% of all entries in Scotland)

Assessment Arrangements 2013

Scotland Rest of UK

Requests for Access

Arrangements

17,263 candidates

(11.3%)

145,430 candidates(6.9% - many arrangements

delegated to schools)

Extra Time 76% 59%

Reader 30% 27%

Scribe 24% 15%

Use of ICT including digital

papers

19% n/a - use of word processing delegated to

schools

Computer readers

2% 0.4%

Coloured paper (excludes coloured

overlays)

3% 2%

Computer Reader (TTS) vs. Human Reader

• TTS offers an independent means to decoding text.• TTS provides greater consistency.• Human readers can be more flexible (can read

anything).• Some learners prefer to use TTS.• Studies have shown candidates are more likely to

check questions with TTS than using a human reader, resulting in higher scores (Dolan et al, 2005)

• Computer Readers are allowed in ALL exams, including all GCSE, A-level & Functional Skills exams including those testing reading skills “since it allows the candidate to independently meet the requirements of the reading standards”

Does text-to-speech help reading?

A 6th year pupil at Denny High in Falkirk was assessed using the Neale Analysis, reading with and without text-to-speech. Reading herself, her

comprehension age was 6 years 9 months. With text-to-speech, it was over 13 years.

2002 need for Digital Question Papers with TTS identified

2003-04 research into specification; development of papers

2004-05 evaluation in 6 schools2005-06 pilot trials #1 - 31 students used

digital papers in 105 examinations 2006-07 pilot trials #2 - 80 candidates used

490 digital papers in 200 examinations

2014 Rest of UK – PDF papers will be provided by exam boards (not interactive); UKAAF guidance

The Scottish Experience:Research, Development and Trial

Type your answers

Click to tick

Use on-screen drawing tools

Read questions with text-to-speech

SQA Digital Question Papers

Software and AppsWindows• Acrobat Reader (free)• Text-to-speech:

– ClaroRead, Co:Writer, Ivona MiniReader , Penfriend, Read and Write Gold, Adobe Read Out Loud, etc

iPad • ClaroPDF (£0.69 + £1.49 for extra voices)• PDF Expert (£6.99)• Adobe Reader (free)

www.bdatech.org/what-technology/text-to-speech/exams/computer-readers

Digital Paper Requests 2008-13

2008 2009 2010 2011 2012 2013

Number of requests 514 1,167 2,000 2,832 3,694 4,291

Number of centres 46 73 101 149 173 188

Number of candidates 204 422 675 1,069 1,327 1,677

Mean number of requests per centre 11.17 15.99 19.80 19.01 21.35 22.82

Mean number of candidates per centre

4.43 5.78 6.68 7.17 7.67 8.92

Mean number of requests per candidate

2.52 2.77 2.96 2.65 2.78 2.56

Digital Question Papers 2008-2013

2008 2009 2010 2011 2012 20130

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

Number of requestsNumber of can-didates

Scottish assessment:Costs of readers/scribes

• 19,058 reader requests; 14,905 scribe requests

• ~ 36,000 individual exams• ~ 54,000 hours• Say average £20/hour for

reader/scribe?= £1,080,000

• Say £10/hour for invigilator?= £540,000

• TOTAL = £1.62m in Scotland last year

1. Become confident with the technology and text-to-speech tool they will be using in the exam.

- Use PDFs of past papers to gain experience on how to navigate the papers

- Use TTS in class

2. Centre applies for digital paper1. In Scotland via the online AAR system.2. In the rest of the UK through the Modified Paper Route

3. Digital exam paper arrives.1. In Scotland on CD with papers2. In the rest of the UK exam boards differ but most allow

for download 1 hour before exam from secure website

How do schools and candidates “use” a computer reader?

Resolving issues with use of digital papers & text-to-speech

1. Voice quality

Problem Solution

“many of the students commented that the synthetic voice was of poor quality and was difficult to understand”

Nisbet, P.D., Aitken, S., Shearer, N. (2004) Trial of External Papers in Accessible PDF for Candidates with Additional Support Needs. http://www.adapteddigitalexams.org.uk/Downloads/Reports/

1.Scottish ‘Heather’ voice licenced for use by Scottish Schools (2007).

2.Scottish ‘Stuart’ voice developed and licenced for Scottish schools (2011).

2. Pronunciation

Problem Solution

“Text to speech doesn’t read every word accurately, 1928 would be read as one thousand nine hundred and twenty-eight. Words are mispronounced, in particular names and places, which can affect the smooth reading of longer passages. Words containing an apostrophe are not read correctly.”

Nisbet, P.D. (2010) SQA Digital Papers 2010 Report.

1.2011 and 2012 papers (154 papers; 2,044 pages; 235,205 words!) analysed for mispronunciations

2.308 terms identified.3.CereProc updated the

voices to fix the pronunciation.

4.Or use a TTS tool with a pronunciation editor.

3. Unreadable image textProblem SolutionSome text is an image and cannot be read by the computer:

1.SQA desktop publishers now replace image text with text boxes with selectable, readable text.

2.By 2012, analysis showed that almost all image text elements were readable.

3.2014 – UKAAF guidance on accessible PDF exam papers

Impact on practice

Reliant on Readers?

Stuck with Scribes?

or

Independent with ICT?