Too much testing, too little learning – getting the balance right

Tim Oates

Group Director

Assessment Research & Development

Criteria relating to assessment – ‘Cambridge Approach’

Reliableconsistent measurement

Validmeasures precisely what it claims to measure

Sound construct basemeasures something consistent with curriculum aims

Consequential validitythe uses to which the assessment is put are technically and ethically sound

Beneficial impactthe full range of effects are beneficial

Utilitycost, resource  

Problems in UK assessment

Underdeveloped formative assessmentAssessment dominating curriculum thinking Relentless transformation into high stakes Creep in functionEscalation of purposes  

http://www.cambridgeassessment.org.uk/Images/109848-cambridge-approach.pdf

Or google ‘Cambridge Assessment Cambridge Approach

In assessment, the concept of ‘construct’ is vital

Multiply three digit numbers Understands and is inventive with metaphorReads a wide range of books for pleasureDiffusion across a membraneUnderstand the concept of percentage and calculate pc Use the concept of inequality to analyse social relationsUnderstands conservation of massMeasures accurately to quantify oxidationVerbal reasoningExternalising behaviourDiagnoses malignant melanomaLands with one engine on fire and undercart locked in ‘up’ position

 

Distributing responsibility for producing public goods of education

• Discipline-specific knowledge, skills and understanding in broad range of disciplines

• Orientation to learning, ‘learning to learn’

• Physical and mental well-being

• Personal and social identity

• Personal capitals (Bynner et al)

• Social capitals (Schuller et al)

• Cultural capitals

• Moral, civic and political understanding, including international awareness

• Facility in technology

Distributing production of public goods across the key elements of the curriculum

• Taught curriculum – subjects• Taught curriculum – cross curriculum elements • ‘Taught’ curriculum – extra-curriculum elements

• Extra-curriculum elements – guided (school trips etc) and unguided (student clubs etc)

• Institutional participation – student councils etc, learner voice• Support elements – IAG etc

• Ethos – values and value-driven practices• Culture – lived experience of the institution

Distributing responsibility for different facets of the curriculum

Aims Central Government

Content Government Agency (eg Ofsted)

Pedagogy Local Government/intermediate bureaucracy

Assessment Governing Bodies

Evaluation Head teachers

Middle tier management

Teachers and assistants

Other staff

Pupils

Parents

Other organisations – health organisations, police etc

Exam boards and assessment organisations

High Expectations

Low Support High Support

Low Expectations

The Zeitgeist and context in 2007

Performativity culture – increasing elaboration of accountability measures – refinement and extension

High support and high expectations

Very high level of centralised innovation and direction through non-statutory instruments

Constant underlying shift in the locus of control

A reduction of ‘curriculum thinking’ to ‘qualifications thinking’ – ‘syllabus to ‘specification’ – assessment begins to dominate

You need restriction in any system, and some have improved by carefully locating restriction in the right place (eg selection to teacher training in Finland, plus dirigiste implementation of comprehensive schooling) – did we, are we, restricting the right things?

Re-conditioning the Zeitgeist

Density of assessment versus the weight of assessment

The recommendation to remove ‘levels’

The purpose of assessment: a focus on constructs

 

Shifting the sense of who is responsible for what

The ‘powerful knowledge’ thesis

Empirical evidence on personal and social capitals

Transnational comparisons of the focii of national curricula and the implicit and explicit imperatives in education

‘Could do better’ Cambridge Dec 2010

Assessments – what do we have?

1National curriculum tests and tasks - which most pupils in State schools are expected to take

2Public examinations - which most pupils are expected to take

3Optional tests and progress tests - which schools can elect to enter pupils for

4Tests other than national curriculum tests (available from commercial companies) - which many schools choose to use

5In-course or school-based assessments - which are set throughout a pupil's provision 

Detail

KS11 maths test1 reading test1 spelling test1 writing task

KS23 maths (non-calculator; calculator; mental arithmetic); 3 English (reading, writing, spelling & handwriting); 2 science

KS31 reading & writing1 Shakespeare2 Science3 maths

GCSEAverage number taken = 9.5Mode of 2 papers per subject (some have more, others fewer)

Advanced level AS in four subjects, for many one of the three units is coursework. Mode approx 10 papers.

A2 as for AS, for three subject mode approx 8

Number of tests/ exams

KS1 4

KS2 8

KS3 7

GCSE 19

AS 10

A2 8

Total 56

2002Daily Mail 105

Trends to 2002

•Modularisation in A level

•Increasing elaboration of National Assessments (eg mental maths)

•Refinement in the form of the national tests

•Enforcement of League Table measures (performance tables and targets)

Trends from 2002

•Increase in the battery of national assessments (eg ICT KS3)

•Development of new test forms (Single Level National Tests)

•State-initiated formative assessment (Assessing Pupil Progress)

•Shift from external testing in KS1 to teacher assessment  

 

DetailEarly Years

Reported as a profile of 13 scales, using a score of 0-9, covering cognitive, social, physical and emotional development

Yr 1 (age5) Phonics screening check Statutory test, based on reading 40 words and non-words, taken in one week window in June.

KS1 Yr 2 (age7) statutory teacher assessment – levels for reading, writing, speaking and listening; overall level for mathematics; and a level for each attainment target in Science. P-scales for pupils with special educational needs.

KS2 Yr 6 (age 11) English reading test Level 3-5 (plus Eng level 6 reading test); English grammar, punctuation and spelling test Level 3-5 (plus Eng level 6 test); Mental maths test; Maths test A; Maths test B; (plus level 6 paper 1, paper 2)

KS3 Yr 7, 8, 9, Optional tests Eng, Maths; yr 9 Science; Yr 9 (Age 14) – statutory teacher assessment in core and non-core subjects (13 subjects) – scale 1-8 in 10 subjects.

Modular GCSE Average number of GCSE or equivalent taken = 10.9Mode of 2.5 papers per subject (some have more, others fewer) plus controlled assessment

Advanced level AS in four subjects, for many one of the three units is coursework. Mode approx 10 papers.

A2 as for AS, for three subject mode approx 8

Number of tests/examinations

EYFS 0

KS1 1

KS2 5 (7 for higher ability)

KS3 0

GCSE 27

AS 10

A2 8

Total 51 (56 – 2002)

2012

Trends to 2012

Rise in shift to GCSE ‘equivalent’ qualifications, attributed to schools striving to meet national targets

Crisis in the practical administration of National Tests

Cessation of expansion of National Testing – abandonment of development of KS3 ICT tests

Increase in attention to formative assessment including introduction of Assessment of Pupil Performance (oriented to determining ‘levels’ through formative assessment processes)

Exploration of innovations in marking (eg rank ordering /paired comparison for national tests in English)

Reduction in National Tests (removal of KS3 tests, reduction of KS2 tests from Sci Eng Math to Eng and Math only)

Elaboration of targets and measures including introduction of EBac (English Baccalaureate) as a school performance measure

Trends from 2012

Introduction of phonics screening test

Linear qualifications promoted and modular examinations discouraged or abandoned

Contracting model for GCSE-level qualifications (move to single board per subject)

Abandonment of APP as a national initiative

Greater HE involvement in design and operation of A Levels encouraged

 

2014 Benchmarking – a means of measuring progression (Bew recommendation)Phonics check KS2 tests

Nick Gibb 2010 – ‘..In Primary, apart from KS2 and the phonics check, I have no interest in the assessment which is done…’

Be very careful to read this in the right way – it confirms the professionalism of teachers and deliberate contraction of the role of the State

High autonomy in formative assessment

Constructs are all-important

The National Curriculum 1995

Chemical Reactions

i. that when chemical reactions take place, mass is conserved;j. that virtually all materials, including those in living systems, are

made through chemical reactions;k. to represent chemical reactions by word equations;l. that there are different types of reaction, including oxidation and

thermal decomposition;m. that useful products can be made from chemical reactions, including

the production of metals from metal oxides;n. about chemical reactions, e.g. corrosion of iron, spoiling of food, that

are generally not useful;o. that energy transfers that accompany chemical reactions, including

the burning of fuels, can be controlled and used;p. about possible effects of burning fossil fuels on the environment.

Science - key stage 3Materials and properties

National Curriculum 2007

Chemical and Material BehaviourIn their study of science, the following should be covered:

a. chemical change takes place by the rearrangement of atoms in substances;

b. there are patterns in the chemical reactions between substances;

c. new materials are made from natural resources by chemical reactions;

d. the properties of a material determine its uses.

The National Curriculum – School Curriculum distinction

The ‘powerful knowledge’ thesis

What’s wrong with a knowledge-based curriculum?

2010

Levels

3 contrasting, co-existing models

1 the score on a compensation-based test2 best fit 3 threshold

Poor construct integrityContradictions between school and State Poor communication with parents Undue pace – expectations of Ofsted Labelling – contrary to TGAT

The lessons from transnational comparisons

High attainment, high equity and high enjoyment is possible

Don’t assess everything which moves

High density formative assessment is valuable to pupil (externalisation of inner processes) teacher (information on progress, misconception and concept development) and parent (how can I best help my child)

‘Has Alex developed a sufficient understanding of conservation of mass to move to next segment of the learning progression?’ – hence year by year in Primary

Spiral curriculum – application of concepts – Singapore Different models of ability and progression – Stigler and Stevenson

The ACHIEVE experience

ECERTS-EECERTS-R

An ideal?

High intensity and density 

Formative function

High levels of apparent duplication and redundancy

Resource-intensive

Highly regulated with high accountability

Evidence-driven

Med School #2Written examinations (short answer, extended matching, multiple choice, modified essay questions)Computer-based tests including data interpretation testsShort in-course examinationsWritten accounts of problems and casesPoster, clinical and other presentationsLogbook and workbook recordingDirect observation of practice including graded performanceClinical examinations – OSCEs Formative assessments in courseStudent-selected componentsPortfolio

Med School #4Module assessment pattern Written examinations (short answer, extended matching question) Practical assessment of communication and clinical skillsCase presentationsCase reports (from family studies) Patient portfolio Research project Personal development portfolio

Med School #7On line examinations (extended matching item, extended response, short answer, multiple choice, data interpretation, drag and drop graphical questions)Written assignments (essay, case reports) Verbal reports and presentationsOn line ‘progress test’OSCEsRecord of Clinical Experience (RoCE)Portfolio Workplace-based assessment tools for specific skills and knowledgeDirect Observation of Procedural Skills (DOPS)Mini-Clinical Evaluation Exercise (Mini-CEX)Case-based discussion (CBD) Ward simulation exercise (filmed) Formative assessment

Med School #19Modular assessment pattern Written examinations and assessments of performance during rotationsOSCEsFormative assessment of clinical skillsFormative assessment of clinical knowledgeSummative written assessment of laboratory medicinePortfolio 

Clinicians, lecturers, tutors and students all emphasised that there was ‘…no real difference between learning and assessment….’

Student: ‘…it is all so intensive…you are reading all evening and night just to get through the mountain of material, and you could be asked a question on an obscure bit of it at any time and in any part of the course; an exam, a question on the ward, by another student…’

Student: ‘….you know you’re going into an OSCE and you know there’s no time to think…you prepare for it, of course, but it could call on knowledge from any part of your course, from any part of your experience…’  

Living in a levels-free world

Wroxham does it Finland, Singapore do itSoft landing as use decays Focus on deep, secure learning of key constructsImplement learning progressions (Schmidt and Prawat)

What we may have from 2014consultation 11 Oct close

Year by year statement of contentEach school publishing its school curriculum and assessment schemeLevels no longer used

Assessment modelStatutory tests at KS1 M&E – discussion of precise form Phonics screening check end of Yr1 – with cut score Statutory tests at KS2 M&E (scale score and decile reporting) KS2 reported against prior attainment measure – baseline assessment

Non-modular GCSE conditioned by accountability measures Non-modular A level VQs

Let’s not fool ourselves

We have one of the most diverse systems in the world(institutional forms, local structures, size of schools, school transfer, ideas about education….)

This was true in 2000 and remains trueMany of the structural shifts began years ago (GM, Academies)

What should we restrict and how? State role in National Curriculum, in assessment

My ideal Educationally focussed measurement

Potent and valid formative assessmentDiagnostic assessment – CAT, PIPS, VESPARCH

High density, low weightHigh autonomy in selection and use (assessment schemes)A pull down bank of items

Independent measurement for monitoring national standards

A switch to high equity and high attainment through attainment measures not progress measures

No regression to thresholds (the grade D phenomenon)

Professionalisation of assessment expertise – use of local collaborative mechanisms for development and promotion of good practice – getting the right unit of collaboration