Using Quality Management (QM) to improve language testing

Nick Saville and Michael MilanovicLTRC 2011

Outline – see handout

• Introduction - QMS

• Historical perspectives and definition of QM

• Key points in a QMS approach:

• People

• Processes

• Quality Control (QC) and Quality Assurance (QA) in

practice

• Illustrations of QC and QA in operational systems

• Conclusion - linking quality and validation

Quality management system

“Getting it right every time”

– Test developers need to adopt managerial practices which

enable their organisation to implement error-free processes

QMS provides a basis for this

– processes continually improved and standards raised

– in keeping with validation as an ongoing activity

Key point - take home message• convergence between the twin concepts of quality and validity

Definition of QMS

Distinction is made between

a) quality management (QM) - overarching concept

b) quality control and quality assurance (QC and QA) -

procedures for checking and assuring quality

QM concerned with the management of

processes which lead to improvements being

implemented

Definition of QMS – meeting needsIn QM, meeting client needs is an important consideration

In language testing, clients are the test users:

– Primary users: test takers

– Secondary users: sponsors of test takers or users of

results to make decisions

See the Code of Fair Testing Practices (1988)

Testing organisations need to guarantee "fitness for purpose”:

to meet client needs for assessment in specific contexts of use, such as the

workplace

Historical perspective - origins

QM - origins in the manufacturing sector

• Now applied across many types of organisation

• Only recently within educational systems

– E.g. Wild and Ramaswamy (eds), 2008

Improving Testing: Applying Process Tools and Techniques to

Assure Quality

Historical perspective - founders

Founders - Taylor (1911), Shewhart (1931, 1939)

Shift from maintaining standards to improving standards

• Continual improvement axiomatic within all QM systems

– processes defined in terms of inputs and outputs

– links managed effectively between different parts of an

organisation

– reduce defective items before the final stage of a process

– cut out high cost of post hoc inspections to check on quality

Historical perspective - TQM

W. Edwards Deming (1900-1993)

• leading figure in the movement

• first coherent "quality management systems“

(QMS)

• Total Quality Management - TQM

• Deming, 1986: Out of the Crisis

Historical perspective - Quality standards

A mechanism of accountability

• dates back over 100 years

• British Standards Institution (BSI), London, 1901

• original quality mark 1903

BSI “kite mark”

Historical perspective – ISO standards

International Organization for Standardization (ISO)

Quality Standards - ISO 9001: 2008 accreditation

Based on auditing

Certifies that processes are being applied consistently and effectively to meet stated objectives of an organisation

Two key points in a QM approach

People • the importance of leadership and staff development

In our field this includes specific training in assessment literacy

Processes

• the importance of defining processes and managing

them effectively

Key points - People

A key asset in achieving error-free processes:

• Managers ensure that staff are empowered to carry out

their responsibilities

• QC & QA procedures ensure all involved contribute to

quality

• Importance of:

– Expertise

– Training

– Ongoing professional development

Key points - Processes

Core processes - the assessment cycle

– Planning and design

– Development - e.g. of systems for test assembly

– Delivery - routine test assembly and administration

– Processing – marking, grading, issue of results

– Review and evaluation

Key points - process interactions map

Figure 1 inputs and outputs

Key points – Plan, Do, Check, Act

• Mistakes, inaccuracies, human error, malpractice, cheating, etc. – all threaten validity

• Iterative processes provide feedback

–to evaluate effectiveness

–to correct errors

–to make improvements in subsequent iterations

Key points - Figure 2

time

Operational System

Development

developing the assessment

decision to provide an assessment

reporting results

assembling tests

administering tests

marking tests

grading tests

eevviiddeennccee

ee vv ii dd ee nn cc ee

aim: produce test specifications

aim: produce test materials

aim: provide test takers and other stakeholders with the results and any other information needed for an appropriate interpretation

aim: put the performance of each test taker into a meaningful category so the test result can be understood

aim: provide an accurate and reliable score for each test taker

aim: collect information about each test taker’s ability

Aims,

Inputs,

Outputs

QC and QA in practice – check, act

• QC synonymous with checking• systematic checks ensure that all aspects of a “work flow” meet the

standards as defined and described in documentation for each stage of the overall process

• checks are carried out by the people who have the responsibility for doing the tasks (not by a separate "quality control department")

• QA carried out to monitor, evaluate and improve• to ensure that all processes are maintained to the required standard

• to monitor and adjust processes within the system

• to evaluate whether core processes are working adequately

QC in practice (1)Illustration 1 - Assembling test materials

• production of test specifications at the development stage and instructions for item writers for assembling tests - essential prerequisite

• checking against specifications - at various stages of the

operational cycle before the assessment is finalised and ready to be administered

• system to collect, store and process test materials • particularly important if a large number of items and tasks are involved

e.g. may require an item banking system

QC in practice (1)Illustration 1 - Assembling test materials

A QC system for managing test materials might include:

• an ID and description of each task or set of materials

• a checklist to track progress and to provide a complete record of the stages completed, changes made and information about decisions taken

• test materials which are stored and are readily accessible with relevant documentation and management information accrued during the editing process

• rules for confidentiality and security in handling materials

QC in practice (2)

• Illustration 2 - Administering tests

• in large-scale operations the administration of the assessment will be delegated to personnel at the testing venue

• the test developer needs to be confident that assessments are administered in a standardised way

Consistency is important

Uncontrolled variation undermines validity

• administrative procedures need to be clearly and comprehensively described and produced in a format which can be used under operational conditions:

QC in practice (2)QC should include:• physical setting

• storage and handling of secure and confidential materials/information

• type and number of personnel needed to oversee the administration• to ensure safety and guarantee test security

• procedures for recruitment, training and management of invigilators /proctors and examiners

• management of the interaction between test takers and administrators• the checking of their identities and eligibility, seating arrangements, provision of

accommodations to meet special needs or requirements

• management of the assessment procedures themselves • providing instructions, handing out and collecting test papers and answer sheets,

monitoring for malpractice (including all kinds of cheating), ensuring that timings are respected

• handling of unforeseen eventualities specific circumstances • illness, fire alarms, power cuts, minor accidents, disruption from outside the venue

• Failure to maintain quality of any aspect of the system threatens validity

• E.g. Undetected problems may lead to construct-irrelevant variance

• This may affect all test takers, groups of test takers, or individuals and lead to unfair outcomes

see Kunnan on Fairness, (2000; 2004)

Linking quality and validation

Linking quality and validation

• Validation is the process of accumulating evidence to

support inferences made using test results

• Synergy between test validation and QM concept of

continual improvement

• Accrual of evidence at all stages in the cycle

Figure 3

Linking quality and validation

Operational system

Development

developing the assessment

decision to provide an assessment

reporting results

producing test materials

administering tests

marking tests

grading tests

periodic review

ee vv ii dd ee nn cc ee

time

eevviiddeennccee

Figure 3 Assessment cycle showing periodic review

Follow the dotted red line _ _ _

Conclusion - putting principles into practice

QMS can help

to bring about change and improvement in well-managed and principled ways

to connect good practice in language assessment with operational procedures which are both transparent and accountable

ISO 9001 in conjunction with an international Code of Practice provides the basis for error-free processes and achievement of auditable professional standards

References – on handoutCouncil of Europe/ALTE (2011) Manual for language test development and examining. For use with the CEFR, Strasbourg: Language

Policy Division, available online www.coe.int/t/dg4/linguistic/ManualtLangageTest-Alte2011_EN.pdf

Deming, W. E. (1986). Out of the Crisis. Cambridge: Cambridge University Press.

Downing, S.M. (2006).Twelve Steps for Effective Test Development. In Downing, S.M.; Haladyna, T.M. 2006.

Fulcher, G. and Davidson, F. (2007). Language Testing and Assessment – an advanced resource book, Abingdon: Routledge.

Hatch, M. J. with Cunliffe, A. L. (2006). Organization Theory: modern, symbolic and post-modern perspectives. (2nd edition). Oxford: Oxford University Press.

Kane, M (2006). Validation. In R. L. Brennan (Ed). Educational measurement (4th edition). Washington, DC: American Council on Education/Praeger.

Kemp, S. (2006). Quality Management Demystified. New York, McGraw Hill.

Kuijper, H. (2003). QMS as a Continuous Process Of Self-Evaluation and Quality Improvement for Testing Bodies. www.alte.org/qa/index.php

Kunnan, A. J. (2004). Test Fairness, in M. Milanovic and C. Weir (Eds). Studies in Language Testing 18:

Rose (2010) Research Notes, 39 (2-7). Downloadable from: www.cambridgeesol.org/rs_notes/offprints

Saville, N. (2005).Setting and monitoring professional standards: A QMS approach. Research Notes 22, 2-5. Cambridge: Cambridge ESOL.

Saville, N. (2010). Auditing the quality profile: From code of practice to standards. Research Notes, 39, 24-28. Cambridge: Cambridge ESOL.

Shewhart, W.A. (1931). Economic control of quality of manufactured product. New York: D. Van Nostrand Company.

Shewhart, W.A. (1939). Statistical method from the viewpoint of quality control. Washington, The Graduate School, the Department of Agriculture.

Taylor F.W. (1911). The Principles of Scientific Management. New York: Harper and Brothers Publishers,.

van Avermaet, P. (2003). QMS and The Setting of Minimum Standards: Issues of Contextualisation Variation between The Testing Bodies. Retrieved from www.alte.org

van Avermaet, P., Kuijper, H; and Saville, N. (2004). A Code of Practice and Quality Management System for International Language Examinations, Language Assessment Quarterly 1(2&3),137-150.

Wild, C.L. and Ramaswamy, R. (2008). Improving testing. Applying Process Tools and Techniques to Assure Quality. London: Routledge.

Thank You!

Saville.n@CambridgeESOL.org