Measurement and Evaluation in Education programmes

What we think we know

Measuring CSI in South African Schools

You must measure what matters

What you measure will matter

You cant measure what matters most

What is Quality?

Quality is not something that simply happens because people are well intentioned and work hard. Quality should be expressed in school aims and policies to promote pupils’ learning. Thus it has to be set as a goal and managed. (Potterton and Northmore 2004) Judith Chapman and David Aspin (1994) have grappled with some of the thorny issues in the quality debate: they have adopted a pragmatic approach, useful for our purposes here. Chapman and Aspin have not limited "quality" to any one specific meaning, despite extensive research, but they have identified a set of "core values" which may be said to be typical of quality schools. Listed below are some of these core values, which we would like to promote in our schools:

Core Values (from Chapman and Aspen 1994)

schools should be capable, democratic and just; learners should have the opportunity to acquire, apply and practise the different kinds of knowledge, skills and attitudes that will prepare them for life; schools should actively demonstrate concern for, and promote high standards of excellence in all aspects of school life, both at an individual and an institutional level; schools should expose learners to a humane outlook on life and instil crucial values as an integral part of each individual's personal and social development; schools should develop in learners a sense of independence and self-worth as human beings, giving them confidence in their ability to contribute to society in different ways; schools should prepare learners to have a concern for the cultural as well as the economic enrichment of the community in which they will play a part.

You must measure what matters

Symptoms and Causes 1.What is the decision this measurement is supposed to support? 2.What is the definition of the thing being measured in terms of observable

consequences? 3. How, exactly, does this thing matter to the decision being asked? 4. How much do you know about it now (i.e., what is your current level of

uncertainty)? 5.What is the value of additional information?

(Hubbard 2014 p47)

What are the symptoms of a ‘good’ school?

1. Curriculum Provision and Resources 1.1. Structure of the curriculum 1.2. Courses and programmes 1.3. Support for learning 2. Student Achievement 2.1. Student performance and quality of learning 3. The Quality of Teaching and Teacher Development 3.1. Planning 3.2. Methods and techniques 3.3. Support of students

3.4. Development opportunities

4. School Systems, Structures and Safety 4.1. Policies and procedures 4.2. Infrastructure 4.3. Safety and security

5. Governance, Management and Ethos 5.1. Leadership and management 5.2. Governance and parent relations 5.3. Ethos

(Extracted from ‘Looking in a Mirror’ Potterton and Northmore 2004)

1. Keep the question precise and short. Wordy questions are more likely to confuse. 2. Avoid loaded terms. A “loaded term” is a word with a positive or negative connotation, which the survey designer may not even be aware of, that affects answers. 3. Avoid leading questions. A “leading question” is worded in such a way that it tells the respondent which particular answer is expected. Example: “Should the underpaid, overworked sanitation workers of Johannesburg get pay raises?” 4. Avoid compound questions. Example: “Do you prefer the seat, steering wheel, and controls of car A or car B?” 5. Reverse questions to avoid response set bias. A “response set bias” is the tendency of respondents to answer questions (i.e., scales) in a particular direction regardless of content.

What you measure will matter

So how brave are you willing to be? Four Useful Measurement Assumptions 1. Your problem is not as unique as you think. 2. You have more data than you think. 3. You need less data than you think. 4. An adequate amount of new data is more accessible than you think.

Hubbard 2014 (p32)

You cant measure what matters most

When education is organized to get quick returns on business investment, and to increase immediate returns by lowering that investment, it favors a teaching force that is young, flexible, temporary, inexpensive to train at the beginning, un-pensioned at the end (except by teachers’ own self-investment), and replaceable wherever possible by technology. Teachers aren’t just completely free individuals. They are creatures of circumstance, products of their working environment. Today, there’s not just one way to teach. What it feels like to be a teacher and do the job every day varies. It depends on what you want to achieve in it, whether you are capable of meeting your own and other people’s expectations, how your job is designed, what your contacts and relationships with colleagues are like, and what sorts of conditions you work in.

Fullan and Hargreaves 2012

Benjamin Zander - The art of possibility

Rule Number 6

The search for the effective school is like the hunt for the unicorn, a quest for a mythical entity. While few of those who have been involved in the research have believed there existed such a paragon, the term 'the effective school' has nonetheless often been used in the singular, and the notion of 'best' practice often seems to subsume that virtuous state. The inherent problem of the idealised school and the best practice model is expressed by Hampden-Turner and Trompenaars in this way: ‘The attempt to avoid conflict, dilemma and ambiguity by putting one's faith in the bottom line, the unicorn's pure and mythic extremity.’ Conflict, dilemma and ambiguity are, of course, at the very centre of learning, individual and organisational, and it is this constant grappling with complexity that makes schools interesting and dynamic places. Effective schools, in their myriad forms, never stay still long enough to be pinned down.

Macbeath 1999 p147

The only problem is that it does not work like that and cannot work like that because there are laws of learning, the first of which is that learners construct knowledge for themselves. The same applies to organisations. Learners and learning organisations construct learning in their own context and by dealing with the contingencies which they face in their daily work. The external reference point provided by school effectiveness research is vital but it can be made sense of only in context and only when there is enough prior learning to render it engaging and meaningful.

Macbeath 1999 p147

Some references

Aspin, D. N., Chapman, J. D. and Wilkinson, V. R. (1994) Quality Schooling: A Pragmatic Approach to Some Current Problems, Topics and Issues (School Development Series). London: Cassell Academic. Hargreaves, A. and Fullan, M. (2012) Professional Capital: Transforming Teaching in Every School. New York, NY: Teachers’ College Press Hubbard, D. W. (2014) How to Measure Anything: Finding the Value of Intangibles in Business. United States: John Wiley & Sons. MacBeath, J. E. C. and National Union of Teachers Members (1999) Schools Speak for Themselves, Again: The Case for School Self-Evaluation. 1st edn. New York: Routledge Falmer Potterton M and Northmore C. (2004) Looking in a Mirror. Johannesburg: The Catholic Institute of Education