TEC 2013 Hyderabad

The role of programme evaluation in curriculum development

OR‘How are we doing?’

Richard KielyUniversity of St Mark & St John, UK

My life in programme evaluation

1. Medium of instruction evaluation (Pupil)

2. Native-speaker teacher evaluation (Teacher)

3. Evaluation for development consultancy (Consultant)

4. International CLIL project (Evaluator)

5. Teacher education programme impact study (Researcher)

Some principles

1. Evaluation cannot just test the theory

2. Evaluation has to understand learning processes as well as outcomes

3. Evaluation has to engage with all stakeholders

4. Evaluation has to facilitate action

5. Evaluation should make sense of the programme for everyone.

This presentation

• Overview of language programme evaluation and its potential in curriculum and professional development.

• Evaluation purposes, designs and methods

• Identities and roles of programme stakeholders

• Using the processes and outcomes of evaluation for improvement of programmes, and

• Management of change issues.

Overview

• Programme evaluation is ‘a set of strategies to document and understand the programme. It involves research activity (conventional studies or action research by which teachers learn about and transform aspects of their practice) and assessment data (conventional measures of outcomes). In addition to these, evaluation has to engage with the social, cultural and historical identity of the programme, as a product of the institution, as a phase in the biographies of participants, and as a context of personal investments of individual stakeholders’.

Kiely 2009: 114

Programme Evaluation

TYPE 1 TYPE 2Experimental/comparative

Case study

Probability Possibility Policy-oriented Practice-oriented

Programme theory (explicit)

Programme theory (implicit)

Product-focus Process-focusExternal Internal

Programme Evaluation

TYPE 1 TYPE 2Experimental/comparative

Case study

Probability Possibility Policy-oriented Practice-oriented

Programme theory (explicit)

Programme theory (implicit)

Product-focus Process-focusExternal Internal

Discuss: Any of these

features new to you?

Evaluation purposes, designs and methods

• Purposes: accountability, development and quality assurance• Designs: the links between data,

theory and action• Methods: ways of getting data.

Evaluation designs – Type 1Template 1 Template 2

Measurement of outcomes (language

tests; teacher performance; teacher

qualifications;

Surveys of attitudes; preferences; aspirations;

wants; needs

Evaluation designs – Type 2Template 1 Template 2 Template 3

Quality Assurance

Programme support

Teacherresearch

Evaluation designs – Type 2Template 1 Template 2 Template 3

Quality Assurance

Programme support

Teacherresearch

Buzz group discussion:What kind of activities do you think would contribute to these designs?

Evaluation designs – Type 2

Template 1 Template 2 Template 3Quality

AssuranceProgramme

supportTeacher research

End of course questionnaires

Mentoring Action research

Surveys Peer observation Reflective PracticeFocus Groups Reading circles Exploratory practice

Audits Assessment workshops

Study for research degrees

Assessment data Meetings

Evaluation designs

Template 1 Template 2 Template 3Quality Assurance

Programme support

Teacher research

Design issuesHow do we find the right amount of activity?How do we get a focus on learning?How do we add value to learning opportunities?

Identities and roles of programme stakeholders

Teachers

Identities and roles of programme stakeholders

Students

Teachers

Identities and roles of programme stakeholders

Students

Teachers

Teacher educators

Identities and roles of programme stakeholders

Students

Teachers

Teacher educators

Leaders and managers

Identities and roles of programme stakeholders

Students

Teachers

Teacher educators

Leaders and managers

Remote stakeholders

Identities and roles

Immediate stakeholders Remote stakeholders

Students Teachers ManagersCustomers Transformers Sponsors Learners Transmitters ParentsParticipants Advisors EmployersPractitioners Assessors

Student role - learning

• Using the processes and outcomes of evaluation for improvement of programmes

WORKSHOP

Student role - learning

• Using the processes and outcomes of evaluation for improvement of programmes

WORKSHOP

Buzz group discussion:

How can teachers

workshop evaluation

process with students?

Student role - learning

• Processes

• Awareness raising• Focus groups• Structured discussions• Reflections as part of assessment

Student role - learning

• Engaging learners in communicative encounters, especially if their aim is to explore emotional content and experiences, can become too bound up in itself unless this activity also reaches an evaluation stage. Trying to understand what has happened while undertaking a particular task, why it was suggested by the teacher, and contributing actively to the evaluation of learning arrangements, sequences, resources and input materials by means of reflection and meta-communicative discourse – all these are considered indispensable learner activities in ELT.

Legutke and Thomas (1991:65)

Student role - learning

• Engaging learners in communicative encounters, especially if their aim is to explore emotional content and experiences, can become too bound up in itself unless this activity also reaches an evaluation stage. Trying to understand what has happened while undertaking a particular task, why it was suggested by the teacher, and contributing actively to the evaluation of learning arrangements, sequences, resources and input materials by means of reflection and meta-communicative discourse – all these are considered indispensable learner activities in ELT.

Legutke and Thomas (1991:65)

Autonomy

Student role - learning

• Engaging learners in communicative encounters, especially if their aim is to explore emotional content and experiences, can become too bound up in itself unless this activity also reaches an evaluation stage. Trying to understand what has happened while undertaking a particular task, why it was suggested by the teacher, and contributing actively to the evaluation of learning arrangements, sequences, resources and input materials by means of reflection and meta-communicative discourse – all these are considered indispensable learner activities in ELT.

Legutke and Thomas (1991:65)

Autonomy

Agency

Student role - learning

• Engaging learners in communicative encounters, especially if their aim is to explore emotional content and experiences, can become too bound up in itself unless this activity also reaches an evaluation stage. Trying to understand what has happened while undertaking a particular task, why it was suggested by the teacher, and contributing actively to the evaluation of learning arrangements, sequences, resources and input materials by means of reflection and meta-communicative discourse – all these are considered indispensable learner activities in ELT.

Legutke and Thomas (1991:65)

Autonomy

Agency

Motivation

Using the processes and outcomes of evaluation for improvement of programmes

Teachers in a community of practice

Using the processes and outcomes of evaluation for improvement of programmes

• Collaborative development (peer observation; mentoring; coaching)

• Workshops on materials; ICT; assessment formats; test data

Using the processes and outcomes of evaluation for improvement of programmes

• Collaborative development (peer observation; mentoring; coaching)

• Workshops on materials; ICT; assessment formats; test data

Buzz group discussion:Do these happen in your context? If not,

why not?

Teacher role – professional learning

• Sponsored professionalism: based on qualification and recognition

• Independent professionalism: ‘a commitment to careful and critical examination of the assumptions and practices embedded in sponsored professionalism’

(Leung 2009:53).

Teachers and Change

• Int: So after the evaluation […..] Will you do it differently next time?

• Millie: I don’t think I’ll do it differently. […] we have developed some good strategies and some quite good materials, that will continue to develop. So it is not a change in direction, but perhaps going further in the same direction.

Kiely 1998: 194

Teachers and Change (Kiely 2001:257)

Feedback   Resistance   Reflection   InnovationCYCLE 1

Anna gets feedback from students suggesting less attention to explaining words in class

    

    

    

 

CYCLE 2    

    

    

Teachers and Change (Kiely 2001:257)

Teachers and Change (Kiely 2001:257)

Feedback   Resistance   Reflection   InnovationCYCLE 1

Anna gets feedback from students suggesting less attention to explaining words in class

    

Anna interprets this as selfish thinking on the part of students

    

    

 

CYCLE 2    

    

    

Teachers and Change (Kiely 2001:257)

Teachers and Change (Kiely 2001:257)

Feedback   Resistance   Reflection   InnovationCYCLE 1

Anna gets feedback from students suggesting less attention to explaining words in class

    

Anna interprets this as selfish thinking on the part of students

    

Anna rationalises that this might not be the best way of using classroom time

    

 

CYCLE 2    

    

    

Teachers and Change (Kiely 2001:257)

Teachers and Change (Kiely 2001:257)

Feedback   Resistance   Reflection   InnovationCYCLE 1

Anna gets feedback from students suggesting less attention to explaining words in class

    

Anna interprets this as selfish thinking on the part of students

    

Anna rationalises that this might not be the best way of using classroom time

    

Anna develops a pedagogy which focusses on comprehension of ideas rather than individual words 

CYCLE 2    

    

    

Teachers and Change (Kiely 2001:257)

Teachers and Change (Kiely 2001:257)

Feedback   Resistance   Reflection   InnovationCYCLE 1

Anna gets feedback from students suggesting less attention to explaining words in class

    

Anna interprets this as selfish thinking on the part of students

    

Anna rationalises that this might not be the best way of using classroom time

    

Anna develops a pedagogy which focusses on comprehension of ideas rather than individual words 

CYCLE 2Anna gets feedback from students suggesting more attention to explaining words in class

    

    

    

Teachers and Change (Kiely 2001:257)

Teachers and Change (Kiely 2001:257)

Feedback   Resistance   Reflection   InnovationCYCLE 1

Anna gets feedback from students suggesting less attention to explaining words in class

    

Anna interprets this as selfish thinking on the part of students

    

Anna rationalises that this might not be the best way of using classroom time

    

Anna develops a pedagogy which focusses on comprehension of ideas rather than individual words 

CYCLE 2Anna gets feedback from students suggesting more attention to explaining words in class

    

Anna resists suggestions that her focus on ‘broad swathes of meaning’ should change

    

    

Teachers and Change (Kiely 2001:257)

Teachers and Change (Kiely 2001:257)

Feedback   Resistance   Reflection   InnovationCYCLE 1

Anna gets feedback from students suggesting less attention to explaining words in class

    

Anna interprets this as selfish thinking on the part of students

    

Anna rationalises that this might not be the best way of using classroom time

    

Anna develops a pedagogy which focusses on comprehension of ideas rather than individual words 

CYCLE 2Anna gets feedback from students suggesting more attention to explaining words in class

    

Anna resists suggestions that her focus on ‘broad swathes of meaning’ should change

    

Anna:‘This group say they want it, so I try to do it for them’, and includes a short activity in Week 6

    

Teachers and Change (Kiely 2001:257)

Teachers and Change (Kiely 2001:257)

Feedback   Resistance   Reflection   InnovationCYCLE 1

Anna gets feedback from students suggesting less attention to explaining words in class

    

Anna interprets this as selfish thinking on the part of students

    

Anna rationalises that this might not be the best way of using classroom time

    

Anna develops a pedagogy which focusses on comprehension of ideas rather than individual words 

CYCLE 2Anna gets feedback from students suggesting more attention to explaining words in class

    

Anna resists suggestions that her focus on ‘broad swathes of meaning’ should change

    

Anna:‘This group say they want it, so I try to do it for them’, and includes a short activity in Week 6

    

Anna spends more time on vocabulary in Weeks 9 & 10, and provides vocabulary tasks to texts in Weeks 11 & 12

Teachers and Change (Kiely 2001:257)

Teachers and Change

• Change is gradual, perhaps not always visible to the naked eye;

• Change occurs at the teacher’s pace;

• Change is negotiated;

• Change happens.

Key guidelines from principles

1. Evaluation cannot just test the theory

2. Evaluation has to understand learning processes as well as outcomes

3. Evaluation has to engage with all stakeholders

4. Evaluation has to facilitate action

5. Evaluation should make sense of the programme for everyone.

Use evaluation to ….

1. …. explain what is going on

2. … answer the ‘How are we doing?’ question

3. … allow all voices to be heard

4. … act for improvement

5. … raise awareness and explain.

ReferencesKiely, R. (2012) Designing evaluation into change management processes. Overview chapter in Tribble, C. (Ed) Managing Change in Language Education. London: The British Council pp 75-91Kiely, R. & P. Rea-Dickins (2009) Evaluation and learning in language programmes. In Knapp, K. and B. Seidlhofer with H. Widdowson (eds) Handbooks of Applied Linguistics: Volume 6: Handbook of foreign language communication and learning. Amsterdam: Mouton de Gruyter. Pp. 663-694Kiely, R. (2011) Understanding CLIL as an innovation. Studies in Second Language Learning and Teaching. Vol 1, No 1, pp 153-71.http://ssllt.amu.edu.pl/images/stories/volume.1/SSLLT_11_153-171_Kiely.pdfKiely, R. (2009) Small answers to the big question: Learning from language programme evaluation. Language Teaching Research Vol 13, No 1: 99-116Kiely, R. (2006) Evaluation, innovation and ownership in language programs. Modern Language Journal, Vol 90, No 3 pp: 597-602Kiely, R. & P. Rea-Dickins (2005) Program Evaluation in Language Education. Basingstoke: Palgrave Macmillan (Series editors Chris Candlin and David Hall) [Second edition – 2013 – in preparation].Legutke, M. & H. Thomas (1991) Process and Experience in the Language Classroom. Harlow: Longman

Thank you

rkiely@marjon.ac.uk