Teaching and Learning Research Programme Annual Conference Papers

5th Annual Conference, 22-24 November 2004 Cardiff Marriott Hotel

First steps and second thoughts on method: beginning to research the learning of new teachers

Jim McNally, Nick Boreham, Peter Cope and Peter Gray University of Stirling

Ian Stronach and Brian Corbin

Manchester Metropolitan

NB: This paper was presented at an internal TLRP conference; if you wish to quote from it please contact the authors directly for permission. Contact details for each project and thematic initiative can be found on our website (www.tlrp.org).

First steps and second thoughts on method: beginning to research the learning of new teachers Paper presented to the TLRP annual conference at Cardiff, November 2004 from the Phase 3 Project on Enhanced Competence-based Learning in Early Professional Development (the ‘EPL’ Project) Ref: RES-139-25-0122 Abstract This paper is an account of our progress in addressing the challenge of developing a quasi-experimental method of assessing early professional learning (EPL), based on a model from data gathered by teachers as ethnographers in their own schools. From the early doubts and debates in internal discussions about the research design, we present in this paper the steps taken so far and our thoughts about these. Progress on the five indicators of early professional learning in the proposal are described and one of these, namely ‘jobsatisfaction for new teachers’, is considered in some detail. We also report on our emerging understanding of the role of teachers as ethnographers: their selection, insights, early findings and ethical concerns. They are submitting their first set of research interviews and observations at the time of this abstract submission, so we have little data on which to report but we shall include a summary of their emerging data at the conference. It is also our intention to discuss the extent to which these findings build on the provisional theorization in the original proposal and also begin to shape the model of EPL. Jim McNally, Nick Boreham, Peter Cope and Peter Gray (Stirling) Ian Stronach and Brian Corbin (Manchester Metropolitan) Contact: email: j.g.mcnally@stir.ac.uk tel: 01786 466265

2

First steps and second thoughts on method: beginning to research the learning of new teachers In taking our first methodological steps forward since the proposal, our thinking has predictably evolved in ways unforeseen. There have been lengthy meetings and contestations but, in forcing ourselves to face internal critique and think again, we think we have made useful progress on some fronts, even if we had to step backwards on to the ropes at times. This paper reports our position so far on EPPI-style systematic review, teachers as ethnographers and ‘jobsatisfaction’ as an indicator of new teacher performance – these in some detail – and the other four indicators more briefly. We conclude with an overall reflection on the methodology we are evolving - quantitative indicators based on qualitative substance and the whole question of naturalistic experimentation - as a meaningful base of evidence on which we can eventually warrant our claims. A summary of the project (‘regard’ version) is attached for reference at the end of the paper. 1 Beginning a systematic review Strictly adhered to, the methodology of EPPI-style systematic reviewing includes the specification of an ‘answerable’ research question plus filtering criteria for inclusion and exclusion of possible literature. The key criteria concern content relevance to the specified question and the soundness of research methodology used, and judgements about these are often made on the basis of titles or abstracts (Gough and Elbourne 2002). Sources can include electronic databases, more traditional ‘handsearching’ of academic texts and ‘grey’ literature such as reports. These are not restricted to sources involving random-controlled trials (RCTs).The merits claimed are transparency and a comprehensive ‘mapping’ of the field, two features which are held to be lacking in the more traditional ‘narrative review’ literature review. The guiding research question for EPL’s commitment to a systematic literature review is: ‘What forms of support for beginning teachers appear to be effective?’ Initial discussions with EPPI-Centre have established that a broad approach will be drawn on, to include continuing professional development (CPD) literature concerning early professional learning. The review aims are three-fold: to inform emerging empirical findings and help guide theory development; to enable a critical review of the systematic review process itself; and to contribute to the robustness of the findings by providing breadth to the large body of contextualised empirical data the research will produce. An example of an EPPI-style review relevant to the EPL project is provided by a recent publication on NQT induction, the research question of which is ‘ How does current research characterise the impact of NQT Induction programmes on new teachers in relation to enhancing teaching expertise, professional development, job satisfaction and retention rates?’ (Totterdell et al 2004). It reports that of the 621 references screened, 51 provided what was regarded as a ‘descriptive map’ and only two focused on new teacher induction and CPD. The review concludes there was a ‘lack of sound research’ to address the review

3

question. However, it does provide a valuable starting point for issues of induction and retention and recommends further investigation of the field. The paucity of existing literature resulting from strictly applied EPPI- style filtering decisions has been noted elsewhere (eg Torrance and Coultas 2004). One major casualty of its methodological strictures is qualitative research, especially on the basis of methodological issues. This is a serious problem for the EPL’s commitment to investigating the role of informal culture, the (learning) experiences of new teachers and the varied contexts of learning. Given that we want to combine qualitative with quantitative approaches our review has been broadened. Our early database searches, including ERIC and BEI, have been subjected to our own initial filtering criteria, concentrating on UK studies 1998-2004 and using a variety of what we judge to be relevant search terms rather than a single specific question. At this stage in particular, exceptions to our initial filtering criteria include sources of particular methodological or theoretical interest. As just one example of what is not included in the EPPI-Centre (Totterdell 2004) ‘descriptive map’, and using ‘new teachers’ and ‘induction’ as search terms, we judge a case study of beginning teachers and their informal learning by Williams (2003) as worth closer reading. It is also the case that a conceptual understanding of early development in teaching benefits from ideas outside the conventional, largely insular literature on teacher education. There is explanatory power in ideas such as the importance of friendship (White 1990); the formation of human bonds (Almond, 1988); psychosocial support and integration (Jacobi 1991); mentoring in adult development (Levinson et al 1978). We developed the grounded concept of ‘relational conditions’ as a way of explaining the social context within which individual beginners lived the text of their own development between extremes of ‘total abandonment’ and ‘rigidly controlled, stifling support’ (McNally et al 1997) – an explanation that is consistent with accounts in cultural psychology of the reciprocity between the ontogenesis and sociogenesis of knowledge (Valsiner and van de Veer 2000). Besides our overall guiding research question (as above), there are three others stated in the project proposal, concerning the use of competence statements, non-formal learning and the development of a model of early professional learning. These ‘express the direction of our intended research’ and as such are too rich to generate direct systematic review ‘hits’. In addition, the size and research backgrounds of the research team are further resources for expert judgements about available literature sources. The ongoing development of the ‘jobsat’ indicator is one example. The development of ways of including ethnographic or interpretative research sources in the systematic review process is currently of considerable interest, as is the broader issue of combining research methods from traditionally separate perspectives (eg. Gorard and Taylor 2004). For example, Dixon-Woods (2004) reviews ten different approaches to such qualitative/quantitative combination in the context of systematic review. In her view, none of these provide a recipe which eliminates questions of judgement. We see our work as part of that development.

4

2 Teachers as ethnographers in their own schools Rationale for use of teachers as ethnographers Understanding how new teachers learn is limited by conventional interview strategies. In one or two interviews, however searching, it is difficult to get beyond a vague yet dominant picture of an informal relational process of belonging and becoming. Our own work in this area led us to making the case for more progressive focusing. This position is supported by the experience of Eraut (2000), who also concluded that a strategy reliant on interviews by outsiders was limited in its potential for eliciting ‘evidence of processes that, if not entirely tacit, do not come readily to mind’. The nature of early professional learning has a greater chance of clearer exposition, therefore, through more sustained contact with the learners and their context. This has led us to the deployment of teachers as ethnographers of new teachers in their own schools. The workplace presence of the T-Eths presents naturalistic opportunities for gathering richer evidence on the activities and social transactions in which new teachers engage. Not only are they uniquely close to the action as researchers, their closeness to the lived experience of learning to teach has the advantage of allowing day-to-day access to studying the nature of individuals’ learning through a broader range of professional settings e.g. post-lesson discussion, staffroom conversations, and also different types of interview e.g. lesson-focused, biographical. Given a protected, flexible time allocation they are in a unique situation to exploit their native status and to be participant observers in a range of situations.

In taking this methodological step forward step we hope to develop a new theoretical understanding of EPL. Our method reflects some characteristics of Flyvbjerg’s (2001) espoused approach to social science - closeness to reality, practice before discourse, cases in context, for example – and, although it is not purely phronetic research, we are exploring his view that a focus on concrete cases and narrative is ‘perfectly compatible’ with attempts at empirical generalisations. The ethnographic case studies of new teachers in their schools will provide specific knowledge in context. Our potential contribution has to be tempered, however, by the negligible influence of researchers so far on teacher education policy. As Hagger and McIntyre (2000) remind us, generalisations from research can offer insight and guidance but the practice of teachers is about subtle judgements in unique situations. Selection process In selecting the teachers we drew largely on our own prior experiences as teachers, researchers and interviewers in educational contexts. There was little guidance in the literature though cautionary observations from Ruddock to Boreham at TLRP conference were noted. Our view was that we wanted credible teachers who had the respect and trust of their colleagues in school. We did not regard research experience as essential and recognized that many teachers act as ethnographers in less explicit ways. Those who applied would be applying to do a job for which they were not specifically trained and for which we did not expect them to be trained. For this reason and also because of our uncertainty in this relative novel experience for us, we did not go as far as writing a ‘person

5

specification’. We wanted good teachers but the kind of good teacher who was also able to stand outside the inevitable absorption of the job. We felt we had to remain open. The advertisement brought a larger response than expected (39 applicants). It was not easy to form a short leet on the basis of written applications ranging from two short pages to over ten. In fact the CVs were not very helpful, oriented as they were to teaching jobs and previous experience, none of which could be of any specific relevance. The one exception actually had a Ph.D. in a related area but, to his advantage, he like us wondered whether this might be a problem. With the exception of a few that seemed to express a cursory interest, interest appeared to be genuine and, as we felt it important to get a more of a sense of each person, an initial long leet of 33 was drawn up. It was subsequently suggested to us that in the immediate ‘post McCrone’ period in Scotland, many teachers were looking for fulfillment other than through promotion. The ‘EPL project’ seemed to reflect an area of great interest in supporting beginners in teaching. The long leet interviews allowed us to think further about what we wanted and, perhaps more, about what we did not want. The candidates were generally fluent and articulate with a range of admirable qualities. Genuine interest in the project, some insight into the beginner’s experience rather than what policy dictated, emerged as possibly the main criteria (along with an impression of credibility and respect amongst colleagues, already mentioned above), although individual intuition and discussion within the interview panel played a large part in clarifying who showed these. A short leet of 13 was eventually agreed and further interviews took place. These allowed us to develop a conversation with these applicants and we began to form firmer impressions. Any of these 13, and some beyond that, were ‘appointable’ so we ranked these ‘independently’, then collectively, then independently again, before whittling it down to six. Collective episodes proved to be important in reconciling the differences and disagreements amongst ourselves, and in learning how others interpreted the interviews, from psychological profiling to impressionistic and intuitive. We also began at this stage to see some seven or eight in the job as individuals interacting well with each other as a working team. The six were appointed on 1.5 days each. Since then one – the youngest - has been promoted and has had to reluctantly give up. She has been replaced by another from the short leet. Another is with child and will therefore be assisted by a co-opted teacher from her school. Looking back from our better acquaintance now with the teacher-ethnographers it is possible that we were operating some kind of implicit ‘perspec’:

capable, respected colleague approachable – a ‘friendly face’ - and part of a wider school community access to people and situations thinker but not paralysed by reflection – ‘doer’ who could also think not closed mind or dogmatic offers some insight into new teacher experience ability to engage in explanatory dialogue genuine interest and not a pure career move energy and commitment to see the job through operates well within a group of peers

6

Though we had decided that three years of teaching experience was probably a minimum requirement, the six have some 140 years of teaching altogether with ten years as the least. (The one with 3 years was lost to promotion within a month of being appointed.) None of them is in or seeks, as far as we know, a post in senior management i.e. as Depute Head. Preparation It was our intention to introduce the appointed teachers to ethnographic methods. This was done in four two hour sessions after school from March to June in preparation for gathering data at the start of the new school session in August. For these sessions we drew specifically on Hamilton (1999) and Coffey (1999), more generally on Spradley (1979) and Glaser (1978) and used some case study excerpts as a basis for more reflexive discussion. Thus we covered interview approaches, the need for sensitivity to the emergence of data, theoretical refinement, styles of reporting and also an awareness of their ethnographic selves. To a large extent they were naturally attuned to much of this. They were well aware of the possibility that their very closeness to the action as teachers, particularly if they had some mentoring responsibility themselves, would become an ethical issue. Already they have begun to conduct interviews in their own styles and appear to be eliciting data comfortably, with a knowledge of the immediate context, as in this excerpt from the new teacher of drama in a temporarily decanted department.

T-Eth: We are going to spend maybe 20 minutes maximum talking about your first wee while and any experiences you have had from the start of term. Again it’s confidential and anything you choose to tell me will be between us. I’m really interested in anything, any experiences, either in classes or in the department or inthe school that you want to talk about. I obviously have some ideas about the things that might come up, so we can maybe work backwards and forwards between the two of us. We are not talking about things that need to be earth shattering either, it’s not huge events all the time, it’s the bread and butter of daily life, so I mean, if there is anything… maybe starting with first impressions?

NewT: It was a bit of an upheaval I suppose being over there and then coming back and not sure. [Note - The drama department was in use as a venue for a theatrical performance in the first week of term and the department was decanted into the main school for that week.] For the classes as well, they weren’t sure if they were coming here or coming there, and because it’s quite a short period then if they cameover here and…..

Reflexivity From the outset the potential ethical dilemmas of their dual role - researching new teachers while also providing possible support - have been acknowledged and discussed. What the introductory academic text of ethnography is not quite able to address is the actual primary identity of the intended researchers as teachers in a school. It was

7

anticipated that there would be changing perceptions by colleagues (and by themselves) of their ‘new’ identity once they assume this rather singular position as researchers of some of these same colleagues. How would they be seen, for example, by new teachers in the staffroom? (a question we asked at interview). The position of part-time teacher and other-time observer could well cast them in a surveillance role. Being seen as respected and trustworthy by school colleagues does become a fairly important criterion if we accept this scenario of identity stress. It places an importance on the selection process that we had perhaps not adequately realized at the time. Potential as an ethnographer is something that can be supported, in contrast to stability of identity through established reputation, over which we have no control. The first tangible indication of this was on their first day free from teaching duties for their research. They themselves ‘felt odd’ in suddenly not doing ‘teachery things’ and of course their immediate colleagues were not slow to notice this ‘day off’. Such comments appear to have been delivered as friendly banter with some actual expression of interest in the research itself. They have had to organize themselves for this different kind of work, to protect the time given to them. One has actually ‘lost’ her room, an ironic outcome of her free day as this is something she has found is cherished by new teachers themselves. The presence of experienced teachers as researchers in their own schools means that probationers are afforded the opportunity to relate stories about the induction process to an understanding and receptive audience. Two ethical issues seem to be emerging from this developing relationship. One is about the power of the teacher-researchers to intervene in problematic situations they gain knowledge of by virtue of their secondment to the project. Secondly, the teacher-researchers themselves tend to be displaced by their research activities, possibly into a marginal category, not true outsiders but “boundary-spanners” who connect the school to the world of HE. This could have implications for their own career and personal/professional development as they become involved in the flows of power between the multiple stakeholders in school education Reflexivity in the literature is also a central concern, for the project in general and for the teacher-ethnographers as it relates specifically to their role. Current writing at MMU is looking at the ways in which teacher-researchers appropriate research methodological texts on reflexivity (Stronach et al, 2004; Somekh and Lewin 2004; Piper and Stronach, 2004). These appropriations are undetermined and creative in ways that hitherto have been neglected by the literature. We take the view, therefore, that we need to think in more interesting ways about what is going on when one reading enters another and re-emerges with a claim to methodological warrant. We hope to practice reflexivity as a singular and situated activity without the need to adhere to any given ‘model’ for inquiry. For our teacher-researchers inquiring into the ‘the familiar’ and ‘distance-near’, there is the head-on issue of how their professional knowledge (assumption, bias, wisdom, or craft) articulates with the development of a research evidence-base; but there are other questions too which we hope to have explored in due course. How can transformative forms of research attitude and perspective be developed? What are the opportunities and constraints in developing such ‘dual’ consciousness? What are the optimal ways of educating and training

8

professionals in such strategies? What is the effect of such approaches on professionalism and the learning outcomes for students and others? Early findings Findings actually begun in the interviews as they began to offer insights and reflections on the experience of new teachers, some of which have fed into our discussions prior to conducting the research interviews, for example the induction scheme itself and the the importance of relationships. As we scrutinise the third tranche of data we have a sense of confirmation of some previous ideas – support of colleagues, acceptance by children in classrooms, emotional aspects of beginning, orientation in new context, and so on. The national (Scottish) induction scheme itself (see SEED leaflet 2004) has revealed some unfortunate casualties in inappropriate placements of individuals, and an overload of anachronistic information from some local education authorities. One crucial bridge has been encountered: how to get at what and how new teachers learn. The T-Eths themselves have found that simply asking this elicits little other that an embarrassed muteness in the newts who seem surprised at their own inability to articulate anything specific. This did not come as a surprise to us. However, the suggestion from one of the T-Eths is that we ask them ‘how have you/do you think you have changed?’. At present we await the outcome of this and will be able to say more in six weeks when the conference takes place. 3 The construct of job satisfaction One of the learning outcomes of the induction year is job satisfaction. It is a common observation that some people are satisfied with their work while others are dissatisfied with it. In fact, most people like some aspects of their work and dislike certain other aspects. This factor has major economic implications, as high levels of job dissatisfaction are associated with resignations - early departure from the profession by dissatisfied teachers is a major problem in England, although less so in Scotland. Ensuring job satisfaction is also important for ethical reasons - as a matter of principle, beginning teachers should find their first year as satisfying an experience as possible. So for these reasons, we are seeking to measure the extent to which our intervention in the induction year affects probationers’ job satisfaction. For the purposes of job satisfaction research, a job is assumed to consist of several elements known as facets, job factors, job dimensions or job characteristics. For example, a teacher’s job might include the facets helping children overcome their problems and administrative paperwork. A review of the job satisfaction literature indicates that most of the published research uses a standard type of job satisfaction instrument. This instrument lists job facets and asks the respondent to rate his or her level of satisfaction or dissatisfaction with each of them separately. Scores for satisfaction/dissatisfaction are usually obtained using 4, 5 or 7 point rating scales, the commonest response format being very satisfied, moderately satisfied, neither satisfied nor dissatisfied, moderately dissatisfied, very dissatisfied. Occasionally, more emotive rating scales are used e.g. Delighted … Terrible. The number of facets included in published job satisfaction

9

instruments varies from 5 to 75. From time to time, researchers have experimented with more complicated types of instrument, but these have not survived in mainstream research. There is a choice between generic and occupation-specific types of instrument. Generic instruments can be used for any job and thus the facets are defined at a general level - e.g. “autonomy”, “financial reward”. In instruments designed for specific occupations, the job facets reflect the details of the occupation concerned e.g. “relationships with parents” might feature in an instrument designed for teachers. For the TLRP project, we designed an instrument specifically oriented to teaching in the induction year. An important theoretical contribution to our understanding of job satisfaction is Herzberg’s (1966) study of the motivation of industrial engineers and accountants in Pittsburg. This research found that the job factors which gave people satisfaction were different from those which made them feel dissatisfied. Herzberg called the former ‘motivators’ and the latter ‘hygiene factors’. Job satisfaction arose from factors which were intrinsic to the work itself (“the motivators”). But when people were dissatisfied, this was usually due to factors extrinsic to the work itself. Importantly, improving the latter did not make people feel satisfied - just less dissatisfied. Herzberg’s two-factor theory has been challenged many times since it was first published, but it has stood the test of time. It has, for example, been broadly confirmed in at least four studies of teacher job satisfaction. Previous research on job facets that relate to satisfaction/dissatisfaction Generic studies One recent review of 8 meta-analyses and review articles by Van Saane et al. (2003) identified 11 generic job factors:

Work content – variety, challenge, routine, role ambiguity, complexity Autonomy – individual responsibility, control over the decision making process Growth/development – personal growth, training/educational opportunities Financial reward – salary, fringe benefits Promotions – likelihood of promotion Supervision – having the support of one’s supervisor, recognition of your efforts, fair treatment, feedback Communication – opportunities to talk about problems Co-workers – competence, professionalism of relations with them Meaningfulness of the work itself Work load – tedium, interpersonal conflict, stress, time pressure Work demand – emotional demands, extra tasks

Studies of job satisfaction in teaching These studies reveal more specific facets. Dinham and Scott (1998) carried out exploratory and confirmatory factor analysis of a 75-item job satisfaction scale given to 892 Australian teachers. The items were taken from interviews with teachers, and thus reflected direct

10

teacher experience. The researchers identified an 8 factor model, naming the factors as follows:

School leadership Promotion for merit Infrastructure School reputation in the community One’s status as a teacher Student achievement Workload Professional self-growth

They found that the most satisfying job dimensions were self-growth and pupil achievement. The job dimensions about which participants felt dissatisfied included poor school leadership, poor infrastructure, and low reputation of the school in the community. Fraser et al. (1998) studied 250 primary and secondary teachers in Scotland during 1988-91. Facets of teaching (taken from a US study) were rated on a 4-point scale of satisfaction. Separate results are not reported for probationers, but were said to be broadly similar to the results for experienced teachers. The facets were ranked in descending order of satisfaction among 5-year old teachers:

Friendliness of staff Autonomy over my teaching Intellectual challenge Support on discipline Pupil motivation and behaviour Salary Recognition of my efforts Prospects for career advance Balance between work/personal life [facets below were rated dissatisfying overall] Colleagues’ view of teaching Availability of resources Influence over policies/procedures Relevance of staff development to my needs Workload Proportion of time spent on admin Society’s view of teachers

Finally, Nias (1981) interviewed 100 early career teachers in primary schools. This was a naturalistic study in which the teachers spoke freely about their likes and dislikes in their own words, rather than rating facets presented to them. However, Nias did obviously translate much of what she was told into her own language (that of a teacher educator).

11

Satisfiers: Being with children, Caring for children and helping them, Feeling of personal competence, Personal growth/extension of self. Negative satisfiers (aspects of the teaching role which were dissatisfying) : Lack of support, Lack of opportunities for personal development, Personally objectionable aspects of teaching (e.g. cleaning paint pots), Low status in the community. Dissatisfiers: Staffroom pettyness, Unfriendly colleagues, Poor physical working conditions, Lack of promotion/career prospects, Physical/mental fatigue. Construction of a measure of job satisfaction The definition of job satisfaction in this research converges closely on the way this construct is defined in the general occupational psychology research and in the extensions of this to teaching. Generation of a list of job facets The six teacher researchers involved in the project, all of whom were currently interviewing probationary teachers as part of the research, were asked to generate a list of the dimensions of a probationary teacher’s job about which they felt satisfaction or dissatisfaction. Satisfiers and dissatisfiers were generated separately. This resulted in 57 items (which contained numerous duplications). Consensus meeting Two teams of three teacher educators and a teacher researcher examined the 57 items and sought to eliminate duplication. Each team proposed its own list of job facets. The teams and the researchers then compared the two lists and agreed on a consensus list. Comparison with previous research The consensus list was compared with each of the four comparator studies cited above. This revealed a high degree of overlap, i.e. most of the job facets identified in previous studies had emerged in our own exercise. However, three facets cited in previous literature were not represented in the list generated by the teacher researchers: Financial reward/salary (occurs in both Van Saane (2003) and Fraser et al. (1998)) Ability to exert influence over policies/procedures (occurs in both Fraser et al. 1998 and Nias 1981)) Promotion/career prospects (occurs in all four comparators) Extended list of facets

12

It is understandable why these facets would not be major sources of job satisfaction/dissatisfaction among teachers in their probationary year. Nevertheless, we decided to cover them to ensure as much convergent validity as possible with previous studies. New items were specially constructed (e.g. “Your salary as a probationary teacher”) or adapted from the original list generated by the teacher researchers (e.g. “The availability of permanent posts in your subject” to deal with career prospects). Revision by teacher researchers The extended list of facets was assembled into a scale and reviewed by the six teacher researchers. The main changes they made were increases in the specificity of some of the facets and minor revisions to the wording of particular items. They also decided to adopt a response scale of Very satisfied, Satisfied, Neither satisfied nor dissatisfied, Dissatisfied, Very dissatisfied. Pretesting JOBSAT currently consists of 40 facets of teaching in the probationary year plus one item to rate satisfaction with the job as a whole. It is being pretested on over 200 probationary teachers in Scotland. This will be followed by statistical analysis of scale properties and the production of a final version. 4. Constructing the other indicators The indicators in this quasi-experimental design represent major qualitative dimensions of EPL and performance. Based on our current knowledge of EPL, we consider that these are in the areas of a) the degree of job satisfaction or extent to which they feel comfortable with their role as teachers (as described above); b) interaction with colleagues or the extent of socio-professional integration; c) children’s descriptions of new teachers. Taking account of professional performance and judgement would require two additional areas of d) pupil attainment in context and e) expert judgement of teaching performance.

Pupil attainment What is interesting about this indicator is the way in which its complexity is masked by its apparent simplicity. Attainment is assessed as a matter of course by schools and teachers and, on the face of it, determining the impact of teachers on attainment should not be problematic. What we want to try and do is to see whether an intervention aimed at improving the development of teachers in their first year can be detected in an improved attainment by the pupils they teach. This is much more difficult than it sounds. We can’t use external exam results because not all of our sample will present pupils for exams at the end of their first year. Even if they did, we would not have a baseline and so we would have to compare overall attainment of pupils in the intervention group with a control group – much more difficult than comparing a measured improvement in attainment if that were possible.

13

The other problem with using external exams is knowing how much of the pupils’ attainment to attribute to the new teacher. Pupils will have learned some of their course material from previous teachers whose effectiveness might be expected to carry through. The US based “Teachers for a New Era” project will use standardised tests, often state mandated, which are administered regularly as a matter of course (University of Washington (2003). But such SATs are not part of the culture in the UK and developing a set which would cover all the subjects which will be taught is a daunting task. Interestingly, the Teachers for a New Era project will also use samples of classwork and videos of children working, both of which form part of our proposal. The problem with each of these is how to make them comparable across different schools. Taking this all into account, we conclude, therefore, that we need multiple sources of attainment data which will be subjected to a professional judgement, in which the judges review a variety of evidence. The evidence would probably be:-

The outcomes (marks) from routine assessment carried out by the newt. Samples of pupil work which will allow the judges to tie the above assessments to

the pupil’s actual attainments. The grades from any external exams carried out by the newt’s pupils Marks and samples of work carried out by the pupils in other subject areas. This will

enable the judges to make a judgement about whether the attainment in the newt’s subject is the same, better or worse than might be expected from the pupils’ performance in other areas in the same school.

A judgement from the new teacher’s department on the attainment of the pupils. We will need to focus this on pupil attainment rather than newt’s teaching performance.

Some video evidence, possibly created in collaboration with pupils in the school. At this stage, we think that the evidence for each new teacher might be put into three categories (par for the course for that context, better or worse) by 5 judges acting independently. Inter-rater reliability will be computed. There is still a fair bit of development required. We need to find out what evidence we can reasonably expect to get from the school involved and how it will be assembled and who will assemble it. Given the number of new teachers, judges will have a limited amount of time to spend on each assessment. We are about to discuss the problem in more depth with our teacher researchers and to tap into their experience of assessment on the ground. This is the next development priority for our project. Expert judgement This will also involve 5 expert judges working independently who will rate newt teaching performance on the basis of a sample video of his/her teaching. Again, we will compute inter-rater reliability. Judges could be teacher fellows (experienced teachers who also have experience of assessing student teaching because of their work on the ITE programme). The videos will be about 30 minutes long. We have yet to decide whether these will be

14

based on a single lesson or on some kind of compilation. Judges will be asked to put newts into perhaps three categories, possibly in a similar kind of scheme to ‘pupil attainment’ i.e. average, better or worse. Judges would be asked to give a rating and a short written justification. Assuming that we get reasonable inter-rater reliability, it may be of interest to see whether the reasoning behind the grading is comparable. A similar rating would be obtained from the newt’s department. We should refer here to the National Commission for the Protection of Human Research Subjects who were able to agree on a set of acceptable procedures but totally unable to agree on their justifications for them (Toulmin, 2001, p.132). Their judgements were in agreement in practice but their reasonings were divergent, dependent on their backgrounds. There is some support, therefore, for exploring the possibility that judges may agree gradings but for quite different reasons. Development would be required for this. How would the video be made up? If it is more complex than a single lesson, how would the compilation be edited. One possibility is to video two or three lessons and ask the newt to submit a video of what they think is a good example of their teaching? Another question is who does the videoing? One possibility is a split screen of the teacher’s and pupils’ records. Split screen may depend on clever editing techniques and, though we may manage this technically, it may be hard to watch. Gaining the consent of the new teachers, headteachers and parents and children may be a greater obstacle to this approach. We are discussing in particular a compilation by pupils on the theme of ‘what my classroom is like’. The idea needs to be piloted in the next few months or so on a small sample of new teachers from other schools and a timetable drafted. Children’s descriptions Our current position is that we would develop an indicator based on an item bank of pupil statements. This would be similar to some existing approaches but honed through user engagement (as with ‘pattain’) to new teachers specifically. We are also considering whether some kind of pupil video of their classroom might be used in combination with an item bank approach. The teacher –researchers could be well placed to do some initial video recording later in the year as they establish good relations with new teachers. Once we have footage we could process it quickly through a mock panel. There are significant logistical implications for equipment and collation. One of the team has experience of ‘Time Code’ (‘an unmissable masterpiece of directing and acting’), filmed on four digital cameras, each following a different character, all in one take, with each recording is played on a different corner of the screen. Apparently it is possible to follow the character whose action seems most relevant or to scan the whole picture. There has been some research done on the generic issues around video as qualitative data (which we have still to locate and analyse) but it may be that this kind of evidence can be used systematically in the assessment of teacher performance (and we are thinking ultimately of the potential benefits as self-assessment). A recent evaluation (BECTA 2002) of the introduction of digital video to a group of 50 schools in England produced positive findings. Crucially, for our purposes here, the production of moving images of classroom activity by pupils introduces a reflexive element into the process of assessing attainment. The study suggested that behaviour and other

15

aspects of the classroom environment were significantly enhanced following some forms of video-based intervention. Interaction with others Initial discussions have led to agreement that this should take the form of a mapping exercise directed at sources and types of support. A graphical device of this kind would avoid questionnaire fatigue and may produce more useable results. In-depth network analysis would be problematic but may well not be required for our purposes. 5 Concluding comment In giving general consideration to the methodology we are evolving - quantitative indicators based on qualitative substance and the whole question of naturalistic experimentation – we have to ask whether it can provide a meaningful base of evidence on which we can eventually warrant our claims. The research design is first and foremost naturalistic in that it depends on humans as instruments, so our initial qualitative findings are subject to criteria appropriate to naturalistic investigation, for example transferability, dependability and confirmability (Lincoln and Guba, 1985). When these findings as grounded theory are then coupled with quantitative means and put to the test, the research design becomes problematic. The defining tension lies in the co-existence of qualitative and quantitative methods within the one project. The credibility of any conclusion will surely rest on their compatibility and ultimate co-productiveness. Two aspects of the design, each at opposite ends of the methodological spectrum, have been discussed above, namely the deployment of teachers as ethnographers in their own schools and the development of indicators. While these are still evolving there is a sense of grappling with a high-risk strategy. Yet, as Gorard (2002) argues:

there are therefore no real rules about research methods (and certainly no best methods), and being a little cavalier in producing results may be fine as long as one isthen a little conservative when presenting them

t- t

He also makes a case for naturalistic experimentation when interventions are ‘part of the normal policy process’. Thus, if 10 schools, for example, were to adopt our model of EPL, then they can be construed as an experimental group; another ten that allow us to observe their ‘normal’ practice become the control group in a natural experiment. While this may have inferior validity to a true experimental design, it is nevertheless closer to practice and can still satisfy the conventional scientific criterion that protocols are transparently reported. Pure experimental designs are simply not possible in educational settings, if only for practical and ethical reasons. Bearing in mind the strong narrative base for our theory development, however, one solution suggested by Gorard to the problem of delivering firmer evidence within educational research is:

…to treat all such narrative approaches as working towards an explanation that can then be (con)tested (by converting to an intervention study on the nex project iteration). But this is seldom done, perhaps because of the dominant mono me hod

16

culture in UK educational research, or perhaps because it is cosier not to have to test our 'knowledge' in a relatively brutal way …...

It may be possible then by combining methods to arrive at a clear and robust warrant as a foundation for evidence-informed practice not only by predicting benefits of particular conditions but also because of the causal explanation which lies at the heart of any warrant for change. Nor is our conception of reflexivity within the project ‘anti-Science’ if Science is held to be ‘a self-conscious, reflexive, open-ended process of knowledge construction’ (Tambiah 11990: 68). The project itself, as it grows, is taking us away from current obsessions with universalistic prescription and a priori methodological specification. We see a potential in reflexivity to open up educational research to creative, non-arbitrary development. We are continually aware too of the need for findings to connect with teacher thinking if there is to be any impact. That is why practitioners are central to the research, not only in carrying it out but in mediating the knowledge we obtain to make it their own and also in its wider dissemination. The attempt to construct indicators of children’s perspectives on new teachers, the defining element of their work, may also be a form of warrant which engages their interest. ‘User engagement’ in the project extends beyond the teacher-researchers to a large group of probationers and colleagues, both in the case study schools and in the schools in those local authority areas which will be involved in the test phases. It will also extend to pupils themselves, and possibly to parents where we think some kind of co-learning process with parents and probationers may be a way of measuring the external impact of the intervention. Flyvbjerg (2001) has alerted us to the importance of power within the methodological deliberations of the social sciences and so if we are to make this engagement function as more than just a one-way flow of data, power has to be devolved to these other stakeholders or actors.

17

References

Almond, B (1988) Human Bonds Journal of Applied Philosophy 5, 1, 3-16 **BECTA 2002 Coffey, A. (1999) The Ethnographic Self: Fieldwork and the Representation of Identity London, Sage Publications Dinham, S. and Scott, C. (1998) A three domain model of teacher and school executive career satisfaction. Journal of Educational Administration, 36, 362-378. Dixon –Woods, M. (2004) Synthesising qualitative and quantitative evidence for policy and practice, Paper presented at ESRC/RCNB seminar on Systematic Reviewing, University of Sheffield, June 2004. Eraut, M. (2000) Non-formal learning, implicit knowledge and tacit knowledge in professional work. In Coffield F. (ed.) The necessity of informal learning ESRC Report (Bristol, Policy Press) Flyvbjerg, B. (2001) Making Social Science Matter Cambridge University Press Fraser, H., Draper, J. and Taylor, W. (1998) The quality of teachers’ professional lives: teachers and job satisfaction. Evaluation and Research in Education, 12, 61-71. Glaser, B. (1978) Theoretical Sensitivity Mill Valley, CA: Sociology Press Gorard, S. (2002) Warranting research claims from non-experimental evidence Occasional Paper Series Paper 48 Cardiff University School of Social Sciences Gorard, S. and Taylor, C. (2004) Combining methods in educational and social research, Maidenhead, Open University Press. Gough, D. and Elbourne, D. (2002) ‘Systematic research synthesis to inform policy, practice and democratic debate’ Social Policy and Society, 1 (3): 225-236. Hagger H. and McIntyre D. (2000) What Can Research Tell us about Teacher Education? Oxford Review of Education 26, 3&4, 483-494 Hamilton, M. (1999) Ethnography for classrooms: constructing a reflective curriculum for literacy Curriculum Studies 7, 3, 429-444 Herzberg, F., Mausner, B. and Snyderman, B. (1959) The Motivation to Work. New York: Wiley. Jacobi, M. (1991) Mentoring and Undergraduate Academic Success: A Literature Review Review of Educational Research 61,4,505-32 Levinson, D. et al (1978) The Seasons of a Man’s Life Knopl, New York Lincoln, Y.S. and Guba, E.G. (1985) Naturalistic Inquiry California, Sage Publications McNally, J., Cope, P., Inglis, W. & Stronach, I. (1997) The Student Teacher in School: Conditions for Development Teaching and Teacher Education 13, 5, 485-498 Nias, J. (1981) Teacher satisfaction and dissatisfaction: Herzberg’s ‘two factor’ hypothesis revisited. Brit. J. of Sociology of Education, 2, 235-246. Piper H, Stronach I (eds) (2004) Educational Research: difference and diversity. Cardiff: Ashgate. Scottish Executive (2003) Teacher Induction Scheme 2004/05 booklet SEED Edinburgh **Smeyers, P. (2001) Qualitative versus quantitative research design: a plea for paradigmatic tolerance in educational research Journal of Philosophy of Education 35, 3, 477-495 Somekh B, Lewin C (eds) (2004, forthcoming) Research Methods in the Social Sciences London: Sage.

18

Spradley, J.P. (1979) The ethnographic interview Holt, Reinhart and Winston Stronach I, Garratt D, Pearce C, Piper H (2004) Reflexivity, the picturing of selves, the forging of method (submitted June to Educational Researcher) Torrance, H. and Coultas, J. (2004) Do summative assessment and testing have a positive ornegative effect on post-16 learners’ motivation for learning in the learning and skills sector? London, LSRC.

Toulmin S (2001) Return to Reason Harvard University Press, London. Totterdell, M., Bubb, S., Woodroffe, L. and Hanrahan, K. (2004) The impact of NQT induction programmes on the enhancement of teacher expertise, professional development, job satisfaction or retention rates: a systematic review of research rates on induction. In: Research Evidence in Education Library, London: EPPI-Centre, Social Science Research Unit, Institute of Education. **Tambiah 11990: 68). University of Washington (2003) Teachers for a New Era, Proposal to the Carnegie Corporation of New York. Valsiner, J. and van de Veer, R. (2000) The Social Mind: The construction of an idea Cambridge CUP Van Saane, N., Sluiter, J., Verbeek, J., and Frings-Dresen, M. (2003) Reliability and validity of instruments measuring job satisfaction – a systematic review. Occupational Medicine, 53, 191-200. White, P. (1990) Friendship and Education Journal of Philosophy of Education 24,1,81-91 Williams, A. (2003) Informal learning in the workplace: a case study of new teachers, Educational Studies, 29(2/3): 207-219.

19

Project Summary Enhanced Competence-Based Learning in Early Professional Development (January 2004 to December 2008) In this project, led by Stirling in partnership with Manchester Metropolitan University, the overall aim is to improve the learning of new teachers and other new professionals. This will be done by developing and disseminating a research-based, practical model of early professional learning (EPL) which integrates outcome-oriented competencies with non-formal learning, context and identity. Some specific objectives are to take account of individual differences and differences in context and to develop a multiple testing instrument for assessing the performance of new teachers and beginners in other professional contexts. Practitioners have a central role in this project, from the gathering of in-depth data in local sites to the construction and dissemination of the model. Initially, a group of four teachers will be trained as a central part of the research team and will be principally responsible as teacher-researchers for gathering the foundation of qualitative data on the learning experience of new teachers within their own schools. Ultimately, there will be direct contact with over 500 teachers in some 40 schools and with a range of other professions giving grounds for optimism that the project will make a serious contribution to policy in the field of early learning in the professions. EPLSummary-Regard j -m oct03 http://www.tlrp.org/proj/phase111/mcnally.htm

20