school context research: design choices, tradeoffs and payoffs

School Context Research: Design Choices, Tradeoffs and Payoffs

MILBREY W. MCLAUGHLIN & JOAN E. TALBERT

Stanford University

What factors in the school setting influence teaching and learning? More specifically, how do conditions in the multiple contexts of teaching--students in a class, colleagues inside and outside school, administrators, educational policies, local political, economic or social factors--matter to the ways teachers define their job, expectations for their students, and professional activities? In 1987, the Centre for Research on the Context of Secondary School Teaching at Stanford University (hereafter CRC) embarked on a mission to integrate various lines of work on teaching contexts and conduct new, multi-method research on context conditions affecting teaching and learning in diverse school settings. 1

The CRC's work aimed to inform educators, researchers, and policymakers about contexts that matter for teaching and learning, and levers for improving the conditions and quality of school. We conducted a three-year field research project in 16 secondary schools with diverse students and community contexts, diverse school missions and district cultures, and contrasting state and sector policy contexts. A distinctive feature of our research is its bottom-up, teachers'- eye perspective on teaching within these diverse embedded contexts, a view that contrasts with that of researchers and policy analysts who look at practice from the outside-in, considering teachers and their work from established social science or policy analytic frameworks.

An overview of CRC research strategy and major findings Research strategy The Centre's research program combined development of a core data base and lines of analysis with the conduct of special, focused studies. The core data base comprises extensive longitudinal data for 16 sites, located in 7 districts within two states collected during the period Fall 1989 through Spring 1991.

AUSTRALIAN EDUCATIONAL RESEARCHER VOLUME 21 No 2 AUGUST 1994 63

64 M c L A U G H L I N & T A L B E R T

It includes:

• qualitative and quantitative field data on classroom, department, school, district and state teaching contexts developed through interviews, site records, school and classroom observations;

• survey data for all teachers in each school (N=877) at three time points (Spring 1989, 1990, 1991)

• qualitative and quantitative data for 48 students in 4 schools.

The field survey data included items replicated from U.S. national teacher surveys as part of the High School & Beyond program and the National Educational Longitudinal Study NELS:88 program. 2 This strategy permits us to locate our field sites in national distributions and to conduct various kinds of 'bridging' analyses using the field and national survey data.

The special research projects built upon the core data base to provide in-depth studies of specific teaching contexts and extend core lines of analysis on: students, subject matter, academic departments, and professional organisations as contexts of teaching. Another special study pursued potentials of the National Educational Longitudinal Study, 1988 (NELS:88) data base for assessing qualitative dimensions of student learning and so for bridging analyses of the educational outcomes of teachers and teaching conditions addressed by our field research.

Major findings and conclusions Students are the most salient context fo r teaching. Teachers agree that students are the context that matters most to what they do in the classroom and how they think about their work (see for example, McLaughlin and Talbert 1993a). They also agree that today's students differ in many ways from students of the past and not-so-distant past. Contemporary students bring different languages and cultures, family pressures, attitudes about and support for academic learning to the classroom. Today's students present challenges which many American teachers feel unprepared or unable to meet. Teachers' responses to the same students vary. Teachers' responses to today's students vary substantially between and within schools (see McLaughlin and Talbert 1993b). Among teachers participating in the CRC research, three broad patterns of adaptation to today's students were evident, namely:

• enforce traditional standards

• lower expectations

• change practices.

S C H O O L C O N T E X T R E S E A R C H 65

The first two adaptations generally located 'the problem' in the student, and led to disappointing outcomes for both students and teachers. Teachers successful in engaging contemporary students generally have moved from traditional, teacher- controlled pedagogy to work interactively with students and encourage an active student role.

Figure I. CRC Data Base and Bridges to National Surveys

CRC Data Base

CRC Survey Data Teacher Surveys

• Whole-school samples

• Three waves: 1989 1990 1991

CRC Field Data Fall, 1988-Summer, 1991

• Teacher interviews • Student interviews

& case studies

• Administrators & staff interviews

• School • District

• Class observations • Record data

NCES National Longitudinal Surveys

High School ~ i i &Beyond

~ i : 1 1984 ATS ~ . ~ Teacher Survey i .... I.t.ems & Data

NELS:88 1988, 1990 1992 Survey

Items & Data:

• Teachers • Students • Subject tests

Key:

~ ~ | Replicated survey items

Iterative data collection and analysis

Professional communities mediate teachers' adaptations. Teachers in our sample who made effective adaptations to today's students had one thing in common: each belonged to an active professional community which enabled them to reflect on their practice and transform their teaching (see McLaughlin 1993). CRC research found that teachers in strong professional communities tend to feel


efficacious teaching classes of low-achieving students, as well as high-achieving students (see Raudenbush, Rowan & Cheong 1992). Further, we found that teachers' beliefs about content, pedagogy and students are shaped by the strength and character of the professional communities in which they work (see Talbert & Perry 1994). Teachers define standards for their classroom practice and appropriate instructional activities through interactions with other educators. Professional communities of secondary school teachers differ from one another in terms of:

• boundaries and inclusiveness

• strength

• culture and habit.

Professional communities are located in multiple, embedded contexts. Professional communities which shape teachers' practice and beliefs evolve within multiple, embedded levels of the policy system (see Talbert & McLaughlin 1994).

• Schools constitute important contexts and differ strikingly in terms of the strength and character of their professional community. Even within the same district, schools have very different levels of collegiality, faculty innovativeness, and learning opportunities as perceived by teachers (see Little & McLaughlin 1993, Rowan, Raudenbush & Kang 1991).

• Subject cultures and subject area departments also constitute important contexts for secondary school teaching (see Grossman & Stodolsky 1994). CRC research finds that departments within the same high school can differ enormously from one another in the opportunities they provide teachers for collegial support, learning, and examining practices for today's students (see Siskin 1994, 1991). We found more variation between departments within a single school than between schools on measures of workplace climate and professional proclivities (see Talbert & McLaughlin 1994). Because department boundaries encompass all elements of the classroom teaching core--student, content and teacher--they comprise the professional community context of greatest import in most schools.

• Districts matter. The strength and character of the district culture makes important and particular contributions to teachers' professional lives and communities, contributions distinct from school or department influences (see McLaughlin 1992). The relevance of district context for professional community lies in the overarching sense of professional identity, inclusion, and pride it can foster. District community character finds its way into the classroom in terms of teachers' willingness to expend the effort necessary to undertake challenging reforms, or to continue in teaching.


• State policy provides context for teaching. State systemic reform efforts provide the content for discourse and instruction that enables strong professional communities to learn and change; strong professional communities are conduits for state policy (see McLaughlin & Talbert 1993b, Talbert & Perry 1994).

Professional contexts and communities outside the policy system matter for teachers. From teachers' perspectives, the communities of practice that matter are not only those defined by the formal policy system; they include other formal and informal organisations such as professional associations, collaboratives, networks, or subject matter organisations (see, for example, Lieberman & McLaughlin 1992). Each of these embedded communities offers a strategic site for reform; teachers' maps of embedded context expand policy opportunities because they identify new agencies, organisations, and opportunities for learning and reform.

This paper retraces our journey through the methodological issues and strategic choices we confronted in developing the CRC research program. We describe the tradeoffs and payoffs associated with design choices in terms of what we were able to learn about contexts that matter for teaching and learning.

Conceptual challenges and paradigm problems We began our research program by reviewing major lines of context-effects research in order to identify context conditions that affect the nature and outcomes of teaching (see Talbert, McLaughlin & Rowan 1993, McLaughlin, Talbert & Bascia 1990).

The challenges associated with developing a strong and coherent research program around the issue of teaching context were complex. Many were rooted in conventional disciplinary conceptions of teaching and learning and the research paradigms associated with them. We found little precedent in the social science literature for constructing a research design that adopted our bottom-up perspective and made problematic the question of which and how contexts of teaching matter. Answers to these questions were strongly embedded in the traditions of research relevant to our mission.

It was apparent immediately that little agreement existed among researchers about salient and significant aspects of teaching context. What IS context? Is it policy? Parents? School organisation? Demographics? Instructional strategies? Each research tradition had its own favourite set of context variables. And, what are the teacher and student outcomes of context differences most important to understand? Again, there was little discussion on this important question.

Traditional curriculum and instruction research was of limited assistance because it tended to focus on the 'technology' of teaching and not to see the student or school contexts ( for example, see Gage 1978). Much of the research


on curriculum focused on teaching, but not on learning, and almost never on the context for teaching and learning (see Walberg's 1991 review article, Wittrock's 1986 collection of articles considering research on teaching, Centra and Potter 1980). Much of the sociological research on teachers and the profession addressed particular aspects of school context such as the collegial environment and problems of teacher isolation, but had little to say about students, the process or consequence of instruction, and the broader contexts of teaching (see Lortie 1975).

Other research traditions also were limited from our perspective. Policy analytic strategies that purported to start from the insider's perspective--the so- called 'backward mapping' strategy for capturing realities and views from the bottom of the system (see Elmore 1979-80)--were compatible in terms of their starting point; but we came to understand that this approach also violated in important respects our commitment to the teachers'-eye view. The 'maps' employed were not teachers'; they were maps of policymakers and policy analysts, tracing the formal system. Teachers' maps of their relevant environment, we soon learned, included colleagues networks and non-formal agencies and organisations as well. So while the overall approach was consistent with our analytic mission, the specific conceptual framework was not.

Policy analysis of a quantitative persuasion and social science research in the school-effects tradition also were problematic for both conceptual and methodological reasons. Their input/output analyses tended to stop at the school (see Brookover, Beady, Flood, Schweitzer & Wisenbaker 1979, Brophy & Good 1986 as examples). Few such studies looked inside the school to see how particular inputs were used, or how features of the school environment mattered to teaching and learning. The school-effects models implied that relevant aspects of context were already known; at question was their effect size (see Teddlie & Stringfield's 1993 report of a 10 year study and analytic review of school effects research, also Coleman & Hoffer 1987). Teaching and teacher variables generally were absent from these input/output models. Further, the student outcomes measures used in this research have not been taken as problematic in ways that research based in current conceptions of teaching and learning (for example, teaching and learning for understanding) prompted us to take very seriously (see Cohen, McLaughlin & Talbert 1993). In particular, analyses of students' scores on standardised achievement tests were silent on questions of their conceptual understanding or critical thinking.

This line of research takes the school as the focus for analysis, an approach consistent with the top-down bias of context-effects research in which the administrative structure of the system defines the important layers of the organisation, but an assumption we considered an empirical issue. In addition, the random sampling of schools typical of this research tradition decontextualised schools and precluded serious attention to the multiple contexts of teaching. The


quantitative social science tradition, to review, essentially took none of the issues central to the CRC mission as problematic: teachers were missing; student outcomes were framed primarily in terms of standardised achievement scores; the unit of analysis, the school, was taken for granted; the contexts in which these units and teaching and learning are embedded were unexamined. For a centre charged to understand context effects, the empirical and theoretical traditions to which we turned presented fundamental problems and little concrete guidance. We believed that to understand how contexts matter for teaching it would be essential to start with the needle rather than the haystack (see Hawkins 1974, p. 65).

Strategic design choices: tradeoffs and payoffs The fundamental challenge and opportunity for the CRC thus turned around understanding what context was and why it mattered--and suspending judgment about appropriate units of analysis and relevant social science concepts. In quantitative terms, we took as an empirical issue such fundamental questions a s :

What was the dependent variable? What are the units of analysis? What teacher variables are important for what student outcomes? Or in qualitative terms, how and why and when does context matter for teaching and learning? As we began our research, we framed our methodological task as identifying and developing a research strategy that would allow us to integrate the diverse paradigms and disciplinary perspectives relevant to contexts of teaching and learning. We were committed to taking the teachers' perspective and to integrating quantitative and qualitative research methods to pursue these core research questions.

To this end, we made a number of major design choices--each of which had important tradeoffs and payoffs for our Centre's work. These choices fell into three broad areas:

• research organisation:

- a core data base v. a collection of related studies

• sample:

- many sites v. few

- purposive v. random

- embedded v. distributed

• analytic perspective:

- longitudinal v. cross sectional

- integrated v. compartmentalised methods


Research organisation Core data base v. collection of studies. A critical initial issue for CRC research strategy involved investment of research resources. Should the Centre invest in a single core data base to support the entire CRC research program, or should we follow the more typical research centre strategy of supporting a collection of related studies? A core strategy would involve all researchers in the same sample of schools for the life of the Centre, rather than supporting each researcher to go his or her own way in the conduct of studies relevant to the CRC mission.

Reliance on a core data base posed significant risks from a research management perspective. A huge investment of CRC resources in a single core could fail if it turned out to lack sufficient theoretical diversity or empirical variation, or if unexpected problems down the road with continued site access significantly altered the character of the core sample. A strategy of sponsoring a variety of research projects developed and carried out independently spreads the risk. As with any such 'federated' approach, the expected result would be some good studies, as well as some disappointments; but the risk would be distributed. A core data base would put all of the research eggs in one basket.

A core data base also presented related issues of depth versus breadth. Investment in a single core data base curtails degrees of freedom to pursue new issues or sites. Once invested in a core sample, it would be extremely difficult if not impossible to follow an opportunity sample or emerging ideas in a new, expanded sample.

A core strategy also raised extremely difficult access problems. It is one thing to request access to a district or site for a one-time, narrowly focused research project; it is quite another to ask permission for an entire Centre research team to pursue for three years a number of yet-undefined questions about school context.

The appeal and strength of the core strategy as a way to get at context questions overcame these reservations. A core strategy holds something constant- -context--as researchers of different disciplinary and methodological persuasions probe issues about which and how contexts matter. Grounding the CRC's research in a core data base provided opportunity to involve multiple perspectives or lenses in examination of the same realities. The core sample strategy also held important integrating potential and opportunity to examine context issues in depth; this was particularly important given that the nature of salient contexts was problematic and central to our mission.

In order to ensure sufficient in-depth analyses of contexts which emerged as significant, we invested in a set of satellite studies during the second year of our research. Each pursued key issues and puzzles that arose in the initial year of our field research within a particular, relevant subset of the core field sites.

Special studies begun in the second year of the CRC field research addressed a number of special topics for in-depth examination--students, subject matter, departments, unions, and workplace supports for teaching for understanding.


Topics for these special studies emerged during our first year of research as critical context issues from teachers' perspectives, and were little understood in either the research or policy communities. For example, teachers said students were the context that was most consequential as they made decisions about their teaching, yet policy studies and context research generally takes students as 'outcome' or 'target.' Following this line of inquiry, we found that students are indeed the context for teaching that matters most, and that teachers' conceptions of learners are in practice social constructions that vary significantly from school to school, and even within school. Figuratively and literally the same student, we saw, will be constructed differently in terms of teachers' subjective impressions of them rather than so-called 'objective' facts. Those subjective constructions of student, in turn, reflected norms about students and learning outcomes operating in teachers' very diverse up-close professional communities.

All special studies were conducted in the core school sample. This strategy provided considerable analytic overhead, since researchers responsible for these special studies could draw on completed and continuing data collection by the Centre, piggy-backing on the larger effort. The special studies contributed in important ways to the CRC effort. As anticipated, they provided in-depth analyses of critical questions that were beyond the scope of the overall research program and its more general focus and integrative interests. They enabled a dialogue between the focused inquiry and the core research; they generated findings and propositions that we were able to pursue in other sites and in the annual survey. In this way, these focused studies contributed to the development of grounded theory about contexts that mattered for teaching and learning.

This strategy also provided almost endless room for correction through researchers visiting sites at different times, viewing sites through different lenses, and even just picking up missing data. Internal validity and 'quality control' were enhanced enormously though the involvement of multiple researchers exploring different issues in the same site.

Sampling strategies What kind of sample for the core data base would provide us with the best opportunity to understand context effects on teaching and learning? On this question, we confronted three major choices: many v. few sites; purposive v. representative; embedded v. distributed sites.

Many or few? The first choice was dictated in part by our decision to support a core data base that would involve to some degree all CRC researchers, rather than a collection of related studies in disparate sites. The benefits of shared knowledge and in-depth understanding of a site could not be achieved within the scope of CRC operations and the limitations of researchers' schedules by a large sample of schools. We chose to limit our sample to a manageable number of sites.


Purposive or representative ? Further, we decided to draw that sample based on what we did know about context factors that mattered. We developed a purposive sample large enough to provide variation on context dimensions of theoretical interest--eg, school sector, student body characteristics, state policy culture--but small enough to allow CRC researchers to visit many participating schools, thereby multiplying the perspectives and understandings associated with any given site. Our strategy of bridging field and national teacher survey measures and data, illustrated below, enabled us to pursue the issue of sample representativeness and to substantially extend the analytic strength of the field sample and data.

Embedded or distributed? Site location raised another sampling decision. Should the sites be distributed across communities or embedded in similar contexts? Few studies consider embedded contexts and instead adopt a 'school' or 'project' or 'classroom' model to examine issues of teaching and learning, thereby framing context in terms of a specific location or 'layer' of the policy system. This approach does not allow examination of the ways in which the multiple contexts together shape teaching and learning--how state, regional, community, school, or department factors combine to create context for teaching and learning in a classroom. A distributed sample could have enhanced Variation on factors of theoretical interest, but these sites would have been decontextualised in ways that mattered fundamentally to the CRC research program.

The embedded sampling plan we developed aimed at understanding the influence of various state and local context factors operating singly and in combination. We chose our 16 school sites from 7 different districts, located in 4 metropolitan areas in two states (see Figure 2). We chose 'locations' to maximise variation on theoretically interesting dimensions associated with each. The state contexts from which we sampled varied significantly in the 'strength' and character of the state education policy system and involvement in educational reform. California has a strong central state education policy presence and had been active for years in developing curricula frameworks, student standards, and other reform policies that provided extensive and specific direction to districts, schools, and teachers. California's aggressive efforts to reform educational practice make it a leader in what is now called 'systemic reform.' Michigan, on the other hand, had been much less active at the state level, deferring to local actors decisions of policy and guidelines. Also distinguishing the two states are economic conditions and student demographics. While Michigan is by no means a wealthy state, its support for public education exceeds California's and its schools are not confronted with the challenges of burgeoning numbers of students with limited English proficiency.


Figure 2. Embedded Field Sample

Districts (N=7)

S c h o o l s Public (N=13)

Private (N=3)

A

0 0 •

• •

CALIFORNIA MICHIGAN

B C A B C D

& ?' A 0 0 • • • • • • • •

I I I I Metropolitan Areas

T e a c h e r s (N=877)

CA Public: 443 CA Private" 92

MI Public: 342

Within each state, we targeted two metropolitan areas that represent substantially different economic contexts--relative scarcity and relative wealth--in terms of urban communities. Within each metropolitan area we selected one urban public school district, one suburban school district and/or an independent school. The schools were selected to represent 'typical' schools serving the range of district students.

This embedded strategy paid off in enabling us to understand system effects as well as the significance of different institutional routines and responses within the same policy context. Our state sampling strategy allowed us to see how different state policy environments played out at the local level. We saw, for example, that the California mathematics framework provided important content around which math teachers could organise to discuss mathematics and new instructional practices--the stuff of strong professional community and adaptation of practice. Strong communities of mathematics teachers in Michigan, absent content to organise discussion, instead coalesced around maintaining traditional practice and showed low levels of adaptation to students not doing well in their classes. Michigan teachers lacked the strong push for changed content and pedagogy generated by the California reform framework.


Our decision to follow an embedded strategy at the metropolitan/district levels turned out to be a critical element of our research. An initial reason in sampling multiple sites from a metropolitan area was to better understand the significance of local labour market factors for teacher mobility. A far more important analytic benefit resulted from this strategy; multiple sites within a district allowed us to identify district-level influences on school sites, and to begin to understand the ways in which district does and does not matter for teaching and learning. We found, for one, that district policies and practices affected different schools and classrooms in different ways. A district policy about student course taking--requiring all students to take university-eligible course work--for example, operated as a constraint in one setting with a significant number of immigrant students and teachers ill-prepared to provide instruction in college preparatory courses; a neighbouring school welcomed this new policy as an opportunity to pull up their lower-end courses.

We also found, for example, that teachers distinguish between school-level and district professional community factors. The culture and habits of the school influenced teachers' sense of commitment to students, innovativeness, and conceptions of subject matter. District level factors, however, were the greatest influence on teachers' sense of professional commitment and satisfaction with being a teacher. For example, teachers working in a district considered hostile and demeaning to teachers were generally unhappy with the profession, even those teachers who expressed loyalty and commitment to their school and students. Conversely, teachers employed in a district rated extremely high on district professional community indicators expressed high levels of professional satisfaction, even if they were unhappy with their school working conditions.

The CRC's embedded sample permitted us to see how layers of context interact and are mutually reinforcing in some cases, and effectively unrelated in others. A variegated, differentiated and strategic understanding of context emerged from this embedded sampling strategy, a perspective that would have been invisible or misspecified in a decontextualised or distributed sample.

Analytic strategies Longitudinal v. cross-sectional data? Should we collect crossectional snapshots of diverse school contexts or invest in a longitudinal look at context? We elected a longitudinal design, despite the significant investment in field relations required by this choice, and the diminished degrees of freedom to expand. A longitudinal sample allowed us to:

• enhance internal validity

• pursue analytic connections

• see change and stability over time


integrate qualitative and quantitative analyses in ways that enhanced both internal and external validity.

The benefits of this choice to the CRC's research agenda were enormous. We were able to see how schools and teachers adapted over time to changes in the policy environment and student demographics. For example, we were able to see how incredibly robust were teachers' professional communities, even in the face of administrator turnover. However, we also saw how quickly workplace culture responded when leadership gave priority and attention to community building. One school, for example, moved from being a 'dead' professional community to one charged with vitality, self-examination and inquiry in the space of a year after the principal challenged faculty to rethink their practices in light of significant student demographic shifts and established supports and resources for a professional community to tackle the problem. We also saw a department change from a demoralised, toxic environment to a robust community of practice in the year following change in department leadership--from a curmudgeonly veteran teacher who hoarded supplies and kept prize assignments to himself to a younger teacher who immediately moved barriers of assignment, physical location, and 'turf to forge a strong department community. Longitudinal research also benefited the CRC because it enabled us to gain the trust of teachers and administrators over time; as they gained confidence in us, they became more and more valuable as informants and respondents.

There were substantial disadvantages to this design choice. It constrained us from adding other, interesting field sites in order to pursue issues that arose in our analysis of each wave of data. Also, we encountered enormous and on-going site relations costs. The succession of principals meant that we had to renegotiate access multiple times in more than half of our schools. Being in the schools for a three year period raised problems of feedback and contamination. What kind of feedback to give a faculty when there are additional years of data collection? We responded to the need to give faculty some kind of feedback on an annual basis by offering sessions that used CRC survey data to provide faculty a descriptive profile of their school in terms of the CRC survey data, the core sample, and the school's 'location' in a national sample. These descriptions turned out to generate extremely useful data as teachers and administrators asked questions about their profile or responded to our questions about their contexts.

Bridging field and national samples and data. One of our initial design choices for Centre research was to exploit, as possible, national survey data in order to enhance the power of in depth field research. As shown in Figure 1, we included teacher survey items from two national longitudinal educational survey programs in our CRC survey questionnaires. The teacher responses for these strategically selected 'bridging' items established an analytic crosswalk between our 16 field sites and approximately 900 teachers and large national samples of


schools and teachers. In many ways, this bridging strategy has given us the best of both worlds--the ability to become knowledgeable about a small number of sites selected as examples of diverse context conditions while also determining national norms on key research variables.

These analytic bridges enable us to 'locate' our field sites and respondents in a representative sample. By comparing the local and national distribution of responses to replicated questionnaire items and scales, we can determine how representative our schools, teachers, and administrators are in national terms (or in terms of regional or metro-status contexts, for example) on measures of such variables as collegiality, leadership, professional autonomy, goal congruence, instructional goals. To illustrate, teachers' leadership 'ratings' of principals in our field sites (on a scale replicated from a HS&B teacher survey), showed some upward bias in the leadership conditions of CRC schools and showed us just where in the national distribution our worst and best cases fall (see Figure 3). This kind of yardstick is useful for determining the range of conditions our field sites represent on a particular context or teaching variable and in 'locating' school, department, or teacher cases within the national distributions.

Figure 3. CRC School Scores on a Principal Leadership Index: Location within the national distribution for public schools ~

30

20

O! 06

12 10

I1 i0i02 07 08

0 14 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 80 84

Principal Leadership

1 The principal leadership index is a 84-point scale derived from teacher survey responses. School scores for the CRC sites are means of teacher scores. The national distribution of school scores on the index was estimated using data from the 1984 Administrator and Teacher Survey of HS&B schools.

SCHOOL CONTEXT RESEARCH 77

These bridges also provided a yardstick we can use to understand the size of observed differences in our sample. Were significant differences found on a particular (bridged) variable within our field sample meaningful in the context of national variation? For example, when we saw differences between department scores on a collegiality scale replicated from the HS&B teacher survey, we could ask how big these differences were in terms of national norms. Our bridging strategy allowed us to know that subject departments within a single school ranged from the very top of the top quartile of U.S. schools to the bottom quartile of the national distribution (see Figure 4; national quartile cutpoints are shown by dashed lines). This anchoring of our field data in this case encouraged us to take very seriously the strength of department boundaries for teacher community and the significance of distinctive department cultures in teachers' worklives.

Figure 4. Within-School Differences in Department Community: A Case Study

°,.,~

. , . . , ~ . 0

© L~

Key:

I

ENG SS MATH SCI F. LANG ART VOC

Cutpoint of top quartile of national sample of high schools ~, ~, Department Averages

School Average Cutpoint of bottom quartile of - Subject Averages" national sample of high schools -- - Natlonal Survey

Teacher N = 121

Technical Note" This analysis uses a Collegiality Index combining 5 survey items used in the 1984 High Schools. Beyond (HS&B) national survey (Alpha = .84). The figure shows: the average score of the index for all teachers in one CRC high school (School 10), average scores for teachers in seven different subject area departments within the school, and national norms for the respective subject areas (based on (HS&B data for teachers classified according to their primary subject assignment).


By taking departments seriously as a unit of analysis, we uncovered critical differences among departments in such areas as conceptions of students as learners, faculty innovativeness, collegiality, and aspects of teachers' professionalism. Secondary school departments, we discovered, were the workplaces that mattered most as contexts for teaching and learning in all but a few secondary school settings.

The bridging strategy also enhanced our ability to interpret national scores and analyses, because we could understand them in the well-known context of our field sites. What does a high score on the principal leadership scale really mean, for example? Since a number of principals in our sample were rated well above the national mean (see Figure 3), we were able to understand that the score by itself--as a construct--does not convey much about style or nature of leadership. The strong principals in our field sample were extraordinarily diverse in terms of leadership style and strategies. Something more than traditional survey measures of leadership, we concluded, was needed to understand the situational aspects of leadership. How leadership is rated conveys nothing about what it is. Further, the principals in our sample who fell around or just below the mode for U.S. high schools were weak or ineffective leaders. Their 'normality' in national terms tells us a great deal about the quality of school leadership in the country generally. It suggests that those who fall in, say, the bottom quartile are actively nonsupportive of teachers and that our sample may underestimate school differences in teacher community and teaching. One could conclude that the overall capacity for educational leadership among high school principals across America should be of considerable concern to policymakers hoping to engender education reform.

Finally, our bridging strategy allows us to pursue particular field-based findings with the national data base emerging from the NELS:88 program. For example, a 'teaching learning community' scale that has been especially powerful in our field research provides an important bridge to the national survey data. CRC analyses of student learning in mathematics between 8th and 10th grades have shown that the 10th grade math teacher's reports on colleague support for his/her learning is a strong predictor of student outcomes, with controls for student background, prior performance in math and reading, and class track. Bridges to the national survey thus enhance our analytic capacity and potential for corroborating and extending conclusions from our field-based research.

Integrating qualitative and quantitative field methods. We were committed from the outset of our research to moving beyond the qualitative/quantitative debates to develop useful strategies for integrating these kinds of data and methods in ongoing, field-based research. We wanted both to push epistemological issues of integrated methods and to establish a multi-method [in addition to a multi-disciplinary] research community. As a matter of Centre policy, all CRC researchers--even quantitative analysts accustomed to working


only at their computers with survey data--did some field work; conversely, qualitative researchers became acclimatised to using CRC survey data through the quantitative site survey reports.

We engaged in iterative development of qualitative and quantitative research over the three years of data collection. We used interviews and observation, for example, to identify important areas or lines of inquiry for the next year's questionnaire. The survey data served to validate and elaborate the interview- based findings. For example, when interview data indicated that teachers adapted practices in significantly different ways to students who differed in background or performance from the 'traditional' student of the past--even within the same school, we included items on teachers' adaptations of expectations, coverage and content.

We used the survey results to give descriptive feedback to our sites in the intermediate years of data collection and to frame follow-up questions to faculty- -probes of apparent inconsistencies or puzzles. Survey data allowed us to address issues of within-site sample bias. Examining an individual teacher's score on a measure in relation to the faculty's distribution on that measure permitted us to understand where particular respondents 'sat' in terms of such factors as innovativeness, views of school leadership and community, conceptions of subject matter and students. We used these assessments to make our interview and observation sample more representative of the school as a whole.

The quantitative survey data also provided complementary evidence for emergent findings from field research. For example, it was obvious after only a short time in the field that the district contexts within which our sample of schools operated differed substantially as a workplace for teachers. But how much? A district professional community measure developed for a subsequent teacher survey allowed us to quantify these differences between districts; we could show that two of our California districts differed from one another on the measure by almost two standard deviation units, impressive by quantitative standards. We also used the survey to pursue our observation that teachers' sense of efficacy differed as much on a class-by-class basis for individual teachers as it did among teachers more generally. Using a survey question that asked teachers to rate their success with each of their classes enabled us to quantify this variance.

Likewise, qualitative data elaborated survey data and findings. For example, we were able to make sense of the very different classroom goals teachers reported for their classrooms in our first survey. We pursued with interviews the seemingly inconsistent finding that teachers whose classrooms we judged to set high expectations for students' academic competence and skills rated 'good work habits' as their top goal. A focus on work habits, we thought, was found in classrooms with weak academic content. Teachers explained this apparent anomaly to us in terms of how they responded to the challenges and different

80 M c L A U G H L I N 8,: T A L B E R T

needs of students in their classes today, and their belief that good work habits were the way to academic excellence. To focus on academic achievement without attention to these instrumental skills was, in their judgment, an invitation to failure.

The integrative strategies we used were a powerful response to 'metho- centrism' and illustrate the capacity for on-going interaction of methods and data, and the resulting benefits to construct validity, internal and external validity. This approach also demonstrated the many ways in which quantitative methods can be a partner in the development of grounded theory. The result of this approach is a unique data base that combines qualitative and quantitative data collected over a three year period for a stable, core sample of schools and respondents. The Centre's bridging strategy links this data base with nationally representative samples, thereby adding significant explanatory power both to our sample, and to national measures.

The strategic choices we made in terms of perspective and sample were essential. Unless we had made problematic many of the taken-for-granteds in traditional social science or policy analysis, we likely would have missed some of the CRC's most important findings. Most especially:

• The crucial role of departments. Taking the school as the salient unit of analysis would have masked the central importance of teachers' department workplace community. We needed to look inside schools to understand which and how context effects teaching and learning.

• Teachers' variable adaptations to today's students. Considering students only as 'learning outcomes,' narrowly conceived, would have missed both the fundamental way in which students are the context consideration that matters most to teachers' classroom, and the qualitatively different learner outcomes associated with different classroom environments. Understanding that teachers' respond differentially to students based in their subjective construction of them as learner rather than to an objective reality underscored the variable nature of teachers' adaptations to the 'same' students and the critical role of the norms of practice that operate in teachers' up-close professional community.

• Teachers' multiple embedded contexts. Using a traditional policy frame to map backwards from teachers' position in the education system would have blinded us to teachers' maps, and the many formal and informal networks, associations, and communities that shape their practice and mediate policy. The embedded contexts in which teachers move represent multiple strategic sites for reform and opportunities for teachers' learning; they can both constrain and facilitate policy goals because they mediate policy. Likewise, a random or distributed sample, stripped of its larger social, political and organisational context, would have masked or mis-specified the context effects at various levels of the system and the conditions under which they constrain or enable teaching and learning.


Implications for education research The centrality of teachers' multiple and embedded contexts to classroom practice, together with the limits of 'average effects' findings for formulating policy, suggest several important implications for education research. Education researchers need to think strategically about the contributions and limits of survey research and field-based research to understanding and improving teaching and learning. Past research concerned with school and teacher effects has focused too much (and sometimes mistakenly) on estimating average, 'independent' effects of particular variables across widely diverse settings and too little on examining the conditions and processes whereby the multiple contexts of teaching combine to influence teaching and learning for better or worse.

Survey research, with refined measures of both outcomes and context conditions, can provide important information on the distributions of key teaching and learning variables and their context correlates in particular historical periods. Breakdowns of large survey samples into particular kinds of settings can be helping in assessing conditional relations. In addition, we have found that survey data can be useful in developing summary institutional 'profiles' that depict diversity in educational environments and suggest strategic differences and commonalities among schools and departments with common clientele, goals or management issues.

Field-based research furnishes critical interpretative analysis and understanding about why and how diverse aspects of school context influence teaching and learning. This important complement and explanation to survey findings can also attend to ways in which broader influences such as state or local curriculum policies, demographic shifts, or shifts in the broader political economy work through and within the school context to shape classroom activities and outcomes. These important issues extend beyond the reach of survey research but are essential to policymakers' and practitioners' understanding of the complex interactions that comprise teachers' multiple contexts.

Education research that attends to context is essential to efforts aimed at rethinking or reforming practice. Practitioners benefited little from past studies that presented only aggregate statistics and decontextualised summary findings. Further, education research must itself be sensitive to context. Most lines of research on promising practice or school effects ignore those contexts that teachers say are most critical to their beliefs and practices--students and subject area. Teaching does not take place in generic classrooms stripped of subject matter concerns or mindless of the backgrounds, needs, and interests of the students who come to school.

Teachers and administrators ask for evidence that promising new practices or reform ideas 'can work here' and concrete, contextualized information about how


to transform their practices. Without research that considers specific salient contexts of teaching and learning, practitioners have little guidance in 'how to get there from here' or insight about factors central to change in their specific classroom or school contexts. Research sensitive to teachers' multiple and embedded contexts can play a strategic role in supporting systemic change in practice by describing, interpreting and broadcasting contextualized examples of the practices reformers pursue.

Research of this stripe moves away from sole focus on questions of 'what works' to examine aspects of the contexts of practice that enable or constrain desired education outcomes. This kind of systemic perspective can help to identify the different [and often overlooked] levers and resources available within different parts of the education system broadly construed, and illustrate the ways in which they can work together to enhance teaching and learning.

The Centre for Research on the Context of Secondary School Teaching (CRC) was funded by a five-year grant from the United States Department of Education, Office of Educational Research and Improvement [Grant # G0087C0235]. This grant was made to Stanford on the basis of a competitive Request for Proposal (RFP); the federal government has supported national research centres since the early 1970's as a way to focus research resources and efforts on topics identified as in the interest of federal education policy and the field generally. Within the parameters of the RFP, competitors have considerable discretion in the research they propose to carry out under the auspices of a federal research centre. The RFP issued in support of the CRC's mission specified the general topic of the Centre's work--research on the contexts of secondary school teaching. The RFP also specified some aspects of the methodology--combined qualitative and quantitative methods and use of the National Educational Longitudinal Survey: 1988 (NELS:88). The RFP did not specify how these links should be made or strategies for integrating research methods. In fact, the RFP's suggestion that the national survey data would be used by the new Centre to analyse context effects on student learning outcomes turned out not to be feasible, since the second wave of NELS:88 data were not available to the CRC until after our 5 year grant expired. Within the broad research area of secondary school context, the design choices and decisions about methodology discussed in this paper were made in the process of developing our proposal and carrying out our research program. High School and Beyond (HS&B) is a national longitudinal survey research program begun in 1980 in a nationally representative samPle of U.S. high schools, and tenth and twelfth grade students within them. The HS&B program continues to folilow the education and work progress of these students. In t984, a special survey was conducted in a 50% sample of


teachers and administrators within these schools. The National Educational Longitudinal Survey of 1988 (NELS:88) began in 1988 with a nationally representative sample of eighth grade students and their schools. It continues to follow these students into high school and beyond. NELS:88 teacher survey data for this program include only English, math, science and social studies teachers.

References Brookover, Wilbur C., Beady, Charles, Flood, Patricia, John Schweitzer &

Joseph Wisenbaker (1979) School systems and student achievement: Schools can make a difference, Praeger Publishers, New York.

Brophy, Jere E. & Good, Thomas L. (1986) Teacher behavior and student achievement, in Merlin C. Wittrock, ed., Handbook of research on teaching, Macmillan, New York.

Centra, John A. & Potter, David A. (1980) School and teacher effects: An interrelational model, Review of Educational Research, vol. 50, no. 2, Summer, pp. 273-291.

Cohen, David K., McLaughlin, Milbrey W. & Talbert, Joan E. (eds.) (1993) Teaching for understanding: Challenges for policy and practice, Jossey Bass, San Francisco.

Coleman, James S. & Hoffer, Thomas (1987) Public and private high schools: The impact of communities, Basic Books, New York.

Edmonds, Ronald (1979) Effective schools for the urban poor, Educational Leadership, vol. 37, no. 10, pp. 15-24.

Elmore, Richard F. (1979-80) Backward mapping: Implementation research and policy decisions, Political Science Quarterly, vol. 94, no. 4, Winter, pp. 601- 616.

Gage, Nathaniel L. (1978) The scientific basis of the art of teaching, Teachers College Press, New York.

Grossman, Pamela L. & Stodolsky, Susan S. (1994) Considerations of content and the circumstances of Secondary school teaching, in Linda Darling- Hammond, ed. Review of Research in Education, vol. 20.

Hawkins, David (1974) The informed vision: Essays on learning and human nature, Agathon Press, New York.

Lieberman, Ann & McLaughlin, Milbrey W. (1992) Networks for educational change: Powerful and problematic, Phi Delta Kappan, vol. 73, no.9.

Little, Judith Warren & McLaughlin, Milbrey W. eds. (1993) Teachers' Work: Individuals, colleagues, and contexts, Teachers College Press, New York.

Lortie, Dan C. (1975) Schoolteacher: A sociological study, University of Chicago Press, Chicago.


McLaughlin, Milbrey W., Talbert, Joan E. & Bascia, Nina eds. (1990) The contexts of teaching in secondary schools: Teachers' realities, Teachers College Press, Columbia University, New York.

McLaughlin, Milbrey W. (1992) How district communities do and do not foster teacher pride, Educational Leadership, vol. 50, no. 1, pp. 33-35.

McLaughlin, Milbrey W. (1993) What matters most in teachers' workplace context? Judith. W. Little and Milbrey. W. McLaughlin, eds., Teachers' work: Individuals, colleagues and contexts, Teachers College Press, New York, pp. 79-103.

McLaughlin, Milbrey W. & Talbert, Joan E. (1993a) How the world of students and teachers challenges policy coherence, on Susan. H. Fuhrman, ed. Designing coherent education policy: Improving the system, Jossey-Bass, Inc., San Francisco, pp.220-249.

McLaughlin, Milbrey W. & Talbert, Joan E. (1993b) Contexts that matter for teaching and learning, Center for Research on the Context of Secondary School Teaching, Stanford, University, Stanford, CA.

Raudenbush, Stephen W., Rowan, Brian & Cheong, Yuk Fai (1992) Contextual effects on the self-perceived efficacy of high school teachers, Sociology of Education, vol. 65, pp. 150-167.

Rowan, Brian., Raudenbush, Stephen. W. & Kang, Sang Jin (1991) Organization design in high schools: A multilevel analysis, American Journal of Education, vol. 99, no. 2, pp. 238-266.

Siskin, Leslie S. (1994)Realms of knowledge: Academic departments in secondary schools, Falmer Press, New York.

Siskin, Leslie S. (1991) Departments as different worlds: Subject subcultures in secondary schools, Educational Administration Quarterly, vol. 27, no. 2, pp. 134-160.

Talbert, Joan E., McLaughlin, Milbrey W. & Rowan, Brian (1993) Understanding context effects in secondary schools, Teachers College Record, vol. 95, no. 1, pp. 45-65.

Talbert, Joan E. & McLaughlin, Milbrey W. (1994) Teacher professionalism in local school contexts, American Journal of Education, vol. 102, no. 2, pp. 123-153.

Talbert, Joan E. & Perry, Rebecca (1994) How department communities mediate math and science education reforms, Center for Research on the Context of Teaching, Stanford University.

Teddlie, Charles & Stringfield, Sam (1993) Schools make a difference: Lessons learned from a l O-year study of school effects, Teachers College Press, New York.

S C H O O L C O N T E X T R E S E A R C H

Walberg, Herbert J. (1991) Productive teaching and instruction: Assessing the knowledge base, in Hersholt C. Waxman and Herbert. J. Walberg, eds. Effective teaching: Current research, McCutchan Publishing Corporation, San Francisco, pp. 33-62.

Wittrock, Merlin C. (1986) Handbook of research on teaching, Macmillan, New York.

school context research: design choices, tradeoffs and payoffs

Documents