positioning and validating the supervision framework
TRANSCRIPT
Positioning and Validating
the Supervision Framework
Jaap Scheerens, Tina Seidel, Bob Witziers,
Maria Hendriks & Gerard Doornekamp
faculty of
BEHAVIOURAL SCIENCES University of Twente
Department of
Educational Organisation and Management
Positioning and Validating
the Supervision Framework
Positioning the supervision frameworks for primary and secondary education of the Dutch Educational Inspectorate in current educational discourse and validating core indicators against the knowledge base of educational effectiveness research
Jaap Scheerens, Tina Seidel, Bob Witziers, Maria Hendriks & Gerard Doornekamp
Positioning and Validating the Supervision Framework/ J. Scheerens, T. Seidel, B. Witziers, M. Hendriks & G. Doornekamp. Enschede/Kiel: University of Twente, Department of Educational Organisation and Management / IPN: 2005 – 430 p. ISBN: 90-365-2343-5 This project was financially supported by a grant from the Dutch Educational Inspectorate.
Colofon Reproduction: Grafisch Centrum Twente No. of prints: 250 Ordering address: University of Twente Faculty of Behavioural Sciences Department of Educational Organisation and Management
P.O. Box 217 7500 AE Enschede the Netherlands tel. 053 489 4579 © 2005 Alle rechten voorbehouden No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means: electronic, photocopying or otherwise, without permission in writing from the publishers.
Ten geleide
De Wet op het Onderwijstoezicht (WOT) draagt de Inspectie van het Onderwijs op
scholen te beoordelen op de mate waarin deze de verwachtingen van de Nederlandse
samenleving waarmaken. Wat die verwachtingen precies zijn en de vraag wanneer deze
‘voldoende’ worden waargemaakt wordt bepaald in een proces waarbij meerdere partijen
betrokken zijn. In de eerste plaats de onderwijsinstellingen, omdat zij de grootste
verantwoordelijkheid dragen voor de invulling en realisering van kwaliteit. Daarnaast
heeft de samenleving een groot belang bij een goed functionerend onderwijssysteem en
bij opbrengsten die recht doen aan uiteenlopende belangen op het economische,
maatschappelijke en individuele vlak. In de Memorie van Toelichting bij de WOT wordt
in dat verband verwezen naar de direct betrokkenen (zoals leerlingen/deelnemers),
afnemers (zoals vervolgonderwijs en werkgevers) en naar de maatschappij in bredere zin.
Tenslotte heeft ook de overheid een belangrijke rol waar het gaat om het waarborgen van
voldoende gemeenschappelijkheid, het garanderen van voldoende kwaliteit en het
bevorderen van optimale kwaliteit. De zorg voor onderwijs is dus een gezamenlijke
verantwoordelijkheid; de Memorie van Toelichting spreekt in dat verband van “een meer
open systeem […] waarbinnen de opvattingen over kwalitatief goed onderwijs – in debat
met veld en samenleving – […] worden uitgewerkt”.
Deze gemeenschappelijke verantwoordelijkheid voor het onderwijs en de ruimte die dit
biedt voor variëteit alsmede het primaat van de instellingen die het onderwijs verzorgen,
waren belangrijke overwegingen achter de herziening van de wettelijke regeling voor het
onderwijstoezicht in de WOT en de invoering in 2003 van nieuwe vormen voor het
toezicht.
De hoofdlijnen van het toezicht zoals die in de WOT zijn vastgelegd worden uitgewerkt
in toezichtkaders. Deze komen volgens de bepalingen van de WOT tot stand in op
consensus gericht overleg met diverse partijen. Een dergelijk op consensus gebaseerd
kader draagt er volgens de MvT aan bij dat de uitoefening van het toezicht “consistent,
transparant, controleerbaar en bediscussieerbaar” is; bovendien vormt een dergelijk kader
“in feite een codificatie van overeenstemming”, waar in het overleg met de betrokken
partijen naar is gestreefd.
De ruimte voor variëteit en de ruimte voor uiteenlopende partijen om elk hun rol te
spelen maakt een heldere bepaling van de algemeen geldende kwaliteitseisen en een
stevige fundering daarvan van belang. Daarmee ontstaat draagvlak voor de uitoefening
van het toezicht en wordt gerichte communicatie mogelijk over de rationale en de
operationalisering van de gestelde kwaliteitseisen. De legitimering van de kwaliteitseisen
die in toezichtkaders zijn neergelegd, vindt in formele zin plaats door goedkeuring van de
toezichtkaders door de minister. Daarnaast is echter ook een stevige fundering in
wetenschappelijke kennis en maatschappelijke opvattingen van belang; het laatste krijgt
vorm door regelmatig overleg met vertegenwoordigers van het onderwijsveld, het eerste
vult de Inspectie van het Onderwijs in door een beroep te doen op de wetenschap.
Het is niet voor het eerst dat de fundering van toezichtkaders aan de orde is. Zo omvatte
het toezichtkader dat uitgangspunt was voor de eerdere toezichtvormen Regulier en
Integraal Schooltoezicht (RST en IST) een verantwoording gebaseerd op wettelijke
voorschriften, op kennis uit wetenschappelijk onderzoek en op opvattingen van
betrokkenen bij onderwijs. De kenmerken van goed onderwijs die de inspectie voor de
uitoefening van het toezicht hanteerde, werden uit deze invalshoeken afgeleid,
inzichtelijk gemaakt en verantwoord.1
In 2003 werden nieuwe vormen van toezicht ingevoerd, waaronder het Jaarlijks
Onderzoek (JO) en het Periodiek Kwaliteitsonderzoek (PKO). Als verantwoording van
het JO en PKO fungeert tot nu toe vooral de Memorie van Toelichting bij de WOT.
Daarin worden, naast een uitwerking van het in de wet gehanteerde kwaliteitsbegrip, de
kwaliteitskenmerken toegelicht, waarbij zowel wettelijke voorschriften als
wetenschappelijke kennis als uitgangspunt fungeren.
Eind 2003 werd besloten tot bijstelling van de eerder dat jaar ingevoerde toezichtkaders.
De reden daarvoor waren onder meer de praktische hanteerbaarheid en de gelaagde
opbouw van de voor het JO en PKO gekozen opzet, waarin de relatie tussen observaties
en oordelen op onderdelen onevenwichtig bleek. Deze vernieuwing leidde tot aangepaste
toezichtkaders die vanaf 2005 uitgangspunt vormen voor het toezicht op het primair en
voortgezet onderwijs en de expertisecentra, in de vorm van een herzien JO en PKO. De
kern van de herziening bestaat uit een vermindering van het aantal meetpunten en een
beperking tot twee niveaus waarop oordelen worden geformuleerd.
Vraagstelling
Met de ontwikkeling van de herziene toezichtkaders is opnieuw behoefte aan nadere
onderbouwing en verantwoording ontstaan. Zoals aangegeven, zijn aanzetten voor een
dergelijke onderbouwing en verantwoording beschikbaar in de vorm van de
verantwoordingsdocumenten die aan het IST en RST ten grondslag lagen. Op een meer
1 Zie onder meer de nota Toezichtskader primair onderwijs, Inspectie van het Onderwijs (Utrecht, 1999).
algemeen niveau bevat ook de Memorie van Toelichting bij de WOT een uitwerking van
de kwaliteitskenmerken en kwaliteitsaspecten waarop het JO en het PKO zijn gebaseerd.
Vanuit de wens de herziene toezichtkaders maximaal te laten voldoen aan de eerder
genoemde eisen van consistentie, transparantie, controleerbaarheid en
bediscussieerbaarheid is, als toespitsing van de genoemde legitimaties, op twee punten
nadere onderbouwing en verantwoording gewenst. Enerzijds betreft het onderbouwing
vanuit wetenschappelijke kennis die rechtstreeks op de (details van de) herziene kaders is
toegespitst en op het niveau van observaties en indicatoren een gedetailleerde
verantwoording geeft van de elementen van goed onderwijs waarvan naar het oordeel van
de inspectie sprake moet zijn en die tot uitdrukking komen in het oordeel dat zij over een
instelling geeft. Anderzijds dienen de toezichtkaders gerelateerd te worden aan de
centrale functies van onderwijs, te weten de voorbereiding op het vervolgonderwijs en de
arbeidsmarkt, de maatschappelijke vorming en de persoonlijke vorming. Daarmee wordt
gestreefd naar een dubbele verantwoording, ten aanzien van zowel het hoe als het
waarom. De hoe-vraag heeft betrekking op de voorwaarden, activiteiten en processen die
van belang zijn om de gestelde doelen te bereiken; de waarom-vraag betrekt daarbij naast
de relatie met de centrale functies van onderwijs tevens de functie die het
kwaliteitsoordeel moet vervullen.
De Inspectie van het Onderwijs heeft de Universiteit Twente gevraagd bouwstenen voor
een dergelijke nadere verantwoording aan te reiken en een studie te verrichten ter nadere
onderbouwing en verantwoording van de vernieuwde toezichtkaders. Het voorliggende
rapport presenteert de uitkomsten van deze studie en geeft een gedetailleerd overzicht van
relevante kennis. Daarmee beschikt de Inspectie van het Onderwijs over belangrijke
elementen voor de nadere verantwoording van de vernieuwde toezichtkaders.
Ik nodig de lezer van harte uit om als daar aanleiding toe is te reageren en ook kritiek met
ons te delen.
De Inspecteur-generaal van het Onderwijs,
mevrouw mr. drs. C. Kervezee
Table of contents INTRODUCTION 1 PART I THE INSPECTION FRAMEWORK CHAPTER 1 THE INSPECTION FRAMEWORKS AND RESEARCH QUESTIONS ADDRESSED 5 IN THIS STUDY – J. SCHEERENS PART II PROCESS INDICATORS AND EDUCATIONAL EFFECTIVENESS CHAPTER 2 THE INSPECTION FRAMEWORK SEEN AGAINST THE BACKGROUND OF 19 CONCEPTIONS OF EDUCATIONAL QUALITY – J. SCHEERENS CHAPTER 3 THE STATE OF THE ART OF SCHOOL EFFECTIVENESS AND INSTRUCTIONAL 33 EFFECTIVENESS RESEARCH AS A BASIS FOR COMPARISON OF THE
INSPECTORATE’S QUALITY INDICATORS – J. SCHEERENS CHAPTER 4 SCHOOL EFFECTIVENESS RESEARCH SYNTHESIS – B. WITZIERS, 47
G. DOORNEKAMP, R. STEEN CHAPTER 5 THE INDICATORS ON TEACHING AND LEARNING COMPARED TO THE 125 REVIEW OF RECENT RESEARCH ARTICLES ON SCHOOL AND INSTRUCTIONAL EFFECTIVENESS – T. SEIDEL, R. STEEN PART III PROPORTIONAL SUPERVISION CHAPTER 6 CRITERIA FOR ASSESSING SCHOOL SELF-EVALUATION APPROACHES 243
– M. HENDRIKS PART IV THE CHANGES CONTEXT OF SUPERVISION CHAPTER 7 RETHINKING SOCIETAL FUNCTIONS OF EDUCATION, IMPLICATIONS FOR 293 OUTCOME AND PROCESS INDICATORS – J. SCHEERENS CHAPTER 8 PRIORITIES IN THE INSPECTION FRAMEWORK AND THE DISCUSSION 307 ABOUT EDUCATIONAL GOVERNANCE – J. SCHEERENS CHAPTER 9 CONCLUSIONS AND RECOMMENDATIONS – J. SCHEERENS 323 ANNEX 1 QUANTITATIVE META-ANALYSIS OF SCHOOL AND INSTRUCTIONAL
EFFECTIVENESS– B. WITZIERS, R. STEEN, J. SCHEERENS 347
REFERENCES 367 SUMMARY IN DUTCH 395
1
Introduction
The purpose of this study is to verify the scientific basis of the indicators that are part of
the Dutch Inspectorate’s Supervision Frameworks for primary and secondary education
and to relate the Supervision Frameworks to central functions of education, such as
qualification, integration and personality development.
With respect to the first part of this general purpose, a more straightforward linkage to
scientific knowledge is possible than is the case for the second part (relating to societal
functions). Since the rationale of the Inspection Frameworks is clearly inspired by the
means-to-end thinking inherent in educational effectiveness research, where process
indicators are selected for their positive association with learning outcomes, the
knowledge base that has been generated by this body of research is a likely source of
validation. The basic question of this kind of validation is to check whether the set of
process indicators on teaching and learning in the Inspection Frameworks, corresponds
well with the set of variables that has received empirical support in educational
effectiveness research. In order to do so it is of vital importance to make up the balance
as far as the state of the art of educational effectiveness research is concerned. This
means that the international research literature should be captured as fully as possible and
that methods should be employed to summarize and synthesize the research results in a
clear and concise way. In order to do so, techniques for meta-analyses are being
employed.
Concerning the second part of the purpose of this study, namely to relate the Supervision
Frameworks to central functions of education, the issue is positioning rather than
validation. This issue is addressed by considering the question to what extent the
substance and mode of application of the Supervision Frameworks correspond with
recent trends in educational discourse. Clearly in this area universal truths do not exist,
and the matching of contents and strategic application of the Inspection Frameworks to
these often ideological and speculative perspectives is of a looser kind. However, also in
this domain, our frame of reference will be based on empirical research results as much
as possible. For example, the claims of “new public management” and associated
conceptions of systemic reform in education will be compared to empirical results on
effective reform and school improvement approaches in education.
With respect to one of the core quality aspects that is part of the Inspection Frameworks,
namely the inspection of quality care and school self-evaluation, methodological
standards and knowledge based on research on the use of evaluations will be applied.
This part of the analysis is meant to speak on the feasibility of “proportional
2
supervision”, which means that the degree of external school inspection is proportional to
the degree to which schools succeed in appropriate forms of quality care and school self-
evaluation.
The core questions addressed in this study are the following:
Are the process indicators on teaching and learning in the Inspection Frameworks
supported by the knowledge base on school and instructional effectiveness?
How feasible is the idea of proportional supervision, given the possibilities and state of
the art of school self-evaluation in the Netherlands?
Do the Inspection Frameworks manifest defensible choices with respect to outcome and
process indicators, and strategic applications, given current perspectives on educational
governance and modern interpretations of the core societal functions of education?
The approach that is chosen in this study does not pretend to be comprehensive as far as a
scientific positioning and validation of the Inspection Frameworks is concerned. For
example we have not attempted to provide a legal analysis of the role of the Inspectorate,
nor have we addressed methodological and measurement aspects of the normative
application of the indicators.
PART I
THE INSPECTION FRAMEWORK
5
Chapter 1
The Inspection frameworks and research questions addressed
in this study
Introduction
In this chapter the Inspection Frameworks for primary and secondary education are
described, some aspects of the legal context are mentioned and the research questions
of the study are presented.
Description of the Dutch School Inspection Frameworks
Schools in the Netherlands are quite autonomous. In comparison to other countries a
very high amount of decisions about instruction, human resources management,
financial management and curriculum are made by schools, c.q. their competent
authorities (OECD, 2003). School autonomy includes school self evaluation and quality
care. Within the given legal frameworks schools independently determine quality
targets and norms as well as the way in which these are to be measured and assessed.
On the basis of their own judgements, the schools determine to what extent quality
improvements are required, as well as the contents of such improvements. Supervision
(Dutch: toezicht) by the Education Inspectorate is to function as an addition to the self
regulatory mechanisms by which school are expected to carry out their self evaluation
and quality management.
The central question of school supervision is: “What is the quality of education in a
particular school like?” (Inspectorate, 2002, p. 9) This core question is differentiated
according to three sub questions:
1) What is the school’s quality care like?
2) What is the school’s quality in teaching and learning?
3) What is the quality of the learning results? (ibid, p.9)
These questions indicate three domains of the supervision framework.
The four domains are divided in the following quality aspects; as indicated in the table
below, based on the Inspection Framework for Primary Schools.
6
Quality domain Quality aspects per domain
Quality care 1. Systematic quality care by the school
2. Testing
Teaching and Learning 3. Subject matter coverage
4. Time
5. Stimulating and supportive teaching
and learning process
6. Safe, supportive and stimulating
school climate
7. Special care for children with learning
difficulties
Outcomes 8. Learning results in basic subjects
For each of the quality aspects a number of indicators have been specified, for example:
- The school knows its own starting situation
- The school systematically evaluates the quality of learning outcomes and
teaching and learning processes
- The school guarantees the quality of testing at the end of the primary school
period
- Subject matter coverage is such that it prepares the pupils for secondary
education
- Subject matter coverage is integrated
- Learning time is sufficient for the students to have them master the subject
matter
- The school programs sufficient teaching time
- The teaching activities are well structured and effective
- The teachers take care of their teaching being adaptive to the learning needs of
the students
- School staff and pupils interact in a positive way
- The school stimulates the involvement of parents
- The school guarantees safety
- The school provides a pleasant and stimulating environment for the students
- The school provides a pleasant and stimulating working environment for its staff
In 2005 an updating and overhaul of the Frameworks took place. The complete list of
indicators of the Framework for secondary schools is presented below.
7
Quality Aspect 1:
The school takes care of the assurance and improvement of the quality of education.
1.1 The school knows its entrance situation, including the specific needs of the student
body.
1.2 The school systematically evaluates the quality of its performance in terms of
learning results.
1.3 The school systematically evaluates the quality of learning, teaching and
counselling.
1.4 The school has formulated measurable improvement targets.
1.5 The school carries out improvement activities in a systematic way.
1.6 The school guarantees the quality of learning and teaching.
1.7 The school guarantees the quality of the school examination and of other evaluation
instruments.
1.8 The school reports about the realized quality of education to interested parties
(parents, students, competent authorities, funding agencies and sponsors).
Quality Aspect 2:
The conditions for quality care are in place
2.1 School management initiates and steers the quality care.
2.2 Quality care is connected to the school’s vision with respect to learning and
teaching as stated in the school plan.
2.3 The school management takes care of a professional school culture.
2.4 The school takes care of an effective communication about the quality of education.
2.5 Staff, school management, pupils, parents and competent authorities are all of them
being involved in the school’s quality care.
Quality Aspect 3:
The subject matter offered is aimed at a broad development of the pupils and on
preparing them for further education and the labour market
3.1 The school has provided the foundations of subject matter offering in the lower
grades.
3.2 The school guarantees that the actual subject matter offering covers the examination
program.
3.3 The program (total of subject matter that is offered) is being connected to important
societal and actual themes.
8
3.4 Subject matter that is offered in one grade connects to the previous and subsequent
grades.
3.5 There is coherence between the subject matter offerings in the various subjects.
3.6 Subject matter offerings are adapted to the educational needs of individual pupils.
3.7 The school offers knowledge about the different cultures that are present in the
Netherlands, on a regular basis, with reference to the corresponding norms and
values.
3.8 Schools with over 20% of students that are weak in language, adapts Dutch
language tuition to the needs of these students in all subjects.
3.9 The school offers content on civic education, social cohesion and norms and values.
Quality aspect 4:
The students get sufficient time to master the subject matter
4.1 The intended teaching time corresponds to the legal norms.
4.2 The structural (i.e. planned) amount of lessons “not given” is minimal.
4.3 The incidental amount of lessons “not given” is limited.
4.4 Non permitted absence of students is limited.
4.5 The teachers use the intended teaching time in an efficient way.
4.6 The school varies the amount of time for teaching and learning relative to the
educational needs of the students.
Quality aspect 5:
The pedagogical approach of the teachers induces a safe and stimulating learning
environment
5.1 The teachers stimulate the pupils’ self confidence.
5.2 The teachers treat the pupils in a respectful way.
5.3 The teachers stimulate that pupils treat one another with respect.
5.4 The teachers realize a productive atmosphere.
Quality aspect 6:
The didactic approach of the teachers supports pupils’ learning
6.1 The teachers provide insight in goal, use and connectivity of the lesson activities.
6.2 The teachers provide clear explanations.
6.3 The teachers check whether the pupils have understood explanations and
assignments.
6.4 The teachers create a meaningful context for the learning process.
9
6.5 The teachers stimulate the students to think (e.g. by posing challenging questions).
6.6 The teachers provide substantive feedback to the pupils.
6.7 The teachers provide insight into the pupils’ learning processes (simulating
reflections on learning strategies).
6.8 The teachers take care of pupils being involved in the educational activities.
6.9 The didactic approach is functional with respect to the pupils’ learning process.
6.10The teachers adapt their didactic approach to the differences between the pupils.
6.11The teachers use the analyses of pupils’ achievement for the way they shape their
instruction.
6.12The teacher’s use of language is adapted to the language needs of the pupils.
Quality aspect 7:
The students play an active and independent role during instructional activities
7.1 Students are confronted with stimulating activities and assignments.
7.2 The students reflect on their own learning processes.
7.3 The pupils are given responsibility for their own learning processes to a sufficient
degree.
7.4 The pupils learn to work together in an effective way.
Quality aspect 8
The school has a safe and stimulating climate
8.1 The pupils manifest involvement in school life.
8.2 The staff manifests involvement in school life.
8.3 The parents feel involved in school life.
8.4 Staff and students interact in a respectful way inside and outside classes.
8.5 Staff and students feel safe at school.
8.6 The pupils, staff and parents experience the school leadership as supportive and
stimulating with respect to the atmosphere at school.
Quality aspect 9
Guidance and counselling is aimed at a full development of the students’ capacities
9.1 Guidance and counselling is aimed at a good development and at the pupils’ well-
being.
9.2 The school uses a consistent system of instruments and procedures for monitoring
progress and development of the pupils.
10
9.3 The school uses information from primary school in the guidance and counselling
of the pupils.
9.4 The school supports pupils and parents with respect to career choices during the
school programme.
9.5 The school supports the pupils and parents with respect to choices regarding
further education and the labour market.
9.6 Special staff functions (like deans and mentors) see to it that pupils receive
continuity in guidance throughout the career at school.
Quality aspect 10:
Students with special needs receive the care that they require
10.1 The school has early diagnosis of pupils who need extra care.
10.2 The school analyzes the kind of care that selected pupils require.
10.3 Special care is carried out in a systematic, planned way.
10.4 The school assesses the effects of special care.
10.5 Teachers and special staff for care co-operate well.
10.6 The school involves the parents of pupils requiring extra care in the special
activities.
Quality aspect 11
The pupils attain the achievement results relative to their capacities
11.1 The pupils attain the achievement level that is to be expected given
national averages.
11.2-11.5 The pupils stay close to the minimum amount of time to finish the program
for each of the program variants in Dutch secondary education.
11.6 Pupils who take part in practical education develop according to an
individual learning route.
11.7-11.10 The pupils obtain the marks that may be expected of them on the final
examinations, relative to national averages, for each of the program
variants of Dutch secondary education.
11.11 The pupils perform according to expectation in further education or the
labour market.
11.12 Same as 11.11, specialised for students in practical education.
11.13 Student competencies are at a level that may be expected.
Indicators are applied by establishing their importance for the corresponding quality
aspects and by being rated on four point scales (good, sufficient, insufficient and bad).
11
These ratings are quantified and compared to pre-fixed norms, that ultimately lead to
judgements of the quality in each quality aspect and domain. For more detailed
information see Janssens, 2005.
The legal context
In this report current ideas about the quality of education and the societal function of
education are discussed for their possible implications for criteria and standards of
school inspection. The degree to which the inspectorate is free to adapt and implement
certain of these possible implications depends on legal frameworks, however. In this
chapter the degrees of freedom the inspectorate has to adapt its own framework of
criteria and standards are briefly touched upon.
Two kinds of legal frameworks are considered: frameworks that consider the official
goals and targets of education, as well as the formal assessment of these on the one hand
and frameworks that regulate the ways inspection should take place on the other.
The first kind of frameworks are relevant to the question of WHAT is to be inspected,
whereas the second kind of frameworks refer to the question HOW inspection is to take
place.
More concrete questions with respect to the WHAT of inspection that are to be
addressed in this chapter are:
- the degree to which the end terms of primary and secondary education determine
the outcomes that the inspectorate should assess;
- the degree to which the examination programs in secondary education determine
the outcomes that the inspectorate should assess;
- the degree to which process indicators like, for example, prescribed teaching
time per subject and teaching methods could be seen to have formal legitimacy,
given the principle of freedom of education.
More concrete questions with respect to the HOW question of school inspection are:
- the degree to which checking “appropriateness” and quality are formally
prescribed and can be clearly distinguished:
- the degree to which the WOT is clear in defining and separating assessment/
evaluation roles versus supportive roles of the inspectors;
- the degrees of freedom the WOT specifies for the inspectorate to define its own
inspection frameworks.
Requirements to WHAT should be inspected
Two legal frameworks set the agenda for inspection: the educational laws (such as the
act on primary education and the act on secondary education) and the Supervision Act.
12
The educational laws contain legal requirements schools should comply to. These are
for primary education:
- The school uses the attainment targets as goals that have to be achieved at the end
of primary school. These targets contain content specific goals aimed at Dutch,
Frisian (Fries) (for schools in that area), English, mathematics, orientation on
human and environment, gymnastics and cultural orientation. The general
attainment targets are related to work attitude, work according to a plan, use of
several learning strategies, self image, social behaviour and new media.
- Education should enable pupils to go through an uninterrupted process of
development that is adjusted to the progress pupils make.
- Pupils that need extra care receive individual guidance that is adapted to their
needs.
- Schools use the multicultural society as a frame of reference.
- Schools offer at least 3520 hours of education in the first four school years and at
least 4000 hours in the last four school years. Pupils should, in principle, be able to
finish primary education in eight years. They receive at most 5,5 hours of education
a day.
- The school provides for a registration system of progress of pupils that need extra
care.
- Schools should describe their policy in quality care and improvement of education
in a school plan.
Secondary education has several types of schools. Each of those has to meet other types
of legal requirements, but the following provide a common core:
- Courses and attainment targets in basic education (basisvorming)
- Length of basic education (basisvorming)
- Minimum number of hours in the first years of secondary education
- Education in a multicultural society
- School plan
- School guide/prospectus
- Reporting progress of pupils
- Final exams
The Supervision Act describes the quality aspects the Inspectorate is obliged to use in
school inspections. These are:
- Learning results
- Progress of development of pupils
- Supply of subject matter
- Pedagogical climate
13
- School climate
- Teaching
- Pupil care
- Content, level and execution of tests, tasks and exams
The Supervision Act also specifies that the framework of inspection that is developed
by the Inspectorate to asses both legal requirements as quality aspects, should be
developed in close cooperation with umbrella organisations2 in the educational field.
The framework can not be fixed if this cooperation/consultation has not taken place.
Requirements to HOW inspections take place
Types of inspections
The Supervision Act regulates three types of inspections. The first and most frequent
one is a periodic quality inspection (pko). The Supervision Act states that the
Inspectorate should yearly inspect the education of every school, except of special
circumstances. A periodic quality inspection leads to an assessment of the quality of
education, which exists of the legal requirements and the quality aspects as described
above.
Types of school inspections also include:
- Additional inspection (no): when a periodic quality inspection gives reasonable
doubt of shortages in educational quality, the Inspectorate enacts an additional
inspection in which the causes of underperformance are also investigated.
- Inspection into quality improvement (okv) the school has realised. This type of
inspection follows only after additional inspections have shown shortages in
educational quality.
- Incidental inspection: inspection into educational quality or compliance of legal
requirements that is, apart from the other three types, initiated by the Inspectorate or
the Minister.
Proportional inspection
A requirement for both periodic quality inspections, as additional inspections and
inspections into quality improvement, is the use of results of quality assurance and self-
evaluation of schools by the inspectorate. The inspectorate has to use these results to
form judgements about school quality, under the condition that school self-evaluations
2 Umbrella organizations unite interest groups and pressure groups in education, according to main actor categories, such as school boards, parents and teachers. The interest groups are organized according to denomination.
14
provide reliable information about the indicators included in the framework and if the
school sets high enough standards of educational quality.
Publication of findings
The findings of school inspections should be made public.
Underperformance
The inspectorate should distinctively inform schools about failing to comply with legal
requirements or failing to meet the quality standards. If schools fail in one or both of
them, the Ministry is the only institution formally responsible and competent for taking
actions. The inspectorate is only allowed to advise about possible actions. Formal steps
include:
- Financial sanctions or determining specific conditions schools have to meet to
receive governmental financing. These sanctions can only be taken in case of non-
compliance of legal requirements.
- Financial or personal support of school boards in case schools fail to meet the
quality aspects.
A problem the Educational Council pointed out, is the extent to which legal
requirements and quality aspects overlap. Quality aspects, as worked out in previous
versions of the framework for inspection, refine the legal requirements as they are
otherwise hard to measure. Separating the two types of actions testing norms of
“appropriateness” and of quality is therefore difficult.
The inspectorate has no formal role in improving underperformance of schools. The
Memorandum of Explanation, guiding the Supervision Act, specifies however, how a
number of instruments are intended to more or less prevent non-compliance or
underperformance. Proportional inspection is expected to stimulate schools to develop
quality assurance measures and to evaluate their performance, which could lead to
improvement. The intent of publishing the findings of school inspections is to inform
the environment of the school about school quality. The expectation is that parents will
use this information to address the school of their children about possible
improvements. The assessment of schools should also motivate them to improve as it is
considered to be an insight into possible improvements.
Research questions addressed in this study
The general purpose of this study is to verify the scientific basis of the Supervision
Framework.
15
This general orientation is narrowed down to two basic aspects. The first raises the
question whether the inspection framework is in line with current thinking about the
basic societal functions of education and corresponding implications for the goals and
means or methods of schooling.
The second basic aspect addresses the quality concept that is at the basis of the
framework of quality aspects and indicators and the empirical support for the “process
indicators” on teaching and learning that are an important part of the framework. An
interesting related question is the question whether the kind of “additional quality
control” that the inspectorate carries out (where additional refers to the school’s own
basic responsibility for quality care) is in line with the evolving institutionalization of
educational governance in the Netherlands. An illustrative question in this context is the
one about output versus process monitoring, where it might be argued that the external
monitoring of outcomes should be seen as sufficient in a radically decentralized
educational system, and the monitoring of process indicators on learning and
instructions would be outside the scope of the Inspectorate’ s concern.
Studying the scientific foundations of the Inspection Frameworks means, first of all,
confronting the existing approaches with available ideas. Of necessity, giving the
normative nature of education, these kind of ideas have often a kind of speculative,
ideological or even utopian character. Ideas about new end terms and objectives in
education, centred around the competency concept, ideas about more expansive versus
more targeted concepts of educational quality and ideas about new kinds of school
organization including an expansion of the scope in roles of teachers and school heads.
When comparing these ideas to the explicit and implicit assumptions of the inspection
framework no strong conclusions will be drawn about the appropriateness of the current
Inspection Frameworks. Educational science, or rather education discourse about
alternative concepts can merely have the function of providing alternative perspectives
that, in most cases are far from established and uncontested.
In the part of the study that addresses the quality aspects regarding school self
evaluation and school quality care, as well as the quality aspect learning and instruction
a more empirical approach is feasible. In the part on school quality care evaluation
methodological aspects may be addressed and empirical studies on the actual
functioning of school self evaluations can be used. Next, when it comes to process
indicators on schooling and teaching, the results of empirical school and effectiveness
research can be used, to verify the question to which these processes are positively
associated with learning outcomes in basic school subjects. This part is the main body
of this study, where it is based on extensive review of the research literature.
16
Sources:
- Wet op het onderwijstoezicht
- Brochure ministerie van OCW over kerndoelen
- Advies van de Onderwijsraad: Deugdelijk toezicht
- http://wetten.overheid.nl/
PART II
PROCESS INDICATORS
AND EDUCATIONAL EFFECTIVENESS
19
Chapter 2
The Inspection Framework seen against the background of
conceptions of educational quality
Introduction
The concepts of quality and quality assessment are very central in the Inspection
Framework. Given the fact that a lot of conceptual confusion is hidden by these broad
umbrella terms, it is considered necessary to provide a conceptual framework that is
able to clarify different emphases and position the current Framework.
A basic conceptual framework
Educational quality can be clarified on the basis of a conceptual framework that
describes education.
The most frequently used way to do this is to depict education as a productive system,
in which inputs are transferred into outcomes. The central “black box” can be defined at
various levels, as far as education is concerned this could be the national education
system, the school or the classroom.
I am well aware of the fact that the choice of this framework is already a narrowing
down of the scope of the quality issue, although, as will be shown, it is still global and
allows for a range of different priorities (this will be further illustrated in subsequent
sections). An alternative approach would have been one that is more evolutionary, more
concentrated on endogenous growth and with a stronger emphasis on organisations as
self-referential systems (Luhmann, 1995).
Steps in elaborating the model of education as a production system are:
a) including a context dimension, that functions as a source of inputs and constraints
but also as a generator of the required outputs that should be produced;
b) differentiating outcomes in direct outputs, longer term outcomes and ultimate
societal impact;
c) recognising the hierarchical nature of conditions and processes, putting public
education down as an example of “multilevel governance”.
The model depicted in Figure 2.1 shows this framework.
20
Figure 2.1: A basic systems model on the functioning of education
When examining this basic framework, at least six ways in defining education quality
can be chosen, by emphasising certain parts, aspects or relationships from the basic
framework.
a) The productivity view
According to this view, the success of the systems is seen as depending on the
attainment of the aspired outputs/outcomes. For example in the sense of a satisfactory
quantity of school-leavers that have attained a specific level, or in terms of an
acceptable level of employment of students with a certain diploma. According to this
view output/outcome/impact indicators are predominant or even the only type of quality
indicators that need to be monitored. It should be noted that in this rather formal and
abstract presentation there is not yet any decision on what kind of outputs and outcomes
are prioritised and ultimately measured. In fact a broad range of skills and personal
attributes might be considered. These may range from basics like literacy and numeracy,
to “intra-personal” skills like motivation and perseverance and learning to learn, “inter-
personal skills” like capacity for teamwork and leadership, and other skills like
problem-solving capacity and computer-literacy might be considered (cf. OECD, 2000,
p. 19).
b) The instrumental effectiveness view
According to the instrumental effectiveness view there is a clear perspective for the
selection of context, input and process indicators, namely their expected effect on
outcomes. To the extent that effectiveness or production functions can be completely
specified, in other words outcomes can be totally predicted, context, input and process
context
outputs inputs Process or throughput
System level School level Classroom level
21
indicators could replace outcome indicators. The value of certain levels and forms of
inputs and processes is determined by there instrumental potential. Clearly the
instrumental perspective offers a more dynamic handle for policy, as it considers not
only given constraints but also factors that are policy malleable.
c) The adaptation perspective
This view “transcends” the instrumental effectiveness perspective by not only looking at
the question how to do things right, but first of all considering the question on how to
do the right things. In other words the adaptation perspective would lead to a critical
analysis of educational goals. Conditions that allow for a continuous sounding of
changing contextual conditions for the education province would receive emphasis as
means, while labour market outcomes or “social capital” could be considered as ends,
according to this view. The adaptation perspective would also cover defining a part of
the school curriculum as meant to adapt to the specific environmental context and local
and national culture. As such, this perspective could be seen as an important
complimentary perspective to the instrumental effectiveness view, which takes more or
less universal standards in basic subjects for granted.
d) The equity perspective
When inputs, processes and outcomes are analysed for their equal or “fair” distribution
among participants in education with different characteristics, equity is the primary
facet of judging educational quality.
e) The efficiency perspective
This perspective can be seen as a further demand on the productivity and instrumental
effectiveness view, by considering the highest possible outcomes at the lowest possible
costs.
f) The disjointed view
Combinations or relations between the various elements of Figure 2.1 were central in
the previous views that represent a particular perspective on education quality. An
alternative view is to consider each element “on its own” and judge whether it is
manifested in an acceptable way, or at an acceptable level. In this way one could, for
example, consider levels of teacher training, as a (minimum) requirement for being
allowed to function as a teacher, class sizes could be judged in terms of being
22
acceptable for being “manageable” units for teachers and students, and teaching
strategies could be rated according to norms of good practice.
The disjointed view is descriptively the simplest one, although in an evaluative sense it
is perhaps the most arbitrary one.
When considering the way these perspectives on educational quality are being reflected
in the composition and use of indicator systems it appears that the last perspective, the
“disjointed view” is predominant (for further discussion see the section on measuring
educational quality).
What is school quality?
In this section the level of the school as an organisation will be used to specify educational
quality in more operational terms. As will be shown in the section on measuring quality,
even if the monitoring system is primarily oriented at the macro or system level, it is
important to “drop down” to the school level and include data on aggregation levels that
are lower than the system level.
Two conceptual frameworks will be used to elucidate choices with respect to quality: (i)
the basic model from systems theory introduced in the above and “filled out” by using
results from empirical school effectiveness research, and (ii) perspectives on organisational
effectiveness.
The model of empirical educational effectiveness research
Below is a somewhat different presentation of the systems model that was introduced in
the previous section. Here the central box is defined at the level of an organization, in
our case, a school. The functioning of the organization is again seen in terms of inputs
flowing into the central box into and by outputs being “somehow” produced (see Figure
2.2).
input organisation as a black box output
Figure 2.2: The organisation as a black box.
In Figure 2.2 it is assumed that within the black box processes take place that transform
inputs into outputs. When it is attempted to further describe these processes in terms of
which process characteristics are most effective in obtaining desired levels of outputs, the
model of Figure 2.1 becomes more elaborate. This model is often used as a conceptual
framework to summarise the results of school effectiveness research.
23
In Figure 2.3 an example of such an ordered summary is shown (cf. Scheerens, 1989).
Figure 2.3: A summary of the findings from school effectiveness research, from Scheerens,
1989.
The notion of quality inherent in integrated school effectiveness models like the one in
Figure 2.3 is that:
a) outputs are the basic criteria to judge educational quality;
b) in order to be able to properly evaluate output, achievement or attainment measures
should be adjusted for prior achievement and other pupil intake characteristics; in this
way the value added by schooling can be assessed;
Context achievement stimulants from higher administrative levels development of educational consumerism ‘co-variables’, such as school size, student-body
composition, school category, urban/rural
Process
school level degree of achievement-oriented
policy educational leadership consensus, cooperative planning
of teachers quality of school curricula in
terms of content-covered and formal structure
orderly atmosphere evaluative potential classroom level time on task (including
homework) structured teaching opportunity to learn high expectations of pupils’
progress degree of evaluation and
monitoring of pupils’ progress reinforcement
Inputs
teacher experience
per pupil expenditure
parent support
Outputs
student achievement, adjusted for:
previous achievement
intelligence SES
24
c) in selecting variables and indicators to assess processes and context one should look
for those factors that have been shown to be correlated with relatively high output,
adjusted in terms of “added-value” as described above;
d) the model is a multi-level model, uniting effectiveness enhancing conditions at system,
school, classroom and individual student level.
It should be noted that educational effectiveness models do not prescribe the types of
outputs that should be used to assess quality. In principle all types of outputs, cognitive or
non-cognitive could be inserted in the right-hand box of Figure 2.3. In the actual practice
of school effectiveness research, however, cognitive outcomes, mostly in terms of
achievement in core-subjects like reading, arithmetic, and language, have predominated.
The process factors shown in the middle section of Figure 2.3, might well be somewhat
different if non-cognitive outcomes or less subject-matter tied cognitive outcomes would
have been used in the actual research studies.
It should also be noted that there is still quite a lot of uncertainty about the selection of
process factors such as indicated in the figure. The available knowledge-base is far
removed from a situation where it would be possible to make precise predictions on the
likely added value of schooling, given the state of certain processes and inputs (in a
subsequent section of this paper this will be documented further).
To the degree that educational effectiveness models provide an acceptable operational
definition of quality, they can also be used as a guideline in the design of instruments for
school evaluation and school self-evaluation. The previously mentioned points: a (focus on
outcomes), b (proper adjustment of outcomes) and c (measure process characteristics
associated with high added value) mentioned in the above can be read as guidelines to
make choices with respect to instrumentation.
However, a broader perspective on quality can be considered. Such a broader perspective
can be obtained from multiple orientations towards organisational effectiveness that will be
discussed in a subsequent section. First, some more clarification will be given about the
way equity and effectiveness perspectives can be defined and, to some extent, also
combined.
Equity and school effectiveness3
Historically the “school effectiveness movement” was strongly oriented towards
students in disadvantaged neighbourhoods, inner city schools in the US and the UK. 3 In this section gender related issues of educational inequality receive no specific emphasis, as the position of girls is primarily seen as an issue of participation in basic education. Yet, some conditions of schooling (e.g. the absence of separate toilets for boys and girls) might put girls in a disadvantaged position. Potential biases of this kind could be seen as a relevant issue for further study.
25
Moreover, evaluations of major compensatory programs in the US, such as Head Start
and Follow Through, gave a strong impetus to research in which alternative
instructional approaches were compared for their effectiveness for disadvantaged
learners. Results were strikingly similar to the results of school effectiveness studies at
the time, cf. Ralph and Fennessey (1983), Scheerens (1987). To the extent that these
early studies have a strong impact on the knowledge base on educational effectiveness
as of today, one might say that there is a certain bias on “what appears to be working”
for students at the low end of the achievement distribution. When considering the
analytic model of empirical school effectiveness research, there are several ways in
which equity related issues can be brought into the picture.
First of all equity of achievement outcomes can be studied by considering measures of
between student and between school variation. Since the primary interest of educational
effectiveness studies is to assess the impact of malleable factors over and above student
background characteristics achievement results at the individual student level are
“adjusted” for the impact of these variables. At the same time there appears to be an
additional impact of the average of relevant student background characteristics, such as
socio-economic status on achievement. To the degree that these student level
background conditions at individual or school level have a larger impact on
achievement school systems can be judged as comparatively selective versus equitable
(the recent debates about the PISA results for a country like Germany illustrate this
issue).4
Secondly, school effectiveness research, or surveys on effectiveness enhancing school
conditions, can provide information about the extent to which the levels of inputs and
process are the same (or different) in all schools and all provinces within countries. In
other words, how equitably are the resources or processes allocated or distributed across
schools and provinces within a country? When the teaching force in each school is
described, for example, can it be said that all pupils – in whichever school they are –
have the same quality of teachers? Or the same provision of resources, and so on? It is
important to view the levels of input and process provision and equity at the same time.
If the achievement levels are all low (and much lower than they should be) but there is
very little variation among schools, then we know that the schools all have the same
very low level. The patterns of variation in school provisions are relevant information
for educational policy. If the levels of school resources are very different among schools
within provinces then it could be seen as the job of the provincial authority to do
4 The German Pisa results are characterised by relatively low average achievement levels; a large variation between students and between schools, and a relatively strong impact of SES related background characteristics on achievement. The categorical organisation of the German school system is seen as one of the causes of this state of affairs.
26
something about this. If there are large differences among provinces but few differences
among schools within provinces then the national authority must do something to
ensure more equity among provinces.
Assuming that there is some measure of the socio-economic status (SES) composition
of the student body within schools, then it would also be possible to examine if it is the
poorer SES schools that have fewer provisions and the higher SES the higher levels of
provision. Similar kinds of questions could be raised with respect to other grouping
variables of schools, like: urban/rural and private/public.
A third way of thinking of equity is based on the philosophy of compensation and
“positive discrimination”. Related, and more neutral sounding, terms in education are:
adaptive teaching and differentiation within classrooms. The basic idea is that
something extra is done for students that are in some way or another “disadvantaged”.
Examples are smaller classes, extra tutoring, adaptive teaching approaches, increased
learning time. School surveys can capture such measures or programs in a descriptive
way. The distribution of “extras” for disadvantaged learners across schools within
countries could also be seen as a specific example of equity in the sense of the first
meaning, stated above (equitable distribution of inputs).
A sophisticated combination of “school effectiveness” and “equity in schooling” is
addressed in studying what is known as “differential effectiveness”. This branch of
school effectiveness research specifically addresses the question which kind of school –
and instructional conditions work best for disadvantaged as compared to more
“advantaged” students.
Multiple criteria to assess organizational effectiveness
In organization theory the question about the “goodness” or quality of an organization is
usually framed by referring to the effectiveness of the organization. The term effectiveness
literary means “goal attainment”. So the implicit assumption is that organizations are
oriented to certain goals or objectives. This assumption is generally accepted, at least as a
useful working hypothesis. It is also less restricted than it may seem at first sight. In fact
quite different types of goals or types of “effectiveness criteria” are considered. An
effectiveness criterion could be defined as any kind of dimension that expresses a desirable
characteristic of central importance in evaluating the functioning of an organization.
Examples could be the examination results of the students, but also indications that the
parents of the pupils are satisfied with the school.
Organizational theorists often adhere to the thesis that the effectiveness of organizations
cannot be described in a straightforward manner. Instead, a pluralistic attitude is taken with
respect to the interpretation of the concept in question. By that it is assumed that it depends
27
on the organization theory and the specific interests of the group posing the question of
effectiveness, which interpretation will be chosen (Cameron & Whetten, 1983, 1985;
Faerman & Quinn, 1985). The main perceptions on organizations, which are used as
background for a wide range of definitions of effectiveness, are summarized in the table
below. For a fuller description of the different organization theories see Scheerens, Glas
and Thomas (2003).
Table 2.1: Organizational effectiveness models
Theoretical background
Effectiveness criterion
Level at which the effectiveness question is asked
Main areas of attention
(Business) economic rationality
Productivity Organization Output and its determinants
Organic system theory
Adaptability Organization Acquiring essential inputs
Human relations approach
Involvement Individual members of the organization
Motivation
Bureaucratic theory; social psychological homeostatic theories
Continuity Organization + individual
Formal structure
Political theory on how organizations work
Responsiveness to external stakeholders
Subgroups and individuals
Independence, power
The diversity of views on effectiveness, which organizational theory makes, leads to the
question which position should be taken. Should we indeed operate from a position of
there being several forms of effectiveness, should a certain choice be made, or is it possible
to develop from several views, one all-embracing concept on effectiveness?
A possible position with regards to this question is the one where productivity, in terms
of quantity and quality of school output, is seen as the ultimate criterion. According to
this view the other criteria are seen either as pre-conditions (adaptivity and
responsiveness) or “means” (criteria referring to organizational conditions such at
teacher satisfaction) (Scheerens & Bosker, 1997). It should also be noted that the
perspectives that stress adaptability and responsiveness of the organization to the
environment do not take particular goals or desirable process characteristics for granted
but pose the very question about which goals the organization should strive for.
A final observation that should be made is that views on organizational effectiveness
primarily breath a prescriptive perspective: these are the things that are important (the
28
effectiveness criteria) and that should be aimed for. At the same time the basic
organization theories that are referred to, also have a descriptive interpretation: this is
how things are. In these more descriptive interpretations we find elements that
emphasize a less idealistic, a more down-to-earth and probably a more realistic view on
the functioning of organizations. For example the political model does away with the
assumption that all the members of the organizations will simply support the
organization’s goals. Instead, each individual member of the organization may pursue
his or her own goals, which might coincide with the official goal, but then again, it
might not. The more recent system dynamics developments that emphasize self-
reference and self-organization put question marks behind the realism in the view as if
the school simply has to adapt to external constraints and policy changes. One could say
that the organizational effectiveness perspective, in all its differing forms, emphasizes a
rationalistic, action-oriented view in which the malleability of organizational
functioning comes first. The more descriptive interpretations point at systemic inertia,
the importance of routine, and the consequences of internal interactions.
Quinn and Rohrbaugh (1983) have united the four most prominent organization
theoretical perspectives in one model, the so called competing values framework. This
framework depends on two dimensions: the internal versus external orientation of the
organization, and the flexibility of the organization versus a preoccupation with control.
Figure 2.4: Typology of effectiveness models. Source: Quinn & Rohrbaugh (1983).
output quality
HUMAN RELATIONS MODEL means: cohesion, morale ends: human resource
development
OPEN SYSTEM MODEL means: flexibility, readiness ends: growth, resource acquisition
means: information management, communication ends: stability, control INTERNAL PROCESS
means: planning, goal setting ends: productivity, efficiency RATIONAL GOAL
flexibility
control
internal external
29
From this framework additional process indicators of school functioning may be generated
(see the subsequent section on the monitoring of quality). The ideas for additional process
indicators that come from this more comprehensive treatment of organisational
effectiveness are summarised in Figure 2.5 (Process indicators induced from the narrower
model of school effectiveness research are also included.)
Human relations model Open system model
Quality of work life indicators - entrepreneurship
- respect - collegiality
- participation in decision-making - capacity for self-evaluation and learning
- professional interaction - overt school marketing activities
- performance feedback - parental involvement
- opportunity to use skills - boundary-spanning positions
- resources - external change agents
- congruence personal/organisational goals - student enrolment figures
- resources (buildings, equipment)
Internal process model Rational goal model
- planning documents (school effectiveness research)
- disciplinary rules - educational leadership
- management information systems - success-oriented ethos
- formalisation of positions - monitoring of student's progress
- continuity in staffing and leadership - time on task
- integrated curricula - content-covered (opportunity to learn)
- attendance rates
- lessons "not given" (broader set of educational goals)
- non-gradedness
- team teaching
- individualisation, differentiation
- continuous learning route
- time spent on social, emotional, creative
and moral development
- "learning to learn" activities
- diagnostic testing
Figure 2.5 : Additional factors for process indicators generated form the Quinn and
Rohrbaugh framework
30
Conclusion: the position of the Inspection Framework
As a school evaluation framework the Supervision Framework of the Inspectorate can be
classified as being associated with the effectiveness perspective on educational quality.
The process indicators of the Inspection Framework can be seen as effectiveness
enhancing conditions that are inspired by empirical school effectiveness research and by an
international consensus on good teaching practice. From the latter perspective Vernooy,
2002, cited by Janssens, 2005, lists the following key variables:
The schools realise the goals set in advance;
The teachers have high expectations of the pupils' abilities;
A positive teaching-learning climate prevails;
The school is focused on getting all pupils to acquire the basic skills;
The school head and team feel responsible for the results and pupil wellbeing;
The school reserves sufficient time for language/reading;
The school uses good, co-ordinated teaching materials;
Professional development is a continual process;
There is a strong focus on prevention and the pupil's results are closely monitored;
Attention is given to the effectiveness of the language/reading lessons;
The school follows a strong team-oriented approach;
The school head is good at:
Expressing a clear vision;
Placing a strong focus on the results of pupils;
Monitoring improvements;
Being closely involved in what happens in the classroom.
When examining the actual quality indicators from the Inspection Framework the
following observations can be made:
- the reference point, in the sense of the effect criteria, or outcome indicators, are test
scores in basic school subjects;
- the majority of process indicators is defined at classroom level; there seems to be a
strong emphasis of the primary process of teaching and learning;
- indicators that represent school self-evaluation and quality care have a double
significance in the framework; they form a set of effectiveness enhancing factors in
their own right, but at the same time they are evaluated from a meta-perspective, with
the possible result that school quality care procedures actually replace direct inspection
by the Inspectorate (this latter idea is the so called proportional supervision;
31
- some of the classroom level indicators might be given a school level interpretation, as
in the case of programmed time, a supportive climate and content coverage (but the
quantitative use of the framework seems to be limited to judgements at classroom
level);
- organizational and managerial conditions that feature as more remote effectiveness
enhancing factors in multi-level school effectiveness models are treated as “possible
background and causes” of quality conditions at the level of the primary process;
examples are: school management, aspects of the school organization, school policies,
professionalization of staff and contacts with parents (Inspectorate, 2002, p. 9);
- there is little emphasis on resource input factors (like facilities, teaching equipment,
computers and the school budget, class size and pupil teacher ratio);
- issues of special care for pupils with learning difficulties and students that are
disadvantaged because of low socio economic status, cultural capital or of minority
background are brought together under the criteria that, broadly speaking, refer to
adaptive teaching;
- school composition, an important issue in recent school effectiveness studies, is not
treated;
- the broader view of organizational effectiveness, as represented in the Quinn and
Rohrbaugh framework, is not represented in the Inspection Framework. There is no
attention for externally oriented school policy, like providing information to parents
and local constituencies, connections with the local business world, nor is there much
attention for aspects of human resource management at school, and for a close
monitoring of formal procedures (this last aspect being strongly represented in “ISO
type” quality management systems;
- the softer areas of school culture and climate are represented in the inspection
framework, with attention for positive interactions, safety and discipline and feeling
supported and stimulated.
The overall strong association with the perspective of school effectiveness allows for a
investigation in the empirical support for the process indicators in the Framework, seen as
effectiveness enhancing conditions, instrumental to the attainment of cognitive outcomes
in basic subjects. In the subsequent chapters the research evidence on school and
instructional effectiveness will be summarized, and updated by a recent review and
research synthesis.
32
33
Chapter 3
The state of the art of school effectiveness and instructional
effectiveness research as a basis for comparison of the
Inspectorate’s quality indicators
Introduction
The logic of using the knowledge base on school and instructional effectiveness as a
basis for examining the foundations of the process indicators in the Inspection
Framework for primary and secondary education was already explained in Chapter 2.
Process indicators, according to this perspective are to be selected on the basis of their
association with value-added performance. In this chapter a general overview is given.
In the subsequent chapters a more detailed review of the research literature over the
last decade (1995-2005) will be presented. The chapter provides a first global
assessment of the fit of the process indicators on teaching and learning with the
knowledge base on educational effectiveness.
The overall design of educational effectiveness studies
The elementary design of school effectiveness research is the association of
hypothetical effectiveness enhancing conditions of schooling and output measures,
mostly student achievement. The basic model from systems theory that was introduced
in chapter one is helpful to clarify this basic design (see Figure 3.1). The major task of
school effectiveness research is to reveal the impact of relevant input characteristics on
output and to “break open” the black box in order to show which process or throughput
factors “work”, next to the impact of contextual conditions. Within the school it is
helpful to distinguish a school and a classroom level and, accordingly, school
organizational and instructional processes.
34
Figure 3.1: A basic systems model on the functioning of education
Research tradition in educational effectiveness varies according to the emphasis that is
put on the various antecedent conditions of educational outputs. These traditions also
have a disciplinary basis. The common denominator of the five areas of effectiveness
research that will be distinguished is that in each case the elementary design of
associating outputs or outcomes of schooling with antecedent conditions (inputs,
processes or contextual) applies. The following research areas or research traditions will
be considered in summarizing the research results obtained in developed countries:
1) Research on equality of opportunities in education and the significance of the
school in this.
2) Economic studies on education production functions.
3) The evaluation of compensatory programs.
4) Studies of unusually effective schools
5) Studies on the effectiveness of teachers, classes and instructional
Numerous reviews on school effectiveness have been published since the late seventies.
Early reviews are those by Anderson (1982), Cohen (1982), Dougherty (1981),
Edmonds (1979), Good & Brophy (1986), Kyle (1985), Murnane (1981), Neufeld et al.
(1983), Purkey & Smith (1983), Ralph & Fenessey (1983), Rutter (1983), and Sweeney
(1982). During the nineties reviews were published by Cotton (1995), Creemers (1994),
Levine & Lezotte (1990), Reynolds et al. (1993), Sammons et al. (1995), and Scheerens
(1992). Scheerens and Bosker (1997) presented the results of a meta analysis of
educational effectiveness research and Scheerens (2000) updated this review by
including educational effectiveness studies in developing countries. Research syntheses
on educational production functions, as those by Hanushek (1997) and Hedges et al.
context
outputs inputs Process or throughput
school level
classroom level
35
(1994), Scheerens and Bosker’s meta- analysis, and meta-analyses on instructional
factors by Fraser, Walberg, Welch and Hattie (1987) show negligible impacts of
resource related factors and “surface” measures of teacher qualifications (such as
highest formal qualification and teacher experience), small to negligible effects of
school organizational variables such as educational leadership, coordination,
achievement oriented policy and climate, evaluation practices, time on task and
opportunity to learn and medium size effects for aspects of structured teaching, such as
providing feedback and reinforcement (Scheerens & Bosker, 1997, p. 305). Studies
carried out in developing countries show that resource input variables have considerably
more impact (Hanushek, 1995, Scheerens, 2000).
Summary of meta-analyses (studies carried out before 1995)
In Table 3.1 (cited from Scheerens and Bosker, 1997) the results of three meta-analysis
and a re-analysis of an international data set have been summarized. The results
concerning resource input variables are based on the re-analysis of Hanushek’s (1989)
summary of results of production function studies that was carried out by Hedges, Laine
& Greenwald, 1994. As stated before this re-analysis was criticized, particularly the
unexpectedly large effect of per pupil expenditure. The results on “aspects of structured
teaching” are taken from meta- analyses conducted by Fraser, Walberg, Welch and
Hattie, 1987. The international analysis was based on the IEA Reading Literacy Study
and carried out by R.J. Bosker (Scheerens & Bosker, 1997, Ch. 7). The meta-analysis
on school organizational factors, as well as the instructional conditions “opportunity to
learn”, time on task”, “homework” and “monitoring at classroom level”, were carried
out by Witziers and Bosker and published in Scheerens & Bosker, 1997, Ch. 6. The
number of studies that were used for these meta-analyses varied per variable, ranging
form 14 to 38 studies.
36
Table 3.1: Review of the evidence from qualitative reviews, international studies and
research syntheses
Qualitative reviews
International analyses
Research syntheses
Resource input variables:
Pupil-teacher ratio
Teacher training
Teacher experience
Teachers’ salaries
Expenditure per pupil
School organizational factors:
Productive climate culture
Achievement pressure for basic subjects
Educational leadership
Monitoring/evaluation
Cooperation/consensus
Parental involvement
Staff development
High expectations
Orderly climate
Instructional conditions:
Opportunity to learn
Time on task/homework
Monitoring at classroom level
Aspects of structured teaching:
-cooperative learning
-feedback
-reinforcement
Differentiation/adaptive instruction
+
+
+
+
+
+
+
+
+
+
+
+
-0.03
0.00
0.02
0.04
0.00
-0.02
0.08
0.20
0.04
0.15
0.00/-0.01 (n.s.)
-0.01 (n.s.)
0.02
-0.03
0.04
-0.07
0.20
0.14
0.05
0.15
0.03
0.13
0.11
0.09
0.19/0.06
0.11 (n.s.)
0.27
0.48
0.58
0.22
37
Early reviews of research on teaching
In the sixties and seventies the effectiveness of certain personal characteristics of teachers
was particularly studied. Medley & Mitzel (1963); Rosenshine & Furst (1973) and Gage
(1965) are among those who reviewed the research findings. From these it emerged that
there was hardly any consistency found between personal characteristics of the teacher like
being warm hearted or inflexible on the one hand, and pupil achievement on the other.
When studying teaching styles (Davies, 1972), the behavioural repertoire of teachers was
generally looked at more than the deeply-rooted aspects of their personality. Within the
framework of "research on teaching" there followed a period in which much attention was
given to observing teacher behaviour during lessons. The results of these observations,
however, in as far as they were related to pupil achievement, seldom revealed a link with
pupil performance (see Lortie, 1973, for instance). In a following phase more explicit
attention was given to the relation between observed teacher behaviour and pupil
achievement. This research is identified in the literature as "process-product studies".
Lowyck, quoted by Weeda (1986, p. 68), summarises variables which emerged "strongly"
in the various studies:
1. Clarity: clear presentation adapted to suit the cognitive level of pupils.
2. Flexibility: varying teaching behaviour and teaching aids, organising different activities
etc.
3. Enthusiasm: expressed in verbal and non-verbal behaviour of the teacher.
4. Task related and/or businesslike behaviour: directing the pupils to complete tasks,
duties, exercises etc. in a businesslike manner.
5. Criticism: much negative criticism has a negative effect on pupil achievement.
6. Indirect activity: taking up ideas, accepting pupils' feelings and stimulating self-activity.
7. Providing the pupils with an opportunity to learn criterion material - that is to say, a
clear correspondence between what is taught in class and what is tested in examinations
and assessments.
8. Making use of stimulating comments: directing the thinking of pupils to the question,
summarising a discussion, indicating the beginning or end of a lesson, emphasising
certain features of the course material.
9. Varying the level of both cognitive questions and cognitive interaction.
Weeda (1986, p. 69) observes that in the study from which these nine teaching
characteristics were drawn, there was much criticism regarding methodology/technique.
He divides the later research studies focused at instructional effectiveness into two areas:
- pedagogic studies aimed at tracing certain environmental factors and teaching behaviour
that can influence levels of performance of certain groups of pupils;
38
- instructional psychology research aimed at establishing the interaction between teaching
variables and pupil characteristics; the so-called aptitude-treatment-interaction studies.
A central factor within the first area is that of effective teaching time. The theoretical
starting points of this can be traced back to Carroll's teaching-learning model (Carroll,
1963). Chief aspects of this model are:
- actual net learning time which is seen as a result of:
perseverance and opportunity to learn;
- necessary net learning time as a result of:
pupil aptitude, quality of education and pupil ability to understand instruction.
The mastery learning model formulated by Bloom in 1976 was largely inspired from
Carroll's model.
The findings of the aptitude-treatment-interaction studies were generally judged to be
disappointing. There were scarcely any interactions discovered, which was later confirmed
by a replication study. De Klerk (1985) regarded the fact that the ATI had failed to reveal
any simple interaction between pupil characteristics and instruction method as a challenge
to do more refined empirical research on more complex interaction patterns.
Stallings (1985) summarised research literature on effective instruction - in as far as it was
concerned with primary education - under the headings: effective net learning time, class
organisation and management, instruction, assessment and teacher expectations.
When studying net learning time it emerged that simply making the school day longer did
not necessarily lead to better levels of performance. More important, ultimately, is how
effectively time is spent. Stallings and Mohlman (1981) established that effective teachers
spent 15% of the school day on organisation and management; 50% on interactive teaching
and 35% on monitoring pupils' work. Aids for an effective use of instruction time include
all types of lesson planning. Under the classification class organisation and management
Stallings discusses streaming and maintaining order. Studies on streaming or working with
ability groups as compared to whole class instruction indicate that this type of teaching
works more positively with the more gifted pupils and that with less able groups - taking
the average result of the large numbers of surveys - hardly any effect was found (also
according to Kulik & Kulik, 1982; Van Laarhoven & De Vries, 1987; Reezigt, 1993, and
Slavin, 1987). Moreover, from various types of studies it emerges that in classes where
there is disruptive behaviour, pupil performance is lower: disruption, naturally enough, is
at the cost of effective learning time.
The question what makes good teaching should be looked at on different levels. For direct
question-and-answer type knowledge other teaching strategies are called for than for
problem-solving and acquiring insight. For learning tasks which greatly depend on
memory, a highly ordered and consistent approach is the most effective. For the acquiring
39
of insight too a clear presentation of the information offered is important as are questions
to check whether pupils have actually absorbed a specific insight. With regard to problem-
solving, some empirical support is available which shows that it is desirable that pupils
take much initiative themselves. Collins & Stevens (1982) mention five teaching strategies
to support learning in the sense of problem-solving: a) a systematic variation of examples;
b) counter examples; c) entrapment strategies; d) hypothesis identification strategies; e)
hypothesis evaluation strategies.
From studies on teacher assessments and expectations of pupils it seems that self-fulfilling
prophecies can occur. If a teacher has once formed negative expectations of certain pupils
(s)he is likely to give them less attention and expose them less to more difficult and
challenging tasks. Obviously this is even more of a disadvantage if the initial assessment
was a wrong one. Thus it is imperative that teachers should try and avoid negative
stereotyping of pupils (Van der Hoeven-Van Doornum, 1990).
In a review of literature on effective teaching at secondary school level Doyle (1985) deals
broadly with the same categories as Stalling's, namely "time on task" and "quality of
instruction". Because in secondary education the total teaching spectrum from which a
choice must be made is far greater than in primary education, the variable "opportunity to
learn" is associated here with the concept of effective net learning time. "Opportunity to
learn" is generally understood in the sense of offering pupils a range of subjects and tasks
that cover educational goals. In educational research, opportunity to learn concentrates on
the extent to which classroom exercises correspond with the content of the tests for
monitoring performance.
As far as the quality of instruction is concerned, there is a stronger emphasis in secondary
education on learning higher cognitive processes like insight, flexibly adopting knowledge
and problem-solving. Doyle considers the effectiveness of direct teaching, which he
defines as follows:
1. Teaching goals are clearly formulated.
2. The course material to be followed is carefully split into learning tasks and placed in
sequence.
3. The teacher explains clearly what the pupils must learn.
4. The teacher regularly asks questions to gauge what progress pupils are making and
whether they have understood.
5. Pupils have ample time to practice what has been taught, with much use being made of
"prompts" and feedback.
6. Skills are taught until mastery of them is automatic.
7. The teacher regularly tests the pupils and calls on the pupils to be accountable for their
work.
40
The question whether this type of highly structured teaching works equally well for
acquiring complicated cognitive processes in secondary education can be answered in the
affirmative (according to Brophy & Good, 1986, p. 367). However, progress through the
subject matter can be taken with larger steps, testing need not be so frequent and there
should be space left for applying problem-solving strategies flexibly. Doyle also
emphasises the importance of varying the learning tasks and creating intellectually
challenging learning situations. For the latter an evaluative climate in the classroom,
whereby daring to take risks even with a complicated task is encouraged, is a good means.
In addition, Doyle deals with the effect of certain ways of working and grouping, including
individual teaching and working together in small groups. Bangert, Kulik & Kulik's meta-
analysis (1983) revealed that individualized teaching in secondary education hardly led to
higher achievement and had no influence whatsoever on factors like the self-esteem and
attitudes of pupils.
Evaluation studies on special programmes to stimulate working in small groups reveal
that some of these have a positive effect on lower attaining pupils. Generally speaking,
from other reviews of research on the effects of cooperative learning it appears that
there is no conclusive empirical evidence to support the positive influence of this type
of work on performance. Vedder (1985) explained the lack of an unequivocal positive
influence of group work by the possible fact that due to the way pupils work together
there is insufficient cognitive stimulation present.
The results in this summary of reviews and meta-analyses indicate that resource-input
factors on average have a negligible effect, school factors have a small effect, while
instructional have an average to large effect. The conclusion concerning resource -input
factors should probably be modified and “nuanced” somewhat, given the results of more
recent studies referred to in the above, e.g. the results of the STAR-experiment
concerning class-size reduction. There is an interesting difference between the relatively
small effect size for the school level variables reported in the meta-analysis and the
degree of certainty and consensus on the relevance of these factors in the more
qualitative research reviews. It should be noted that the three blocks of variables depend
on types of studies using different research methods. Education production function
studies depend on statistics and administrative data from schools or higher
administrative units, such as districts or states. School effectiveness studies focusing at
school level factors are generally carried out as field studies and surveys, whereas
studies on instructional effectiveness are generally used on experimental designs. The
negligible to very small effects that were found in the re-analysis of the IEA data-set
could be partly attributed to the somewhat “proxy” and superficial way in which the
variables in question were operationalized as questionnaire items. An additional finding
from international comparative studies (not shown in the table) is the relative
41
inconsistency of the significance of the school effectiveness correlates across countries,
also see Scheerens, Vermeulen and Pelgrum, 1989, and Postlethwaite and Ross, 1992.
More recent contributions to the study of instructional effectiveness
In this section the results of some more recent contributions and reviews are briefly
summarized, on the basis of work by Anderson, 1991, 2004; Baumert et al., 2001;
Brophy, 2001; Klieme & Rakoczy, 2003; Muijs & Reynolds, 2001; NCS, 2002; OECD,
2003 (a fuller description is given in Scheerens, 2003). In these reviews, a strong
corroboration of the main characteristics of effective instruction as laid out in earlier
reviews (Scheerens & Bosker, 1997) can be discerned. In addition to this consolidation
in the knowledge base there are a few additional newer trends. These are the following:
- a reconsideration of personal characteristics of effective teachers,
- more attention to the teaching of higher order skills, self-regulated learning and
“constructivist” approaches,
- a strong re-statement of the fact that teaching is about facilitating learning, by
considering learning activities and student engagement.
In the United States the issue of effective teacher characteristics is receiving much
attention in the debate about standards for teaching competency (Darling-Hammond,
2000). Empirical studies indicate that subject matter mastery and verbal skills are
important assets of teacher effectiveness. In the United Kingdom, Hay McBerr (2000,
cited by Anderson, 2004) identified twelve characteristics, in the sense of relatively
stable traits, associated with effective teachers. Among others he mentions
characteristics like flexibility, trustworthiness, and commitment. An interesting feature
in Hay McBerr’s list is the “drive for improvement”. This trait is similar to the
“relentlessness” that is emphasized in Slavin’s “Success for All” program (1996) and
what Anderson and Pellicer (1998) have called “zero tolerance to failure”. The
dimension of confidence is associated with the “high expectations” factor in the school
and classroom climate, as one of the frequently identified factors of effective schooling.
Klieme and Rakoczy (2003) distinguish a similar dimension, which they call
“achievement press” (Leistungsdruck).
There is one other dimension in which more recent contributions return to an aspect that
was also present in the very early publications on teaching effectiveness, as Gagne’s
conception about “the conditions of learning” (Gagne, 1972) and the Caroll model,
(Caroll, 1963). This is attention for student engagement and learning strategies as the
ultimate “mediator” between teaching activities and student outcomes. In the OECD
study on “student approaches to learning” a range of variables related to engagement is
discerned, variables like “self-efficacy”, “instrumental motivation” and subject matter
42
interest (OECD, 2003). As learning strategies a distinction is made between
memorization strategies, elaboration strategies and control strategies. Control strategies
have a significant association with reading performance. Control strategies refer to
students’ ensuring that their learning goals are reached. “These strategies involve
checking what one has learned and working out what one still has to learn, allowing
learners to adapt their learning to the task at hand” (OECD, 2003, p. 13). In a way these
control strategies are the pendant of the main features of “structured teaching” and
direct instruction, where it is the teacher who actively orders and controls the teaching
and learning situation. When putting these two orientations, structured teaching on the
one hand, and students effectively employing control strategies next to one another the
following types of associations can be discerned:
- structured teaching happens as a substitute for student control strategies,
- structured teaching happens as an additional support for student control strategies,
- structured teaching happens as a model and example to enhance student control
strategies,
- structured teaching happens as a suppressor of student control, because students are
not given sufficient leeway to develop and manifest this behavior themselves.
Weaker students in primary and secondary education are more likely to benefit from the
first two alternatives, whereas the last two alternative combinations are more probable
when dealing with better students in secondary education (where obviously the third
alternative is a positive and the fourth a negative example).
It is beyond the scope of this paper to give much more detail on these reviews; a bit
more will said about the paper by Baumert et al. 2001. These authors interpret
instruction as an opportunity structure for insightful learning. “This means that
instructional materials, task selection, and instructional processes are analyzed from the
perspective of whether they foster or obstruct active individual knowledge acquisition.
Dimensions of this opportunity structure include the safeguarding of the social action
framework by means of appropriate classroom management; pacing and range of
learning opportunities (quantity of instruction); general instructional quality, in
particular the didactical quality of the structure and realization of the instruction; and
the quality of teacher-student and student-student relations.”
They go on to say that:
“With respect to general properties of classroom management and the quantity and
quality of instruction, robust findings from previous research give a good indication of
which aspects of mathematics instruction need to be assessed.” (in the context of the
OECD PISA-study)
43
“- Important properties of classroom management include: clearly defined rules and
procedures, prevention of disturbances, effective responses to critical events, and
routinization of basic social acts in the classroom.
- Important aspects of learning opportunities and pacing (and hence the quantity of
instruction) include: learning opportunities with respect to the test items, appropriate
ratio of material covered to lesson time (pacing), faithfulness to objectives and
relevance of the instructional materials, pressure to perform and interaction tempo.
- The basic properties of instructional quality include: level of difficulty, clarity and
structure in the presentation of material, adaptivity and individualization of
instruction, remediality, participation in instructional activities, monitoring of
student activities, and general constructivist properties of insightful learning.”
The following dimensions of the quality of teacher-student relations are considered: the
teacher's ability to motivate students, social orientation, and diagnostic competence in
the social domain, as well as the students' general satisfaction with their subject teacher.
About the quality of student-student relations in learner groups they propose the
following dimensions: cohesion and formation of cliques, competitiveness, mutual
assistance, aggression, and violation of norms. They also underline the significance of
subjective norms with respect to the academic or non-academic orientation of the
student body.
Again, in this contribution there seems to be some degree of combination of the
“traditional” aspects of structured teaching and constructivist ideas on learning and
instruction (Baumert et al., ibid).
The results of these more recent reviews, in the sense of the most important
instructional conditions that were referred to, are summarised in Table 3.2 below. The
table includes the main observation categories of a classroom observation schedule that
is currently being used in an internationally comparative study of SICI, an international
organisation of educational Inspectorates in Europe.
44
Table 3.2: Summary of recent reviews and the observation categories of the Dutch
Inspectorate
The observation categories of the Dutch Inspectorate do not correspond exactly to the
indicators of the more encompassing Supervision Framework for primary schools, but
they are certainly closely related. The observation categories are described as behavioral
categories that have proven to be reliable observation categories in national and
international settings.
Conclusion
There appears to be a fair correspondence between the indicators that represent the
quality aspect learning and instruction and main factors that have received empirical
support in school and instructional effectiveness research. This is reflected in the
summary table below.
Teaching (Anderson) enacted curriculum classroom physical environment classroom climate classroom organisation & management actual teaching pre-conditions (lesson planning) communication with students stimulating involvement
Dutch inspectorate
learning time support in climate challenge in climate structure in teaching activating students teaching learning strategies attainment/teacher focus on attention classroom organization
Brophy
opportunity to learn curricular alignment supportive classroom climate achievement expectations cooperative learning goal-oriented assessment coherent content; clear explanations thoughtful discourse establishing learning orientations sufficient opportunities for practice and application scaffolding student’s task engagement modeling learning and self-regulation strategies
Baumert et al. quantity and quality of instruction teacher student relations student student relations
45
Most important factors from review
Opportunity to learn
Learning time
Structured teaching
Stimulating engagement
Task-oriented climate
Mutual respect
Orderliness, safety
Monitoring and questioning
Feedback and reinforcement
Modelling learning/self-regulation
Categories in Observation check list
Learning time
Clear and structured teaching
Activating
Challenge
Support (mutual respect)
Orderly, functional learning environment
Evaluates whether objectives are reached
Feedback
Learning to use learning strategies
Classroom organisation
46
47
Chapter 4
School effectiveness research synthesis
Introduction
In this chapter the results are presented of a research review and research synthesis on
variables associated with school effectiveness. More precisely research studies are
reviewed in which malleable variables, most of them defined at school level, are
associated with outcome variables. The review consists of detailed tables and abstracts
in which the variables, research methods and type of effects are described. In addition
the results are summarized in a quantitative way. This latter practice is known as
research synthesis or meta-analysis, of which different kinds exist. The type of research
synthesis that was used is the so called vote-counting procedure, in which positive,
statistically significant associations between a school variable and an effect variable
are aggregated over research studies. In the same way non-significant or significantly
negative associations are shown. Since, as was shown in Chapter 6, a fair
correspondence exists between the quality aspects and indicators of the Inspection
Framework and the school and instructional characteristics frequently studied in school
effectiveness studies, the results can be used as a basis for judging the scientific support
of (most of) the quality indicators of the Inspection Framework.
It must be noted that there is some overlap between the meta-analysis reported in this
chapter, and the analyses in the subsequent chapter, focused at instructional
effectiveness. Chapter 5 focuses exclusively at instructional variables at classroom
level, whereas school effectiveness studies frequently contain school level variables as
well as instructional variables. This development is fuelled by the rise of so called
comprehensive models of educational effectiveness. These models include both school
level and teacher variables (Creemers, 1994; Scheerens & Bosker, 1997), inspiring
researchers to take into account both type of variables. In the concluding chapter of this
report the results of the two chapters will be integrated.
Methodology
Type of meta-analysis
A meta-analysis can be carried out in many different ways. In general, a quantitative
approach is to be preferred. This approach makes it possible to estimate the ‘true’ effect
sizes of particular variables (e.g. parental involvement). Moreover, such an approach
48
makes use of the fact that effect sizes vary among studies. This variation makes it
possible to investigate which study characteristics (e.g. country in which the study took
place, study conducted either in secondary education or primary education) are
accountable for the variation in the effect size.
A more simple approach is followed for this study. In this study a vote-counting
procedure is carried out. Vote counting implies counting the number of positive and
significant positive outcomes in a given set of studies containing information about the
relationship between a dependent variable and a specific independent variable of
interests. More specifically this implies that each study is examined whether or not the
test statistic exceeds a conventional critical value at a given significance level (usually
α=.05).
The conventional vote-counting procedure has been criticized on several grounds. First,
it does not incorporate sample size into the vote. As sample sizes increases, the
probability of obtaining statistically significant results increases. Second, the procedure
does not allow the researcher to determine which treatment is the best in an absolute
sense. Although information is found about the best treatment, it is unknown what the
margin of superiority is; it does not provide an effect size estimate. Third, the procedure
has a very low power for the range of sample sizes and effect sizes most common in the
social sciences. When effect sizes are medium to small, the conventional vote-counting
procedure frequently fails to detect the effects. Moreover, for medium to small effect
sizes, the power of the conventional vote-counting procedure tends to zero as the
number of studies to be included increases.
Despite of these disadvantages we used the vote-counting procedure as a basic and
robust way to assess the scientific support of the key variables that can be associated
with the indicators from the Inspection Supervision Framework. As stated in the
proposal for this study, we will also explore the possibility of carrying out a more
sophisticated meta-analyses, which will allow us to establish effect sizes, and weigh the
information on the basis of sample sizes. At this stage the feasibility is not yet certain,
and the results will be made available as an eventual annex to this report.
Literature Search Methods
The search methods were similar to the one used in the search for literature on teacher
effectiveness, which are described in detail in the next chapter. The methods included
searches on the Web of Science, ERIC and ERA database, ranging from the years 1995-
2004. Moreover, the literature database of ECER conferences was examined. In the
49
search the following key words were used: schooleffectiviteit, onderwijseffectiviteit,
onderwijsresultaten, effectiviteit, effectiveness, effective teaching, effective instruction,
teacher effectiveness, educational effectiveness, school effectiveness, student
achievement. Finally, recent reviews and books on school effectiveness were checked in
order to find additional relevant literature (‘snowball method’).
The first step of this search resulted in 295 publications. From these 295 publications,
89 appeared not to be useful for our purposes, while from 58 publications it could not be
determined whether or not they contained useful information. These were articles that
appeared to be inaccessible. This implies that 95 articles were left that contained
information relevant for the purposes of our study.
Analysis
In the analysis of the studies information was sought on the direction and significance
of relationships between school and teacher effectiveness variables and student
outcomes.
Although the individual study is the unit of analysis, it must be noted that some studies
dealt with multiple outcome indicators and/or multiple indicators of concepts relating to
teacher and/or school effectiveness. This implied that with respect to the question of
whether the study reports a significant relationship (or not), the relationship of each
indicator on each outcome was taken into account and included and “counted in” in our
conclusions. For example, when a (hypothetical) study is using two indicators on the
factor time (e.g. time spent on mathematics and time spent on reading) and assesses the
impact of each indicator on two outcomes (e.g. mathematic test and reading test), there
are four relationships (2 indicators x 2 outcomes). All these four relationships were
examined for their direction and significance and, consequently, all four results were
included in our final data set.
A similar approach is used when one indicator is used in studies carried out in several
countries.
With regard to the direction and sign significant positive relationships or effects were
marked with “+”, significant negative relationships with “-“, and non significant
associations with “o”. The counting has been applied for cognitive outcome variables
(CO) and non-cognitive outcomes (NCO). Which specific outcomes it concerns, is
discussed in another section of this chapter.
Apart from using information about the direction and significance, other data from these
studies was used as well. The following data were collected: authors of the study,
publication year, domain in which the study took place, the school and instruction
variables central in the study, the output variables used, content summary, the country
50
in which the study took place, school type, number of subjects per analysis level
(school, classroom, teacher, students), employed design, and applied statistical methods.
A framework developed by Scheerens & Bosker (1997, Chapter 4) was used to decide
to which concept a specific indicator belonged. This framework gives an overview of
important concepts in the field of teacher and school effectiveness and at the same it
provides a thorough overview of which indicators can be used to assess each concept
involved.
The results of the analyses are threefold:
1. a summary on the direction of effects for the teaching and school characteristics and
outcome measures;
2. a summary on the direction of effects per publication
3. a content summary of all publications involved in the meta-analysis.
In the main text of this chapter only the results mentioned under the first point are
presented in tabular form. In the appendices the reader can find a summary on the
direction of effects per publication and a content summary of all publications involved.
Effectiveness enhancing conditions at school level
As it has come to the fore in previous sections, the relationship between factors at the
school level and student outcome measures is at the heart of school effectiveness
studies. Over the years many reviews have been written about which school level
factors (appear to) have an impact on student outcome measures. A closer look on these
reviews shows that in this respect there appears to be a consensus among researchers on
which school level factors are conducive for student learning (Scheerens & Bosker,
1997). Factors which are frequently mentioned in the literature concern:
Achievement orientation
Educational leadership
Consensus and cohesion among staff
Curriculum quality/opportunity to learn
School climate
Evaluative potential
Parental involvement
Effective learning time
51
Structured instruction
Independent learning
Differentiation
Feedback and reward
To carry out a meta-analysis on these factors is important from a scientific point of
view. Such an analysis gives insight into the question of what works in education.
However, doing such an analysis carries also weight for the Inspectorate, given the
overlap between de quality aspect mentioned in the Inspectorate Framework (see
Chapter 2) and the factors found in the school effectiveness literature
A description of these variables follows below, the description is based on Scheerens &
Bosker (1997), Chapter 4.
Achievement orientation
The core idea behind this concept is the determination to get from pupils what they are
worth, in terms of aptitude and home environment. Standard setting in such a way that
pupils are challenged, but not demotivated because the standards are either too high or
too low, appears to be the main structural measure in a balanced interpretation of this
concept. The general concept of achievement orientation entails –amongst others- overt
policy choices, teachers attitudes (expectations) and behaviors and structural facilities.
Educational leadership
This concept plays a role since the 1980s and since that time thought to be an important
indicator of effective schools, although in recent years this thought is more and more
disputed. Many researchers have cast doubts about the impact of this type on leadership
on student achievement (Witziers, Bosker & Krüger, 2003).
In general, educational leadership deals with the behaviors of school leaders with regard
to the education system. In this respect two types of behaviors can be distinguished. The
first type of behavior is more operational in nature. School leaders deal explicitly with
teachers and what is going on inside the instructional system of the school. The second
type of behaviors is more strategic in nature. It concerns behaviors more remote from
the classroom and what is happening inside them, but which nevertheless are thought to
be conducive for student learning. Examples concern goal setting, the development of
the school’s mission and involving teachers in the school decision-making.
52
Consensus and cohesion among staff
Traditionally, teachers have worked autonomously inside their classrooms. In such a
setting teachers may work at cross-purposes, while at the same time there may not be
sufficient continuity for pupils, when they pass from one teacher to another. The
concept of cohesion among staff indicates that in effective schools teachers work
together as a team. In this respect the extent to which schools show coherence and
consistency (in their educational approach) is considered to be an important aspect of
effective schools.
Curriculum quality and opportunity to learn
The curriculum can be described as the blue print for the functioning of the primary
process in education. In articulating the curriculum and by setting clear targets, the
curriculum could function as a powerful coordination mechanism, serving the needs of
effective schooling.
The concept of opportunity to learn relates to the extent to which content that is actually
taught corresponds to the test or examination of items used to assess achievement. In
this respect it covers the overlap between curriculum and test.
School climate
School climate is one of the most popular concepts in education research. This also
implies that there are many meanings attached to this concept. In the school
effectiveness literature it mostly relates to two aspects of school life. The first aspect
relates to the extent to which a school is characterized by a safe and orderly climate.
Such a climate is supposed to have a positive impact on learning.
The second aspect concerns to the quality of internal relationships in schools. In this
respect it can refer to the quality of teacher-teacher relations, school leader-teacher
relations, teacher-pupil relations and pupil-pupil relations.
Evaluative potential
The concept of evaluation potential indicates the willingness and possibilities of schools
to employ evaluation as a basis for learning and feedback. The concept relates to
different levels inside the organizations, ranging from the classroom level (e.g.
diagnosing learning difficulties) to the organizational level (e.g. the use of school
diagnosis). When schools engage in these kind of activities, it is believed that school
life improves with potential beneficial effects for student learning.
Parental involvement
In several strands of school effectiveness research parental involvement is deemed to be
an important effectiveness-enhancing factor. One reason is that it leads to continuity in
53
home and school learning. Moreover, it strengthens the ties among the school’s
stakeholders which in turn leads to some kind of community feeling. Parental
involvement is usually indicated either by the actual involvement of parents or the
school’s effort to make this involvement possible.
Classroom climate
The concept of classroom climate is similar to school climate in the sense that it relates
to the existence of a safe and orderly environment or to the existence of good, internal
relationships. However, as the name of the concept already implies, these phenomena
are not measured at the school level, but at the classroom level.
Effective learning time
Scheerens & Bosker (1997) interpret learning time as a measure of the quantity of
exposure to ‘educational’ treatment at school. This definition implicitly implies that this
variable can be measured at several levels. For example, at the school level it can be
measured by taking a look at the time per subject matter in the school’s timetable. At
the classroom level it is measured by assessing the amount of time pupils –actually-
spent on their tasks. Homework is also frequently mentioned in this context, indicating
that giving homework to students increases their learning time.
Structured instruction
In education there are different views with regard to the question of good teaching and
instruction. In school effectiveness the idea prevails that instruction should be well
structured and closely monitored (e.g. Creemers, 1994). An idea which is clearly
influenced by the Carroll model which states ‘that learners must be clearly told what
they are to learn, that they must be put into adequate contact with learning materials and
that the steps in learning must be carefully planned and ordered’ (Carroll, 1989, p. 26).
In the practice of research this often means that researchers assess whether or not
teachers show behaviors which fit the characteristics of the direct instruction-model.
More specifically, it concerns the question of whether teachers show behaviors such as
goal setting, providing clear expectations, testing students on a regular base, giving
feedback and many others.
Independent learning
Independent learning can be considered as a concept which belongs to another paradigm
of good teaching (constructivism). This paradigm stresses the importance of
independent learning, the use of meta-cognitive skills and contextualized learning.
54
Differentiation
Differentiation is an important target of many measures to improve education. One
could think for example of the strong focus on taking care of pupils with learning and
behavioral problems in Dutch schools. Differentiation can be defined as instruction that
is adaptive to the specific needs of subgroups of pupils. In this respect it is opposed to a
concept as ‘whole-group’-teaching. Differentiation also plays a role in school
effectiveness research. In general it is hypothesized that when instruction is geared to
specific needs of students, there will be positive results in terms of student achievement,
although research has not confirmed this hypothesis yet.
Reinforcement and feedback
In school effectiveness research reinforcement and feedback are viewed as important
basic conditions for learning both in a cognitive and in a motivational sense. More
specifically, it is thought that motivation decreases without feedback and that pupils can
only improve when they are informed about their mistakes and errors and about how
they can improve their work.
From the description of these concepts, it is clear that many concepts used in school
effectiveness research are (still) fuzzy in the sense that different, although related,
phenomena inside classes and school are categorized under one concept. Moreover, it
must be noted that there is hardly any consistency among researchers concerning the
definition of concepts and how to operationalise them. This makes it difficult to carry
out a meta-analysis, because it is not always –obviously– clear to which concept a
particular variable belongs. As it is noted before, a framework developed by Scheerens
& Bosker (1997, Chapter 4) was used to decide to which concept a specific indicator
belonged. This framework gives an overview of important concept in the field of
teacher and school effectiveness and at the same time it provides a thorough overview
of which indicators can be used to assess each concept involved.
Characteristics of studies included
Outcomes
Traditionally, school effectiveness research has always used cognitive outcomes to
come to conclusions about what effective schools are. Cognitive outcomes refer to
results of learning with respect to the development of knowledge, measured either by
standardized achievement and competency tests or specific tests on content
understanding or student performance. More specifically, these tests are either in the
area of mathematic tests or in the area of language tests.
Most studies in this review are studies following this tradition, that is, they are either
using mathematic tests and/or language tests. A minority of studies uses other type of
55
outcome measures. It either concerns achievement tests in the area of science or
composite scores.
Non-cognitive outcomes have always been used less in school effectiveness studies.
Non-cognitive outputs comprise motivational and affective and developments of
students such as their well-being. This neglect of non-cognitive outcomes comes also to
the fore in the studies under review. Only a small minority of studies have included this
type of outcomes.
Another characteristic feature of school effectiveness research has always been the
strong reliance on correlational designs. This characteristic is also present in the studies
under review. Most studies under review have used this design. The flaw of this type of
design is obvious. Causal attribution (showing that schools and/or teachers cause the
effect) is difficult to achieve.
Many authors in the school effectiveness literature have hold pleas for the application of
longitudinal and/or (quasi-) experimental studies to counter the weakness of
correlational designs. However, within the studies under review this plea is seldomly
put into practice. The number of studies with either a longitudinal or (quasi-)
experimental design are scarce.
The prevalent way to analyze the data concerns the application of multi-level analyses.
In this respect progress has been made in the sense that the application of multi-level
analysis has become standard in the field of school effectiveness research. This also
implies that studies still analyzing data on an aggregate level are scarce.
Almost all studies include variables relating to student intake characteristics. In this
respect all studies correct the outcomes for these characteristics, most notably
characteristics such as socio-economic status, age, gender, ethnicity etc. Only a few
studies actually take prior achievement into account, implying that only in a few studies
the dependent variable using learning growth (or learning gain) is employed.
Tables 4.1 and 4.2 show the results for the factors included in this part of the analysis.
The difference between the two tables is that Table 4.2 gives more detailed information
than Table 4.1. This latter table provides an overview of the results for each variable,
while Table 4.2 not only shows the results for each variable, but also provides
information about the results of specific indicators.
Results
Table 4.1 and Table 4.2 show the results for the factors included in this part of the meta-
analysis.
56
Table 4.1: Summary on number of school effectiveness studies in which school and
teacher characteristics are related to mathematics/arithmetic, language
(including reading and writing) and science and other subjects
Characteristics
Mathematics / arithmetic
Language / reading / writing
Science and other subjects
Σ - o + Σ - o + Σ - o + Σ 1. Achievement,
orientation, high expectations
20 1 11 8 12 - 10 2 5 2 - 3 37
2. Educational leadership
33 4 20 9 17 1 10 6 11 1 5 5 61
3. Consensus and cohesion among staff
17 1 11 5 12 2 10 - 1 - 1 - 30
4. Curriculum quality/opportunity to learn
9 2 1 6 9 2 2 5 2 - - 2 20
5. School climate 27 - 16 11 27 - 20 7 14 2 3 9 68
a) orderly atmospheres
14 - 7 7 9 - 5 4 7 1 1 5 30
b) climate in terms of effectiveness orientation and good internal relationships
13 - 9 4 18 - 15 3 7 1 2 4 38
6. Evaluative potential
4 2 1 1 4 1 2 1 1 1 - - 9
7. Parental involvement
39 1 23 15 39 2 24 13 4 - - 4 82
8. Classroom climate
10 - 8 2 7 - 6 1 9 - 6 3 26
9. Effective learning time
23 5 12 6 18 3 12 3 4 - - 4 45
10. Structured instruction
10 1 2 7 7 - 5 2 2 - - 2 19
11. Independent learning
1 1 - - 2 1 1 - 1 1 - - 4
12. Differentiation 40 9 26 5 57 10 31 16 8 2 1 5 105 13. Reinforcement
and feedback 2 1 1 - 3 - 3 - - - - - 5
Σ 262 28 148 86 251 22 146 63 76 11 19 46 579Σ: Sum of replications addressing school and teaching characteristics and outcome criteria; -: studies with significant negative effects; o: studies with no effects; +: studies with significant positive effects
Note: Some publications address multiple learning outcomes as well as multiple indicators. Thus, the number of replications given is higher than the number of publications included for the review.
57
The variable most investigated, is the variable Differentiation, (105 possible positive,
significant relationships), followed by the variable Parental involvement (82 possible
positive, significant relationships). Most other factors are examined far less.
With regard to the question of which variable is the most successful, it turns out that
variables relating to Curriculum quality/opportunity to learn and to Structured
instruction are the ones most likely to show a positive and significant relationship with
the student outcomes under scrutiny. In the case of Quality of curriculum/opportunity to
learn 65% of all existing relationships are positive and significant, while this is the case
for 57% of all relationships involving variables relating to Structured instruction. In this
respect these variables appear to be the best variables when it comes down to explaining
student outcomes, although it must be noted that in both cases the number of studies
analysed is rather limited.
Other important variables concern School Climate and Educational Leadership. With
regard to School Climate, the most important aspect is the existence of an orderly
atmosphere and, to a lesser extent, the existence of good internal relationships. In the
case of this variable about 40% of all relationships are positive and significant. School
climate is followed closely by Educational Leadership, in particular if one leaves out
one study which, due to its large number of indicators, has an uneven large impact on
the final result. If one leaves this particular study out, about 40% of all relationships are
positive and significant.
Educational Leadership is followed by Achievement Orientation (35% of all
relationships are positive and significant). Within this category the indicator ‘clear focus
on the mastering of basic subjects’ is the most important indicator, accounting for the
bulk of positive and significant relationships.
All other factors are less successful in the sense that, compared to the other factors
mentioned previously, far less positive and significant relationships are
found. The percentage of positive and significant relationships range from
0% (Feedback) to 28% (Effective learning time).
58
Table 4.2: Number of studies in which school and teacher characteristics are related to mathematics / arithmetic, language (including
reading and writing) and science and other subjects
Mathematics /
arithmetic
Language / reading
/ writing
Science and other
subjects
Source
Σ - o + Σ - o + Σ - o + Σ 2. Achievement, orientation, high
expectations 20 1 11 8 12 - 10 2 5 2 - 3 37
1.1. clear focus on the mastering of basic
subjects 11 3 8 3 2 1 - 14 7, 9, 10, 19, 25, 75,
1.2. high expectations (school level) 1 1 1 1 4 2 2 6 10, 59, 77,
1.3. high expectations (teacher level) 8 1 7 8 7 1 1 1 17 3, 7, 23, 75, 77, 80,
1.4. records on pupils’ achievement - - - -
14. Educational leadership 33 4 20 9 17 1 10 6 11 1 5 5 61
2.1. general leadership skills 5 1 4 1 1 2 2 8 3, 10, 20, 33, 34, 42, 58, 72,
2.2. school leader as information provider - - - -
2.3. orchestrator of participative decision making 5 3 1 1 5 4 1 - 10 27, 76,
2.4. school leader as coordinator - - - -
2.5. teacher leadership - - 2 2 2 73
2.6. time educational / administrative leadership 4 4 4 4 - 8 76,
2.7. meta-controller of classroom processes - 1 1 - 1 35
2.8. counsellor and quality controller of
classroom teachers 3 3 4 1 3 6 3 3 13 5, 23, 36, 60, 73, 74,
59
Mathematics /
arithmetic
Language / reading
/ writing
Science and other
subjects
Source
Σ - o + Σ - o + Σ - o + Σ 2.9. facilitator of work-oriented teams - - - -
2.10. initiator and facilitator of staff
professionalization 1 1 1 1 - 2 11
2.11. principal’s attitudes 4 3 1 1 1 1 1 6 7, 77,
2.12. instructional leadership head of department 11 11 - - 11 7,
15. Consensus and cohesion among staff 17 1 11 5 12 2 10 - 1 - 1 - 30
3.1. types and frequency of meetings and
consultations 1 1 1 1 - 2 10, 77,
3.2. contents of cooperation 11 1 7 3 6 1 5 - 17 17, 23, 39, 75, 76, 78, 81,
3.3. satisfaction about cooperation - - - -
3.4. importance attributed to cooperation - - 1 1 1 3,
3.5. indicators of successful cooperation - - - -
3.6. level of consensus concerning goals 5 3 2 5 5 - 10 10, 23, 76,
16. Curriculum quality / opportunity to learn 9 2 1 6 9 2 2 5 2 - - 2 20
4.1. the way curricular priorities are set - - - -
4.2. choice of methods and text books 1 1 1 1 1 1 3 5,
4.3. application of methods and text books - - - -
4.4. opportunity to learn 8 2 1 5 8 2 2 4 1 1 17 10, 39, 41, 56, 62, 63, 64, 65,
66, 67,
60
Mathematics /
arithmetic
Language / reading
/ writing
Science and other
subjects
Source
Σ - o + Σ - o + Σ - o + Σ 4.5. satisfaction with the curriculum
17. School climate 27 - 16 11 27 - 20 7 14 2 3 9 68
b) orderly atmospheres 14 - 7 7 9 - 5 4 7 1 1 5 30
1) the importance given to an orderly climate 1 1 - 1 1 2 17, 59,
2) rules and regulations 2 2 2 1 1 1 1 5 10, 23, 39, 77,
3) punishment and rewarding - - - -
4) absenteeism and drop out 4 4 1 1 4 4 9 9, 21, 61,
5) good conduct and behaviour of pupils 5 4 1 5 4 1 1 1 11 18, 59, 76,
6) satisfaction with orderly school climate 1 1 1 1 - 2 36,
7) existence of an orderly climate 1 1 - - 1 10,
c) climate in terms of effectiveness orientation
and good internal relationships 13 - 9 4 18 - 15 3 7 1 2 4 38
1) priorities in an effectiveness-enhancing
school climate - - - -
2) perceptions on effectiveness-enhancing
conditions - - - -
3) relationships between pupils - 1 1 - 1 77,
4) relationships between teacher and pupils 1 1 1 1 2 1 1 4 18, 59,
5) relationship between head teacher and pupils - - 2 2 2 12,
61
Mathematics /
arithmetic
Language / reading
/ writing
Science and other
subjects
Source
Σ - o + Σ - o + Σ - o + Σ 6) relationships between staff 1 1 2 1 1 1 1 4 23, 36, 37,
7) relationships: the role of the head teacher 1 1 1 1 2 77,
8) engagement of pupils 4 4 4 4 2 2 10 59, 76,
9) appraisal of roles and tasks 1 1 2 2 - 3 80,
10) job appraisal in terms of facilities,
conditions of labour, task load and general
satisfaction
5 4 1 7 6 1 - 12 23, 61, 76,
11) facilities and building - - - -
18. Evaluative potential 4 2 1 1 4 1 2 1 1 1 - - 9
6.1. evaluation emphasis - - - -
6.2. monitoring pupils’ progress 3 2 1 2 1 1 - 5 7, 61, 75,
6.3. use of pupil monitoring systems - 1 1 1 1 2 77,
6.4. school process evaluation - 1 1 - 1 77,
6.5. use of evaluation results - - - -
6.6. keeping records on pupils’ performance 1 1 - - 1 10,
6.7. satisfaction with evaluation activities - - - -
19. Parental involvement 39 1 23 15 39 2 24 13 4 - - 4 82
7.1. emphasis on parental involvement in school
policy 3 3 8 3 5 2 2 13 6, 28, 31, 68, 69, 70, 71, 77,
62
Mathematics /
arithmetic
Language / reading
/ writing
Science and other
subjects
Source
Σ - o + Σ - o + Σ - o + Σ 7.2. contacts with parents 36 1 20 15 31 2 21 8 2 2 69 1, 5, 9, 16, 18, 60, 76, 77, 79
7.3. satisfaction with parental involvement - - - -
20. Classroom climate 10 - 8 2 7 - 6 1 9 - 6 3 26
8.1. relationships within the classroom 4 4 3 3 7 4 3 14 23, 26, 74,
8.2. order 1 1 2 1 1 - 3 7, 23, 77,
8.3. work attitude 2 1 1 2 2 1 1 5 22, 23, 26,
8.4. satisfaction 1 1 - - 1 39,
8.5. orderly instructional climate 1 1 - - 1 7,
8.6. engagement of pupils in classroom 1 1 - 1 1 2 26,
21. Effective learning time 23 5 12 6 18 3 12 3 4 - - 4 45
9.1. importance of effective learning time 1 1 - - 1 7,
9.2. monitoring of absenteeism - - - -
9.3. time at school 1 1 - - 1 39,
9.4. time at classroom level 8 2 5 1 11 10 1 - 19 10, 75, 77, 80, 81,
9.5. classroom management 4 2 2 1 1 2 2 7 2, 10, 77,
9.6. homework 9 6 3 6 3 2 1 2 2 17 4, 10, 29, 39, 75, 76, 77, 81,
63
Mathematics /
arithmetic
Language / reading
/ writing
Science and other
subjects
Source
Σ - o + Σ - o + Σ - o + Σ 22. Structured instruction 10 1 2 7 7 - 5 2 2 - - 2 19
10.1. importance of structured instruction 4 2 2 4 2 2 1 1 9 24, 76,
10.2. structure of lessons 1 1 - - 1 39,
10.3. preparation of lessons - - - -
10.4. direct instruction 2 2 - - 2 39, 81,
10.5. monitoring 3 1 2 3 3 1 1 7 7, 40, 75, 80,
23. Independent learning 1 1 - - 2 1 1 - 1 1 - - 4 26, 77,
24. Differentiation 40 9 26 5 57 10 31 16 8 2 1 5 105
12.1. general orientation 38 9 25 4 48 10 27 11 6 2 1 3 92 10, 13, 14, 15, 30, 32, 38, 43,
45, 48, 49, 50, 51, 52, 54, 59,
61, 75, 76, 79, 80,
12.2. special attention for pupils at risk 2 1 1 9 4 5 2 2 13 8, 23, 44, 46, 47, 53, 55, 57,
80,
25. Reinforcement and feedback 2 1 1 - 3 - 3 - - - - - 5 10, 23, 77,
Σ: Sum of studies; -: studies with significant negative effects; o: studies with no effects; +: studies with significant positive effects
64
Implications for the School Inspection Framework
Given the overlap between the factors mentioned in the school effectiveness literature
and the Inspection Framework, it is clear that the results also have implications for the
Inspection Framework.
For some quality indicators there seems to be sufficient scientific support. It concerns
quality indictors related to the quality of the curriculum, structured teaching and to the
existence of a safe and orderly climate. For other indicators there is (far) less scientific
support. A more comprehensive interpretation of these results, with respect to assessing
the scientific basis of the quality indicators of the Inspection Framework will be carried
out in the concluding chapter of this report, after the results of the meta-analysis on
instructional variables (Chapter 5) have been presented.
65
Annexes to Chapter 4 Annex 4.1
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
1 Balli, S.J.,
Wedman,
J.F. & Demo,
D.H.
1997 mathemati
cs
Parental
involvement in
homework
20 homework
assignments
Group 1--students
were prompted to
involve family
members, and family
members were
prompted to be
involved; Group 2--
students were
prompted to involve
family members; and
Group 3--no
prompting was given
family involvement
with mathematics
homework among
the three groups
was statistically
significant
USofA Middle
school,
White 6th
grade
74 pupils
and their
families
pretest-
treatment-
posttest /
3
treatments
analysis of
variance
ANOVA,
F(2,66)
13.61, p <
.01
7.2
1 Balli, S.J.,
Wedman,
J.F. & Demo,
D.H.
1997 mathemati
cs
Parental
involvement in
homework
mathematics
posttest 40
assignments
Group 1--students
were prompted to
involve family
members, and family
members were
prompted to be
involved; Group 2--
students were
prompted to involve
family members; and
Group 3--no
prompting was given
higher levels of
family involvement
would be
associated with
higher posttest
achievement was
not supported
USofA Middle
school,
White 6th
grade
74 pupils
and their
families
pretest-
treatment-
posttest /
3
treatments
analysis of
variance
ANCOVA,
F(2,70)
0.15, p .85
7.2
66
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
2 Behets, D. 1997 physical
education
Active learning
time, instruction
and
management
mean score of
three trials of
handstand
rollover
The study examined
teacher and pupil
behaviors and
compared more and
less effective
teachers in a
gymnastic setting.
During four
consecutive 25
minute lessons a
novel gymnastic skill
(handstand rollover)
had to be taught.
significant learning
gain
Belgium /
Flanders
Secondary
schools,
pupils 12-
14 years /
girls
170 pupils
and 9
teachers
PE
pretest-
treatment-
posttest
T-test T-test,
t=8.61, p <
.01
9.5
2 Behets, D. 1997 physical
education
Active learning
time, instruction
and
management
three time
categories:
active learning
time,
management,
and instruction
Teachers were
labeled the most,
moderate and least
effective teachers.
the most effective
teachers scored
significantly higher
for active learning
time and
significantly lower
for instruction time
Belgium /
Flanders
Secondary
schools,
pupils 12-
14 years /
girls
36 lessons
and 9
teachers
PE
pretest-
treatment-
posttest
Kruskall-
Wallis test
Kruskall-
Wallis test.
Active
learning
time:
H=14.2, p
< .01.
Instruction
time:
H=14.3, p
< .01
9.5
3 Heck, R.H. &
Marcoulides,
G.A.
1996 composite
score
Organisational
characteristics
(structure (1),
values(2)and
climate (4))
,managerial
processes
(3)and teacher
attitudes (5)
level of
academic
performance
within school
measured by
national
standardized
tests for
different
academic
domains
Climate (relating to
teacher
relationships) and
teachers attitudes
(perception of
student abilities)
have a positive
and significant
relationship with
outcome variable
Singapore Secondary
school
26
schools,
140
teachers
correlation
al
Lisrel 1=0,2=0,3
=-.02,
4=.35,
5=.52
14; 3.4;
2.1; 1.3
67
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
4 Cooper, H.,
Lindsay, J.
J., Nye, B. &
Greathouse,
S.
1998 Proportion of
completed
homework
TCAP, class
grade
Students, parents,
and teachers
completed a
questionnaire
concerning amount of
homework assigned
by teachers, portion
of assignments
completed by
students, and
attitudes about
homework.
class grades were
predicted only by
standardized test
scores and the
proportion of
homework
completed by
students
USofA,
Tennessee
Lower
grades
(2nd, 4th)
285
students
survey path
analysis
R2(236) =
.28
9.6
4 Cooper, H.,
Lindsay, J.
J., Nye, B. &
Greathouse,
S.
1998 Proportion of
completed
homework
TCAP, class
grade
Students, parents,
and teachers
completed a
questionnaire
concerning amount of
homework assigned
by teachers, portion
of assignments
completed by
students, and
attitudes about
homework.
class grade
predictors also
included parent,
teacher, and
student attitudes
USofA,
Tennessee
Upper
grades(
6th, 8th,
10th, 12th)
424
students
survey path
analysis
R2(250) =
.34
9.6
5 Creemers, B.
& Werf, G.
v.d.
2000 language
(Bahasa
Indonesia)
Primary
Education
Quality
Improvement
Project (PEQIP)
student
achievement:
Bahasa
Indonesia
a = teacher
development, b =
management, c =
books and materials,
d = community
participation
teacher
development and
management are
negative, books
and learning
materials and
community
participation
both meet the
criterion
Indonesia Grade 6 1854
pupils
experimen
tal group
vs. control
group
multi-level
analysis
square
root of
multiple
R2 is the
total effect
size of an
interventio
n, a: -.22,
b: -.10, c:
.15, d: .27
a: na / b:
2.8 / c:
4.2 / d:
7.2
68
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
5 Creemers, B.
& Werf, G.
v.d.
2000 mathemati
cs
Primary
Education
Quality
Improvement
Project (PEQIP)
student
achievement:
mathematics
a = teacher
development, b =
management, c =
books and materials,
d = community
participation
teacher
development and
management are
negative, books
and learning
materials and
community
participation
both meet the
criterion
Indonesia Grade 6 1854
pupils
experimen
tal group
vs. control
group
multi-level
analysis
square
root of
multiple
R2 is the
total effect
size of an
interventio
n, a: .13,
b: .18, c:
.10, d: .18
a: na / b:
2.8 / c:
4.2 / d:
7.2
5 Creemers, B.
& Werf, G.
v.d.
2000 science Primary
Education
Quality
Improvement
Project (PEQIP)
student
achievement:
science
a = teacher
development, b =
management, c =
books and materials,
d = community
participation
teacher
development and
management are
negative, books
and learning
materials and
community
participation
both meet the
criterion
Indonesia Grade 6 1854
pupils
experimen
tal group
vs. control
group
multi-level
analysis
square
root of
multiple
R2 is the
total effect
size of an
interventio
n, a: .30,
b: .00, c: -
.10, d: .18
a: na / b:
2.8 / c:
4.2 / d:
7.2
6 Desimone, L. 1999 mathemati
cs
Parent
involvement (12
types)
NELS:88,
student
achievement:
mathematics
significant
differences
according to
students' race-
ethnicity and family
income
USofA Grade 8 19386
students
secondary
analysis
regression
analysis
(OLS)
adjusted
R2 .29
7.1
6 Desimone, L. 1999 reading Parent
involvement (12
types)
NELS:88,
student
achievement:
reading
significant
differences
according to
students' race-
ethnicity and family
income
USofA Grade 8 19386
students
secondary
analysis
regression
analysis
(OLS)
adjusted
R2 .26
7.1
69
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
6 Desimone, L. 1999 Parent
involvement (12
types)
NELS:88,
student
achievement:
grades
significant
differences
according to
students' race-
ethnicity and family
income
USofA Grade 8 19386
students
secondary
analysis
regression
analysis
(OLS)
adjusted
R2 .22
7.1
7 Grift, W. v.d.,
Houtveen,
Th. &
Vermeulen,
C.
1997 mathemati
cs
Teacher, head
of department
and school
leader variables
as measured by
perceptions of
students,
teachers, head
of departments
and school
leaders
mathematics:
grade received
on last report
Most indicators are
non-significant,
except students'
appreciation of
teachers' teaching
abilities, teacher
perceptions of
orderly
instructional
climate (+),
monitoring (-),
academic press (-
), and principal
perception of task
oriented work
climate (-)
Netherland
s
Secondary
education
121
schools,
109
teachers,
100 head
of
departmen
ts, 107
school
leaders,
2938
pupils
survey multilevel 1.3; 8.2;
8.5; 10.5;
1.3; 9.1;
6.2; 1.3
1.1; 1.3;
2.11;
2.11; 2.12
8 De Fraine,
B., Damme,
J. v.,
Landeghem,
G. v.,
Opdenakker,
M.-C. &
Onghena, P.
2003 language Class
composition
language
achievement
test
classes with a high
average initial
cognitive ability or
a large proportion
of girls result in
higher language
achievement
Belgium /
Flanders
Secondary
school,
2nd year
2569
pupils
multi-level
analysis
? 12.2
70
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
9 Greenberg,
E.
2004 mathemati
cs
School climate
(3 aspects)
NAEP 2000
mathematics
achievement
test
a = student
behaviour, b =
parental involvement,
c = school morale
independent
relationship to
student
achievement after
controlling for
other school and
student factors,
better student
behaviour,
increased parental
involvement, and
higher school
morale are
positively
associated with
student
achievement in
mathematics
USofA Grade 4 /
Grade 8 /
Grade 12
? secondary
analysis
principal
component
analysis
grade 4 a:
R2 .26, b:
R2 .26, c:
R2 .26 /
grade 8 a:
R2 .27, b:
R2 .27, c:
R2 .27 /
grade 12
a: R2 .24,
b: R2 .24,
c: R2 .24
a: 5a.4 /
b: 7.2 / c:
1.1
71
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
10 Werf, G. v.d. 1997 mathemati
cs
school variables
(organization,
policy) teacher
variables
(curriculum,
groping of
pupils, quality of
teaching,
learning time,
opportunity to
learn)
arithmetic (Cito-
test, 40 items)
School variables
are non significant,
except educational
leadership (-).
Positive
relationship
concern the
teacher level
indicators whole
class, instruction,
minimum goals, %
of content covered.
Negative
relationships
concern scheduled
time for arithmetic,
ability grouping,
sequence of
curriculum,
control/correction
of learning tasks,
monitoring,
minutes spent on
organization,
minutes spent on
individual learning
tasks
Netherland
s
Primary
school
560
school,
50.000
pupils
survey secondary multi-level
analyses, analysis of
variance
2.1; 5a.7;
5a.2; 3.6;
3.1; 1.2;
1.1; 6.6;
12.1; 1.1;
13; 13;
9.4; 9.5;
9.6; 4.4
11 Jacob, B.A.
& Lefgren, L.
2004 mathemati
cs /
reading
In-service
training of
teachers
mathematics
and reading
Iowa Test Basic
Skills
training of teachers
has no effect on
student
achievement
USofA Grade 3 to
Grade 6
100288
students,
461
schools
quasi-
experimen
tal design
regression
discontinui
ty
approach
? 2.10
72
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
12 Levačić, R.,
Steele, F.,
Smees, R. &
Malmberg, L.
2003 language,
mathemati
cs, science
School climate:
pupil climate
and leadership
constructs
with/out context
key stage 3:
English,
mathematics,
science,
average Eng.,
math. and
science
Construct is a mixture
of several school
effectiveness
indicators.
a positive and
significant
relationship
between head
teacher leadership
as assessed by
pupils
UK Secondary
schools,
Year 10
519 pupils secondary
analysis
table 6 >
appendix
a7-a12
5b.5
12 Levačić, R.,
Steele, F.,
Smees, R. &
Malmberg, L.
2003 language,
mathemati
cs
School climate:
pupil climate
and leadership
constructs
with/out context
key stage 4:
English,
mathematics,
GCSE/GNVQ
Construct is a mixture
of several school
effectiveness
indicators.
a positive and
significant
relationship
between head
teacher leadership
as assessed by
pupils
UK Secondary
schools,
Year 11
351 pupils secondary
analysis
table 6 >
appendix
a7-a12
5b.5
13 Lou, Y.,
Abrami, P.C.,
Spence,
J.C.,
Poulsen, C.,
Chambers,
B. &
D'Apollonia,
S.
1996 Within-class
grouping
student
achievement
mean of experimental
group minus mean of
control group divided
by pools standard
deviation
students learning
in small groups
within classrooms
achieved more
than students not
learning in small
groups
USofA Elementar
y,
secondary
and
postsecon
dary
schools
16073
students
meta-
analysis
homogenei
ty test
.17 12.1
13 Lou, Y.,
Abrami, P.C.,
Spence,
J.C.,
Poulsen, C.,
Chambers,
B. &
D'Apollonia,
S.
1996 Within-class
grouping
student
achievement
homogeneous
grouping vs.
heterogeneous
grouping
no evidence that
one type of
grouping is
superior for
promoting
achievement of all
students
USofA Elementar
y,
secondary
and
postsecon
dary
schools
16073
students
meta-
analysis
homogenei
ty test
.12 12.1
73
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
14 Lou, Y.P.,
Abrami, P.C.
& Spence,
J.C.
2000 Within-class
grouping
student
achievement
mean of experimental
group minus mean of
control group divided
by pools standard
deviation
a small but
significantly
positive effect of
small-group
instruction
USofA Elementar
y,
secondary
and
postsecon
dary
schools
? meta-
analysis
homogenei
ty test
.16 12.1
15 McEwan,
P.J.
2003 language
(Spanish)
Peer effects
(classroom
mean of
mothers’
education)
achievement
test Spanish
(standardized
scores)
mothers’ education
has important
effects on Spanish
achievement
Chile Grade 8 163078
pupils
secondary
analysis
analysis of
variance
R2 .24,
with fixed
school
effects R2
.35
12.1
15 McEwan,
P.J.
2003 mathemati
cs
Peer effects
(classroom
mean of
mothers’
education)
achievement
test
mathematics
(standardized
scores)
mothers’ education
has important
effects on
mathematics
achievement
Chile Grade 8 163078
pupils
secondary
analysis
analysis of
variance
R2 .22,
with fixed
school
effects R2
.40
12.1
16 Okpala,
C.O.,
Okpala, A.O.
& Smith, F.E.
2001 mathemati
cs
Parent
involvement
(volunteer
hours), SES
(free/reduced
lunch),
instructional
supplies
expenditures
mathematics
achievement
a. = parental
involvement, b. =
free/reduced lunch, c.
= instructional
supplies expenditures
instructional
supplies
expenditures and
parental volunteer
hours not
statistically
significant,
percentage of
students in
free/reduced lunch
was related
negatively
USofA Grade 4 4256
students
analysis T-test T-test, a.
t=0.84, b.
t=-7.62*, c.
t=0.77
7.2
74
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
17 Opdenakker,
M.C. &
Damme, J. v.
2001 mathemati
cs
School
composition,
school process
mathematics
achievement
(LOSO-project)
school composition:
numerical
intelligence, SES,
speaking Dutch at
home, proportion of
girls, educational
level of father /
school process: the
school as an
organisation,
teaching practices
and the goals of the
school
significant
correlations
between school
process variables
‘orderly learning
environment’ (a)
and ‘cooperation
between teachers’
(b) and the school
composition
variables mean
ability and SES at
school level
Belgium /
Flanders
Secondary
schools,
first grade
A
4699
pupils
multi-level
analysis
multi-level
analysis
table IV a: 5a.1 /
b: 3.2
18 Parcel, T.L.
& Dufur, M.J.
2001 mathemati
cs
Family social
and school
social capital (a.
school social
problems, b.
parent-teacher
communication,
c. teachers
care, d. parental
involvement)
Peabody
Individual
Achievement
Test (PIAT) for
mathematics
family social capital:
time and attention
parents spend in
interaction with their
children / school
social capital: bonds
between parents and
school that can
facilitate educational
outcomes
Family social
capital (home
environment,
married mother)
and school social
capital (private
school, physical
environment)
increases
achievement
USofA Grade 1 -
grade 8
2034
students
secondary
analysis
OLS
regression
R2 .423 a: 5a.5 /
b: 7.2 / c:
5b.4 / d:
7.2
18 Parcel, T.L.
& Dufur, M.J.
2001 reading Family social
and school
social capital (a.
school social
problems, b.
parent-teacher
communication,
c. teachers
care, d. parental
involvement)
reading
recognition
PIAT
Assessment
family social capital:
time and attention
parents spend in
interaction with their
children / school
social capital: bonds
between parents and
school that can
facilitate educational
outcomes
Family social
capital (married
mother, location of
child, more
working hours) and
school social
capital (private
school, physical
environment)
increases
achievement
USofA Grade 1 -
grade 8
2203
students
secondary
analysis
OLS
regression
R2 .523 a: 5a.5 /
b: 7.2 / c:
5b.4 / d:
7.2
75
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
19 Phillips, M. 1997 mathemati
cs
Communitarian
climate (shared
values,
democratic
governance,
positive teacher
relationships)
mathematics
achievement
in schools where
teachers caring for
students is high,
test score
relatively low
USofA Middle
school,
grade 9
(grade 7)
5659
students
secondary
analysis
multi-level
analysis
correlation
-.106
1.1
19 Phillips, M. 1997 mathemati
cs
Academic
climate
(teachers'
expectations,
percent taking
algebra, amount
of homework)
mathematics
achievement
learn more
mathematics when
students do more
homework
USofA Middle
school,
grade 9
(grade 7)
5659
students
secondary
analysis
multi-level
analysis
correlation
.241
1.1
20 Pugh, G. &
Telhaj, S.
2003 mathemati
cs
Enmeshment
with external
communities
mathematics
achievement
(TIMSS-1999)
enmeshment effects
suggest attainment
benefits 1. when
school principals take
responsibility for
community relations,
and 2. when schools
are influenced by
certain groups in the
wider community; in
particular, by faith
communities rather
than by trade unions
or business groups.
student attainment
is improved when
school principals
take responsibility
for community
relations, and
when the school is
willing and able to
be influenced by
the values of
groups (church or
religious groups) in
the wider
community
Belgium /
Flanders
Secondary
education,
grade 8
5259
students
secondary
analysis
clustering-
robust
linear
regression
adjusted
R2 .66
2.1
21 Roby, D.E. 2004 School-wide
student
attendance
Ohio Proficiency
Tests (all tests
passed
average)
objective of this study
is for educators to
gain knowledge and
insight concerning
the relationship of
student attendance
and student
achievement.
moderate positive
relationships
between student
achievement and
student attendance
(statistically
significant)
USofA Grades 4,
6, 9, 12
1946 /
1292 / 711
/ 691
students
secondary
analysis
Pearson's
r .57 / .54 /
.78 / .55
5a.4
76
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
22 Rowe, K.K.J.
& Hill, P.W.
1998 mathemati
cs
inattentive
behaviours in
the classroom
mathematics
achievement -
Victorian Quality
Schools Project
(VQSP)
Inattentiveness in
the classroom is
strong positive
predictor of
students’
achievement
progress
Australia Three
cohorts: K-
1, 2-3, 4-5
4558
students
secondary
analysis
multi-level
analysis
? 8.3
23 Hill, P.W., &
Rowe, K.J.
1998 language ,
affective
school-,
teacher/classroo
m level
variables
literacy
achievement
and
attentiveness
With regard to
literacy
achievement, most
indicators are non-
significant, except
multigrade classes
(-), time (+)
appropriateness of
instruction (+), high
expectations(+),
teacher
professional
development (+).
Time (to complete
tasks),
Appropriateness of
instruction, teacher
warmth have a
positive, significant
relationship with
attentiveness
Australia primary
school
59
schools,
365
teachers,
6423
students
longitudina
l design
multi-level
(Mln)
language:
8.1; 8.3;
8.2; 8.1;
1.3; 13;
12.2; 8.1;
8.3; 3.2;
5a.2; 13;
3.6; 5b.9;
2.8; 5b.6;
5b.10 /
affective:
12.2; 8.1
24 Self-Brown,
S.R. &
Mathews, S.
2003 Classroom
structure:
experimental
groups (token
economy,
contingency) vs.
control group
student
achievement
goal orientation
relationship between
the classroom
learning environment
and the student goal
orientation
significant main
effect for
classroom
structure
USofA Elementar
y school,
grade 4, 5
71
students
quasi-
experimen
tal design
two-way
analysis of
variance
F(2,67) =
36.70
10.1
77
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
25 Stipek, D.J.,
Feiler, R.,
Byler, P.,
Ryan, R.,
Milburn, S. &
Salmon, J.M.
1998 reading /
mathemati
cs
Classroom
climate
(emphasizing
basic skills and
less pos. soc.
climate vs. de-
emphasizing
basic skills and
more pos. soc.
climate)
student two
achievement
tests (one for
letters/reading
and another for
numbers/math)
and motivation
the considerable
time devoted to
basic skills
acquisition (letters
and numbers) in
more basic-skills-
oriented
preschools
produced very little
gain on
standardized
achievement test
USofA Preschool
/
Kindergart
en
228
children
experimen
tal design
? 1.1
25 Stipek, D.J.,
Feiler, R.,
Byler, P.,
Ryan, R.,
Milburn, S. &
Salmon, J.M.
1998 reading /
mathemati
cs
Classroom
climate
(emphasizing
basic skills and
less pos. soc.
climate vs. de-
emphasizing
basic skills and
more pos. soc.
climate)
student two
achievement
tests (one for
letters/reading
and another for
numbers/math)
and motivation
some benefits to
an emphasis on
basic skills,
despite its
association with a
less positive social
climate
USofA Preschool
/
Kindergart
en
228
children
experimen
tal design
? 1.1
26 Young, D.J. 1998 mathemati
cs
classroom
climate learning
environment
(involvement,
cooperation,
cohesion,
teacher support,
task orientation,
independence &
autonomy)
mathematics
and science
scales
developed in the
context of
TIMSS
Most indicators
have no significant
impact, except
independence
(negative for math
and science)
Australia secondary
school
28 schools, 3397
students
multi-level math: 8.6;
8.1; 8.3;
11
78
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
26 Young, D.J. 1998 science classroom
climate learning
environment
(involvement,
cooperation,
cohesion,
teacher support,
task orientation,
independence &
autonomy)
mathematics
and science
scales
developed in the
context of
TIMSS
Most indicators
have no significant
impact, except
independence
(negative for math
and science)
Australia secondary
school
28 schools, 3397
students
multi-level science:
8.6; 8.1;
8.3; 11
27 Sweetland,
S.R. & Hoy,
W.K.
2000 reading /
mathemati
cs
Teacher
empowerment
(school climate)
Effectiveness
and New
Jersey’s Eighth
Grade Early
Warning Test
(reading /
mathematics)
empowerment is
defined in terms of
teachers’ power to
control critical
decisions about
teaching and learning
conditions, first the
relationship
between school
climate and teacher
empowerment is
considered and then
the relationship
between teacher
empowerment and
school effectiveness
teacher
empowerment is
related to higher
levels of
effectiveness
(even when
controlling for
SES)
USofA Middle
school,
Grade 8
86 schools secondary
analysis
multiple
regression
analysis
adjusted
R2
effectivene
ss .46
reading
.60
mathemati
cs .62
2.3
28 Townsend,
M. & Choi,
S.F.
2004 reading Parental self-
efficacy
reading
achievement:
Progressive
Achievement
Test of Reading
(school records)
parent self-efficacy
refers to parents'
beliefs to influence
their child's
developmental and
educational
outcomes
significant and
positive
relationship
between parents’
reported self-
efficacy for their
children’s reading
and the actual
reading
achievement of
New
Zealand
Primary
school,
Year 4
(aged 8-9)
83 children secondary
analysis
multiple
regression
analysis
F (2,78) =
4.99
7.1
79
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
their children
29 Trautwein,
U., Koller,
O., Schmitz,
B. &
Baumert, J.
2002 mathemati
cs
Homework FIMS and SIMS
mathematics
achievement
Test
achievement of
students in classes or
schools in which
homework is
assigned has been
compared to the
achievement of
students in classes or
schools in which
students are not
given any homework
homework is
substantially
related to
achievement gains
in mathematics
Germany Grade 7 1976
students
secondary
analysis
multi-level
analysis
R2 total
.59
9.6
30 Van de gaer,
E., Pustjens,
H., Damme,
J. v. &
Munter, A. d.
2004 language
(Dutch) /
mathemati
cs
Class
composition:
single sex vs.
co-educational
classes
Dutch
achievement
and
mathematics
achievement
(LOSO-study)
effects of single-sex
vs. coeducational
classes on the
progress of language
and mathematics of
boys and girls are
investigated
for girls the gender
composition of the
class does not
matter
Belgium /
Flanders
Secondary
education,
2nd year,
girls
language
2158 girls /
mathemati
cs 2135
girls
secondary
analysis
multi-level
analysis
girls:
language
X2 1.078 /
mathemati
cs X2
0.002
12.1
30 Van de gaer,
E., Pustjens,
H., Damme,
J. v. &
Munter, A. d.
2004 language
(Dutch) /
mathemati
cs
Class
composition:
single sex vs.
co-educational
classes
Dutch
achievement
and
mathematics
achievement
(LOSO-study)
effects of single-sex
vs. coeducational
classes on the
progress of language
and mathematics of
boys and girls are
investigated
for boys the
gender
composition matter
only for language
Belgium /
Flanders
Secondary
education,
2nd year,
boys
language
1973 boys
/
mathemati
cs 1974
boys
secondary
analysis
multi-level
analysis
boys:
language
X2 11.293
/
mathemati
cs X2
2.262
12.1
31 Voorhis, F.L.
v.
2003 science Interactive
science
homework
(TIPS classes)
vs.
noninteractive
homework
classes
science report
cards
effects of weekly
interactive science
homework on family
involvement,
achievement and
attitude
students in TIPS
classes earned
higher Report Card
grades than
noninteractive
homework classes
USofA Grade 6
and grade
8
253
students
secondary
analysis
multiple
regression
analysis
R2 .46 7.1
80
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
32 Wong, K.C.,
Lam, Y.R. &
Ho, L.M.
2002 language
(English /
Chinese) /
mathemati
cs
Class
composition:
single sex vs.
co-educational
classes
HKCEE:
English,
Chinese,
mathematics,
EBFS (English +
best five
subjects)
girls benefited from
studying in single-
sex schools
whereas boys
benefited from co-
educational
schools
Hong
Kong
Secondary
schools
(final
grade)
45000
students
secondary
analysis
multi-level
analysis
? 12.1
33 Marks, H.M.
& Printy,
S.M.
2003 mathemati
cs / social
studies
Educational
leadership:
integrated
leadership
academic
achievement:
authentic
student
performance in
mathematics
and social
studies
analysis is grounded
in two conceptions of
leadership —
transformational and
instructional
influence of
integrated
leadership on
school
performance,
measured by the
achievement of its
students, is
substantial
USofA 8 elementary, 8 middle
and 8 high schools
secondary
analysis
multi-level
analysis
.57 2.1
34 Griffith, J. 2003 Four models of
organizations: a.
rational goals, b.
human
relations, c.
open systems,
d. internal
process control
school-
aggregated
student
achievement,
standardized
test scores
each model is
distinguished by its
emphasis on specific
organizational
processes, which in
turn imply what
constitutes desired
output and serves as
basis for determining
organizational
effectiveness
only for b, c and d:
moderately strong
relations to school
achievement
through variables
USofA Elementar
y school,
grade 5
9431
students
secondary
analysis
path
analysis
(structural
equation
modelling)
path
coefficient
s: a. .16 /
b. .32 / c.
.44 /d. .33
2.1
35 Hallinger, P.,
Bickman, L.
& Davis, K.
1996 reading Principal
instructional
leadership
Basic Skills First
Test (reading)
principal instructional
leadership: principal
activity in key
dimensions of the
school's educational
program
no significant
direct effect of
principal
leadership on
student
achievement in
reading
USofA Grade 3,
grade 6
87 schools secondary
analysis
path
analysis
(structural
modelling,
EQS)
? 2.7
81
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
36 Uline, C.L.,
Miller, D.M.
&
Tschannen-
Moran, M.
1998 reading,
mathemati
cs, writing
Instrumental
and expressive
functions of
school
effectiveness
New Jersey's
Eight Grade
Early Warning
Test: reading,
mathematics,
writing
instrumental
functions/activities:
student achievement
(reading, writing,
arithmetic) and
expressive
functions/activities:
(a./b.) trust in
principal and
colleagues, and (c.)
school health
each of the six
individual
measures are
substantially and
significantly related
to effectiveness;
instrumental and
expressive
activities contribute
equally to
effectiveness of
schools
USofA Middle
schools
86 schools secondary
analysis
path
analysis
(estimating
structural
equation
models)
instrument
al activities
beta .32 /
expressive
activities
beta .33
a: 2.8 / b:
5b.6 / c:
5a.6
37 Lee, V.E. &
Loeb, S.
2000 mathemati
cs
Size of the
inner-city
elementary
school
mathematics
achievement
school size has
both a direct and
an indirect effect
(through teachers'
collective
responsibility) on
student learning,
small schools are
favoured
USofA Elementar
y schools
(K-8)
22599
students
secondary
analysis
multi-level
analysis
Direct: ES
= -.54 (sm-
me), ES =
-.31 (sm-
la) /
Indirect:
ES = -.10
(sm-me),
ES = -.14
(sm-la) //
total affect
sm-me .64
SD / sm-la
.45 SD
5b.6
38 Burns, R.B.
& Mason,
D.A.
2002 reading,
mathemati
cs
Combination
classes (two
grade classes)
vs. single grade
classes
pre-test: Letter
Series Test,
standardized;
post-test:
reading and
mathematics
achievement
test scores
(SAT/CAT)
class formation
procedures to create
class compositions
assignment
procedures
affected class
distributional
properties and
achievement
variation within
and between
classes
USofA 22
schools,
200
classes,
single
grade and
combinatio
n classes
District 1:
2650
students;
District 2:
2277
students
pretest-
posttest
multi-level
analysis
? 12.1
82
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
39 Luyten, H. &
Jong, R. de
1998 mathemati
cs
autonomy-
teacher
cooperation,
regulation of
teacher
behaviour by
school
management,
time,
instructional
variables
mathematic test
based upon text
books
instruction
variables have a
small, but positive
relationship
(structured
lessons, clear
instruction, number
of homework
assignments)
Netherland
s
secondary
school
22
schools,
pupils
survey multi-level
(Mln)
3.2; 3.2;
3.2; 5a.2;
9.3; 10.2;
10.4; 4.4;
9.6; 8.4
40 Olina, Z. &
Sullivan, H.J.
2002 science Classroom
evaluation
strategies: no
evaluation,
teacher
evaluation and
self- + teacher
evaluation
written research
reports (see
treatment) /
post-test (21
items)
12-lesson
instructional program
"Learning
Explorations" (basic
concepts of scientific
research)
students in teacher
eval.. and in self-
plus-teacher eval.
conditions
received higher
ratings. Students
in teacher eval.
scored significantly
higher on the post-
test
Latvia 12 classes
high
schools
189
students
quasi-
experimen
tal design
one way
analysis of
variance
written
reports:
F(2,186) =
5.70 /
post-test
F(2,186) =
4.11
10.5
41 Cooper, H.,
Valentine,
J.C.,
Charlton, K.
& Melson, A.
2003 School calendar modified school
calendar by doing
away with the long
summer break while
not increasing the
length of the school
year
effect on
achievement is
quite small (see
also rows 62 to 67)
USofA Grades K-
12
meta-
analysis
(47
studies)
Effect size
schools
with
modified
calendar:
d = .06 /
Effect size
control
schools d
= .11
4.4
83
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
42 Deinum, J.F. 2000 mathemati
cs
Schoolbeleidsfa
ctoren: beleid
t.a.v. tijd, beleid
t.a.v.
gelegenheid tot
leren, beleid
t.a.v. kwaliteit
mathematics
test
in hoeverre zijn
effecten van
schoolfactoren direct
of indirect via
leraargedrag van
invloed op de
leerresultaten van
leerlingen
schoolfactoren
hebben zowel
direct als indirect
via het klasniveau
effect op
leerresultaten
Netherland
s
Secondary
schools
(MAVO)
1031
students
secondary
analysis
multi-level
analysis /
path
analysis
Model C:
X2 = 43.0;
df = 37; p
= 0.22 /
quality:
.15; time: -
.04;
opportunit
y: .08
2.1
43 Berry, C. 2001 reading Monograde
classes vs.
multigrade
classes
McLoad Gap
Test
(standardized
cloze reading
test)
a study that
compares the reading
progress of students
in multigrade schools
with the reading
progress of students
in monograde
schools
students in
multigrade schools
tend to have
higher
achievement
scores in the end
of primary school
leaving
examination than
students in
monograde
schools
Turks and
Caicos
Islands
Primary
education,
grades 3 -
5
? quasi-
experimen
tal design
T-test (?) Group A: +
(sign.) /
group B: +
(sign.) /
group C:
no
difference
12.1
44 Doeleman,
R. &
Westerbeek,
K.
2002 language
(Dutch) /
arithmetic
Onderwijsverbet
ering (invoering
directe
instructie,
uitbreiding
leertijd,
klassenmanage
ment,
zorgverbreding)
woordenschat,
lezen (technisch
en begrijpend),
spelling,
rekenen
effecten van
onderwijsverbetering
sproject KEA op
woordenschat, lezen,
spelen en rekenen
positieve effecten
in de onderbouw
sterker dan in de
bovenbouw
Nederland Basisonde
rwijs,
groep 1
t/m 8
per cohort
ca. 140
leerlingen
time series
met zeven
cohorten
(voor-KEA,
KEA, en
vijf na-
KEA)
parameter
schattinge
n
ontbreken,
KEA en
na-KEA
cohorten
scoren
gemiddeld
hoger dan
de voor-
KEA
cohorten;
in de
bovenbou
12.2
84
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
w vindt
een
terugval
plaats
45 Elsäcker, W.
v. &
Verhoeven,
L.
1997 reading E: voorlezen
aan groepjes
van vijf
leerlingen / C:
klassikaal
voorlezen
mt (mondeling
tekstbegrip) / n
(navertellen)
effecten van
groepsgrootte op
begrip en retentie van
één voorgelezen
verhaal
middelmatig tot
sterke positieve
effecten
Nederland Basisonde
rwijs,
groep 2
E: 72
leerlingen,
C: 72
leerlingen
counter
balanced
design
d(mt) = .37
(+ .17) /
d(n) = .72
(+ .17)
12.1
46 Houtveen,
A.A.M.
2002 reading Begeleiding
t.a.v. de
implementatie
van adaptief
onderwijs bij
begrijpend lezen
(E: wel; C: niet)
bl (begrijpend
lezen), mv
(metacognitieve
vaardigheden),
Ia (leesattitude)
effecten van het
schoolverbeteringspr
oject
"kwaliteitsverbetering
begrijpend lezen"
zwakke pos.
effecten op
vaardigheden (iets
sterker na
introductie
covariaten); geen
effecten op attitude
Nederland Basisonde
rwijs,
groep 5 en
7
E. 476
leerlingen,
C: 450
leerlingen
vergelijkin
g E-C; incl.
voormetin
g; geen
random
toewijzing
klassen
aan
condities
d(bl)= .17-
.25,
d(mv)= 28-
.46, d(la)=
.07- 08 (na
correctie
voor de
genoemde
covariaten
)
12.2
47 Houtveen,
A.A.M.,
Booij, N.,
Jong, R. d. &
Grift,
W.J.C.M.
v.d.
1996 reading Begeleiding bij
het invoeren
van adaptief
onderwijs (E.
wel; C. niet)
tl (technisch
lezen); Ia
(leesattitude)
effecten van adaptief
onderwijs op
technisch lezen
de interventie heeft
weinig of geen
effecten (na
controle voor de
voormetingen een
zwak positief
effect)
Nederland Basisonde
rwijs,
groep 3
E. 319
leerlingen,
C. 137
leerlingen
vergelijkin
g E-C; incl.
voormetin
g; geen
random
toewijzing
van
klassen
aan
condities
d(as-voor)
= .34 (+
.10), d(in-
voor) = -
.40(+ .10),
d(la-voor)
= -.03 (+
.10), d(ti-
na) = 21 (+
.10), d(la-
na) = .06
(+ .10)
12.2
48 Houtveen,
A.A.M., Pijl,
S.J., Pijl,
Y.J., Reezigt,
1998 reading Drie, uit
groepsobservati
es
samengestelde
affectief
functioneren,
vorderingen
voor technisch
effecten van adaptief
onderwijs en
leerlingenzorg op
leerlingen
geen tot zwak
negatieve effecten
van adaptief
onderwijs op
Nederland Basisonde
rwijs,
groep 2 en
3
in totaal
ca. 700 lln.
correlation
ele opzet
met
metingen
beta's voor
de
regressie
van de
12.1
85
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
G.J. &
Vermeulen,
C.J.
maten
(werkwijze,
activiteiten,
interacties)
en begrijpend
lezen
vorderingen en
affectief
functioneren
op twee
momenten
(februari
en juni
1996)
afhankelijk
e
indicatoren
variëren
tussen -
.27 en .12
(alleen
enkele
negatieve
bèta's
blijken
stat. sign.)
49 Mijs, D.,
Houtveen,
A.A.M. &
Vernooy,
C.G.T.
2001 reading BOV-model
(planmatig
werken, effectief
instructiegedrag
,
klassenmanage
ment)
tl en bl
(technisch en
begrijpend
lezen, zowel
vorderingen als
% uitval)
effecten van het
BOV-model op de
leesprestaties
geen
parameterschattin
gen af te leiden
Nederland Basisonde
rwijs,
groep 3 en
4
E: ca.
2100
leerlingen
vergelijkin
g E met
landelijke
normgroep
(na 1. en
na 2.
projectjaar
)
pos.
effecten
van
onbekend
e omvang,
zowel wb.
de
vorderinge
n als wb.
de uitval
12.1
50 Overmaat,
M., Ledoux,
G. &
Koopman, P.
1997 language
(Dutch) /
arithmetic
Tijdsbesteding,
instructievormen
,
leervorderingen
bijhouden,
remedial
teaching,
deskundigheid
leerkracht
taal- en
rekenvorderinge
n, welbevinden,
% verwijzingen
naar SBO
effecten van adaptief
onderwijs op taal- en
rekenprestaties,
welbevinden en
percentage verwijzing
naar SBO
geen of zwak
negatieve
verbanden met
vorderingen, wel
bevinden, of met
het % verwijzingen
Nederland (Speciaal)
Basisonde
rwijs,
groepen 4,
6 en 8
BO: ca.
70.000
leerlingen,
SBO: ca.
5000
leerlingen
(Prima-1)
correlation
ele opzet
bèta's voor
de
regressie
van de
vorderinge
n op niet-
klassikale
instructiev
ormen
bedragen -
.08 (taal)
en -.11
(rekenen)
12.1
86
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
51 Reezigt,
G.J.,
Houtveen,
A.A.M., Grift,
W. v.d. &
Guldemond,
H.
2002 reading Diverse
indicatoren voor
adaptief
onderwijs (o.a.
planmatig
werken,
pedagogische
aanpak,
afstemming
onderwijsaanbo
d)
le (vorderingen
voor lezen), se
(sociaal-
emotioneel
gedrag), we
(werkhouding),
zv (zittenblijven
of verwijzen)
effecten van adaptief
onderwijs en
leerlingenzorg op
leerlingen
inconsistente
effecten op Ie,
geen effecten op
se, we en zv
Nederland Basisonde
rwijs,
groepen 1
t/m 4
tussen 500
en 1000
leerlingen
per
leerjaar
correlation
ele opzet
met
metingen
aan het
begin en
einde
bèta's voor
de
regressie
op de
diverse
indicatoren
variëren
tussen -30
en 25; het
merendeel
is niet
statistisch
significant
12.1
52 Reitsma, P.,
Wesseling,
R. & Stiva, F.
1997 reading E: als C maar
met computer;
C: standaard
voorbereidend
lezen, zonder
computer
as (auditieve
synthese)
effecten van
computergestuurde
oefeningen op
ontluikende
geletterdheid
zwak positief effect
(na controle voor
voormeting middel
matig positief
effect)
Nederland Basisonde
rwijs,
groep 2
E: 39
leerlingen /
C: 69
leerlingen
vergelijkin
g E-C; incl.
voortoets;
geen
random
toewijzing
klassen
aan
condities
d(as-voor)
= -.19(+
.20) / d(as-
na) = .30(+
.20)
12.1
53 Segers, E.,
Verhoeven,
L., Boot, I.,
Berkers, I. &
Vermeer, A.
2001 language De interventie
bestond uit 3 tot
6 maal
gedurende 25
minuten werken
met het
computerprogra
mma
Woordenschat
(een toets op
basis van de 29
via de computer
aangeboden
woorden)
effecten van
computergestuurd
woordenschatprogra
mma op
woordenschatontwikk
eling allochtone
kleuters
sterk positieve
effecten in beide
leerjaren
Nederland Basisonde
rwijs,
groep 1 en
2 (alleen
allochtone
leerlingen)
Totaal 55
leerlingen
voortoets-
interventie
-natoets-
opzet,
zonder
controlegr
oep
d(groep 1}
.63(+ .26) /
d(groep 2}
1.08(+ .30)
12.2
87
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
54 Veenman, S.
&
Raemaekers,
J.
1996 reading /
arithmetic
Nascholingspro
gramma voor
hanteren
combinatieklass
en (E: wel; C:
niet)
tl (technisch
lezen), bl
(begrijpend
lezen), re
(rekenen)
effecten van
nascholing "omgaan
met
combinatieklassen"
op leraren en
leerlingen
nauwelijks of geen
effecten op de
vorderingen van
de lln.
Nederland Basisonde
rwijs,
groepen 3
t/m 8
E: ca. 630
leerlingen /
C: ca. 340
leerlingen
E-C
vergelijkin
g incl.
voormetin
g en
retentieme
ting
(metingen
bij lln.
betreffen
retentie na
2 tot 5
jaar); geen
random
toewijzing
klassen
aan
condities
d(tl) =
.03(+ .07),
d(bl) = .11
(+ .07),
d(re) = -
.06(+ .07)
(het betreft
hier alleen
retentieme
tingen na
twee tot
vijf jaar)
12.1
55 Zoelen, L. v.
& Houtveen,
A.A.M.
2000 arithmetic Begeleiding
t.a.v. de
implementatie
van adaptief
onderwijs bij
rekenen/wiskun
de (E: wel; C:
niet)
v (vorderingen
rekenen); a
(attituden
rekenen); u
(percentage
uitvallers)
effecten van het
schoolverbeteringspr
oject
"kwaliteitsverbetering
rekenen/wiskunde"
zwak pos. effecten
op vaardigheid,
geen effecten op
attitude,
wisselende
effecten op uitval
Nederland Basisonde
rwijs,
groepen 3
en 4
E: 246
leerlingen /
C: 442
leerlingen
vergelijkin
g E-C; incl.
voormetin
g; geen
random
toewijzing
klassen
aan
condities
d(v-voor) =
-.02 - .10;
d(v-na) =
.31 - .33;
d(a-voor) =
-.09 - -.07;
d(a-na) =
.00 - -.07;
d(u-voor) =
.30 - .09;
d(u-na) = -
~ - .34
12.2
56 Annevelink,
E.
2004 language Reduction of
class size from
24 to 15
language
achievement
(passive
vocabulary test)
nagaan wat effect
van groepsgrootte
(small classes, pupil-
teacher ratio, pupil-
adult ratio) is op
leerprestaties
small classes or
classes with lower
PTR or PAR time
spent on task
increases
significantly
Netherland
s
Primary
education,
grade 2
526 pupils
(+303
pupils not
tested at
each
occasion)
multi-level
analysis
effect size:
small
classes -
0.48 /
lower PTR
-0.50 /
4.4
88
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
(negative) lower PAR
-0.47
56 Annevelink,
E.
2004 language Reduction of
class size from
24 to 15
language
achievement
(passive
vocabulary test)
nagaan wat effect
van groepsgrootte
(small classes, pupil-
teacher ratio, pupil-
adult ratio) is op
leerprestaties
small classes or
classes with lower
PTR or PAR time
spent on task
increases not
significantly
(negative)
Netherland
s
Primary
education,
grade 3
526 pupils
(+303
pupils not
tested at
each
occasion)
multi-level
analysis
effect size:
small
classes -
0.31 /
lower PTR
-0.29 /
lower PAR
-0.01
4.4
57 Muijs, D. &
Reynolds, D.
2003 mathemati
cs
Support by
Numeracy
Support
Assistants
(NSAs,
classroom
support
assistants)
within the
framework of
the Gatsby
Mathematics
Enhancement
Programme
Primary
mathematics
test of the
National
Numeracy
Project
one suggested
advantage of the
deployment of
learning support
assistants is to
provide additional
support to low
achieving pupils
no effect of being
supported by a
numeracy support
assistant was
found
UK Primary
education,
Year 1 and
2
E: 180
pupils, C:
180 pupils
quasi-
experimen
tal design
multi-level
analysis
? 12.2
58 Grift, W. v.d.
& Houtveen,
Th.
1999 language,
mathemati
cs
educational
leadership
composite score
(language,
arithmetic,
information
processing test)
educational
leadership has a
small, positive
relationship with
school output
Netherland
s
primary
school
174
schools
2.1
89
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
59 Smyth, E. 1999 composite,
affective
school
organization
(streaming),
pupil
involvement,
disciplinary
climate, pupil
interaction,
academic
climate
(expectations)
exam
performance
streaming
indicators have a
significant
relationship with
performance, just
as indicators
relating to
disciplinary climate
and academic
climate.
Ireland secondary
schools
106
schools,
5235
pupils
survey multi-level
(mln)
12.1;
5a.3; 8.1;
8.1; 5b.4;
1.2; 1.3
59 Smyth, E. 1999 composite,
affective
school
organization
(streaming),
pupil
involvement,
disciplinary
climate, pupil
interaction,
academic
climate
(expectations)
absenteeism,
potential drop-
out
absenteeism is
associated with
flexible subject
choice, positive
interaction among
pupils, teacher
expectations.
Potential drop-out
is associated with
academic climate
and positive
interactions among
pupils
Ireland secondary
schools
106
schools,
5235
pupils
survey multi-level
(mln)
abs: 8.1;
5b.4; 8.1
1.2; 1.3 /
pot.do:
8.1; 5b.4;
8.1 1.2;
1.3
60 Werf, G. v.d.,
Creemers,
B., Jong, R.
d. & Klaver,
E.
2000 mathemati
cs,
language
(Bahasa
Indonesia),
science
PEQIP:
management
evaluation
standardized
achievement
tests
mathematics,
Bahasa
Indonesia and
science
frequency of
classroom
observations and
evaluation of the
quality of the
school and the
teachers explain
differences in
student
achievement
Indonesia Grade 6 1854
students
quasi-
experimen
tal design
multi-level
analysis
? 2.8
90
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
60 Werf, G. v.d.,
Creemers,
B., Jong, R.
d. & Klaver,
E.
2000 mathemati
cs,
language
(Bahasa
Indonesia),
science
PEQIP:
community
participation
standardized
achievement
tests
mathematics,
Bahasa
Indonesia and
science
amount of
voluntary work by
parents and their
engagement in
learning activities
have positive
effects on student
achievement
Indonesia Grade 6 1854
students
quasi-
experimen
tal design
multi-level
analysis
? 7.2
61 Veenstra,
D.R.
1999 language,
mathemati
cs
Schoolkenmerk
en:
teamstabiliteit
Nederlands
(tekstbegrip),
wiskunde
nagaan of verschillen
in prestaties bij
Nederlands en
wiskunde verklaard
kunnen worden door
leerling-, gezins-,
docent- en/of
schoolkenmerken
zwak positief effect
op leerprestaties,
geen effect op
leervorderingen
Nederland Voortgezet
onderwijs
6896
leerlingen
secondair
e analyse
multi-level
analysis
LP ned
.043 / wis
.043
5b.10
61 Veenstra,
D.R.
1999 language,
mathemati
cs
Schoolkenmerk
en: opvang
lesuitval
Nederlands
(tekstbegrip),
wiskunde
nagaan of verschillen
in prestaties bij
Nederlands en
wiskunde verklaard
kunnen worden door
leerling-, gezins-,
docent- en/of
schoolkenmerken
invalregeling en
huiswerkles heeft
positief effect op
prestaties bij
vbo/mavo-
leerlingen,
invalregeling ook
effect op
vorderingen
Nederland Voortgezet
onderwijs
6896
leerlingen
secondair
e analyse
multi-level
analysis
LP inval
ned .192
wis .031 -
hwles ned
.346 wis
.035 / LV
inval ned
.263 wis
.099 -
hwles ned
.140 wis
.140
5a.4
91
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
61 Veenstra,
D.R.
1999 language,
mathemati
cs
Schoolkenmerk
en:
klassensamenst
elling homogeen
vs. heterogeen
Nederlands
(tekstbegrip),
wiskunde
nagaan of verschillen
in prestaties bij
Nederlands en
wiskunde verklaard
kunnen worden door
leerling-, gezins-,
docent- en/of
schoolkenmerken
op Nederlands in
heterogene
klassen lagere
scores (muv vwo-
ers)
Nederland Voortgezet
onderwijs
6896
leerlingen
secondair
e analyse
multi-level
analysis
LP hom
ned .066
wis .066 -
het ned -
.213 wis -
.001 / LV
hom ned
.019 wis
.019 - het
ned -.202
wis -.202
12.1
61 Veenstra,
D.R.
1999 language,
mathemati
cs
Schoolkenmerk
en:
mentoraatsyste
em
Nederlands
(tekstbegrip),
wiskunde
nagaan of verschillen
in prestaties bij
Nederlands en
wiskunde verklaard
kunnen worden door
leerling-, gezins-,
docent- en/of
schoolkenmerken
hoe uitgebreider
het
mentoraatsysteem,
hoe slechter de
prestaties van
leerlingen
Nederland Voortgezet
onderwijs
6896
leerlingen
secondair
e analyse
multi-level
analysis
LP brd/hor
ned -.176
wis -.176 /
LV brd/hor
ned -.262
wis -.262
6.2
62 Kneese, C. 1996 reading,
mathemati
cs
Calendar 60/15 reading,
mathematics
overall direction of
effect: positive
USofA,
California
Grades 3
to 7,
single-
track
606
students
4.4
63 Dunn, E.R. 1996 reading,
mathemati
cs
Calendar 30/10 reading,
mathematics,
other
USofA,
Texas
Grades K
to 6, single
track
112
students
effect size
-.21
4.4
64 Haenn, J.F. 1996 reading,
mathemati
cs
Calendar 45/15 reading,
mathematics
USofA,
North
Carolina
Grades K
to 5, single
track
304
students
effect size
-.03
4.4
65 Paloczy, S.T. 1997 reading,
mathemati
cs
Calendar 45/15 reading,
mathematics,
attendance
USofA,
Texas
Grades 3
to 5, single
track
148
students
effect size
+.43
4.4
66 Reece, J.L.,
Myers, C.L.
2000 reading,
mathemati
Calendar 45/10
and 45/15
reading,
mathematics,
USofA,
Kentucky
Grade 2,
single
218
students
effect size
+.34
4.4
92
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
& Nofsinger,
C.O.
cs,
language
language track
67 Consolie,
P.G.
1999 reading,
mathemati
cs
Calendar 45/15 reading,
mathematics
USofA,
Georgia
Grade 5,
single
track
159
students
effect size
+.79
4.4
68 D'Angelo, N.,
Reents, K. &
Zomboracz,
C.
1997 reading 10-week
program to
improve
language skills
through parent
activity
pre- and post
reading
assessment test
overall
improvement of
reading
USofA Rural,
middle
class,
middle
school
students
19
students
7.1
69 Epstein, J.L.,
Herrick, S.C.
& Coates, L.
1996 language Home learning
packets (2
summers)
pre- and post
tests on
language art
skills
no effect for packet
use
USofA,
Maryland
Primarily
African
American
middle
school
students
244
students
7.1
70 Hampton,
F.M.,
Mumford,
D.A. & Bold,
L.
1998 1-year study of
multifaceted
program
(FAST),
including
parental support
comparisons of
academic
performance of
FAST students
to same grade
peers
FAST students in 4
of 5 classrooms
performed
significantly better
on stand. test than
non-FAST
students
USofA,
Ohio
5
kindergart
en classes
in low-
income
African
American
neighbour
hood
119
students
7.1
71 Zellman, G.,
Stecher, B.,
Klein, S. &
McCafrey, D.
1998 mathemati
cs, reading
8 1.5 PIQE
program
sessions
focusing on
support for
parents
pre- and post
measures of
mathematics
and reading
scores
no effect on
student grades
USofA Children
from 5 low-
income
urban
elementary
schools
2730
students
7.1
93
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
71 Zellman, G.,
Stecher, B.,
Klein, S. &
McCafrey, D.
1998 mathemati
cs, reading
8 1.5 PIQE
program
sessions
focusing on
support for
minority low-
income parents
pre- and post
measures of
mathematics
and reading
scores
no direct evidence
of student gains
USofA Children
from 5 low-
income
urban
elementary
schools
125
students
7.1
72 Hofman,
R.H.,
Hofman,
W.H.A. &
Guldemond,
H.
2001 mathemati
cs
Leadership
styles: a.
modest school-
based
leadership, b.
varied style of
leadership
lacking
consensus, c.
effective,
cohesive and
coordinated
leadership style
national
standardized
mathematical
test (Cito)
average
mathematics
achievement is
significantly higher
in schools
characterized by
the effective
management type
Nederland Secondary
education,
3rd year
(91
schools)
5110
students
secondary
analysis
a. 0.0
(0.0), b. -
9.4 (4.2),
c. 10.8
(3.8)
2.1
73 Leithwood,
K., & Jantzi,
D.
2000 Principal
leadership
student
engagement:
identification
and participation
measure of principal
leadership is based
on directs effects on
teaching and learning
principal
leadership has
weak and
significant indirect
effects on
identification, but
not on participation
Canada Elementar
y schools,
grades 5,
6, 7, and 8
1818
teachers /
6490
students
survey path-
analysis
identificati
on .16 /
participatio
n .10
2.8
73 Leithwood,
K., & Jantzi,
D.
2000 Teacher
leadership
student
engagement:
identification
and participation
measure of teacher
leadership is based
on the extent of
influence on the
school of teacher
leaders acting
individually and in
groups
teacher leadership
has no significant
total effects on
identification and
participation
Canada Elementar
y schools,
grades 5,
6, 7, and 8
1818
teachers /
6490
students
survey path-
analysis
identificati
on .01 /
participatio
n .07
2.5
94
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
74 Silins, H. &
Mulford, B.
2002 Leadership
(system factors)
and
organisational
learning
student
outcomes: level
of participation
in school
The LOLSO project
focused on three
aspects of high
school functioning:
leadership,
organisational
learning and the
impact of both on
student outcomes.
teacher's work and
school size
influence
participation
directly (all
significant)
Tasmania Secondary
schools
over 5000
students /
3500 year
10
students ?
survey least
squares
path-
analysis
total
effects: a.
leadership
satisfactio
n 0.04 / b.
teacher's
work 0.51 /
c. school
size: -0.39
a: 2.8 / b:
8.1 / c: na
74 Silins, H. &
Mulford, B.
2002 Leadership
(system factors)
and
organisational
learning
student
outcomes: level
of engagement
with school
The LOLSO project
focused on three
aspects of high
school functioning:
leadership,
organisational
learning and the
impact of both on
student outcomes.
participation and
SES (both weak)
and teacher's work
(strong) influence
engagement (all
significant)
Tasmania Secondary
schools
over 5000
students /
3500 year
10
students ?
survey least
squares
path-
analysis
total
effects: a.
leadership
satisfactio
n 0.08 / b.
teacher's
work 0.90
/ c.
participatio
n 0.29
a: 2.8 / b:
8.1 / c: na
75 Driessen, G.
& Sleegers,
P.
2000 mathemati
cs,
language
consistency
between
teachers and
instructional
variables
reading and
arithmetic tests
Consensus/approa
ch intensity of
teachers has no
impact on both
outcome variables.
Attention given to
reading strategy
and intensity of
teaching approach
have a positive
impact on reading
score. No
variables are to
math scores
Netherland
s
primary
schools
567
schools,
1714
teachers,
7410
pupils
survey multi-level math: 3.2;
9.4; 9.4;
9.6; 6.2;
12.1;
10.5; 1.3;
1.1 /
language:
3.2; 9.4;
9.4; 9.6;
6.2; 12.1;
10.5; 1.3;
1.1
76 D'Agostino 2000 language,
mathemati
cs
school and
instructional
variables
initial score on
language and
reading tests
School variables
are non significant,
although in some
United
States
primary
school
Cohort 1:
134
schools,
survey multi-level
analyses
performed
language:
5b.10;
5a.5; 3.2;
95
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
(CTBS/4) and
learning rate
instances goal
consensus among
teachers (+) and
teacher
participation in
decision-making (-)
are significant. At
the teacher level
the significant
relationships are
related to teacher
led basic skills (+)
and assigning
homework (+). But
results are not
consistent. There
are different
results per cohort
and per outcome
variable.
3308
pupils
(reading)
2996
(math).
Cohort 3:
124
schools,
3203
(reading)
3203
(math)
on initial
scores and
learning
rate in
each
cohort
3.6; 5b.8;
2.3; 2.6;
7.2; 10.1;
9,6; 12.1 /
math:
5b.10;
5a.5; 3.2;
3.6; 5b.8;
2.3; 2.6;
7.2; 10.1;
9,6; 12.1
77 Hofman,
R.H.,
Hofman,
W.H. &
Guldemond,
H.
1999 mathemati
cs,
affective
Classroom
variables (social
and instructional
climate ( quality
of instruction,
opportunity to
learn), school
level variables
(parental
involvement,
achievement
oriented policy,
evaluation of
education,
emphasis on
basic skills,
mathematic test
and sense of
well-being
With regard to the
cognitive outcome
positive, significant
relationships are
reported
concerning
efficient planning
of instruction
process, clarity of
classroom rules,
parental
involvement,
emphasis on basic
skills, monitoring,
orderly climate.
Homework and
Netherland
s
primary
school
103
schools,
2023
students
survey multi-level math:
5b.3; 9.4;
9.6; 9.5;
8.2; 13;
11; 5b.7;
3.1; 1.3;
7.1; 7.2;
1.2; 6.4;
2.11; 6.3;
5a.2 /
well-
being:
5b.7;
2.11;
5a.2; 6.3
96
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
monitoring,
orderly climate)
and
administrative
variables
(governing
structure and
culture)
teacher
cooperation have a
negative impact.
Emphasis on basic
skills, orderly
climate and
monitoring have a
negative impact on
sense of well-
being. Teacher
participation in
decision-making
has a positive
impact on this
outcome.
78 Kyriakes, L.,
Campbell,
R.J. &
Gagatsis, A.
2000 mathemati
cs
Pupil, classroom
(quality of
teaching, time
on task,
opportunity to
learn) en school
level variables
(Quality of
instruction)
based upon
Creemers'
model of
educational
effectiveness
mathematics
score based on
external
assessment
(written test)
and teacher
assessment
Most variables do
not have a
significant
relationship with
the outcome
variable, except
homework
assigned (+). The
school level
variable Rules
guiding instruction
has a negative
impact
Cyprus primary
school
30
schools,
56
teachers,
1051
pupils
survey multi-level
(ml-win)
3.2
97
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
79 Douglas
Willms, J. &
Somers, M.
2001 mathemati
cs,
language,
efficiency
Classroom
practice
variables
(muiltigrade
classroom, pupil
testing, ability
grouping,
parental
involvement,
classroom
climate
(discipline),
school level
(parental
involvement)
and school
resources
mathematics,
language
(Spanish) and
time to complete
grade
Language: the
most consistent
results concern
classroom
discipline (6+ out
11), parental
involvement at
pupil-level, pupil
testing (4+ out 11).
Results are similar
for math:
classroom climate
(4+_out of 11),
pupil testing (3 out
of 11), parent
involvement (5+
out of 11).
11 South
American
countries
primary
school
Each
country
about 100
schools,
each
school 20
pupils in
grade 3
and 20
pupils in
grade 4
survey multi-level
(HLM)
math:
12.1;
12.1; 7.2;
7.2 /
language:
12.1;
12.1; 7.2;
7.2
80 Meijnen,
G.W.,
Lagerweij,
N.W. & Jong,
P.F. de
2003 language,
mathemati
cs
Instructional
variables
(opportunity to
learn, basic
cognitive skills,
structured
approach,
adaptive
instruction) and
teacher
variables
(expectations
and perceived
influence on
student
development
the most
relevant ones)
growth in
reading
comprehension,
word decoding,
mathematics
With regard to
growth in word
decoding and
reading
comprehension
only student
variables are
significant. For
math time spent on
math (+), time
spent on word
decoding (-)
Netherland
s
primary
school
28 school, 42 teachers,
282 teachers
math: 9.4;
9.4; 9.4;
9.4; 12.2;
10.5;
12.1;
5b.9; 1.3;
1.3 /
language:
9.4; 9.4;
9.4; 9.4;
12.2;
10.5;
12.1;
5b.9; 1.3;
1.3
98
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
81 Jong, R. d.,
Westerhof,
K.J. &
Kruiter, J.H.
2004 mathemati
cs
Classroom
variables (task
directedness,
meta-cognition,
quality of giving
students a turn,
advance
organisers,
instruction time,
opportunity to
learn
(Homework),
grouping,
curriculum) and
School level
variables
(Grouping
(school track),
curriculum,
departmental
rules)
mathematics
tests based
upon textbooks
(s) used by
schools
task directness,
opportunity to
learn (homework
assignments) and
departmental rules
have a positive,
significant
relationship with
math
Netherland
s
primary
school
28
schools,
56
classes,
25 pupils
on
average in
each class
survey 10.4; 9.4;
9.6; 3.2
99
Nr. Authors Year Domain School effectiveness variables
Outcome variables
Summary Direction of effects
Country School type
Nr. of students
Design Methods Sterkte van het verband
Factors
82 Muijs, D. &
Reynolds, D.
2000 mathemati
cs
Teacher
behaviours and
classroom
organisation
variables
National
Foundation for
Educational
Research's
Numeracy tests
(three tests, one
mental, one
written (A and
B)).
In total there are
seven significant
relationships
possible. Effective
teaching has a
consistent impact
on math progress.
Time on task in
two out of seven
cases. A
constructivist
approach has a
negative
relationship in one
out of seven
cases, and one
negative. Whole--
class teaching has
a positive impact in
out of seven
cases.
United
Kingdom
primary
school
16 school,
24
teachers in
year 1, 26
in year 3,
28 in year
5, 2128
pupils
survey multi-level 10.4;
10.4;
12.1; 11
100
Annex 4.2 Class size: linking teaching and learning Annevelink, E.
Enschede: s.n.
2004
Annevelink heeft bij 46 basisscholen in totaal 526 leerlingen gevolgd van groep 1 tot en met
groep 3. Zij maakt in haar onderzoek onderscheid tussen groepsgrootte, leerling-leerkracht-ratio
(LLR) en leerling-volwassene-ratio (LVR). Bij de laatste maat kunnen bijvoorbeeld ook ouders of
onderwijsassistenten een rol spelen in de klas. Dit onderscheid brengt bijvoorbeeld aan het licht
dat de groepsgrootte in groep 1 geen invloed heeft op de prestaties van de leerlingen, maar de
LLR wèl. In de hogere groepen 2 en 3 ziet de onderzoeker geen invloed op de prestaties. Dit is
verrassend en anders dan vaak wordt verwacht.
Uit het onderzoek, waarvoor de leerlingen zijn geobserveerd en toetsen zijn afgenomen voor
taalprestaties, blijkt ook niet dat leerlingen die bijvoorbeeld storend gedrag vertonen of zich
terugtrekken, aantoonbaar meer voordeel hebben van een kleinere groep. Wel maakt
Annevelink de kanttekening dat zij weinig echte achterstandscholen heeft onderzocht. Deze
scholen hebben meer mogelijkheden om hun formatie te versterken, en daar zullen meer kleine
groepen en lage leerling-leerkracht-ratio’s voorkomen.
Hoewel de prestaties niet aantoonbaar beter zijn in kleine groepen of bij lage ratio’s, zijn de
voordelen van meer interactie en 1-op-1 contacten natuurlijk wel van kracht. Gaat een leerling
naar een volgende klas en komt hij of zij daar een heel andere grootte tegen, of een hogere
LLR, dan wordt dit als complex ervaren, zeker in de lagere klassen. Ook uit ander onderzoek
blijkt dat scholen hun formatie niet altijd optimaal zetten. Het besluit van de minister om extra
formatie niet langer te oormerken voor de laagste klassen, maakt dit volgens Annevelink alleen
nog maar complexer.
Family involvement with middle-grades homework: Effects of differential prompting
Journal Of Experimental Education, 66(1), 31-48
Balli, S. J., Wedman, J. F. & Demo, D. H.
1997
A middle-grades homework intervention was investigated to determine if variations in prompting
families to be involved with mathematics homework would influence their level of involvement.
The extent to which family involvement was a predictor of student achievement in mathematics
was also examined, as were the relationships among family involvement, student achievement,
and parent education level. Families in the 2 prompted groups were significantly more involved
with mathematics homework than were families in the no-prompt group. Level of family
involvement was not significantly related to student achievement on the posttest. However,
students across the 3 groups whose parent(s) held a 4-year college degree scored significantly
higher on the posttest than did students neither of whose parents held a college degree, even
though reported levels of family involvement were nearly identical across parent education
levels. Qualitative data elicited in follow-up interviews with family members indicated that
"quality of involvement" with homework merits examination in future research.
101
Comparison of more and less effective teaching behaviors in secondary physical education
Teaching And Teacher Education, 13(2), 215-224
Behets, D.
1997
The purpose of this study was to examine teacher and pupil behaviors and to compare more
and less effective teachers in a gymnastic setting. The subjects were nine physical education
teachers and their regular class. During four consecutive 25 minute lessons a novel gymnastic
skill had to be taught. Videotaped lessons were analyzed for teacher and pupil behaviors in
three time categories: active learning time, instruction, and management. Teacher effectiveness
was measured in terms of gains in pupil learning on the gymnastic skill from pre to post test.
Results indicated that the most effective teachers scored significantly higher for active learning
time and significantly lower for instruction time. During activity the most effective teachers spent
significantly more time in observing. No major differences were found for leachers' location and
movement patterns, and for teacher feedback statements. Several significant differences were
found for the instructional variables, all indicating that the most effective teachers spent
significantly less time and attention in providing information to the pupils. This study confirms
that effective teaching is characterized by a lot of practice time and limited instruction and
management, or that physical education is 'learning by doing'. Copyright (C) 1997 Elsevier
Science Ltd.
Achievement effects of multigrade and monograde primary schools in the Turks and Caicos Islands
International Journal of Educational Development, 21(6), 537-552
Berry, C.
2001
This article reports on a study that compares the reading progress of students in multigrade
schools with the reading progress of students in monograde schools. The research was
conducted in a small island state in the Caribbean. The results of the study indicate that
multigrade schools are particularly effective at promoting the reading progress of low-achieving
students. It is hypothesised that this is partly because of differences in the approach to
instruction in multigrade and monograde classrooms. Whereas monograde classrooms tend to
be characterised by undifferentiated whole-class teaching, in multigrade classes students have
more opportunity to engage in small-group work. The implications of the findings are explored
for policy, practice and research.
Class composition and student achievement in elementary schools
American Educational Research Journal, 39(1), 207-233
Burns, R. B. & Mason, D. A.
2002
Principals and teachers sometimes intentionally use class formation procedures to create class
compositions. This study examined the class distributional properties of 200 elementary school
classes in two school districts. Fifty-six classes were combination classes-classes with students
from two grade levels. The two districts differed in their policies toward gifted and talented
classes and in principals' preferences for heterogeneous student assignment. Students
completed pretest measures of ability and independence and posttest measures of
achievement. Hierarchical linear modeling procedures were used to estimate composition
102
effects. The major findings were that (a) principals and teachers assigned higher ability and
more independent students to combination classes; (b) these purposeful assignment
procedures affected the class distributional properties and how achievement variation was
allocated within and between classes; and (c) composition effects were observed for both ability
and independence in the district with a commitment to gifted and talented classes, and these
composition effects were strongest in combination classes. It is argued that evidence on both
student assignment rules and substantive mechanisms for compositional effects are needed to
properly interpret regression coefficients relating outcome to pretest in classroom studies.
Achievement, attendance, and discipline in a year round elementary school Dissertation Abstracts International, 60-11, 5830
Consolie, P. G.
1999
Relationships among attitudes about homework, amount of homework assigned and completed, and student achievement Journal Of Educational Psychology, 90(1), 70-83
Cooper, H., Lindsay, J. J., Nye, B. & Greathouse, S.
1998
Students (n = 709), parents, and teachers (n = 82) completed a questionnaire concerning
amount of homework assigned by teachers, portion of assignments completed by students, and
attitudes about homework. Student achievement measures were also collected. Weak relations
were found between the amount of homework assigned and student achievement. Positive
relations were found between the amount of homework students completed and achievement,
especially at upper grades (6-12). At lower grades (2 and 4), teacher-assigned homework was
related to negative student attitudes. At upper grades, teachers with more positive attitudes
toward homework and those whose students performed more poorly on standardized tests
reported assigning more homework. A path analysis for lower grades indicated that class
grades were predicted only by standardized test scores and the proportion of homework
completed by students. At upper grades, class grade predictors also included parent, teacher,
and student attitudes.
The effects of modified school calendars on student achievement and on school and community attitudes
Review Of Educational Research, 73(1), 1-52
Cooper, H., Valentine, J. C., Charlton, K. & Melson, A.
2003
This review synthesizes studies of the effects of modifying the academic calendar in Grades K-
12 to do away with the long summer break while not increasing the length of the school year.
The synthesis indicated that the quality of evidence on modified calendars is poor. Within this
weak inferential framework, the average effect size for 39 school districts was quite small, d
=.06, favoring modified calendars. Studies that used statistical or matching controls revealed an
effect size of d = H. Modified calendars were associated with higher achievement for
economically disadvantaged students. Students, parents, and staffs who participated in
modified calendar programs were positive about their experiences. Policymakers can improve
acceptance of modified calendars by involving communities in the planning and by providing
103
quality intersession activities.
Economic Viewpoints in Educational Effectiveness: Cost-effectiveness Analysis of an Educational Improvement Project School Effectiveness And School Improvement, 11(3), 361-384
Creemers, B. & Werf, G. v. d.
2000
Cost-effectiveness analysis is not only important for decision making in educational policy and
practice. Also within educational effectiveness research it is important to establish the costs of
educational processes in relationship to their effects. The integrated multilevel educational
effectiveness models provide opportunities to conduct more refined cost effectiveness analyses
than were carried out in the past. A cost-effectiveness analysis of an educational improvement
project in Indonesia illustrates that combining the knowledge base and methodology of
educational effectiveness research and cost-effectiveness analysis provides fruitful possibilities
for future theoretical and practical developments in both approaches.
Instructional and School Effects on Students' Longitudinal Reading and Mathematics Achievements
School Effectiveness And School Improvement, 11(2), 197-235
D'Agostino, J. V.
2000
For this study, Prospects, a data set on schools and students in the United States collected
during the early 1990s, was used to examine the effects of instructional and school
organizational characteristics on the longitudinal mathematics and reading achievements of
students from either a first- or third-grade cohort. Three schooling models were tested using
hierarchical linear modeling (HLM) while controlling for parental socioeconomic (SES) status.
Factors and variables that represented instructional and school features were derived from
teacher and principal responses to survey items. These features had direct and interactive
effects on mathematics achievement, supporting both an environmental and interactive model
of schooling. Further, schools characterized by teacher collegiality, support for innovation,
principal leadership, goal agreement, and community support contained teachers who
employed important instructional strategies more effectively, and students who had the highest
mathematics gains over the observed period.
Improving reading achievement through the use of parental involvement and paired reading D'Angelo, N., Reents, K. & Zomboracz, C.
Chicago: Saint Xavier University
1997
The effect of schools and classes on language achievement British Educational Research Journal, 29(6), 841-859
De Fraine, B., Damme, J. v., Landeghem, G. v., Opdenakker, M.-C. & Onghena, P.
2003
This study addresses the effects of secondary schools and classes on language achievement in
Flanders, Belgium. The results of a three-level analysis (students within classes within schools)
104
indicate that the group composition at the class level is very important. In classes with a high
average initial cognitive ability or a large proportion of girls, the language achievement is higher.
These compositional effects are discussed with reference to type 'A' and type 'B' effects. The
analyses show that group composition is more important than educational practices in
accounting for differences in language achievement. With whom one is taught has a larger
impact than how one is taught. Indications of differential effectiveness of classes related to prior
achievement were found, with greater variations in effectiveness between classes for pupils of
low prior achievement.
Schoolbeleid, instructie en leerresultaten Deinum, J. F.
Groningen: GION
2000
We know that conditions at school level have an effect on achievement of students. But we
don't know how they affect that achievement. This study tries to give an answer to the question
whether conditions at school level affect activities in the classroom, how these activities
influence student achievement and if there is a direct effect of those school level conditions on
achievement.
Linking parent involvement with student achievement: Do race and income matter?
Journal Of Educational Research, 93(1), 11-30
Desimone, L.
1999
Parent involvement in children's learning at school and at home is considered a key component
of school reform, but more information is needed about how the effects of this involvement vary
for students from disparate racial-ethnic and economic backgrounds. Data from the National
Education Longitudinal Study of 1988 (Ingels, Abraham, Karr et al., 1990; Ingels, Abraham,
Resinski et al., 1990) were used to examine the relationship between 12 types of parent
involvement and 8th-grade mathematics and reading scores. Ordinary least-squares regression
indicated that statistically significant differences existed in the relationship between parent
involvement and student achievement according to the students' race-ethnicity (i.e., Asian,
Black, Hispanic, and White) and family income (i.e., low and middle), as well as according to
how achievement was measured, type of involvement, and whether it was reported by the
student or parent.
Evaluatie KEA; verslag van het kleinschalig Experiment Achterstandsbetrijding in Rotterdam 1991-2000 Doeleman, R. & Westerbeek, K.
Rotterdam: CED-Groep
2002
105
Consistency of Teaching Approach and Student Achievement: An Empirical Test School Effectiveness And School Improvement, 11(1), 57-79
Driessen, G. & Sleegers, P.
2000
This article describes the results of a study into the relations between school, teacher, class,
and student characteristics in Dutch elementary schools. Central to the study were the socio-
ethnic background of the students, socio-ethnic class composition, language and math test
results, teaching approach, and consistency of teaching approach within the school. The major
question was whether student achievement levels vary according to the consistency of the
teaching approach after controlling for socio-ethnic background at both the individual and class
levels. The sample consisted of 7,410 grade 8 students and in total 1,714 teachers from 567
schools. The results of multilevel analyses showed consistency of teaching approach to be of
no relevance to achievement levels. The most important factor appeared to be the socio-ethnic
background of the students. Ethnic minority students perform less well than native Dutch
working-class students, who in turn perform less well than the other students studied. In
addition, students in classes with a relatively high number of so-called disadvantaged students
perform less well independent of their individual socio-ethnic background.
The effect of calendar configuration on elementary students' achievement gains
Dissertation Abstracts International, 57-10, 4200
Dunn, E. R.
1996
Kleuters leren meer van voorlezen in kleine groepen
Pedagogische studiën: tijdschrift voor onderwijskunde en opvoedkunde, 74(2), 117-129
Elsäcker, W. v. & Verhoeven, L.
1997
Effects of summer home learning packets on student achievement in language arts in the middle grades
Journal of Early Adolescence, 15(114-144
Epstein, J. L., Herrick, S. C. & Coates, L.
1996
Climates for Learning Greenberg, E.
San Diego: American Institutes for Research
2004
The results presented in this paper support previous studies which found that school climate is
multi-dimensional rather than uni-dimensional Using principle component analysis, we identified
three distinct school climate factors based on school administrators’ responses to the NAEP
2000 school background survey. The questions on the NAEP school survey did not allow us to
measure other factors that might influence school climate, such as the leadership style of the
principal or the manner in which adults in the school interact with students and with each other.
106
We suspect that many of these factors may be important aspects of school climate as well.
Therefore, there may be additional independent components of school climate that were not
identified in this study.
The findings presented in this paper suggest that school climate cannot be dismissed as simply
being the product of fixed school conditions. The school characteristics we were able to
measure —school size, school poverty level, type of school, and urbanicity of the community in
which the school was located—accounted for less than half the variation in school climate
among the schools in our sample. If school climate is to some extent independent of school
characteristics, it may indeed be possible for administrators and teachers in an individual school
to positively influence the climate in that school.
The findings in this paper also suggest that further study on the relationship between school
climate and student achievement is merited. Because the data used in this paper are cross-
sectional and do not include any baseline measure of student achievement, we cannot attribute
causality to any of our findings. The connection between student achievement and school
climate may run in either direction: higher achievement levels may lead to a better school
climate or a better school climate may lead to higher achievement letters. The data used in this
paper do not allow us to distinguish between these two very different scenarios.
However, the fact that better student behavior, increased parental involvement, and higher
school morale are positively associated with student achievement in mathematics, even after
controlling for other school and student characteristics that are known to be related to student
achievement, suggests that there is at least some possibility that improving school climate may
result in an increase in student achievement. Confirming the direction of this relationship would
require conducting additional studies that start with a baseline measure of student achievement
and then relate improvements in achievement to differences in school climate.
Schools as organizational models: Implications for examining school effectiveness
Elementary School Journal, 104(1), 29-47
Griffith, J.
2003
In the present study, 4 recurrent models of organizations found in the management and
organizational psychology literatures were used to describe schools. Each model is
distinguished by its emphases on specific organizational processes, which in turn imply what
constitutes desired output and usually serves as a basis for determining organizational
effectiveness. Survey responses relating to school processes and outputs were obtained from
fifth-grade students (N = 9,431) and school staff (N = 3,291) in 117 elementary schools. To
derive school-aggregated progress in student achievement, I obtained school archival data on
student performance on standardized achievement tests for the current time period and 1 year
earlier. The open systems and human relations models provided the best-fit statistics and
explained the most variance in achievement progress, followed by the rational goal model.
Model-to-data fit statistics were similar when percentages of students enrolled in the free and
reduced-price meals system were considered. Research implications regarding what constitutes
effective schools, in addition to practical implications regarding how school staff might organize
and manage their schools, are discussed.
107
Educational leadership and pupil achievement in primary education
School Effectiveness And School Improvement, 10(4), 373-389
Grift, W. v. d. & Houtveen, A. A. M.
1999
Educational leadership can be defined as the ability of a principal to initiate school
improvement, to create a learning-oriented educational climate, and to stimulate and supervise
teachers in such a way that the latter may execute their tasks as effectively as possible. In
1989, 1993, and 1998, teacher perceptions of educational leadership of principals in Dutch
elementary education were evaluated by means of a Rasch-scale. It turned out that the
educational leadership of Dutch principals has grown considerably. In both 1989 and 1993 the
effect of educational leadership on pupil achievement was investigated. While in 1989 no
significant relationship was found, the results of 1993 show a significant relationship between
educational leadership and average pupil achievement over 3 successive years corrected for
school environment.
Instructional Climate in Dutch Secondary Education
School Effectiveness And School Improvement, 8(4), 449-462
Grift, W. v. d., Houtveen, T. & Vermeulen, C.
1997
Evaluating the promise of single-track year-round schools Haenn, J. F.
New York: 1996
School context, principal leadership, and student reading achievement Elementary School Journal, 96(5), 527-549
Hallinger, P., Bickman, L. & Davis, K.
1996
In this article, we explore the nature and extent of the school principal's effects on reading
achievement in a sample of 87 U.S. elementary schools. Our study responded to prior critiques
of the Literature in school administration by formulating and testing a multidimensional model of
principal effects on student learning. By using principal and teacher questionnaires and student
test scores, we examined relations between selected school context variables (student SES,
parental involvement, principal gender, and teaching experience), principal instructional
leadership (principal activity in key dimensions of the school's educational program),
instructional climate (school mission, opportunity to learn, teacher expectations), and student
reading achievement. Results showed no direct effects of principal instructional leadership on
student achievement. The results did, however, support the belief that a principal can have an
indirect effect on school effectiveness through actions that shape the school's learning climate.
We also found that principal leadership itself is influenced by both personal and contextual
variables (SES, parental involvement, and gender). The study confirmed the appropriateness of
viewing the principal's role in school effectiveness through a conceptual framework that places
the principal's leadership behavior in the context of the school organization and its environment
and that assesses leadership effects on student achievement through mediating variables.
108
Parental involvement in inner-city schools: The project FAST extended family approach to success
Urban Education, 33, 410-427
Hampton, F. M., Mumford, D. A. & Bold, L.
1998
School Culture and Performance: Testing the Invariance of an Organizational Model School Effectiveness And School Improvement, 7(1), 76-95
Heck, R. H. & Marcoulides, G. A.
1996
Modelling student progress in studies of educational effectiveness
School Effectiveness And School Improvement, 9(3), 310-333
Hill, P. W. & Rowe, K. J.
1998
It is argued that a crucial requirement in studies of educational effectiveness is the modelling of
change or growth in student learning. To illustrate one approach to achieving this end, results
are presented from multivariate multilevel analyses of three-waves of data for three Grade level
cohorts of students From a longitudinal study designed to explain variation in elementary school
students' progress in literacy achievement. The article provides estimates of the influence of
prior achievement and social background factors including 'critical events', on students'
progress, examines the extent to which progress can be accounted for by the grouping effects
of students within classes and schools over successive years, and provides estimates of the
effects of explanatory variables at the student- and class/teacher-levels.
The effectiveness of cohesive schools
International Journal of Leadership in Education, 4(2), 115-135
Hofman, R. H., Hofman, W. H. A. & Guldemond, H.
2001
This paper presents the findings of research into leadership and management of secondary
education using configuration theory. Based on data from almost 100 secondary schools
(school leaders and department heads), three different styles of management are distinguished.
These leadership or management styles are based on the extent to which school leaders and
departments heads make use of a set of six coordination mechanisms based on Mintzberg's
configuration theory. Multilevel analysis has been used to determine the extent to which these
leaderships styles affect student performance. Results show that one leadership style fits the
definition of an effective type of management. In schools with such a leadership style, students
reach higher achievement levels in mathematics than students in the other schools.
Social and Cognitive Outcomes: A Comparison of Contexts of Learning
School Effectiveness And School Improvement, 10(3), 352-366
Hofman, R. H., Hofman, W. H. A. & Guldemond, H.
1999
This study determines effects of social learning contexts (classroom, school and boards) on
social and cognitive outcomes of primary school pupils. Central to this research are the
109
differential effects of attending private and public schools for pupils' math achievement and
sense of well-being at school. The relationship between pupil backgrounds, sector,
characteristics of classrooms, schools and governing bodies on the one hand and pupil
outcomes, on the other, have been analyzed using multilevel analysis (VARCL). The sample
consisted of 103 schools and 2023 pupils (grade 8, age 11). After controlling for socioeconomic
backgrounds, the results show that indicators of cognitive and social effectiveness are mainly
climate factors. Basic elements of cognitive school effectiveness (math) seem counter-
productive with respect to pupils' sense of well-being at school.
Begrijpend leesonderwijs dat werkt: evaluatie van het adaptieve schoolverbeterings-project "Kwaliteitsverbetering Begrijpend Lezen" Houtveen, A. A. M.
Utrecht: ICO-ISOR
2002
Adaptief onderwijs en leerlingresultaten
Pedagogische studiën: tijdschrift voor onderwijskunde en opvoedkunde, 73(6), 422-433
Houtveen, A. A. M., Booij, N., Jong, R. d. & Grift, W. J. C. M. v. d.
1996
Adaptief onderwijs: stand van zaken in het WSNS-proces Houtveen, A. A. M., Pijl, S. J., Pijl, Y. J., Reezigt, G. J. & Vermeulen, C. J.
De Lier: Academisch Boekencentrum
1998
The impact of teacher training on student achievement - Quasi-experimental evidence from school reform efforts in Chicago
Journal Of Human Resources, 39(1), 50-79
Jacob, B. A. & Lefgren, L.
2004
While there is a substantial literature on the relationship between general teacher
characteristics and student learning, school districts and states often rely on in-service teacher
training as a part of school reform efforts. Recent school reform efforts in Chicago provide an
opportunity to examine in-service training using a quasi-experimental research design. In this
paper, we use a regression discontinuity strategy to estimate the effect of teacher training on
the math and reading performance of elementary students. We find that marginal increases in
in-service training have no statistically or academically significant effect on either reading or
math achievement, suggesting that modest investments in staff development may not be
sufficient to increase the achievement of elementary school children in high-poverty schools.
110
Empirical evidence of a comprehensive model of school effectiveness: A multilevel study in mathematics in the 1st year of junior general education in the Netherlands
School Effectiveness And School Improvement, 15(1), 3-31
Jong, R. d., Westerhof, K. J. & Kruiter, J. H.
2004
In the field of school effectiveness and school improvement, scholars as well as practitioners
often complain about the absence of theory to guide their work. To fill this gap, Creemers (1994)
developed a comprehensive model of educational effectiveness. In order to gain empirical
evidence, we tested some of the main components of the model in lessons of mathematics in
the lst year of lower general education in The Netherlands. The results show that the main
factors in the model-time spent, opportunity to learn, and the quality of instruction-are most
important in predicting achievement. No evidence was found for the relationships between
levels.
Impact of the year-round calendar on student achievement in Alameda Unified School district Kneese, C.
1996
The significance of the classroom effect in primary schools: An application of Creemers' comprehensive model of educational effectiveness
School Effectiveness And School Improvement, 11(4), 501-529
Kyriakides, L., Campbell, R. J. & Gagatsis, A.
2000
This article presents findings of an attempt to test Creemers' model of educational effectiveness
by using data derived from an evaluation study in Mathematics in which 30 schools, 56 classes
and 1,051 pupils of the last year of primary school of Cyprus participated. More specifically, we
examine whether the pupil, classroom and school variables show the expected effects on pupils'
achievement in Mathematics. Research data concerned with pupils' achievement in
Mathematics were collected by using two different forms of assessment (external assessment
and teacher's assessment). Questionnaires were administered to pupils and teachers in order to
collect data about most of the variables included in Creemers' model. The findings support the
main assumptions of the model. The influences on pupil achievement are multilevel and the net
effect of classrooms was higher than that of schools. Implications for the development of
research on school effectiveness are drawn.
School size in Chicago elementary schools: Effects on teachers' attitudes and students' achievement American Educational Research Journal, 37(1), 3-31
Lee, V. E. & Loeb, S.
2000
This study explores whether teachers and students are influenced by the size of the inner-city
elementary school to which they belong. Focusing on teachers' attitudes about their
responsibility for student learning and students' 1-year gains in mathematics achievement
scores, we used data from almost 5,000 teachers and 23,000 sixth and eighth-grade students in
264 K-8 Chicago schools. The data were collected through 199 7 surveys and annual
111
standardized tests. We employed hierarchical linear modeling (HLM) to estimate school effects.
On both outcomes, small schools (enrolling fewer than 400 students) are favored compared
with medium-sized or larger schools. in small schools, teachers have a more positive attitude
about their responsibility for students' learning and students learn more. Even after taking size
into account, learning is also higher in schools with higher levels of collective responsibility.
Thus, we conclude that school size influences student achievement directly and indirectly,
through its effect on teachers' attitudes.
Principal and Teacher Leadership Effects: a replication
School Leadership and Management, 20(4), 415-434
Leithwood, K. & Jantzi, D.
2000
Survey data from an achieved sample of 1818 teachers and 6490 students in one large
Canadian school district were used to explore the relative effects of principal and teacher
leadership on student engagement with school. Results demonstrated greater effects on
student engagement of principal as compared with teacher sources of leadership. The effects of
principal leadership were weak but significant, whereas the effects of teacher leadership were
not significant. Both forms of leadership were mediated by many of the same elements of the
school organisation.
The relationship between school climate & head teacher leadership, and pupil attainment: evidence from a sample of English secondary schools Levačić, R., Steele, F., Smees, R. & Malmberg, L.
Edinburgh: 2003
This paper reports a quantitative study of the relationship between measures of school climate
and pupil attainment. Much school improvement work and leadership training are based on the
belief that school climate affects pupil attainment. This study provides only limited support for
these beliefs. In 2002 a DfES funded pilot study was undertaken in 20 primary and 20
secondary schools in England on the relationship between pupil attainment and school
resourcing. In order to control for other factors that might affect pupil attainment questionnaires
on school climate and headteacher leadership were administered. These were closely related to
the School Climate Assessment Instrument developed by Grosin and McNamara at the
University of Stockholm and the constructs mirror closely the "eleven factor for effective
schools" derived from the literature by Sammons et al (1995). They are similar to the processes
of effective schools in Teddlie and Reynolds (1999). The headteacher leadership constructs are
also similar to those developed by Hay McBer for the Leadership Programme for Serving
Heads. There were different instruments for pupils and staff. The following school climate
constructs were derived: teachers' interest and respect teachers' pro achievement beliefs pupils'
attitudes to learning home-school relationships homework check-up rewards pupil-teacher
relationships pupil participation staff unity parental support for teachers headteacher leadership.
The school climate constructs were derived from factor analysis and included as explanatory
variables in estimating a two level mixed model of pupil attainment. Pupil level data were
supplied from the PLASC dataset giving a sample size of 2700 to 3200. Measures of prior
attainment, gender, ethnicity, EAL, and free school meals at the pupil level were included as
well as school context variables. Pupil attainment was measured by GCSE/GNQV total score
and GCSE English and maths grades and by KS3 maths, English and Science. In addition for
around 300-400 pupils measures of affective outcomes using the Australian Education
112
Research Council's instrument for the social objectives of schooling were obtained. Most of the
school climate variables were insignificant. Of the adult school climate constructs positive
student attitudes and parental support for teachers had a significant and positive relationship
with KS3 and KS4 attainment. The pupil- teacher-relationships was and positive and significant
for KS3 English and maths. The only pupil significant school climate constructs were homework
check up at KS3, pupil participation (negative at KS3). Head teacher leadership performed the
best, being significant and positive for 6 out of 7 KS3 and KS4 attainment measures. The most
interesting finding is the consistency with which headteacher leadership as assessed by the
pupils was positively related to attainment whereas the headteacher leadership constructs
reported by staff were mainly insignificant or inconsistently signed. Very little of the variance in
affective outcomes was at school level. More school climate variables were significant and
positively related to affective than to cognitive attainment. The intra-school correlations for
school climate variables were low indicating little common agreement amongst the members of
the school community in their perceptions of school climate (author abstract).
Within-Class Grouping: A Meta-Analysis
Review Of Educational Research, 66(4), 423-58
Lou, Y., Abrami, P. C., Spence, J. C., Poulsen, C., Chambers, B. & d'Apollonia, S.
1996
The effects of within-class grouping on student achievement and other outcomes were
quantitatively integrated using one set of 145 effect sizes exploring grouping versus no grouping
and a set of 20 effect sizes related to homogeneous versus heterogeneous ability grouping.
Overall, results favored homogeneous grouping. (SLD)
Effects of within-class grouping on student achievement: An exploratory model Journal Of Educational Research, 94(2), 101-112
Lou, Y. P., Abrami, P. C. & Spence, J. C.
2000
In this meta-analysis, the authors attempted to develop a parsimonious model of factors that
account for the significant variability in the findings on the effects of within-class grouping on
student achievement. Two weighted least squares regression models were tested using 103
independent findings from 51 studies at elementary through postsecondary grades. Results
indicate that the most important study features that accounted for 48% of the total variance
include outcome measure source, teacher training equivalence, grouping basis, type of small-
group instruction method, grade level, and relative ability of students. Goodness-of-fit statistics
indicate that the model fits the data and that the remaining variance may be explained by
sampling errors.
Parallel Classes: Differences and Similarities. Teacher Effects and School Effects in Secondary Schools
School Effectiveness And School Improvement, 9(4), 437-473
Luyten, H. & Jong, R. d.
1998
113
Principal Leadership and School Performance: An Integration of Transformational and Instructional Leadership
Educational Administration Quarterly, 39(3), 370-397
Marks, H. M. & Printy, S. M.
2003
Focusing on school leadership relations between principals and teachers, this study examines
the potential of their active collaboration around instructional matters to enhance the quality of
teaching and student performance. The analysis is grounded in two conceptions of leadership—
transformational and instructional. The sample comprises 24 nationally selected restructured
schools - 8 elementary, 8 middle, and 8 high schools.
In keeping with the multilevel structure of the data, the primary analytic technique is hierarchical
linear modeling (HLM). The study finds that transformational leadership is a necessary but
insufficient condition for instructional leadership. When transformational and shared instructional
leadership coexist in an integrated form of leadership, the influence on school performance,
measured by the quality of its pedagogy and the achievement of its students, is substantial.
Peer effects on student achievement: evidence from Chile
Economics Of Education Review, 22(2), 131-141
McEwan, P. J.
2003
This paper reports estimates of peer effects on student achievement, using a 1997 census of
eighth-grade achievement in Chile. The-data allow detailed measures of peer characteristics to
be constructed for each classroom within a school. The paper addresses the endogeneity of
peer variables by including school fixed effects that control for unobserved family and student
characteristics. The estimates suggest that the classroom mean of mothers' education is an
important determinant of individual achievement, though subject to diminishing marginal returns.
Additional specifications using family fixed effects are not suggestive that estimates are biased
by within-school sorting. (C) 2002 Elsevier Science Ltd. All rights reserved.
Instruction Characteristics and Cognitive Achievement of Young Children in Elementary Schools
School Effectiveness And School Improvement, 14(2), 159-187
Meijnen, G. W., Lagerweij, N. W. & Jong, P. F.
2003
This ongoing Dutch study into the school careers of young children in elementary schools,
facilitated by the Educational Priority Policy, has focused on the influence of school and class
organization on linguistic and cognitive development. In this article, the development in word
decoding, reading comprehension, and math of 3rd graders (6 years of age) is analyzed. The
data are gathered in 28 elementary schools by means of tests, questionnaires, and logbooks. A
multilevel analysis shows that for word decoding and reading comprehension differences in
development caused by differences in instruction are very small. For math, on the contrary,
there is a substantial proportion of variance at the class level. Instruction characteristics and
years of experience by the teachers seemed to be important.
114
Op weg naar adaptief onderwijs bij beginnend lezen Mijs, D., Houtveen, A. A. M. & Vernooy, C. G. T.
Utrecht: ISOR
2001
The effectiveness of the use of learning support assistants in improving the mathematics achievement of low achieving pupils in primary school Educational Research, 45(3), 219-230
Muijs, D. & Reynolds, D.
2003
The use of learning support assistants in schools has become increasingly common in England,
partly as a result of government support for this strategy. One suggested advantage of the
deployment of learning support assistants is to provide additional support to low achieving
pupils. However, so far research on the effects of this strategy is very limited. In this study the
effect of using trained learning support assistants to help pupils underachieving in mathematics
is examined using a quasi-experimental design whereby pupils receiving support were matched
to those not receiving support on prior achievement and pupil background factors.
School effectiveness and teacher effectiveness in mathematics: Some preliminary findings from the evaluation of the mathematics enhancement programme (primary) School Effectiveness And School Improvement, 11(3), 273-303
Muijs, D. & Reynolds, D.
2000
In this study the effect of teacher behaviours and classroom organisation on pupils' progress in
mathematics was studied in years 1, 3 and 5 of primary schools in the UK participating in a
mathematics intervention programme. Data on a total of 78 teachers and 2,128 pupils was
collected. Teacher behaviours were measured using a classroom observation instrument
developed for the project, and pupils were tested in March and July of 1998 using a curriculum-
appropriate Numeracy test developed by the National Foundation for Educational Research.
Background data on pupils was also collected at both testing occasions. Using multilevel
modelling techniques it was found that teacher behaviours were able to explain between 60%
and 100% of pupils' progress on the Numeracy tests. Amount of time spent teaching the whole
class was not related directly to pupils' progress, but structural equation models were tested in
which time spent teaching the whole class was found to be related to effective teaching
behaviours and thus indirectly to pupil progress. The implications of the study for British
educational policies and for educational research more generally are discussed.
Parental involvement, instructional expenditures, family socioeconomic attributes, and student achievement Journal Of Educational Research, 95(2), 110-115
Okpala, C. O., Okpala, A. O. & Smith, F. E.
2001
The influence of parental involvement, socioeconomic status of parents, and instructional
supplies expenditures on mathematics achievement scores of Grade 4 students in a low-income
county in North Carolina were examined. An educational production function framework was
used to analyze the influence of educational resources on mathematics achievement scores.
115
Pearson product-moment correlation and ordinary least squares regression were used to
determine the overall strength of each relation and the variables with the greatest impact on
mathematics achievement. Results indicated that instructional supplies expenditures per pupil
and parental volunteer hours were not statistically significant in explaining mathematics test
scores. Furthermore, results showed that the percentage of students in free/reduced-price lunch
programs was related negatively to students' academic performance in mathematics. This
finding supports the notion that economic circumstances are correlated with academic
achievement.
Effects of classroom evaluation strategies on student achievement and attitudes
Etr&D-Educational Technology Research And Development, 50(3), 61-75
Olina, Z. & Sullivan, H. J.
2002
This study investigated the effects of teacher evaluation and the combination of teacher
evaluation and student self-evaluation on student performance and attitudes. Participants in the
study were 189 Latvian high school students and their six teachers. The six teachers were
assigned to one of three treatment conditions: (a) no evaluation, (b) teacher evaluation, and (c)
self-evaluation plus teacher evaluation. All groups completed a 12-lesson instructional program
on how to conduct experiments and produce research reports. Students in the teacher-
evaluation group received teacher evaluation on their initial research reports. Students in the
self-plus-teacher evaluation group self-evaluated their reports and received teacher evaluation
on them. The no-evaluation group received no formal evaluation instructions. Students in the
teacher-evaluation and the self-plus-teacher evaluation groups received significantly higher
ratings on their final projects than those in the no-evaluation group. However, the no-evaluation
group had more favorable attitudes toward the program than the other two groups, while the
self-plus-teacher evaluation group was significantly more confident of their ability to
independently conduct future research experiments.
Relationship between school composition and characteristics of school process and their effect on mathematics achievement British Educational Research Journal, 27(4), 407-432
Opdenakker, M. C. & Damme, J. v.
2001
This study explores the relationship between school composition and characteristics of school
process and investigates their effect on mathematics achievement in Belgian (Flemish)
secondary education by means of multilevel analysis. Attention is paid to the differential
effectiveness of both types of school characteristics. The study confirms that there are important
relationships between school composition and school process variables in secondary education.
The analyses of the effect of both variables on achievement revealed that these variables have
important net and joint effects on achievement independent of initial ability. We found that the
addition of school composition variables to models with school process variables caused a
decline in the effect of important school process variables. This outcome has important
consequences for school effectiveness research, school improvement and teacher training.
116
Adaptief onderwijs in PRIMA-I: stand van zaken en effecten Overmaat, M., Ledoux, G. & Koopman, P.
Amsterdam: SCO-Kohnstamm Instituut
1997
The impact of the elementary school year-round education pilot program on student achievement and behavior in the Harlandale Independent School District Dissertations Abstracts International, 58-07, 2539
Paloczy, S. T.
1997
Capital at home and at school: Effects on student achievement Social Forces, 79(3), 881-911
Parcel, T. L. & Dufur, M. J.
2001
We investigate the effects of both family and school capital on student math and reading
achievement. We use the National Longitudinal Survey of Youth (NLSY) merged Child-Mother
Data for 1992 and 1994, to which indicators of capital in the children's schools for 1993-94 and
1994-95 have recently been added. We study children who attended first through eighth grades
in both 1992 and 1994, with samples of 2034 for math achievement and 2203 for reading
recognition. Findings suggest that school capital effects are modest in size while family capital
effects are stronger; combinations of school and family capital boost or modify additive findings.
We sketch directions for future research and discuss the usefulness of analyzing school and
family capital as parallel concepts.
What makes schools effective? A comparison of the relationships of communitarian climate and academic climate to mathematics achievement and attendance during middle school American Educational Research Journal, 34(4), 633-662
Phillips, M.
1997
Over the past few decades, scholars and policymakers have been perplexed about why
students learn so little in some schools. Many researchers and reformers currently claim that
school effectiveness hinges on communal organization. They contend that shared values and
activities, positive adult social relations, positive teacher-student relations, and democratic
governance enhance students' school engagement and their academic achievement. Yet a
competing theory-that of academic press-posits a more direct link between school processes
and academic outcomes. This theory suggests that schools are effective when they offer
demanding curricula and employ teachers whose educational expectations for their students are
high. The present article used hierarchical modeling to compare the merits of these two
theories. Analyses of longitudinal data on three cohorts of students (N > 5,600) from 23 middle
schools indicated that communal organization was not related to mathematics achievement or
attendance. Academic press, on the other hand, was positively related to both mathematics
achievement and attendance. These results call for somewhat greater skepticism about the
currently popular communitarian solution to school ineffectiveness.
117
Attainment effects of school enmeshment with external communities: community policy, church/religious influence, and TIMSS-R mathematics scores in Flemish secondary schools Pugh, G. & Telhaj, S.
Hamburg: 2003
This paper analyses school enmeshment effects, which we define as attainment effects arising
from either the voluntary or structural position of schools in external networks. We use a unique
dataset on Flemish secondary school students from the 1999 repeat of the Third International
Mathematics and Science Study to estimate an educational production function. Our estimates
of enmeshment effects suggest attainment benefits 1. when school principals take responsibility
for community relations, and 2. when schools are influenced by certain groups in the wider
community; in particular, by faith communities rather than by trade unions or business groups.
These enmeshment effects are consistent with the literature on social capital as well as with
recent developments in the economics of identity, while the second group of results contributes
to the literature on the attainment effects of faith schools (in particular, of Catholic schooling). In
addition, other results indicate possibilities for improving attainment in all schools (author
abstract).
Retention of academic skills over the summer months in alternative and traditional schools
Journal of Research and Development in Education, 33, 166-174
Reece, J. L., Myers, C. L. & Nofsinger, C. O.
2000
Ontwikkelingen in en effecten van adaptief onderwijs in de klas en integrale leerlingenzorg op schoolniveau Reezigt, G. J., Houtveen, A. A. M., Grift, W. v. d. & Guldemond, H.
Groningen: GION
2002
Het effect van computer-ondersteunde oefeningen in klanksynthese bij kleuters
Pedagogische studiën: tijdschrift voor onderwijskunde en opvoedkunde, 74(1), 1-20
Reitsma, P., Wesseling, R. & Stiva, F.
1997
Research on School Attendance and Student Achievement: A Study of Ohio Schools
Educational Research Quarterly, 28(1), 3-14
Roby, D. E.
2004
Examines the relationship between school attendance and achievement among grades four,
six, nine and twelve in six of the largest school districts in Ohio. Significance of academic
standards and expectations; Significance of student learning time loss; Recommendation for
further study on student absenteeism.
118
Modeling Educational Effectiveness in Classrooms: The Use of Multi-Level Structural Equations to Model Students' Progress
Educational Research and Evaluation, 4(4), 307-347
Rowe, K. K. J. & Hill, P. W.
1998
Long-standing and enduring problems in quantitative studies of educational effectiveness relate
to fitting models that adequately reflect the complex inter-relationships among multivariate,
multilevel factors affecting students' educational progress, particularly among those that operate
within classrooms. This article illustrates one approach to solving such difficulties by combining
the analytic approaches of multilevel analysis and structural equation modelling in a two-stage
process. The data used are drawn from a longitudinal study of teacher and school effectiveness
for three Grade-level cohorts of 4,558 students clustered within 334 class/teacher groups in 52
elementary schools. The article provides estimates in inter-relationships among achievement
and social background factors including "critical events' on students' progress in mathematics,
together with their affective schooling experiences and inattentive behaviours in the classroom.
Findings are represented indicating the extent to which progress can be accounted for by the
grouping effects of students within classes and schools, highlighting the importance of
accounting for the interdependent effects among variables. Both the substantive and
methodological implications of the findings are discussed.
ICT-ondersteuning van de woordenschat van allochtone kleuters
Pedagogische studiën: tijdschrift voor onderwijskunde en opvoedkunde, 78(5), 287-297
Segers, E., Verhoeven, L., Boot, I., Berkers, I. & Vermeer, A.
2001
Allochtone kinderen in Nederland hebben een aanzienlijke achterstand in woordenschat ten
opzichte van hun autochtone leeftijdgenoten, die in de loop van de basisschoolperiode eerder
groter wordt dan kleiner. In de onderwijspraktijk bestaat grote behoefte aan effectieve
interventieprogramma’s. In dit artikel worden twee interventiestudies beschreven waarin wordt
nagegaan of de inzet van een adaptief woordenschatprogramma op de computer positieve
effecten heeft op de woordenschatontwikkeling van allochtone kleuters. Aan de eerste studie
namen 25 oudste allochtone kleuters deel, aan de tweede studie 30 jongste allochtone kleuters.
Beide groepen kinderen bleken na een training van 75 tot 90 minuten een significante
vooruitgang te hebben geboekt. De oudste kleuters gingen meer vooruit dan de jongste
kleuters. Uit de tweede studie bleken zowel voorkennis als hoeveelheid gedane oefeningen op
de computer van invloed te zijn op de leerwinst. De belangrijkste conclusie van het artikel is dat
jonge kinderen in staat zijn om zelfstandig hun woordenschat uit te breiden met een adaptief en
interactief computerprogramma dat gerichte hulp en feedback geeft.
Effects of classroom structure on student achievement goal orientation
Journal Of Educational Research, 97(2), 106-111
Self-Brown, S. R. & Mathews, S.
2003
The authors assessed how classroom structure influenced student achievement goal orientation
for mathematics. Three elementary school classes were assigned randomly to I classroom
structure condition: token economy, contingency contract, or control. Students in each condition
were required to set individual achievement goals on a weekly basis. The authors assessed
differences in goal orientation by comparing the number of learning vs. performance goals that
119
students set within and across classroom structure conditions. Results indicated that students in
the contingency-contract condition set significantly more learning goals than did students in
other classroom structure conditions. No significant differences were found for performance
goals across classroom structure conditions. Within classroom structure conditions, students in
the contingency-contract group set significantly more learning goals than performance goals,
whereas students in the token-economy condition set significantly more performance goals than
learning goals.
Schools as learning organisations: The case for system, teacher and student learning
Journal of Educational Administration, 40(5), 425-446
Silins, H. & Mulford, B.
2002
An Australian government-funded four-year research project involving 96 secondary schools,
over 5,000 students and 3,700 teachers and their principals has provided a rich source of
information on schools conceptualised as learning organisations. The LOLSO project focused
on three aspects of high school functioning: leadership, organisational learning and the impact
of both on student outcomes. This research has established a relationship between the system
factors of leadership and organisational learning and student outcomes as measured by student
levels of participation in and engagement with school. This paper summarises this research and
reports on a study that empirically tests the relationship between students' participation in and
engagement with school and student achievement using model building and path analysis. The
importance of learning at the system, teacher and student level is discussed in the context of
school restructuring.
Pupil Performance, Absenteeism and School Drop-out: A Multi-dimensional Analysis
School Effectiveness And School Improvement, 10(4), 480-502
Smyth, E.
1999
This article assesses whether second-level schools in Ireland, typically covering pupils 12 to 18
years of age, are equally effective in relation to three different outcomes: examination
performance, absenteeism and potential drop-out among pupils. The article uses data from a
large-scale survey of second-level pupils in 116 schools in Ireland. Analysis is restricted to one
cohort: pupils aged 15-16 years who took a nationally standardised examination, the Junior
Certificate, in 1994. Multivariate multi-level modelling techniques are used to assess the impact
of pupil background and schooling factors on overall examination performance, on absenteeism
levels and on intentions to leave school after the exam. Some consistency is found among
these different dimensions of school effectiveness: pupil absenteeism and potential drop-out
rates are lower in schools which enhance academic progress among pupils. These outcomes
are associated with more positive teacher-pupil relations and a more positive academic climate
within the school.
Good beginnings: What difference does the program make in preparing young children for school?
Journal Of Applied Developmental Psychology, 19(1), 41-66
Stipek, D. J., Feiler, R., Byler, P., Ryan, R., Milburn, S. & Salmon, J. M.
1998
Cognitive competencies and motivation were assessed at the beginning and the end of the year
120
for 228 preschoolers and kindergartners and again at the end of the next year (kindergarten or
first grade) for 93 of the participants. Participants were in classrooms classified as either
emphasizing basic skills in a less positive social climate or de-emphasizing basic skills in a
more positive social climate. Cognitive competencies were assessed with two achievement
tests (one for letters/reading and another for numbers/math) and six subscales from the
McCarthy test. Motivation (perceptions of competence, attitudes toward school, anxiety, affect,
risk taking, expectations for success, independence, and persistence) was assessed in an
experimental setting and by observing children's behavior in their classroom. The results
showed primarily negative effects on both cognitive and motivation outcomes of preschool
programs emphasizing basic skills using structured, teacher-directed approaches in a relatively
negative social climate. For kindergartners both positive and negative achievement and
motivation outcomes were associated with both types of classrooms.
School characteristics and educational outcomes: Toward an organizational model of student achievement in middle schools
Educational Administration Quarterly, 36(5), 703-729
Sweetland, S. R. & Hoy, W. K.
2000
Empowerment is defined and measured in terms of teachers' power to control critical decisions
about leaching and learning conditions. This research first considers the relationship, between
school climate and teacher empowerment and then the relationship between teacher
empowerment and school effectiveness, which includes measures of mathematics and reading
achievement in 86 middle schools. The results support the pivotal importance of teacher
empowerment in rite effectiveness of schools. Finally, a theoretical model is proposed to explain
the linkages between organizational characteristics and student achievement.
Reading achievement in New Zealand: effects of parental self-efficacy and children's motivation Townsend, M. & Choi, S. F.
Manchester: 2004
This study examined the combined effects of parental self-efficacy and children's motivation for
reading on children's reading achievement. Self-concept and reading value were assessed in 83
children in Year 4 classes (aged 8-9) in New Zealand. A measure of reading achievement was
obtained from school records. Parents of the children completed a self-efficacy scale, and a
subsample of parents participated in an interview. Parental self-efficacy was significantly
associated with children's reading achievement, and this association was additive to the effects
of children's motivation (author abstract).
Do homework assignments enhance achievement? A multilevel analysis in 7th-grade mathematics
Contemporary Educational Psychology, 27(1), 26-50
Trautwein, U., Koller, O., Schmitz, B. & Baumert, J.
2002
In discussions of possible remedies for educational deficits in Western industrialized countries,
the issue of homework frequently attracts considerable attention, although there is still a lack of
strong empirical support for the homework-achievement relation. In the present study, repeated-
measurement data collected from 1976 German 7th-graders in 125 classes were analyzed to
121
investigate the role of homework in enhancing mathematics achievement. Intelligence, SES,
motivation, and type of secondary school were controlled. The frequency of homework
assignments had a positive effect on math achievement gains, whereas lengthy homework
assignments had a negative, albeit nonsignificant, effect on achievement gains, However, the
effect of homework length interacted significantly with individual achievement level, suggesting
that extensive homework assignments tended to reduce intraclass achievement variability.
Monitoring of homework completion did not contribute significantly to achievement gains,
Methodological implications for homework research are discussed. (C) 2001 Elsevier Science
(USA).
School effectiveness: The underlying dimensions
Educational Administration Quarterly, 34(4), 462-483
Uline, C. L., Miller, D. M. & Tschannen-Moran, M.
1998
This article explores two underlying dimensions of school effectiveness: instrumental functions
and expressive functions. The study explores Mott's index of perceived organizational
effectiveness as a concise measure that captures both of these underlying dimensions.
Attention to these underlying dimensions helps to make manageable this multifaceted and
complex construct. A model of school effectiveness that accounts for both is presented and
tested.
Effects of single-sex versus co-educational classes and schools on gender differences in progress in language and mathematics achievement British Journal Of Sociology Of Education, 25(3), 307-322
Van de Gaer, E., Pustjens, H., Damme, J. v. & Munter, A. d.
2004
In this study, the effects of single-sex versus co-educational classes and schools on the
progress in language and mathematics of boys and girls at the end of the second year of
secondary education are investigated. Data from the Longitudinaal Onderzoek Secundair
Onderwijs project are used. Multilevel analyses were carried out on a sample of approximately
4000 pupils, 330 classes (190 single-sex), 180 teachers and 50 schools (20 single-sex). The
results indicate that for boys the gender composition of the classes has more impact than the
gender composition of the schools, whereas for girls the gender composition of the schools is
more important. Boys make more progress for language (and not for mathematics) in co-
educational classes even after we have taken into account the selective nature of the classes.
Girls, on the other hand, make more progress for mathematics (but not for language) in single-
sex than in co-educational schools.
Retentie-effecten van een nascholingsprogramma voor effectieve instructie en klassemanagement Pedagogische studiën: tijdschrift voor onderwijskunde en opvoedkunde, 73(5), 357-371
Veenman, S. & Raemaekers, J.
1996
122
Leerlingen, klassen, scholen Veenstra, D. R.
S.l.: s.n.
1999
Interactive homework in middle school: Effects on family involvement and science achievement Journal Of Educational Research, 96(6), 323-338
Voorhis, F. L. v.
2003
The purpose of this intervention study was to examine the effects of weekly interactive science
homework on family involvement in homework, student achievement, and homework attitudes.
Sixth- and 8th-grade students (N = 253) participated in the 18-week study. Six classes of
students completed TIPS (Teachers Involve Parents in Schoolwork) assignments with directions
for family and parent involvement. Four classes completed non-interactive homework (no family
involvement directions). Interactive students reported significantly higher levels of family
involvement than did non-interactive students. Students in both groups who more regularly
involved family members completed more assignments; TIPS students turned in more accurate
assignments than did non-TIPS students. TIPS students also earned significantly higher
science report card grades. The findings of this study support the hypothesis that TIPS
interactive homework affects family involvement in homework, science attitudes, and student
achievement in the middle grades.
Differences in School and Instruction Characteristics between High-, Average-, and Low-Effective Schools
School Effectiveness And School Improvement, 8(4), 430-448
Werf, G. v. d.
1997
Evaluation of school improvement through an educational effectiveness model: The case of Indonesia's PEQIP Project Comparative Education Review, 44(3), 329-355
Werf, G. v. d., Creemers, B., Jong, R. d. & Klaver, E.
2000
Family, classroom, and school effects on children's educational outcomes in Latin America
School Effectiveness And School Improvement, 12(4), 409-445
Willms, J. D. & Somers, M. A.
2001
This study, which is based on a longer report by Willms and Somers (2000), employs
hierarchical linear regression models to examine the relationships between 3 schooling
outcomes (language and mathematics achievement, and time to complete primary schooling)
and family background, as well as various school policies and practices. The analyses employ
123
data from the Primer Estudio Internacional Comparativo (PEIC; UNESCO, 1998), which
includes data for 13 Latin American countries, with samples of approximately 100 schools in
each country, and 40 grades 3 and 4 pupils sampled in each school. The study finds that the
relationship between schooling outcomes and family background varies among countries. The
most successful country, Cuba, has uniformly effective schools, and relatively small inequities
along social class lines and between the sexes. Across all countries, the most effective schools
tend to be those with: high levels of school resources; classrooms which are not multigrade, and
where students are not grouped by ability; classrooms where children are tested frequently;
classrooms and schools with a high level of parental involvement; and classrooms that have a
positive classroom climate, especially with respect to classroom discipline. The article
concludes with a discussion about how we might improve capability to monitor school
performance in low-income countries.
The effects of schooling on gender differences
British Educational Research Journal, 28(6), 827-843
Wong, K. C., Lam, Y. R. & Ho, L. M.
2002
The study on which this article is based examined the gender differences in educational
achievements based on a longitudinal sample of more than 45,000 secondary school students
in Hong Kong who took a public examination in 1997. The results coincided with the findings
from recent British studies that boys did less well than girls in all areas of the school curriculum.
The multilevel analyses of the effects of schooling, after controlling for initial ability, indicated
that schooling did have an effect on gender differences. Girls achieved better results studying in
single-sex schools whereas boys achieved better in co-educational schools. Compared with
other students, it was those boys studying in the arts stream that did the least well in the public
examination. The results are discussed in the context of the methodology of investigating
gender differences and of the substantive questions of school effectiveness.
Rural and Urban Differences in Student Achievement in Science and Mathematics: A Multilevel Analysis
School Effectiveness And School Improvement, 9(4), 386-418
Young, D. Y.
1998
Findings from an evaluation of the Parent Institute for Quality Education Parent Involvement Program Zellman, G., Stecher, B., Klein, S. & McCafrey, D.
Santa Monica: RAND
1998
Naar effectieve schoolverbetering: resultaten van het onderzoek naar de effecten op leerkracht- en leerlingniveau van het adaptief onderwijsproject "Kwaliteitsversterking Rekenen en Wiskunde" Zoelen, L. v. & Houtveen, A. A. M.
Utrecht: ISOR
2000
124
125
Chapter 5
The indicators on teaching and learning compared to the
review of recent research articles on instructional
effectiveness
Introduction
This chapter is based on a review of the literature on instructional effectiveness. This
implies that malleable variables at classroom level, usually controlled by teachers are
at the focus of attention. A broader set of effectiveness criteria is considered in this
chapter, in the sense that, apart from cognitive outcomes, also non-cognitive outcomes
and learning processes are considered. The research results are presented in terms of
proportion of significant effects, detailed overviews for each research article and
abstracts of the articles.
The indicators on teaching and learning compared to the review of recent research
articles on instructional effectiveness
The goal of this chapter is to present the review of research articles on instructional
effectiveness in the last decade (1995-2005) and to position the Dutch Supervision
Framework with regard to the current knowledge base on instruction effectiveness. In
the previous chapter the knowledge base on instructional and school effectiveness has
been pointed out and main instructional factors have been summarized. Thereby, the
elementary design of school and teaching effectiveness research is the association of
hypothetical effectiveness enhancing conditions of schooling and output measures (see
Figure 5.1, cf. Chapter 5).
126
Figure 5.1: A basic systems model on the functioning of education
In this chapter the black box of the classrooms is “opened” in order to show which
teaching factors work next to the impact of context, input and school level throughput.
Furthermore, the previous research was mainly restricted to cognitive outputs. The
present review is going to expand the definition of outputs by additionally relating
teaching conditions to the quality of learning processes as well as non-cognitive outputs
of schooling (Artelt, Baumert, McElvany, & Peschar, 2003; Bransford, Brown, &
Cocking, 2000; OECD, 2003).
In order to give a differentiated picture on teaching conditions and their effects on
student learning the basic systems model is going to be elaborated to a more
differentiated model on instruction effectiveness (Figure 5.2):
Figure 5.2: Model of instruction effectiveness
Teacher professional
competencies
Instruction characteristics
Learning processes
Cognitive & non-cognitive
outputs
Classroom context
context
outputs inputs Process or throughput
School level Classroom level
127
Thereby, four components are distinguished: a) teacher professional competencies as an
input measure, b) instruction characteristics as a learning enhancing measure, c) the
quality of learning processes as a more situative and proximal student throughput
measure, d) cognitive and non-cognitive student outputs as more stable and long-term
student measures. The components are summarised in Table 5.1 and outlined in the
following section of the chapter.
Table 5.1: Components of the instruction effectiveness model
Components Description
Teacher professional
competencies
Teacher characteristics that are assumed to have a direct effect
in the actual teaching situation, e.g. high expectations for
students, beliefs about the nature of learning
Instruction characteristics Broad variety of instruction characteristics as reported in the
research articles on instructional effectiveness. Instruction
characteristics are going to be grouped according to the Dutch
inspectorate’s observation schedule
Quality of learning processes Student throughput measures with a focus on the regulation of
learning activities in the process of knowledge acquisition, e.g.
students’ cognitive engagement, quality of learning motivation,
application of deep learning strategies
Cognitive and non-cognitive
outputs
Student output measures with a focus on stable student
characteristics. Cognitive outputs comprise the students’
achievement, competencies, performance. Non-cognitive
outputs comprise the students’ interest in domains, their
attitudes and beliefs as well as their self-concept of ability.
Teacher professional competencies
Teacher professional competencies are not outlined by the inspectorates in their
instruction observation schedule. From a methodological point of view, teacher
characteristics are difficult to observe in classroom situations in a reliable and objective
manner. However, the review has shown that there is a tendency in recent studies to
investigate the role of professional competencies of teachers for student learning.
Currently, the assessment of teacher competencies could be described as rather “soft”.
The conceptualisations are heterogeneous and definitely need further theoretical and
methodological specification. However, we chose two variables for the review that
might be relevant for the Inspectorate’s observation schedule and that are
methodologically feasible to observe: high expectations for students and beliefs about
the nature of learning.
128
Instruction characteristics
The research on instructional effectiveness comprises a broad variety of different
instruction characteristics. In the previous chapter several possible ways of organising
and grouping instruction characteristics have been presented (Brophy & Good, 1986;
Creemers, 1994; Doyle, 1986; Fraser, Walberg, Welch & Hattie, 1987; Scheerens,
2000; Scheerens & Bosker, 1997). Thereby, the organisation of instruction
characteristics does not directly match with the framework on teaching and learning
processes as outlined by the Dutch inspectorate’s observation schedule. However, due
to the fact that the framework is based on the results of instruction effectiveness
research (Scheerens & Bosker, 1997) a large overlap is given. Thus, the categories as
outlined by the Dutch Inspectorate’s observation schedule are going to represent the
underlying framework for the review. In Table 5.2 the instruction characteristics of the
Dutch inspectorate’s observation schedule are given and the indicators as found in the
current literature review are integrated into this schedule.
Thereby, the instruction characteristics have to be regarded as a kind of “syndrome” or
teaching pattern (Baumert, Blum & Neubrand, 2002; Klieme & Rakoczy, 2003; Seidel,
2003). Thereby, we do not use the term “syndrome” in the medical sense of the word. In
fact, in instruction research a “syndrome” indicates a complex of concurrent teaching
indicators that are circled around a key idea of instruction. Teaching “syndromes”
implicate that instruction characteristics are interrelated, both with respect to the
simultaneous occurrence of instruction characteristics as well as with respect to the
sequencing of characteristics in the process of teaching (Seidel, 2003).
In the following, the key ideas of instruction characteristics as they were found in the
research articles of the last decade are going to be outlined. Thereby, the description of
the Inspectorate framework is used as a basis for describing the key ideas as they were
reported in empirical studies on instruction effectiveness. Since the reported teaching
aspects were rather heterogeneous, we decided on the maximum likelihood of
correspondence to one of the instruction characteristics. Some teaching aspects might
not be associated instantly with the given instruction characteristics. However, we
worked with the idea of teaching “syndromes” and, thus, we argue that the combination
of different teaching aspects should give the best possible description.
(1) The first instruction characteristic refers to learning time and deals with the fact that
instruction has to offer a maximum of time for students to engage in learning activities,
both with respect to instruction time in classrooms as well as at home. Teaching aspects
as they were reported in the research articles on instruction effectiveness were: time on
task, effective use of teaching time, opportunity to learn, homework and mastery
learning.
(2) The second instruction characteristic is classroom organisation. Teaching aspects
that refer to the creation of an organized learning situation are grouped within this
129
category. Teaching factors such as classroom management, discipline and control are
summarized.
(3) Next to the creation of an organized learning situation, the emotional quality of the
instructional setting is taken into consideration. The third characteristic orderly and
functional learning environment comprises factors such as learning climate, classroom
climate, achievement pressure, mastery-orientation, performance orientation. Learning
and classroom climate implicate the necessity for a positive emotional climate between
students and teachers. Achievement pressure is a category used in several German
studies (Baumert & Köller, 2000; Clausen, 2001; Klieme & Rakoczy, 2003; Kunter,
2004) and means that the learning climate is dominated by the pressure of a teacher on
the students to perform well in class. In consequence, instruction situations with the
goal on learning turn into achievement situations and, in turn, students aim to perform
well and/ or better than the others in class (Schulmeiß, 2004; Seidel & Prenzel, 2003).
This teaching aspect is also investigated in studies with a theoretical focus on goal
orientation (Anderman et al., 2001; Nolen, 2003). Thereby, mastery and performance
orientation are distinguished. Classrooms with a focus on mastery orientation represent
learning situations with the goal to master und deeply understand the contents taught in
class. Performance oriented classrooms have a higher degree of competition between
students and encourage students to focus on best performances. The two teaching
aspects of achievement pressure and performance orientation represent the category
“orderly and functional learning environment” in a reverse order. Thus, the results of
these studies have been recoded for the review in the sense of low achievement pressure
and low performance orientation.
(4) The fourth category is clear and structured teaching and comprises well established
teaching aspects such as direct teaching, structured teaching, clarity of instruction, goal-
directed teaching (Rosenshine, 1979). Teaching for basic skills, a concept that is closely
related to the syndrome of clear and structured teaching, has been added to this
category.
Activating is a teaching syndrome that is centred around the idea of offering students
multiple opportunities for active learning (Slavin, 1995). In this sense, the fifth category
comprises teaching aspects such as cooperative learning, situated learning, discovery
learning, peer-tutoring, student experiments, hands-on activities, group work, individual
work, individual learning, and student discussions.
(6) The sixth category learning to use learning strategies refers to the goal of self-
regulation and meta-cognition (Artelt et al., 2003). The research articles of the last
decade include the learning of a variety of learning strategies. Thereby, the research
articles are focused on the teaching of domain specific or specialized learning strategies.
Overall, the category includes studies on teaching cooperative learning strategies,
problem-solving, meta-cognitive strategies, scientific inquiry, thinking aloud, concept
130
mapping strategies, organising/structuring methods, language acquisition strategies,
phonemic awareness, reading strategies, writing strategies, and finally general learning
strategies.
Table 5.2: Instruction characteristics as outlined by the Dutch Inspectorate’s
observation schedule and indicators as given by the review on instruction
effectiveness for the years 1995-2005
Instruction characteristics Indicators
(1) Learning time Time on task, effective use of teaching time, opportunity to learn, homework, mastery learning
(2) Classroom organisation Classroom management, discipline, control
(3) Orderly, functional learning environment
Learning climate, classroom climate, achievement pressure*, Mastery-orientation, performance orientation*
(4) Clear and structured teaching Direct teaching, structured teaching, teacher demonstrations, teaching for basic skills, clarity
(5) Activating Cooperative learning, situated learning, discovery learning, peer-tutoring, student experiments, hands-on activities, group work, individual work, individual learning, discussions
(6) Learning to use learning strategies
Cooperative learning strategy training, problem-solving, meta-cognitive training, scientific inquiry training, thinking aloud training, concept mapping, organising/structuring methods, language acquisition training, phonemic awareness training, reading strategies, writing strategies, formal learning strategy training
(7) Challenge Cognitive activation, orientation towards understanding, active student engagement, authentic contexts, relevance to students, language level, varying representation formats
(8) Support (mutual respect) Quality of teacher-student interactions, student-student interactions, teacher support
(9) Feedback / monitoring Feedback, monitoring, individual frame of reference
(10) Evaluation of goals / attainment
Assessments, tests
* reverse coded for summary
131
(7) Challenge represents a syndrome that is concerned with the teaching goal to provide
a cognitively activating and challenging learning environment (Arnold, Fisher,
Doctoroff & Dobbs, 2002; Baumert et al., 2002; Bund-Länder-Kommission für
Bildungsplanung und Forschungsförderung, 1997; Cantrell, 1999; Gruehn, 1995;
Klieme & Reusser, 2003; Prenzel, 2000; Prenzel & Ostermeier, in press; Seidel, 2003).
In this sense challenge means an instruction that is centred around the aim of activating
student thinking. Teaching aspects such as higher order thinking, cognitive activation,
deep understanding of content, meaningful contexts, authentic instruction, relevance of
contents, appropriate and high level language, variation of different representation
formats have been summarized as providing cognitive challenge for students.
(8) Whereas challenge can be regarded as a current trend in the research articles of the
last decade the eighth category support and mutual respect represents an instruction
characteristic that has already been investigated rather intensively. Articles that
addressed the quality of teacher-student interactions, student-student interactions, and
teacher support have been summarized for this category.
(9) Furthermore, the Dutch inspectorate’s observation schedule refers to two
characteristics with a focus on monitoring and evaluating student learning, as well as
giving feedback to students in order to achieve teaching and learning goals. In the
review on instruction effectiveness these two indicators have been grouped as
feedback / monitoring because current research studies theoretically outline the
scaffolding of student learning by monitoring, evaluation and feedback as a strongly
interrelated teaching syndrome (Bolhuis, 2003; Kobarg, 2004).
(10) Finally, instruction characteristics that address the evaluation of goals and
attainment in instruction have been grouped in the last category. A small number of
studies have been addressing the question of evaluation in the last decade. They mainly
concentrated on issues of assessment and tests and their role in the evaluation process
(cf. Wenglinsky, 2002). Thus, a narrow conceptualization of this concept has taken
place which leaves room for further theoretically more elaborated research.
Next to the instruction characteristics as outlined by the Dutch inspectorate’s
observation schedule the review showed additional characteristics. They are listed in
Table 5.3 and comprise the concept of adaptive teaching, the quality of material used,
the role of practice and the investigation of “integrated instructional concepts”.
(11) Studies in which adaptive teaching has been addressed are centred around the idea
that teaching has to be adaptive to the students’ pre-requisites. Thus, indicators such as
the variable use of teaching methods, adaptive teaching, orientation towards individual
learning processes, provision of choice, and taking into account student pre-requisites
have been grouped for the category adaptive teaching. Thereby, the theoretical
132
conceptualisation of these studies has been rather heterogeneous and further elaboration
of this concept is required.
Table 5.3: Additional instruction characteristics given by the review on instruction
effectiveness for the years 1995-2005
Instruction characteristics Indicators
(11) Adaptive teaching Variable teaching methods, adaptive teaching, orientation towards individual learning processes, choice, taking into account student pre-requisites
(12) Practice Drill, repetitions, applications
(13) Material Quality of curriculum, textbooks, use of computers
(14) “Integrated” instructional concepts
Constructivist instruction, inductive teaching, concept-oriented/integrated instruction
(12/13)The quality of material comprises teaching aspects such as the use of textbooks
and computers as well as in some cases the quality of the curriculum. Practice
summarizes the role of repetition, drill and simple applications for student learning.
(14) Finally, the review has shown that research studies have investigated the effects of
so-called “integrated” instructional concepts, meaning that the quality of teaching is
regarded as an interrelated syndrome of several instruction characteristics. The
integrated concepts are, therefore, investigated as a concept rather than as a composition
of single teaching aspects. This category comprises studies with a focus on
constructivist instruction, inductive teaching, and concept-oriented instruction. From a
theoretical point of view these studies are rather heterogeneous. Therefore, the review is
going to list the effects separate for each concept.
Student throughput: Quality of learning processes
With respect to the quality of learning processes we summarize throughputs that have
been proved as relevant for the process of knowledge acquisition, such as attention,
selection, and integration of new contents in knowledge structures, as well as self-
determined learning motivation and the application of deep learning strategies (Artelt et
al., 2003).
133
Cognitive and non-cognitive student outputs
Non-cognitive and cognitive outputs refer to the results of learning and represent more
stable and cumulative learning developments. Non-cognitive outputs comprise
motivational, affective and conative developments of students such as the development
of stable interests, motivational orientations, attitudes or belief systems. Cognitive
outputs refer to results of learning with respect to the development of knowledge,
measured either by standardized achievement and competency tests or specific tests on
content understanding or student performance.
Empirical evidence for the effectiveness of instruction characteristics
In the next section of the chapter the results of the review on research articles on
instructional effectiveness in the last decade (1995-2005) are presented. First the
literature search methods and the inclusion criteria are outlined. Second, we present a
summary on the results of the qualitative review. Third and last, quantitative findings on
the consistency of findings on instruction effectiveness are outlined.
- Literature Search Methods
Broad searches of the literature on instruction effectiveness were conducted. The
methods included searches on the Web of Science, ERIC and ERA database, ranging
from the years 1995-2005. Furthermore, recent reviews and books on instruction
effectiveness have been searched to include additional relevant literature. The search
was based on the use of following keywords for instruction: effective instruction,
instructional effectiveness, direct instruction, teacher effectiveness, mastery learning,
mathematics instruction, reading instruction, science instruction, classrooms,
mathematics teaching, reading teaching, science teaching. Each instruction keyword
was crossed with the following output keywords: achievement, competencies, interest,
motivation, engagement, attainment. Overall, 317 publications matched the crossed
keyword combinations (instruction characteristics x student output).
- Inclusion criteria
The review is focused on empirical articles on teaching factors in the regular school
system. Thus, studies were excluded that did not primarily investigate teaching from
grade 1-13. However, in instances of long-term investigations from Kindergarten to
primary schooling Kindergarten studies were also included. Publications with a focus
on university teaching have been excluded from the review. Furthermore, studies with a
focus on teaching students with severe learning disabilities were not included. Taking
these criteria into consideration, the number of matching publications was reduced to
260. In a final step we analysed the articles according to the criterion of reporting
empirical findings on instruction effectiveness. Thereby, the number of published
articles was reduced to 107.
134
Findings of the qualitative review
First, a qualitative review has been conducted and summaries on each publication
according to twelve categories were provided: authors, publication year, domain,
instruction variables, output variables, content summary, summary on the direction of
effects, country, school type, number of subjects per analysis level (school,
classroom/teacher, students), design, and applied methods. The results of the qualitative
review are presented in Table 5.9. Furthermore, the abstracts of each publication are
provided in the appendix of this chapter (see page 199).
- Classification of current research studies
The qualitative review has shown that the instruction effectiveness studies applied
rather heterogeneous research designs and were associated with different research
paradigms. We considered four criteria that differentiate between types of studies on
instruction effectiveness and that are helpful for the interpretation of the findings of
these studies. The four criteria are:
Sample size (large vs. small scale)
Research design (experimental, quasi-experimental, correlational)
Points of measurement (cross-sectional, pre-post, longitudinal data)
Analysis methods (difference of means, correlational, multi-level, multi-path)
By combining the criteria, different types of research studies can be classified. Two of
the criteria (research design and analysis method) are listed in Table 5.9. Thus,
information on the nature and the type of each publication is given. In the following,
three examples are given to illustrate characteristic types of research studies:
First, there is a prominent number of cross-sectional large scale surveys with a
correlational design and the application of multi-level analysis methods. Most of these
studies show high methodological standards by using randomized samples, control for
background variables, high-level scaling of output measures and by taking into account
a large portion of different instruction characteristics. However, most of the studies are
based on a cross-sectional research design. Thereby, the strength of the relationship
between instruction characteristics and output measures at one point of time is taken
into consideration. Thus, the studies are limited with respect to empirical evidence on
long-term effects of instruction on learning outputs.
Second, there is a group of quasi-experimental intervention studies with a pre-post-test
design. The strength for these studies is the theoretical focus on specific teaching
approaches with their effects on learning outputs (direct teaching vs. constructivist
teaching). However, the studies are frequently limited with respect to their quasi-
experimental design (e.g. the composition of the samples in each experimental
condition), the lack of control groups, the quality of output measures as well as the
control for student background variables.
135
Third, there is a small number of longitudinal intervention studies with an experimental
and well-controlled pre-post-test design. These studies apply high standards in the
design of experiments that include randomized samples, control of student background
variables, sufficient sample sizes for multi-level analyses, comparisons of multiple
experimental as well as control groups, and long-term investigations. Thus, these
studies provide evidence on effects of teaching approaches on learning outputs.
- Effective instruction characteristics as shown by the qualitative review
Overall, the findings of the qualitative review show a corroboration of the
characteristics for effective teaching as outlined by Scheerens & Bosker (1997). The
most prominent indicators were learning time, classroom organisation, learning
environment, structured teaching, activating, support, feedback, learning to use
learning strategies. The characteristics, as outlined by Scheerens & Bosker for the
publications up to 1994, are also shown for the current review on the years 1995-2005.
In addition, the qualitative review on the last decade provided evidence for distinctive
trends. These can be outlined as follows (cf. Scheerens, 2004; Seidel & Prenzel, 2004):
a) Consideration of professional competencies of teachers. The reconsideration of
personal characteristics of effective teachers is built on assumptions of teacher expertise
research (such as epistemological beliefs or motivation goals), as well as assumptions of
organisation research (such as leadership and high expectations for students). The key
assumption of these studies is a positive influence of high expectations or constructivist
beliefs of teachers on the creation of learning-enhancing conditions in the classrooms
and, in turn, on the quality of student learning. Staub & Stern (2002), for example, show
positive effects of constructivist teacher beliefs on the students’ achievement gains from
grade 2 to 3 in mathematics instruction.
b) Integration of single instructional activities in instruction syndromes / patterns. The
findings of the review indicate a tendency to investigate instructional approaches that
combine a variety of teaching aspects. The compilation of teaching aspects into a
concept or syndrome is legitimated by previous findings on effective instruction
characteristics. In this respect, research takes advantage of the existing knowledge base
on instruction effectiveness and is testing different approaches or models against each
other. For example, the studies published by Guthrie et al. (1999) integrated single
instructional activities and approaches in a concept called “concept-oriented reading
instruction”. In Switzerland, Pauli et al. (2003) investigated an integrated constructivist
approach called ELF (“Erweiterte Lehr- und Lernformen”) and showed its effects on
cognitive and non-cognitive learning outputs.
c) Focus on learning processes. Current models on teaching and learning point out
learning processes as mediating factors between instruction characteristics and learning
outputs. Thus, there is an increasing number of studies that investigate the relationship
136
between instruction conditions and the students’ goal-oriented perceptions, motivations,
emotions and the use of learning strategies. The study of Seidel (2003) is an example
for the investigation of the relationship between instruction characteristics as observed
by video analyses and the students’ reported perceptions of learning conditions, quality
of learning motivation and cognitive learning strategies with respect to the video taped
instruction.
d) Consideration of multiple output criteria and cumulative learning. To a great extent
instructional research was focused on achievement as a single output criteria. However,
the review showed a trend in combining several output criteria with respect to cognitive
(achievement, competencies, performance) as well as non-cognitive outputs (interest in
a domain, attitudes, beliefs) of schooling. For example, the publications of Gruehn
(1995) and Kunter (2004) represent studies that investigated the compatibility of
cognitive and non-cognitive outputs of instruction. Furthermore, as shown by the study
of D’Agostino (2000) cumulative and long-term effects of schooling come to the fore of
multi-level effectiveness studies.
Results of the quantitative review
The next step from the qualitative review to a quantitative review was taken by using a
“vote-counting procedure” (Creemers, 1994; Scheerens & Bosker, 1997; Scheerens &
Creemers, 1996). According to this approach the number of significant (p < .05)
positive or negative associations between instruction characteristics and output
measures has been counted and compared to the number of tests with non-significant
associations.
The “vote-counting” procedure has been started by analysing each publication with
respect to the association between instruction characteristics and learning outputs.
Significant relationships or effects were marked with “+”, negative relationships with “-
“, and non significant associations with “o”. The counting has been applied for learning
processes (LP), non-cognitive (NCO) and cognitive outputs variables (CO). The results
of the first quantitative analysis step are reported in Table 5.8 (p. 150ff). On the basis of
these results a summative evaluation of the vote-counting has been conducted.
The instruction characteristics as independent variables have been grouped according to
the Dutch inspectorate’s observation schedule (see Table 5.2): learning time, classroom
organisation, orderly and functional learning environment, clear and structured
teaching, activating, learning to use learning strategies, challenge, support (mutual
respect), feedback/monitoring, evaluation of goals/attainment. In addition, teacher
characteristics, adaptive teaching, practice, material, and integrated instruction
concepts have been added to the list of instruction characteristics. For each
137
characteristic (e.g. classroom organisation) the single teaching indicators have been
used for a in depth-analysis (e.g. classroom management, discipline, control).
The effects of instruction characteristics on learning outputs are going to be reported by
means of three tables:
In Table 5.4 the number of studies in which instruction characteristics are related to
student output measures is given.
To illustrate the different effects of instruction on learning processes, non-cognitive
and cognitive outputs the number of associations for each of the three chosen
output measures is shown by Table 5.5.
Since several teaching indicators were grouped according to the Dutch
inspectorate’s observation schedule Table 5.6 additionally gives the number of
associations for each single teaching indicator. Thus, differences between
teaching indicators as well as prominent teaching indicators can be shown.
- Number of positive effects
Overall, 1467 associations or effects between instruction characteristics and output
measures have been counted. 615 (42%) have shown positive associations (+) between
instruction characteristics and student learning outputs (Table 5.4). However, 768 (52%)
gave no evidence and 84 (6%) evidence for negative effects of teaching on learning
outputs. With respect to the three output measures differences in the overall direction of
effects were found (Table 5.5): Instruction characteristics were more likely to be
positively related to learning processes (58% positive associations) than to non-
cognitive (45% positive associations) and cognitive output measures (36% positive
associations).
Furthermore, the direction of associations for each of the instruction characteristics was
rather diverse (see Table 5.4). The characteristics with a majority of positive effects
were learning to use learning strategies (73%), support (60%), challenge (53%), and
concept-oriented instruction (51%). Instruction characteristics with inconsistent
evidence were teacher characteristics (47%), learning time (42%), activating (41%),
material (41%), feedback / monitoring (37%), constructivist instruction (37%), clear
and structured teaching (36%), and orderly and functional learning environment (35%).
Doubtful empirical evidence was given for classroom organisation (25%), adaptive
teaching (25%), inductive teaching (20%), evaluation of goals / attainment (17%), and
practice (0%).
- Differences between output measures
The majority of studies focused on cognitive output measures (N = 872). In comparison,
non-cognitive outputs (N = 302) and learning processes (N = 293) have been less
frequently addressed (see Table 5.5). However, the direction of associations differed
between instruction characteristics. Learning time (56%), orderly and functional
138
learning environment (67%), activating (74%), learning to use learning strategies
(78%), challenge (52%), support (61%), feedback/monitoring (72%), and adaptive
teaching (89%) consistently showed positive effects on learning processes.
With respect to non-cognitive outputs the vote-counting points out a consistent positive
relationship for learning time (69%), activating (62%), learning to use learning
strategies (61%), challenge (59%), support (42%), and concept-oriented instruction
(80%).
Inconsistent findings were shown for cognitive outputs. Learning to use learning
strategies (73%) and challenge (50%) are the only two characteristics with consistent
positive evidence. However, many instruction characteristics predominantly give
inconsistent evidence for effects on cognitive outputs. These are learning time (60%
non-significant associations), classroom organisation (65% non-significant
associations), orderly and functional learning environment (72% non-significant
associations), clear and structured teaching (65% non-significant associations),
activating (60% non-significant associations), support (67% non-significant
associations), feedback/monitoring (81% non-significant associations), evaluation of
goals (78% non-significant associations), adaptive teaching (76% non-significant
associations), practice (58% non-significant associations), material (48% non-
significant associations), and constructivist instruction (57% non-significant
associations).
- Differences between instruction characteristics
Finally, findings for each instruction characteristic as outlined by the Inspectorate’s
observation schedule are going to be illustrated in more detail.
Learning time: The category learning time comprised indicators such as time on task,
effective use of teaching time, opportunity to learn, homework and mastery learning.
Overall, 99 studies had a focus on learning time. They have shown rather inconsistent
effects on student output measures (Table 5.4): 42% of the studies pointed out a positive
association between learning time and student outputs and 52% showed no significant
relationship. However, in only 6% of the cases a significant negative effect has been
shown. With respect to the three selected output measures (Table 5.5) the effects of
learning time are rather consistently positive for learning processes (56%) and non-
cognitive outputs (69%), but not for cognitive outputs (32% positive associations).
Classroom organisation: Indicators such as classroom management, discipline and
control were summarized for the second syndrome of classroom organisation. 25% of
the 76 counted associations were positively related to student output measures (Table
5.4). The majority of the studies (66%) gave no empirical evidence for a effect of
classroom organisation on student outputs. Furthermore, no differences in the direction
of associations between the three selected output measures have been observed (Table
139
5.5). To summarize, inconsistent empirical evidence is given for effectiveness of
classroom organisation.
Table 5.4: Number of studies in which instructional characteristics are related to
student outputs
Direction of effects
Instruction Characteristics Σ - o +
Learning Time 99(100 %)
6(6 %)
51 (52 %)
42(42 %)
Classroom organisation 76(100 %)
7(9 %)
50 (66 %)
19 (25 %)
Orderly, functional learning environment 117(100 %)
5(4 %)
71 (61 %)
41(35 %)
Clear and structured teaching 154(100 %)
6(4 %)
92 (60 %)
56(36 %)
Activating 216(100 %)
15(7 %)
112 (52 %)
89(41 %)
Learning to use learning strategies 135(100 %)
1(1 %)
36 (26 %)
98(73 %)
Challenge 213(100 %)
10(5 %)
90 (42 %)
113(53 %)
Support (mutual respect) 68(100 %)
0(0 %)
27 (40 %)
41(60 %)
Feedback / Monitoring 62(100 %)
3(5 %)
36 (58 %)
23(37 %)
Evaluation of goals / Attainment 70(100 %)
2(3 %)
56 (80 %)
12(17 %)
Teacher characteristics 19(100 %)
0(0 %)
10 (53 %)
9(47 %)
Adaptive teaching 72(100 %)
5(7 %)
49 (68 %)
18(25 %)
Practice 43(100 %)
15(35 %)
28 (65 %)
0(0 %)
Material 46(100 %)
4(9 %)
23 (50 %)
19(41 %)
“Integrated” instruction concepts
Constructivist instruction 19(100 %)
0(0 %)
12 (63 %)
7(37 %)
Inductive teaching 5(100 %)
3(60 %)
1 (20 %)
1(20 %)
Concept-oriented instruction 53(100 %)
2(4 %)
24 (45 %)
27(51 %)
Σ 1467(100 %)
84(6 %)
768 (52 %)
615(42 %)
Note: Some publications address multiple learning outputs as well as multiple instruction characteristics. Thus, the numbers given are higher than the number of publications included for the review.
140
Table 5.5: Number of studies in which instructional characteristics are related to
cognitive, non-cognitive and learning process variables
Characteristics
Learning Process Non-Cognitive
Outputs
Cognitive Outputs
Σ - o + Σ - o + Σ - o + Σ
Learning Time 23 10 13 13 1 3 9 63 5 38 20 99
Classroom
organisation
4 1 2 1 26 1 18 7 46 5 30 11 76
Orderly, functional
learning environment
21 3 4 14 36 1 24 11 60 1 43 16 117
Clear and structured
teaching
41 2 22 17 17 1 8 8 96 3 62 31 154
Activating 23 6 17 39 1 14 24 154 14 92 48 216
Learning to use
learning strategies
27 6 21 18 7 11 90 1 23 66 135
Challenge 67 5 27 35 54 22 32 92 5 41 46 213
Support (mutual
respect)
36 14 22 20 5 15 12 8 4 68
Feedback 18 5 13 17 9 8 27 3 22 2 62
Evaluation of goals 16 1 14 1 54 1 42 11 70
Teacher characteristics 6 5 1 13 5 8 19
Adaptive teaching 9 1 8 18 14 4 45 5 34 6 72
Practice 4 1 3 8 1 7 31 13 18 43
Material 9 3 3 3 4 4 33 1 16 16 46
“Integrated” concepts
Constructivist
instruction
2 1 1 3 3 14 8 6 19
Inductive teaching 2 1 1 2 1 1 1 1 5
Concept-oriented
instruction
7 3 4 5 1 4 41 2 20 19 53
Σ 293 16 107 170 302 8 159 135 872 60 502 310 1467
Orderly and functional learning environment: The third category comprised instruction
characteristics such as classroom climate, learning climate, no achievement pressure,
mastery orientation, no performance orientation, no competition between students.
Again, the 117 investigations on orderly and functional learning environment gave
inconsistent empirical evidence for effects on student outputs (Table 5.4). 61 % of the
141
counted associations have shown no significant effects, 4 % negative and 36 % positive
effects on student output measures. However, differences between the three selected
output measures were observed (Table 5.5): An orderly and functional learning
environment is, above all, positively associated with the quality of learning processes
(67% positive associations). Inconsistent effects are found for non-cognitive (31%) and
cognitive output measures (27%). Classroom/learning climate and no achievement
pressure are two indicators that have been strongly associated with positive effects on
learning processes (100%, Table 5.6).
Clear and structured teaching: Clear and structured teaching included indicators such
as structured teaching, direct teaching, goal-directed teaching, clarity, teacher
demonstrations, and teaching for basic skills. Overall, the review has shown 154
associations in the last decade for clear and structured teaching (Table 5.4), but the
empirical evidence has turned out to be rather inconsistent (4% negative, 60% no
significant, 36% positive effects). No differences between the three output measures
have been observed, but most of the studies have focused on cognitive outputs or on
learning processes (Table 5.5). The moderate effects for clear and structured teaching
can be put down to mixed associations for direct teaching, teacher demonstrations and
teaching for basic skills. Positive effects were observed for goal-directed and clear
teaching: 56% positive relationships for learning processes, 75% for non-cognitive
outputs, and 41% for cognitive outputs (Table 5.6). Thus, goal-directed and clear
teaching has turned out to be the most relevant aspect, showing consistent positive
effects on student learning.
Activating: Cooperative learning, situated learning, discovery learning, peer-tutoring,
group work, student experiments, hands-on activities, individual student work, and
student discussions were summarized for the category activating. Activating was one
the most intensively investigated category of the review (216 associations, Table 5.4).
Overall, inconsistent effects for activating have been found: 41% of the reported
relations show positive signs, 7% negative and 51% give no significant evidence. The
pattern of mixed signs of associations is due to the results on cognitive outputs (Table
5.5). Consistent positive effects for activating became evident for the quality of learning
processes (74% positive relations) as well as for non-cognitive outputs (62%). However,
the number of studies with a focus on learning processes and non-cognitive outputs is
much smaller than the number of studies for cognitive outputs (154 compared to 23
resp. 39). Compared to the other indicators for activating (Table 5.6) cooperative
learning has consistently shown positive effects on all student output measures (90% for
learning processes, 67% for non-cognitive outputs, 74% cognitive outputs). Individual
learning is an indicator with a high proportion of studies that show negative effects on
cognitive output measures (48%). Mixed results on cognitive outputs were found for
situated/discovery learning, peer-tutoring, student work and student discussions.
142
However, situated/discovery learning, peer-tutoring and student work/hands on
activities have shown positive associations with learning processes and non-cognitive
outputs. To summarize, activating shows inconsistent effects on student learning, but
with distinctive differences between the chosen indicators. Thereby, cooperative
learning has turned out to be the most powerful indicator within the teaching syndrome
of activating.
Learning to use learning strategies: Learning strategies have also been investigated
quite intensively over the last decade (Table 5.4), showing the teaching of a variety of
learning strategies (cf. Table 5.6). Overall, the category learning to use learning
strategies has reached the highest proportion of reported positive effects on student
output measures (73%). Furthermore, the teaching of learning strategies was positively
related to all three output measures, showing similar association patterns for learning
processes, non-cognitive as well as cognitive output measures (Table 5.5). The
consistent positive effect has been shown for a variety of different domain-specific
learning strategies. The teaching of rather general and formal learning strategies has
turned out to be least effective (Table 5.6). Overall, the review has shown consistent
positive effects of learning to use learning strategies on student outputs. This is of
special relevance since learning strategies have not yet been reviewed as intensively as
characteristics such as learning time or clear and structured teaching.
Challenge: Next to learning strategies challenge has turned out to be another trend in
research on instruction effectiveness (Table 5.4). Overall, 213 cases were counted for
indicators such as cognitive activation, teaching for understanding, active student
engagement, providing authentic contexts, relevance of learning contents, language
level and use of various representation formats. With respect to student output
measures, challenge shows rather consistent positive effects (53% positive
associations), with the same association pattern for the three selected output measures.
Challenge consistently turned out to be positively associated with the quality of learning
processes as well as non-cognitive and cognitive output measures (Table 5.5). Again,
differences between the grouped indicators were shown (Table 5.6): the quality of
learning processes relates positively with cognitive activation/teaching for
understanding and authentic contexts/relevance. Non-cognitive outputs have shown
distinctive positive associations with authentic contexts/relevance and cognitive outputs
with cognitive activation/teaching for understanding. Mixed results were found for
active student engagement.
Support (mutual respect): The review has shown consistent positive effects of support
on student learning (Table 5.4): 60% of the counted investigations pointed out positive
associations. The results are predominantly due to the consistent positive associations
between support and the quality of learning processes (61%) and non-cognitive outputs
143
(75%, Table 5.5). However, inconsistent results were found for cognitive outputs (33%
positive associations, 67% no significant relations).
Feedback/Monitoring: Inconsistent associations between feedback and learning outputs
have emerged from the review (37% positive associations). The mixed association
pattern could be shown for non-cognitive as well as cognitive output measures.
However, the picture changes for learning processes, pointing out rather consistent
positive associations between feedback and the quality of learning processes (72%
positive associations).
Evaluation of goals/Attainment: Research on the last category of the Dutch
inspectorate’s observation schedule was rather limited and comprised aspects such as
the frequency and quality of assessments and tests. Thereby, the relationship between
assessments and learning outputs was inconsistent (17% positive, 80% non significant
associations), with the same association pattern for non-cognitive and cognitive outputs.
Effects of assessments on learning processes have not been addressed in the reviewed
research articles. Thus, the scope of the findings was rather narrow and limited with
respect to the assessment and evaluation of student outputs. A thorough interpretation of
the relevance of this instruction characteristic needs further empirical evidence.
Next to the categories of the Dutch inspectorate’s framework further analyses have been
conducted for the instruction characteristics teacher characteristics, adaptive teaching,
practice, material, and “integrated” teaching concepts. With respect to these areas, three
aspects are going to be elaborated in more detail: the relevance of teacher
characteristics, adaptive teaching and constructivist teaching as an example for
integrated concepts.
Teacher characteristics: Only a small number of studies has addressed the professional
competencies of teachers (19 investigations). The small number of studies has shown
rather inconsistent results for the effectiveness of teacher characteristics (47% positive
associations, 53% no significant relationships). Thereby, differences between non-
cognitive and cognitive outputs were observed: consistent positive effects were found
for cognitive outputs (8 out of 13), no effects for non-cognitive outputs (5 out of 6) and
no investigations have been conducted on learning processes. To summarize, high
expectations and beliefs of teachers are consistently found to be relevant for cognitive
student outputs. However, the number of studies was rather limited and further
empirical evidence is required.
Adaptive teaching: The research findings have shown inconsistent empirical evidence
for adaptive teaching (25% positive associations, Table 5.4). Thereby, the category
comprised a variety of indicators such as the use of various teaching methods, adaptive
teaching, open tasks/choice and taking into account student pre-requisites. With respect
to the three output measures (Table 5.5), adaptive teaching has shown rather positive
144
effects on the quality of learning processes (89%), however, rather small effects were
found on non-cognitive (22%) and cognitive outputs (13%). The inconsistent findings
are due to the results of the two indicators adaptive teaching and taking into account
student pre-requisites (Table 5.6). Contrary to this, positive associations were found
between open tasks/choice and the quality of learning processes. To summarize,
inconsistent empirical evidence is given for the effectiveness of adaptive teaching.
However, the concept of adaptive teaching has been investigated rather heterogeneously
and might need further theoretical and empirical specification.
Constructivist teaching: The integrated concept of constructivist teaching has been
addressed in a limited number of studies (N = 19). However, many studies addressed
teaching aspects that are strongly connected to the idea of constructivist teaching (e.g.
the concept of cognitive activation, teaching for understanding, authentic contexts or the
learning of self-regulated learning). For the review, the studies that did not address
constructivist teaching as an integrated concept but as a combination of different
teaching aspects have been counted for each of the mentioned instruction characteristics
and are not listed within this category. Overall, the 19 studies with a focus on the
integrated concept of constructivist teaching have shown inconsistent results with
respect to their effectiveness (37% positive associations). No differences in the
association pattern have been observed between the three output measures.
145
Table 5.6: Number of studies in which single instructional indicators are related to
cognitive, non-cognitive and learning process variables
Learning
Processes
Non-Cognitive
Output
Cognitive
Output
Σ - o + Σ - o + Σ - o + Σ
Learning Time 23 10 13 13 1 3 9 63 5 38 20 99
Time on task, effective use of
teaching time
21 10 11 8 2 6 19 2 11 6 48
Opportunity to learn 2 2 5 1 1 3 14 2 10 2 21
Homework 27 16 11 27
Mastery learning 3 1 1 1 3
Classroom organisation 4 1 2 1 26 1 18 7 46 5 30 11 76
Classroom management 1 9 6 3 21 4 12 5 31
Discipline 2 2 1 11 7 4 10 1 6 3 23
Control 1 1 6 1 5 15 12 3 22
Orderly, functional learning
environment
21 3 4 14 36 1 24 11 60 1 43 16 117
Classroom climate, learning
climate
11 11 27 1 18 8 58 1 42 15 96
No achievement pressure
(recoded)
3 3 3 2 1 2 1 1 8
Mastery orientation 3 2 1 4 4 7
No performance orientation
(recoded) / competition
4 1 3 2 2 6
Clear and structured teaching 41 2 22 17 17 1 8 8 96 3 62 31 154
Structured/direct teaching 19 1 12 6 9 1 3 5 55 2 38 15 83
Goal-directed, clear 18 8 10 4 1 3 29 17 12 51
Teacher demonstrations 2 2 2 2 3 1 2 7
Teaching for basic skills 2 1 1 2 2 9 5 4 13
Activating 23 6 17 39 1 14 24 154 14 92 48 216
Cooperative learning 10 1 9 9 3 6 23 6 17 42
Situated / discovery learning 2 2 6 1 5 10 6 4 18
146
Learning
Processes
Non-Cognitive
Output
Cognitive
Output
Σ - o + Σ - o + Σ - o + Σ
Peer-tutoring 2 2 54 34 20 56
Student work, hands-on 2 1 1 13 5 8 28 1 21 6 43
Individual work 9 4 5 9 1 5 3 27 13 14 45
Student discussions 12 11 1 12
Learning to use learning
strategies
27 6 21 18 7 11 90 1 23 66 135
Cooperative learning skills 10 4 6 10
Meta-cognitive strategies 13 2 11 4 4 24 3 21 41
Scientific inquiry learning 7 2 5 16 2 14 23
Thinking aloud training 15 1 14 15
Organising methods 2 2 1 1 3 3 6
Language acquisition training 17 12 5 17
Reading/Writing strategies 2 2 1 1 14 1 4 9 17
General learning strategies 5 4 1 1 1 6
Challenge 67 5 27 35 54 22 32 92 5 41 46 213
Cognitive activation, orien-
tation towards understanding
19 1 7 11 20 11 9 58 22 36 97
Active student engagement 24 3 11 10 10 5 5 12 5 5 2 46
Authentic contexts, relevance 24 1 9 14 24 6 18 11 7 4 59
Language level 7 6 1 7
Various representation formats 4 1 3 4
Support (mutual respect) 36 14 22 20 5 15 12 8 4 68
Quality of interactions,
teacher support
36 14 22 20 5 15 12 8 4 68
Feedback / Monitoring 18 5 13 17 9 8 27 3 22 2 62
Feedback, frame of reference,
monitoring
18 5 13 17 9 8 27 3 22 2 62
Evaluation of goals 16 1 14 1 54 1 42 11 70
Assessments, tests 16 1 14 1 54 1 42 11 70
147
Learning
Processes
Non-Cognitive
Output
Cognitive
Output
Σ - o + Σ - o + Σ - o + Σ
Teacher characteristics 6 5 1 13 5 8 19
High expectations 6 5 1 9 3 6 15
Constructivist epistemological
beliefs
4 2 2 4
Adaptive teaching 9 1 8 18 14 4 45 5 34 6 72
Various teaching methods 2 1 1 2
Adaptive teaching 3 2 1 13 10 3 16
Open tasks, choice 9 1 8 9
Student pre-requisites 15 12 3 30 5 23 2 45
Practice 4 1 3 8 1 7 31 13 18 43
Drill & repetitions 4 1 3 15 4 11 19
Applications 4 1 3 4 4 16 9 7 24
Material/Media 9 3 3 3 4 4 33 1 16 16 46
Textbooks 16 15 1 16
Computers 9 3 3 3 4 4 17 1 1 15 30
“Integrated” instructional
concepts
11 1 4 6 10 1 5 4 56 3 28 25 77
Constructivist instruction 2 1 1 3 3 14 8 6 19
Inductive teaching 2 1 1 2 1 1 1 1 5
Concept-oriented/ integrated
instruction
7 3 4 5 1 4 41 2 20 19 53
Summary on the consistency of findings on instruction effectiveness
Finally, we want to conclude the chapter on instruction effectiveness with Table 5.7 in
which the consistency of findings is summarised. Overall, the review on recent research
articles on instructional effectiveness in the last decade (1995-2005) has shown that the
instruction characteristics as outlined by the Dutch inspectorate’s observation schedule
are based on empirical research findings. None of the instruction characteristics was
lacking empirical evidence.
148
Table 5.7 Consistency of results on teaching effectiveness
Instruction Characteristics No empirical
evidence
Consistent
negative
associations
Inconsistent
associations
Consistent
positive
associations
Learning Time x
Classroom organisation x
Orderly, functional learning environment x
Clear and structured teaching x
Activating x
Learning to use learning strategies x
Challenge x
Support (mutual respect) x
Feedback / Monitoring x
Evaluation of goals / Attainment x
Teacher characteristics x
Adaptive teaching x
Practice x
Material x
“Integrated” instruction concepts
Constructivist instruction x
Inductive teaching x
Concept-oriented instruction x
However, some characteristics that have been investigated in the research are not listed
in the observation schedule of the inspectorates. These are: teacher professional
competencies with respect to high expectations for students and their beliefs about the
nature of learning, adaptive teaching, the role of textbooks and computers (summarised
as material), and practice.
Concerning the consistency of findings the following four instruction characteristics
have turned out to show consistent positive effects on student outputs: learning to use
learning strategies, (cognitive) challenge, support and the integrated approach of
concept-oriented instruction (which also has a strong focus on cognitive challenge).
Even if a large number of instruction characteristics have shown rather inconsistent
findings it does not mean that these characteristics are irrelevant for student outputs.
The inconsistency of findings points out a significant amount of variance between
classrooms in the implementation of these teaching characteristic. Classrooms with a
high implementation of these teaching characteristics might be effective with respect to
149
student outputs, and classrooms with a low implementation might be ineffective. The
review gives no evidence on the quality of implementation. Thus, for a deeper insight
qualitative analyses of the effects of single studies that are either high or low in
implementation would have to be conducted.
In addition, the vote-counting procedure does not take into account the quality of the
reported research studies. A systematic meta-analysis of these findings will give more
substantial evidence on the direction and size of the effects.
150
Summary on the direction of effects per publication
Table 5.8: Direction of effects per publication (CO=Cognitive Outputs, NCO=Non-Cognitive Outputs; LP: Learning processes; +=significant
positive effects (p<.05); o=non-significant effects; -=significant negative effects)
ID
Aut
hors
Lea
rnin
g T
ime
Cla
ssro
om o
rgan
isat
ion
Ord
erly
, fun
ctio
nal l
earn
ing
envi
ronm
ent
Cle
ar a
nd s
truc
ture
d te
achi
ng
Act
ivat
ing
Lea
rnin
g to
use
lear
ning
str
ateg
ies
Cha
llen
ge
Ada
ptiv
e te
achi
ng
Sup
port
(m
utua
l res
pect
)
Feed
back
Eva
luat
ion
of g
oals
Prac
tice
Mat
eria
l/med
ia
Tea
cher
cha
ract
eris
tics
Con
stru
ctiv
ist i
nstr
ucti
on
Indu
ctiv
e te
achi
ng
Con
cept
-ori
ente
d/
inte
grat
edin
stru
ctio
n
115 Adedayo (1998) CO:1+
1 Alexander et al.
(2002) CO:1+
NCO: 2+
2 Allsopp, (1997) CO:1o
3 Anderman et al.
(2001) NCO:
2+/2o
4 Applebee et al.
(2003) CO:1+
5 Arnold, et al.
(2002)
CO:1+
NCO: 2+
6 Ashman & Gillies
(1997) LP:
6+/4o
151
ID
Aut
hors
Lea
rnin
g T
ime
Cla
ssro
om o
rgan
isat
ion
Ord
erly
, fun
ctio
nal l
earn
ing
envi
ronm
ent
Cle
ar a
nd s
truc
ture
d te
achi
ng
Act
ivat
ing
Lea
rnin
g to
use
lear
ning
str
ateg
ies
Cha
llen
ge
Ada
ptiv
e te
achi
ng
Sup
port
(m
utua
l res
pect
)
Feed
back
Eva
luat
ion
of g
oals
Prac
tice
Mat
eria
l/med
ia
Tea
cher
cha
ract
eris
tics
Con
stru
ctiv
ist i
nstr
ucti
on
Indu
ctiv
e te
achi
ng
Con
cept
-ori
ente
d/
inte
grat
edin
stru
ctio
n
92 Baumann et al.
(2003) CO:
2+/1o
117 Baumert & Köller
(2000)
CO:1+
NCO:2+
LP:2o
CO:2o/1
NCO:6o
LP:1+/
3o/2-
CO:2o
NCO: 2+
LP:1+/1
o
CO:2+
NCO:
2+/2o
LP:
1+/3o
CO:2o
NCO: 4o
LP:3o/1-
CO:
1+/1o
NCO:4o
LP:3o/1-
CO:1
NCO:
1o/1
LP:1+/1-
93 Bennacer, H.
(2000) CO:1+/1
-
94 Bianchini, (1997) CO:1+
95 Borsch et al. (2002) CO:8+
97 Brush (1997) CO:1+
98 Burkam et al.
(1997)
CO:1o CO:1+ CO:1- CO:1o/1
- CO:1o CO:1-
99 Byrne &
Fieldingbarnsley
(1995)
CO:
5+/12o
152
ID
Aut
hors
Lea
rnin
g T
ime
Cla
ssro
om o
rgan
isat
ion
Ord
erly
, fun
ctio
nal l
earn
ing
envi
ronm
ent
Cle
ar a
nd s
truc
ture
d te
achi
ng
Act
ivat
ing
Lea
rnin
g to
use
lear
ning
str
ateg
ies
Cha
llen
ge
Ada
ptiv
e te
achi
ng
Sup
port
(m
utua
l res
pect
)
Feed
back
Eva
luat
ion
of g
oals
Prac
tice
Mat
eria
l/med
ia
Tea
cher
cha
ract
eris
tics
Con
stru
ctiv
ist i
nstr
ucti
on
Indu
ctiv
e te
achi
ng
Con
cept
-ori
ente
d/
inte
grat
edin
stru
ctio
n
101 Cantrell (1999) CO:
4+/1o
103 Chang (2002) CO:
2+/1o
104 Chang & Baru-faldi
(1999) CO:1+
105 Chinn et al. (2001) LP:4+
119 Chularut &
DeBacker (2003) CO:1+
NCO: 1+
LP:2+
7 Clausen (2001) CO:6o
NCO:6o
CO:2o
NCO:
1+/1o
CO:2o
NCO:
1+/1-
CO:6oN
CO: 6o
CO:
1+/1o
NCO: 2o
CO:4o
NCO:
1+/3o
CO:2-
NCO: 2o
8 Crinjen & Feehan
(1998)
CO:
1+/1-
120 D'Agostino (2000) CO:
2+/4o CO:
3+/3o CO:2o
10 Davidson et al.
(1996) CO:1+
153
ID
Aut
hors
Lea
rnin
g T
ime
Cla
ssro
om o
rgan
isat
ion
Ord
erly
, fun
ctio
nal l
earn
ing
envi
ronm
ent
Cle
ar a
nd s
truc
ture
d te
achi
ng
Act
ivat
ing
Lea
rnin
g to
use
lear
ning
str
ateg
ies
Cha
llen
ge
Ada
ptiv
e te
achi
ng
Sup
port
(m
utua
l res
pect
)
Feed
back
Eva
luat
ion
of g
oals
Prac
tice
Mat
eria
l/med
ia
Tea
cher
cha
ract
eris
tics
Con
stru
ctiv
ist i
nstr
ucti
on
Indu
ctiv
e te
achi
ng
Con
cept
-ori
ente
d/
inte
grat
edin
stru
ctio
n
11 De Fraine et al.
(2003) CO:3o CO:1o
12 de Jong et al.
(2004)
CO:1+ CO:1+ CO:1+
13 Dowdell (1996) CO:1+
14 Driessen &
Sleegers (2000)
CO:6o CO:
1+/1-
CO:2- CO:2o CO:
3+/11o CO:2o
15 Einsiedler &
Treinies (1997) CO:2+
16 Foorman et al.
(1998) CO:
5+/3o
NCO: 1o
17 Freedman (1997) CO:2+
NCO: 1o
18 Fuchs, D. et al.
(1997) CO:3+
19 Fuchs, L. S. et al.
(2002) CO:1+
154
ID
Aut
hors
Lea
rnin
g T
ime
Cla
ssro
om o
rgan
isat
ion
Ord
erly
, fun
ctio
nal l
earn
ing
envi
ronm
ent
Cle
ar a
nd s
truc
ture
d te
achi
ng
Act
ivat
ing
Lea
rnin
g to
use
lear
ning
str
ateg
ies
Cha
llen
ge
Ada
ptiv
e te
achi
ng
Sup
port
(m
utua
l res
pect
)
Feed
back
Eva
luat
ion
of g
oals
Prac
tice
Mat
eria
l/med
ia
Tea
cher
cha
ract
eris
tics
Con
stru
ctiv
ist i
nstr
ucti
on
Indu
ctiv
e te
achi
ng
Con
cept
-ori
ente
d/
inte
grat
edin
stru
ctio
n
20 George & Kaplan
(1997) NCO: 1+
21 Ginsburg-Block &
Fantuzzo (1998) CO:2+
NCO: 2+
CO:2+
NCO: 2+
22 Gruehn (1995) CO:2+
NCO:
2+/2o
CO:
3+/1o
NCO:
5+/3o
CO:
2+/1o
NCO:
3+/3o
CO:1+
NCO: 2+
CO:1-
NCO:
1o/1-
CO:
1+/2o/1-
NCO:
1+/7o
CO:
1+/1o
NCO:
2+/2o
CO:
1+/1o
NCO: 4o
CO:1-
NCO:
1o/1-
23 Guthrie et al.
(1996) CO:
10+/2o
25 Guthrie et al.
(1999) CO:
3+/2o
LP:1+
28 Guthrie et al.
(2001)
CO:1o
LP:1+
24 Guthrie et al.
(1998) CO:
1o/1-
LP:1+
26 Guthrie et al.
(2000) LP:
2+/3o
155
ID
Aut
hors
Lea
rnin
g T
ime
Cla
ssro
om o
rgan
isat
ion
Ord
erly
, fun
ctio
nal l
earn
ing
envi
ronm
ent
Cle
ar a
nd s
truc
ture
d te
achi
ng
Act
ivat
ing
Lea
rnin
g to
use
lear
ning
str
ateg
ies
Cha
llen
ge
Ada
ptiv
e te
achi
ng
Sup
port
(m
utua
l res
pect
)
Feed
back
Eva
luat
ion
of g
oals
Prac
tice
Mat
eria
l/med
ia
Tea
cher
cha
ract
eris
tics
Con
stru
ctiv
ist i
nstr
ucti
on
Indu
ctiv
e te
achi
ng
Con
cept
-ori
ente
d/
inte
grat
edin
stru
ctio
n
31 Hamilton et al.
(1995) CO:4o CO:
3+/5o CO:4o
32 Hardre & Reeve
(2003) CO:1+
NCO: 1+
LP:2+
122 Hardy, I (2004) CO:3+
33 Helmke & Weinert
(1997) CO:
1+/1o
CO:2o CO:
3+/3o CO:
1+/3o
CO:2+
34 Henderson &
Landesman (1995) CO:
1+/1o
NCO: 4o
35 Hill & Rowe (1998) CO:1+
NCO: 1+ NCO: 1+ CO:1+
NCO: 1+
NCO: 1+ CO:1+
36 Hogan (1999) CO:
4+/1o
37 Hopkins et al.
(1997) CO:1+
38 Houghton et al.
(1995) CO:4+
156
ID
Aut
hors
Lea
rnin
g T
ime
Cla
ssro
om o
rgan
isat
ion
Ord
erly
, fun
ctio
nal l
earn
ing
envi
ronm
ent
Cle
ar a
nd s
truc
ture
d te
achi
ng
Act
ivat
ing
Lea
rnin
g to
use
lear
ning
str
ateg
ies
Cha
llen
ge
Ada
ptiv
e te
achi
ng
Sup
port
(m
utua
l res
pect
)
Feed
back
Eva
luat
ion
of g
oals
Prac
tice
Mat
eria
l/med
ia
Tea
cher
cha
ract
eris
tics
Con
stru
ctiv
ist i
nstr
ucti
on
Indu
ctiv
e te
achi
ng
Con
cept
-ori
ente
d/
inte
grat
edin
stru
ctio
n
39 Houtveen et al.
(1999) CO:1+
40 Jacobson & Lehrer
(2000) CO:1+
41 Jovanovic & King
(1998) NCO:
2+/4o
42 Klieme & Rakoczy
(2003) CO:1+
NCO: 1+
CO:1o
NCO: 1+
CO:1+
NCO: 1+ CO:1+
NCO: 1+ CO:1+
NCO: 1+
CO:1o
NCO: 1+
43 Kramarski &
Mevarech (1997) CO:1+
LP:
4+/1o
44 Kramarski et al.
(2001) CO:7+
LP:
4+/1o
45 Kroesbergen et al.
(2004) CO:
3+/1o
NCO: 1-
LP:1+
CO:
3+/1o
NCO: 1o
LP:1o
157
ID
Aut
hors
Lea
rnin
g T
ime
Cla
ssro
om o
rgan
isat
ion
Ord
erly
, fun
ctio
nal l
earn
ing
envi
ronm
ent
Cle
ar a
nd s
truc
ture
d te
achi
ng
Act
ivat
ing
Lea
rnin
g to
use
lear
ning
str
ateg
ies
Cha
llen
ge
Ada
ptiv
e te
achi
ng
Sup
port
(m
utua
l res
pect
)
Feed
back
Eva
luat
ion
of g
oals
Prac
tice
Mat
eria
l/med
ia
Tea
cher
cha
ract
eris
tics
Con
stru
ctiv
ist i
nstr
ucti
on
Indu
ctiv
e te
achi
ng
Con
cept
-ori
ente
d/
inte
grat
edin
stru
ctio
n
46 Kuklinski &
Weinstein (2001) CO:
5+1o
NCO:
1+/5o
123 Kunter (2004) LP:
4+/2o
CO:
1+/3o
NCO:
3+/1o
LP:
1+/2o
LP:12+ CO: 3o/1
NCO: 4+
LP:
5+/1o
CO: 2o/2
NCO:
2+/2o
LP:
2+/1o
CO: 4o/4
NCO:
4+/4o
LP:
8+/1o
CO:2-
NCO: 2o
LP:
4+/2o
CO: 3o/1
NCO: 4+
LP:9+
47 Kupermintz et al.
(1995)
CO:
2+/5o
CO: 4o CO:
3+/2o
CO:
2+/5- CO:
8+/2o CO:
4o/3-
CO:2o
48 Kupermintz, &
Snow (1997) CO:3+/
11o/4- CO:
16o/2- CO:
6+/3o CO:
12o/6-
CO:9o
49 Kyriakides et al.
(2000)
CO:
2+/1o CO:2o CO:2o
50 Labudde & Pfluger
(1999) CO:1o
NCO: 3+ CO:
3+/3o
NCO:
18+
CO:1o
NCO: 3+
CO:3o
NCO: 9+
158
ID
Aut
hors
Lea
rnin
g T
ime
Cla
ssro
om o
rgan
isat
ion
Ord
erly
, fun
ctio
nal l
earn
ing
envi
ronm
ent
Cle
ar a
nd s
truc
ture
d te
achi
ng
Act
ivat
ing
Lea
rnin
g to
use
lear
ning
str
ateg
ies
Cha
llen
ge
Ada
ptiv
e te
achi
ng
Sup
port
(m
utua
l res
pect
)
Feed
back
Eva
luat
ion
of g
oals
Prac
tice
Mat
eria
l/med
ia
Tea
cher
cha
ract
eris
tics
Con
stru
ctiv
ist i
nstr
ucti
on
Indu
ctiv
e te
achi
ng
Con
cept
-ori
ente
d/
inte
grat
edin
stru
ctio
n
51 Lazarowitz et al.
(1996) CO:
1+/3o
NCO: 3+
LP:
2+/1o
52 Lowther et al.
(2003) CO: 13+
LP:
3+/2-
53 Luyten & de Jong
(1998)
CO:
2+/2o CO:2o CO:
2+/2o
124 Mac Iver et al.
(2003) CO:
6+/12o
55 Marks (2000) LP:6+ LP:6+
56 Martinez &
Martinez (1999)
CO:1o
57 Mathes et al. (1998) CO:
10+/ 33o
59 McGuinness et al.
(1995) CO:
6+/1o
159
ID
Aut
hors
Lea
rnin
g T
ime
Cla
ssro
om o
rgan
isat
ion
Ord
erly
, fun
ctio
nal l
earn
ing
envi
ronm
ent
Cle
ar a
nd s
truc
ture
d te
achi
ng
Act
ivat
ing
Lea
rnin
g to
use
lear
ning
str
ateg
ies
Cha
llen
ge
Ada
ptiv
e te
achi
ng
Sup
port
(m
utua
l res
pect
)
Feed
back
Eva
luat
ion
of g
oals
Prac
tice
Mat
eria
l/med
ia
Tea
cher
cha
ract
eris
tics
Con
stru
ctiv
ist i
nstr
ucti
on
Indu
ctiv
e te
achi
ng
Con
cept
-ori
ente
d/
inte
grat
edin
stru
ctio
n
61 Mevarech (1999) CO:
3+/3o
126 Mevarech &
Kramarski (1997) CO:5+
127 Mevarech&
Kramarski (2003) CO:2+
63 Miller & Meece
(1997)
NCO:
1+/1o/1-
LP:1+
128 Möller et al. (2002) CO:1o CO:2+
65 Muijs &
Reynolds(2000)
66 Muthukrishna &
Borkowski (1995) CO:
3+/6o
NCO:
3+/1o
LP:2+
CO:5+
NCO: 2+
LP:3+
67 Nolen (2003) NCO: 1+ CO:2+
160
ID
Aut
hors
Lea
rnin
g T
ime
Cla
ssro
om o
rgan
isat
ion
Ord
erly
, fun
ctio
nal l
earn
ing
envi
ronm
ent
Cle
ar a
nd s
truc
ture
d te
achi
ng
Act
ivat
ing
Lea
rnin
g to
use
lear
ning
str
ateg
ies
Cha
llen
ge
Ada
ptiv
e te
achi
ng
Sup
port
(m
utua
l res
pect
)
Feed
back
Eva
luat
ion
of g
oals
Prac
tice
Mat
eria
l/med
ia
Tea
cher
cha
ract
eris
tics
Con
stru
ctiv
ist i
nstr
ucti
on
Indu
ctiv
e te
achi
ng
Con
cept
-ori
ente
d/
inte
grat
edin
stru
ctio
n
68 Pauli et al. (2003) CO:1o
NCO: 2o
LP:1+
69 Perry (1998) NCO: 1o
LP:2+
70 Ramsden (1997) CO:1o
71 Reezigt et al.
(2003)
CO:4+/
7o/4- CO:
6+/12o CO:6o CO:1+/
9o/2-
CO: 12o
73 Seidel (2003) LP:
2+/5o LP:
1+/6o LP: 11+/
20o/4- LP:
5+/2o
LP:
3+/4o
74 Seidel et al. (2002) LP:
5+/1o LP:
1+/5o
LP:
3+/3o
75 Seidel et al. (2003) NCO:
1+/1o
LP:
3+/9o
161
ID
Aut
hors
Lea
rnin
g T
ime
Cla
ssro
om o
rgan
isat
ion
Ord
erly
, fun
ctio
nal l
earn
ing
envi
ronm
ent
Cle
ar a
nd s
truc
ture
d te
achi
ng
Act
ivat
ing
Lea
rnin
g to
use
lear
ning
str
ateg
ies
Cha
llen
ge
Ada
ptiv
e te
achi
ng
Sup
port
(m
utua
l res
pect
)
Feed
back
Eva
luat
ion
of g
oals
Prac
tice
Mat
eria
l/med
ia
Tea
cher
cha
ract
eris
tics
Con
stru
ctiv
ist i
nstr
ucti
on
Indu
ctiv
e te
achi
ng
Con
cept
-ori
ente
d/
inte
grat
edin
stru
ctio
n
76 Seidel et al. (2005) CO: 1+
NCO: 1o
LP:
8+/7o
77 She & Fisher
(2002) CO:2o
NCO: 3o
LP:1-
CO:
1+/1o
NCO:
1+/2o
LP:1+
CO:2+
NCO: 3+
LP:1+
CO:2o
NCO: 3o
LP:1o
CO:2o
NCO: 3o
LP:1o
79 Staub & Stern
(2002)
CO:
2+/2o
80 Stevens & Slavin
(1995)
CO:5+
NCO: 1o
LP:1+
81 Stipek et al. (1998) CO:2o
NCO:
3+/3o
CO:2o CO:
1+/1o
NCO:
1+/1o
162
ID
Aut
hors
Lea
rnin
g T
ime
Cla
ssro
om o
rgan
isat
ion
Ord
erly
, fun
ctio
nal l
earn
ing
envi
ronm
ent
Cle
ar a
nd s
truc
ture
d te
achi
ng
Act
ivat
ing
Lea
rnin
g to
use
lear
ning
str
ateg
ies
Cha
llen
ge
Ada
ptiv
e te
achi
ng
Sup
port
(m
utua
l res
pect
)
Feed
back
Eva
luat
ion
of g
oals
Prac
tice
Mat
eria
l/med
ia
Tea
cher
cha
ract
eris
tics
Con
stru
ctiv
ist i
nstr
ucti
on
Indu
ctiv
e te
achi
ng
Con
cept
-ori
ente
d/
inte
grat
edin
stru
ctio
n
82 Stolarchuk & Fisher
(2001)
CO:
1+/2o
NCO:
2o/1-
CO:1o
NCO: 1o
CO:1o
NCO: 1+
CO:1o
NCO: 1o
CO:1o
NCO: 1+
CO:1o
NCO: 1o
130 Sumfleth et al.
(2004)
CO:1+
84 Taylor et al. (2003) CO:1+ CO:2o CO:
1+/1-
CO:
4+/1o
86 Thuen & Bru
(2000)
LP:
1+/1-
LP:1+ LP:2+ LP:1+
87 Tiedemann &
Billmann-M (2004)
CO:
2+/1o
88 Tomoff et al.
(2000)
CO:1- CO:1+
89 Townsend & Hicks
(1997)
NCO:
3+/2o
LP:2+
90 Turner (1995) LP:
3+/1o
163
ID
Aut
hors
Lea
rnin
g T
ime
Cla
ssro
om o
rgan
isat
ion
Ord
erly
, fun
ctio
nal l
earn
ing
envi
ronm
ent
Cle
ar a
nd s
truc
ture
d te
achi
ng
Act
ivat
ing
Lea
rnin
g to
use
lear
ning
str
ateg
ies
Cha
llen
ge
Ada
ptiv
e te
achi
ng
Sup
port
(m
utua
l res
pect
)
Feed
back
Eva
luat
ion
of g
oals
Prac
tice
Mat
eria
l/med
ia
Tea
cher
cha
ract
eris
tics
Con
stru
ctiv
ist i
nstr
ucti
on
Indu
ctiv
e te
achi
ng
Con
cept
-ori
ente
d/
inte
grat
edin
stru
ctio
n
91 Turner et al. (2002) LP:4o2-
106 Udziela (1996) CO:1o
107 Van Horn & Ramey
(2003)
CO:
16o/1-
CO:3+/
13o/1-
CO:1+/
15o/1-
109 Wenglinsky (2002) CO:1+ CO:1+ CO:
1o/1-
111 White &
Frederiksen (1998)
CO:
14+/2o
NCO:
5+/2o
112 Wigfield et al.
(2004)
NCO:
4o/1+
NCO:
4+/1o
132 Willms & Somers
(2001)
CO:
10+ /14o
NCO:1+/
10o/1-
CO:
19o/5
NCO:
12o
CO:
7+/17o
NCO: 1+
/10o/1-
114 Yair (2000) LP:
8+/3o/1-
LP:4+
164
Content summaries on instruction effectiveness studies
Table 5.9: Content summaries on instruction effectiveness studies (1995-2005)
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Adedayo, O.A. 1998 Math direct teaching,
interactive-cooperative,
interactive-individual
Achievement Three teaching methods: a)
traditional lecture/direct
teaching, b) interactive with
individual use of material,
c) interactive with
cooperative with use of
materials. Interactive
methods superior to
traditional, b)>females,
c)>males
+; Positive effects of
interactive teaching
compared to direct
instruction on mathematics
achievement; Gender
effects: females favouring
individual methods, males
favouring cooperative
methods
Nigeria College 165 Quasi-
experimental
ANOVA
Alexander, P. A.
Fives, H.,
Buehl, M. M.,
Mulhern, J.
2002 Science Situated/discovery
learning vs. traditional
science instruction;
Within situated
learning: a) teacher-led,
b) student led
Science
achievement,
interest, beliefs
Three teaching methods;
pre-post-test; science:
lessons on discoveries of
Galileo; comparison
situated/discovery learning
vs. traditional science
instruction; Within situated
learning: a) teacher-led, b)
student led
+; Positive effects of
situated learning on
achievement, interest and
beliefs; positive effect of
teacher-led situated
learning on achievement,
positive effect of student-
led situated learning on
beliefs
USA Middle
school
183 Quasi-
experimental
ANOVA
165
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Allsopp, D. H. 1997 Math Peer tutoring vs.
independent practice
Mathematics
problem solving
skills
Comparison of Peer
Tutoring with independent
student practice; additional
comparison of students at
risk of math failing with no
risk students
-; no differences between
groups in achievement
gains, no differences
between at-risk or not at-
risk students
USA Middle
school
262 Quasi-
experimental
ANOVA
Anderman, E. M.
Eccles, J. S.
Yoon, K. S.
Roeser, R.
Wigfield, A.
Blumenfeld, P.
2001 Math Goal-orientation:
mastery vs.
perfomance-oriented
instruction
Values, interest Amount of variance of
value development
explained by mastery- and
performance-oriented
instructional practices
-/o, performance-oriented
instructional methods
show negative impact on
value development;
mastery-oriented methods
show no effect
USA Middle
school
570 Quasi-
experimental
Multi-
level
Anderman, E. M.
Eccles, J. S.
Yoon, K. S.
Roeser, R.
Wigfield, A.
Blumenfeld, P.
2001 Reading Goal-orientation:
mastery vs.
perfomance-oriented
instruction
values/interest Amount of variance of
value development
explained by mastery- and
performance-oriented
instructional practices
-/o, performance-oriented
instructional methods
show negative impact on
value development;
mastery-oriented methods
show no effect
USA Middle
School
570 Quasi-
experimental
Multi-
level
Applebee, A. N.
Langer, J. A.
Nystrand, M.
Gamoran, A.
2003 Reading Discussion-based
instructional
approaches
achievement:
literacy
performance
Relationship between
literacy performance and
discussion-based
approaches, comparison of
high and low achieving
students
+; discussion-based
approaches have a positive
impact on literacy
performance, effect for
low and high achieving
students
USA Middle and
high school
1412 Quasi-
experimental
Multi-
level
166
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Arnold, D. H.
Fisher, P. H.
Doctoroff, G. L.
Dobbs, J.
2002 Math Cognitive activation:
math related activities
Mathematics
achievement,
interest
Intervention "Head Start
Classrooms"; goal:
engagement in math
activities; comparison
intervention/control
+; cognitive
activation/time on task has
a positive effect on math
achievement and interest
compared to control group
USA Pre-school 112 Quasi-
experimental
ANOVA
Ashman, A. F.
Gillies, R. M.
1997 Other/All Cooperative learning
skill training
quality of
student
behavior/engag
ement in
cooperative
settings,
development of
language
Comparison of student
trained for cooperative
learning skills and non-
trained students
+; trained students show
more positive engagement
in cooperative learning
settings, higher motivation
as well as a positive
development of the
language level
Australia Middle
school
192 Quasi-
experimental
ANOVA
Baumann, J. F.
Edwards, E. C.
Boland, E. M.
Olejnik, S.
Kame'enui, E. J.
2003 Math Vocabulary acquisition
strategy training:
contextual analysis
instruction (MC) vs.
traditional textbook
instruction (TV)
text
understanding
Comparison of contextual
analysis instruction vs.
traditional textbook
instruction and effects on
language achievement
+/o; MC better in transfer
tests, no difference in
overall text comprehension
measures
USA Elementary
School
157 Quasi-
experimental
ANOVA
167
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Baumert, J.
Köller, O.
2001 Math Cognitive activation,
practice, skills,
receptive teaching
Mathematics
achievement,
beliefs, learning
strategies, self-
concept, interest
Relationship between
teaching methods and
cognitive and non-cognitive
student outputs; data of the
German TIMSS Population
III sample
+, cognitive activation has
a positive relationship with
achievement, beliefs,
learning trategies, self-
concept, interest; -
receptive teaching/skill
learning: negative effect
on achievement, beliefs,
deep learning strategies
Germany High school Class
means
Correlational Multi-
level
Baumert, J.
Köller, O.
2001 Science time on task; student
experiments, teacher
experiments, inductive
teaching, computers,
cognitive activation,
application
science
achievement,
beliefs, learning
strategies,
interest
Relationship between
teaching practices and
cognitive and non-cognitive
student outputs; data of the
German TIMSS Population
III sample
cognitive activation:
positive effect on
achievement and interest;
student experiments:
positive effekt on interest,
inductive teaching:
negative effect on
achievement, beliefs,
learning strategies, interest
Germany Secondary II Class
means
Correlational Multi-
level
Bennacer, H. 2000 Reading Classroom
management, discipline
Achievement Investigation of the
relationship between
environment factors and
achievement
-; negative effect of
classroom rules on
achievement, positive
effect of classroom
organisation on
achievement
France Middle
school
Correlational Path
analysis
168
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Bianchini, J. A. 1997 Science student active
participation (talk)
science
achievement
Investigation of the
development of science
achievement in a
curriculum applying
groupwork (pre-post-
comparison); role of
student status in class for
the engagement in
groupwork and
developments
+; positive effect of
student talk on post-test
achievement
USA Middle
school
80 Other Regression
/correlatio
n
Borsch, F.
Jurgen-Lohmann, J.
Giesen, H.
2002 Science Cooperative learning science
achievement
Comparison of a Jigsaw-
cooperative learning
environment with a control
group (traditional)
+; positive effect of
cooperative learning on
achievement; effect
accounts for high and low
pre-knowledge students
Germany Elementary
school
370 Quasi-
experimental
ANOVA
Brush, T. A. 1997 Math Cooperative learning
vs. individual math
computer learning
mathematics
achievement
Investigation of the
Integrated Learning System
(ILS: learning strategy
training + computers) under
two conditions: cooperative
vs. individual math
computer learning
+; positive effect of
cooperative learning on
math achievement
compared to individual
learning
USA Elementary
school
65 Quasi-
experimental
ANOVA
169
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Burkam, D. T.
Lee, V. E.
Smerdon, B. A
1997 Science homework, individual
work, student
experiments, teacher
demonstrations,
computers, climate,
student engagement
science
achievement
Study investigates the role
of hands-on lab activities
on student achievement;
uses the NELS:88 data
base; investigates gender
differences
+/o/-; positive relationship
for climate, no relationship
for homework,
experiments, engagement,
negative for individual
work, teacher demo,
computers
USA High school 12120 Correlational Regression
/correlatio
n
Byrne, B.
Fieldingbarnsley, R.
1995 Reading Learning strategies
training: Phonemic
Awareness Training
Reading
achievement
Follow-up study on
phonemic awareness
training. Comparison of
trained and untrained
students in language
comprehension 2 and 3
years after training
+; positive effect of
phonemic awareness
training on language
comprehension two and
three years after training
Australia Pre-school 115 Experimental ANOVA
Cantrell, S. C. 1999 Reading Cognitive activation,
meaningful reading and
writing instruction
Reading and
Writing
achievement
Implementation study on
meaningful reading and
writing instruction;
comparison of high and low
implementation classes
+; positive effect of
instruction that is oriented
toward meaningful
learning on reading and
writing achievement
USA Elementary
school
41 Quasi-
experimental
ANOVA
Chang, C. Y.
2002 Science teacher-centred
instruction vs. student-
centred instruction
Science
achievement
Comparison of teacher-
centred vs. student-centred
multimedia computer-
assisted instruction (CAI)
+; positive effect of
teacher-centred CAI
instruction compared to
student centred instruction
Taiwan High school 244 Experimental ANOVA
170
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Chang, C. Y.
Barufaldi, J. P.
1999 Science Learning strategies
teaching: problem
solving instruction
Earth science
achievement
Comparison of a problem-
solving-based instructional
model with a control group
with respect to achievement
+; positive effect of
problem solving teaching
on achievement
Taiwan Middle
school
172 Experimental ANOVA
Chinn, C. A.
Anderson, R. C.
Waggoner, M. A.
2001 Reading cooperative learning Learning
processes /
engagement
Comparison of
collaborative reasoning and
recitation instruction and its
effects on student
engagement and cognitive
learning strategies
+; positive effect of
cooperative learning on
student cognitive learning
process
USA Primary
school
84 Quasi-
experimental
ANOVA
Chularut, P.
DeBacker, T. K.
2003 Other/All Learning strategies
teaching: concept
mapping
English as a
foreign
language
achievement,
learning
strategies, self-
efficacy
Comparison of concept
mapping group with control
group; effects of concept
mapping on cognitive,
motivational-affective and
meta-cognitive outputs
+; positive effect of
concept mapping
instruction on
achievement, learning
strategies, self-efficacy
USA High school,
college
97 Experimental ANOVA
171
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Clausen, M. 2001 Math pacing, monitoring,
discipline, speed,
classroom climate;
cognitive activation,
adaptive teaching;
repetitions
Mathematics
achievement &
interest,
development of
achievement
and interest
Study investigates the
relationship between
instructional variables and
mathematics achievement
and interest as well as
developments; Uses
German sample of the
TIMSS 1995 Video Study
positive correlation of
pacing on achievement;
positive r between
individual reference norms
and interest, Negative r
between individualising
and achievement; Positive
r between climate and
interest, negative r
between repetitive practice
and achievement
Germany Middle
school
53 Correlational Regression
/correlatio
n
Crinjen, Alfons
A.M.
Feehan, Michael
1998 Reading Mastery Learning Reading
achievement
Comparison of Mastery
Learning intervention to a
control group
+/-; positive effect of
Mastery Learning on
reading achievement in the
first year of reading
instruction, negative effect
at the end of elemenary
school
USA Elementary
school
363 Experimental Multi-path
D'Agostino, J. V. 2000 Reading Teacher-led basic skills
instruction
Reading
achievement
and
development
Relationship of teacher-led
basic skills instruction and
reading achievement;
Longitudinal data of Title I
project
+; positive effect of basic
skill instruction on
achievement as well as
achievement gains
USA all schools 3308 Correlational Multi-
level
172
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
D'Agostino, J. V. 2000 Math Teacher-led basic skills
instruction
Mathematics
achievement
and
development
Relationship of teacher-led
basic skills instruction and
math achievement;
Longitudinal data of Title I
project
+; positive effect of basic
skill instruction on
achievment as well as
achievement gains
USA all schools 2996 Correlational Multi-
level
Davidson, J.
Elcock, J.
Noyes, P.
1996 Reading computer-assisted
practice
Reading
achievement
Comparison of computer-
assisted practice and
control and its effects on
reading achievement
+; positive effect of
computer-assisted practice
on reading achievement
UK Primary
school
60 Quasi-
experimental
ANOVA
De Fraine, B.
Van Damme, J.
Van Landeghem, G.
Opdenakker, M. C.
Onghena, P.
2003 Other/All adaptive teaching,
feedback, taking into
account student pre-
requisites
Dutch language
achievement
Amount of explained
variance by student,
teaching and school factors;
LOSO-project
o: no effect of instructional
variables on dutch
language achievement; +
positive effect of learning
climate on achievement
Flanders /
Belgium
Secondary
school
2569 Correlational Multi-
level
de Jong, R.
Westerhof, K. J.
Kruiter, J. H.
2004 Math opportunity to learn,
adaptive teaching:
grouping, structure and
clarity
Mathematics
achievement
Amount of explained
variance explained by
teaching factors
+: opportunity to learn,
grouping according to pre-
knowledge, structure &
clarity
Netherland
s
Secondary
school
Correlational Multi-
level
Dowdell, T. 1996 Reading direct instruction Reading
achievement
Comparison of direct
teaching group with control
group; sample: low
achieving students
+; direct instruction had a
positive effect on
achievement gains
USA Middle
School
72 Quasi-
experimental
ANOVA
173
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Driessen, G.
Sleegers, P.
2000 Other/All approach intensity,
reading time,
opportunity to learn,
homework, progress
registration, individual
orientation, attention
reading strategy, tests,
expectations for
students, basic skills
Mathematics
achievement
Relationship between
instructional variables as
well as consistency of
instruction with math
achievement; PRIMA-
study
+/o/-; positive effect of
approach intensity,
feedback, test frequency,
and basic skill teaching on
math achievement;
negative effect of
individual orientation; no
effects of all other
variables
Netherland
s
Elementary
school
7410 Correlational Multi-
level
Driessen, G.
Sleegers, P.
2000 Other/All approach intensity,
reading time,
opportunity to learn,
homework, progress
registration, individual
orientation, attention
reading strategy, tests,
expectations for
students, basic skills
Language
Achievement
Relationship between
instructional variables as
well as consistency of
instruction with language
achievement; PRIMA-
study
o/-; negative effect of
individual orientation in
teaching on language
achievement; no effects of
all other variables
Netherland
s
Elementary
school
7410 Correlational Multi-
level
Einsiedler, W.
Treinies, G.
1997 Science structuring methods Science
achievement
Comparison of classes with
implementation of various
structuring methods and
control classes
-; no differences between
groups with respect to the
effectiveness of structuring
methods
Germany Elementary
school
456 Experimental Multi-
level
174
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Foorman, B. R.
Francis, D. J.
Fletcher, J. M.
Schatschneider, C.
Mehta, P.
1998 Reading direct instruction Reading
achievement:
word reading
Comparison of three
treatment groups: a) direct
instruction, b) embedded,
c) implicit. Sample: Title 1
services, at-risk students
+; direct instruction with
highest achievement gains
USA Elementary
school
285 Experimental Multi-path
Freedman, M. P. 1997 Science hands-on laboratory
experiences
Science
achievment,
science
attitudes
Comparsion of classes with
hands-on laboratory
experiences and control
classes
+/o; hands-on laboratory
experiences show positive
effects on achievement, no
effect on attitudes
USA Experimental ANOVA
Fuchs, D.
Fuchs, L. S.
Mathes, P. G.
Simmons, D. C.
1997 Reading Peer-assisted learning
strategies
Reading
achievement
Effectiveness of a
classwide peer tutoring
program for low and
average achieving students
+; peer-assisted learning
strategies show positive
effects on the development
of reading achievement;
independent of student
types
USA Elementary,
middle
school
Experimental ANOVA
Fuchs, L. S.
Fuchs, D.
Yazdian, L.
Powell, S. R.
2002 Math Peer-assisted learning
strategies
Mathematics
achievement
Effects of a peer tutoring
program for low , average
and high achieving students
+; peer-assisted learning
strategies show positive
effects on the development
of math achievement;
independent of student
types
USA Elementary 323 Experimental ANOVA
George, R.
Kaplan, D.
1998 Science Science activities Science attitude Structure of relationship
between teacher and parent
factors for student attitudes
towards science; sample of
NELS:88
+, science activities have a
positive relationship and
direct effect on science
attitudes
USA all school
types
7980 Correlational Path
analysis
Ginsburg-Block, M.
D.
Fantuzzo, J. W.
1998 Math Peer collaboration,
problem solving
Mathematics
achievement,
motivation,
self-concept
Study investigates effects
of two instructional
methods: peer collaboration
and problem solving on
+; both approaches show
positive effects on
mathematics achievement,
motivation and self-
USA Elementary
school
104 Experimental ANOVA
175
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
mathematics achievement,
academic motivation and
self concept; sample: low
achieving students
concept; no interaction
effects found
Gruehn, S. 1995 Math achievement pressure,
discipline, rules, speed
of interactions, time on
task, practice, cognitive
activation, pacing,
student participation,
individual orientation,
monitoring, feedback,
clarity, classroom
climate
Mathematics
development,
self-concept,
interest
Study investigates the
relationship between
teaching factors and the
development of
mathematics achievement,
self-concept and pleasure to
go to school on the basis of
the German longitudinal
BIJU sample
+/o/-; achievement:
positive r for discipline,
cognitive activation,
pacing, monitoring,
feedback, clarity, climate;
self-concept: positive r for
discipline, monitoring,
feedback, climate; interest:
discipline, monitoring,
feedback
Germany Middle
school, all
types
Correlational Regression
/correlatio
n
Guthrie, J. T.
Van Meter, P.
McCann, A. D.
Wigfield, A.
Bennett, L.
Poundstone, C. C.
Rice, M. E.
et al.
1996 Reading Concept-oriented
reading instruction
(CORI)
reading
achievement,
literacy
engagement
Study investigates the
implementation of the
CORI intervention with
pre-post-test comparisons
+; students in CORI
classrooms showed a
positive development in
literacy engagement
(searching, drawing,
writing, transfer, text
comprehension)
USA Elementary
school
140 Quasi-
experimental
ANOVA
Guthrie, J. T.
Van Meter, P.
Hancock, G. R.
Alao, S.
Anderson, E.
McCann, A.
1998 Reading Concept-oriented
reading instruction
(CORI)
reading
achievement,
literacy
engagement
The CORI context
increased strategy use,
conceptual learning and
text comprehension more
than traditional instruction
+; positive effect of CORI
on achievement and
strategy use
USA Elementary
school
172 Quasi-
experimental
Path
analysis
Guthrie, J. T.
Anderson, E.
1999 Reading Concept-oriented
reading instruction
Reading
engagement/ach
The CORI context
increased strategy use,
+; positive effect of CORI
on achievement and
USA Elementary
school
106 Quasi-
experimental
ANOVA
176
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Alao, S.
Rinehart, J.
(CORI) ievement,
reading strategy
use
conceptual learning and
text comprehension more
than traditional instruction,
when background is
controlled
strategy use
Guthrie, J. T.
Wigfield, A.
Von Secker, C.
2000 Reading Concept-oriented
reading instruction
(CORI)
Reading
motivation and
strategy use
The CORI context
increases reading
motivation and strategy use
+; positive effect of CORI
on reading motivation and
strategy use
USA Elementary
school
162 Quasi-
experimental
Multi-
level
Guthrie, J. T.
Schafer, W. D.
Huang, C. W.
2001 Reading Opportunity to
learn/read
Reading
engagement
Study investigates NAEP
data to analyse the
relationship between
opportunity to read in
classrooms and the students
reading engagement
+; positive effect of
opportunity to read on
reading engagement
USA Elementary
school
577 Correlational Multi-
level
177
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Hamilton, L. S.
Nussbaum, E. M.
Kupermintz, H.
Kerkhoven, J. I. M.
Snow, R. E.
1995 Science materials, experiments,
problem solving,
understanding
Science
achievement
Study investigates the
relationship between
instructional variables and
science achievement;
sample: NELS:88
o; no effects of
instructional variables on
science achievement
USA High school 5041 Correlational Regression
/correlatio
n
Hardre, P. L.
Reeve, J.
2003 Other/All quality of teacher
support, perceived
teacher support
Motivation,
intention to
persist
Study investigates the role
of teacher support for
student motivation and
persistence not to drop-out
+; positive relationship
between teacher support
and self-determined
learning motivation,
perceived competence and
intention to persist
USA High school 483 Correlational Path
analysis
Hardy, I.
Jonen, A.
Möller, K.
Stern, E.
2004 Science representation formats science
understanding
Study compares different
representation formats and
their effects on science
understanding. The two
variables are integrated
format (experimental) and
self-constructed
representations (control)
+; positive effect of the
use of integrated
representations on
conceptual understanding
Germany Elementary
school
98 Experimental ANOVA
Helmke, A.
Weinert, F.E.
1997 Math classroom managem.,
structuredness,
adaptivity, variability
of teaching methods,
social climate, clarity,
climate, active student
participation
mathematics
achievement
gains
Study investigates the
relationship between
instructional factors and
achievement gains in a
longitudinal German
elementary study
o;+; positive relationships
for classroom
management,
structuredness, adaptivity,
variability of teaching
methods, clarity and active
student participation;
Germany Elementary
school
Correlational Regression
/correlatio
n
Henderson, R. W.
Landesman, E. M.
1995 Math Integrated mathematics
instruction
mathematics
achievement,
attitudes
Study investigates an
intervention program on an
integrated mathematics
instruction; comparisons of
+; posive effect of
integration math
instruction on math
achievement; no effect on
USA Middle
school
102 Experimental ANOVA
178
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
experimental and control
groups
math attitudes
Hill, P. W.
Rowe, K. J.
1998 Reading high expectations,
appropriateness/adaptiv
ity, incentives, time on
task, warmth
literacy
achievement,
student
attentiveness
Study investigates
relationship between
several teaching factors and
literacy achievement, as
well as student
attentiveness; furthermore
student attentiveness is an
important mediating factor
for student literacy
achievement
+; literacy achievement:
positive relationship for
expectations,
appropriateness, time;
attentiveness: positive r for
time, appropriateness,
incentives and warmth
Australia Kindergarden
- Middle
school
6678 Correlational Multi-
level
Hogan, K. 1999 Science Thinking aloud
together
metacognitive
knowledge
about
collaborative
reasoning,
science
understanding
Intervention study with
control groups to
investigate the effect of
thinking aloud on students
scientific reasoning and
understanding of science
concepts
+/o; positive effect of the
intervention on
metacognitive abilities; no
effect on science
understanding
USA Middle
school
163 Experimental ANOVA
179
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Hopkins, K. B.
McGillicuddy-
DeLisi, A. V.
DeLisi, R.
1997 Math direct instruction vs.
constructivist
instruction
Math
achievement:
computational
performance
Experimental study of the
effects of directs instruction
vs. constructivist
instruction on students
computational performance
+; direct teaching showed
higher computational
performance than
constructivist instruction;
girls outperformed boys in
both formats;
USA Elementary
school
74 Experimental ANOVA
Houghton, S.
Litwin, M.
Carroll, A.
1995 Reading Peer mediated
instruction
Reading
achievement
Study investigates the
effects of peer mediated
instruction on reading
achievement; investigates
gain over the course of the
intervention
+; positive gains in
reading achievement over
the course of the
intervention; for tutors as
well as tutees
Australia Primary
school
24 Experimental ANOVA
Houtveen, A. A. M.
Booij, N.
de Jong, R.
van de Grift, Wjcm
1999 Reading Adaptive Instruction
approach: includes
optimizing time on
task, direct instruction,
phonics instruction
method, diagnostic
teaching
Reading
achievement
Quasi-experiment on
effects of adaptive
instruction on reading
results of children in first
year of reading instruction
in Dutch primary schools
+; positive effect of
adaptive instruction
approach on reading
compared to control group
Netherland
s
Primary
school
456 Quasi-
experimental
ANOVA
180
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Jacobson, C.
Lehrer, R.
2000 Math Understanding-oriented
math instruction
Mathematics
achievement
Study investigates
differences between
teachers in the
implementation of an
understanding-oriented
instruction in mathematics;
differences in the
implementation go along
with different math
achievements; teacher high
in implementation scored
higher
+; positive effect of high
implementation of
understanding-oriented
math instruction compared
to low implementation
USA Elementary
school
Quasi-
experimental
Other
Jovanovic, J.
King, S. S.
1998 Science Hands-on science
activities
Science
attitudes
Study investigates the
effect of regular hands-on
activities on science
attitudes
+; positive effect of active
participation in hands-on
activities on development
of science attitudes
USA Middle
school
165 Quasi-
experimental
Regression
/correlatio
n
Klieme, E.
Rakoczy, K.
2003 Math Discipline,
achievement pressure,
support, monitoring,
clarity, cognitive
activation
Math
achievement,
interest
Study investigates the
relationship and the amount
of variance explained by
teaching factors on student
achievement and interest;
German national PISA
sample
achievement; positive r for
discipline, support, clarity
and cognitive activation;
interest: positive r for
discipline, support,
monitoring, clarity,
cognitive activation
Germany Gymnasium Correlational Regression
/correlatio
n
181
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Kramarski, B.
Mevarech, Z. R.
1997 Math Metacognitive training Math
achievement,
metacognitive
strategies
Study investigates the
effects of metacognitive
training on students' ability
to construct graphs and
reflect on learning
+; metacognitive treatment
showed positive results on
math achievement and
reflection on learning
Israel High school 68 Experimental ANOVA
Kramarski, B.
Mevarech, Z. R.
Lieberman, A.
2001 Math Metacognitive training Math
achievement,
math
explanations
Study compares 3 methods:
MT in math and english;
MT in math, control;
+; MT in both subjects
outperformed MT in math;
both outperformed control
group; both for math
achievement as well as
math explanations
Israel High school 182 Quasi-
experimental
ANOVA
Kroesbergen, E. H.
Van Luit, J. E. H.
Maas, C. J. M.
2004 Math Constructivist teaching
vs. direct teaching
Mathematics
achievement,
learning
strategies,
motivation
Study compares the effects
of small-group
constructivist and explicit
mathematics instruction in
basic multiplication on
low-achieving students'
performance and
motivation
+; both conditions showed
positive effects compared
to control; direct teaching
outperformed
constructivist teaching in
achievement
Netherland
s
Elementary
school
265 Experimental Multi-
level
Kuklinski, M. R.
Weinstein, R. S.
2001 Reading Teacher expectations Reading
achievement
Study investigates the role
of teacher expectations for
student self-expectations
and reading achievement
+; no relationship between
teacher expectations and
student self expectations;
but positive relationship
for achievement
USA Elementary
school
376 Correlational Path
analysis
182
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Kunter, M. 2004 Math cognitive activation,
relevance, support,
discipline, structure,
monitoring, active
student participation,
individual orientation
Mathematics
achievement,
interest
Effects of instruction
factors on student
achievement and interest
development
achievement: positive
relationship for student
participation and negative
for individual orientation;
interest: positive
relationship for discipline,
structure, monitoring,
participation
Germany Secondary
school
1900 Correlational Multi-
level
Kupermintz, H.
Ennis, M. M.
Hamilton, L. S.
Talbert, J. E.
Snow, R. E.
1995 Math direct instruction,
individual work,
applications, cognitive
activation,
computation,
understanding,
homework
Mathematics
achievement
Study investigates the
relationship between
instructional variables on
math achievement; uses
data of NELS:88 study;
Math0th grade sample
-/o/+; positive r for direct
instruction, cognitive
activation, teaching for
understanding, no effect
for computation,
homework; negative r for
individualization,
applications
USA High school 5460 Correlational Regression
/correlatio
n
Kupermintz, H.
Snow, R. E.
1997 Math individual work,
applications, cognitive
activation, compuation,
discussion, media,
teacher control
Mathematics
achievement
Study investigates the
relationship between
instructional variables on
math achievement; uses
data of NELS:88 study;
Math2th grade sample
-/o/+; positive r for
cognitive activation and
discipline; no relationship
for computation,
discussion, media;
negative r for individual
work, applications
USA High school 2373 Correlational Regression
/correlatio
n
183
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Kyriakides, L.
Campbell, R. J.
Gagatsis, A.
2000 Math clarity, teacher support,
time on task,
opportunity to learn
(homework)
Mathematics
achievement
Study investigates several
aspects of school and
teaching factors on math
achievement; compares
different sources of
information
(teacher/student)
o/+; no effects for clarity,
teacher support, time on
task; positive effect for
opportunity to learn
(measured by amount of
homework)
Cyprus Primary
school
1051 Correlational Multi-
level
Labudde, P.
Pfluger, D.
1999 Science Constructivist
approaches in science
teaching: relevance,
contexts/contents,
communication and co-
operation
Mathematics
achievement,
interest
Study investigates the
predominance of
constructivist aspects in
science teaching in
Switzerland; furthermore it
investigates the relationship
between constructivist
aspects and TIMSS
achievement and interest
scores
achievement: positive r for
relevance, no relationship
for communication and
student experiments;
interest: positive r for all
variables
Switzerlan
d
High school 670 Correlational Regression
/correlatio
n
Lazarowitz, R.
Baird, J. H.
Bowlden, V.
1996 Science Cooperative mastery
learning (jigsaw) vs.
individual mastery
learning
Science
attitudes
Study compares group
mastery learning with
individualized mastery
learning
Student in group mastery
learning score higher on
self-esteem and positive
attitudes than students in
individualized mastery
learning groups. IML also
showed a decline in
attitudes
Israel/USA High school 113 Quasi-
experimental
ANOVA
184
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Lowther, D. L.
Ross, S. M.
Morrison, G. M.
2003 Other/All Laptop computers Writing
achievement;
problem-
solving
Study investigates the 24h
access of students to laptop
computers and its effects on
writing achievement and
problem-solving skills;
experimental study
+; positive effect of
treatment group on writing
assessment as well as
problem-solving skills
USA Middle
school
118 Experimental ANOVA
Luyten, H.
de Jong, R.
1998 Math homework, structure,
clarity, quality of
teacher-student
interactions
Mathematics
achievement
Study investigates school
and teacher effects in Dutch
secondary schools
o/+; effect of homework
assignments and clarity on
math achievement; no
effect of structure and
pleasant lessons
Netherland
s
Secondary
school
956 Correlational Multi-
level
Mac Iver, M.A.
Kemper, E.
Stringfield, S.
2003 Math Direct instruction
curriculum
Mathematics
achievement
4-year study investigated
the implementation of the
Baltimore Curriculum
Project (BCP) that uses a
combination of direct
instruction and core
knowledge
o; no effect of direct
instruction on math
achievement
USA Primary
school
Quasi-
experimental
ANOVA
Mac Iver, M.A.
Kemper, E.
Stringfield, S.
2003 Reading Direct instruction
curriculum
Reading
achievement
4-year study investigated
the implementation of the
Baltimore Curriculum
Project (BCP) that uses a
combination of direct
instruction and core
knowledge
+/o; mixed effects of
direct instruction on
reading achievement
USA Primary
school
Quasi-
experimental
ANOVA
185
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Mac Iver, M.A.
Kemper, E.
Stringfield, S.
2003 Math Direct instruction
curriculum
Math
achievement
4-year study investigated
the implementation of the
Baltimore Curriculum
Project (BCP) that uses a
combination of direct
instruction and core
knowledge
+/o; mixed effects of
direct instruction on
reading achievement
USA Primary
school
Quasi-
experimental
ANOVA
Marks, H. M. 2000 Other/All Authentic
instruction/relevance;
teacher support
Student
engagement
Study investigates several
factors for student
engagement in instructional
activity
+; positive effect of
relevance of
contents/authentic
instructional work and
teacher support
USA Elementary,
Middle, High
school
3669 Correlational Multi-
level
Martinez, J. G. R.
Martinez, N. C.
1999 Math Mastery learning Mathematics
achievement
Study investigates the
effect of mastery learning
on math achievement by
experimental design
o; no differences between
treatment and control
group
USA College 80 Experimental ANOVA
Mathes, P. G.
Howard, J. K.
Allen, S. H.
Fuchs, D.
1998 Reading Peer-assisted learning
strategies
Reading
achievement
Study investigates the
effects of PALS instruction
compared to usual reading
instruction
+; positive effects of
PALS on reading
achievement; greatest
gains for low achieving
students
USA Elementary
school
96 Quasi-
experimental
ANOVA
186
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
McGuinness, D.
McGuinness, C.
Donohue, J.
1995 Reading Phonological
awareness training
Reading
achievement
Study investigates the
effect of a phonological
awareness training on
reading achievement
compared to a control
group
+; positive effect of PA on
reading achievement
USA Primary
school
45 Quasi-
experimental
ANOVA
Mevarech, Z. R. 1999 Math Metacognitive learning
strategy training within
cooperative learning
settings
Math problem
solving tasks
Study investigates effects
of 3 learning settings:
cooperative (control),
cooperative with learning
strategies in direct
instruction, cooperative
with metacognitive training
+; positive effect of
metacognitive training on
problem solving
achievement
Israel High school 174 Quasi-
experimental
ANOVA
Mevarech, Z. R.
Kramarski, B.
1997 Math IMPROVE: method
that combine learning
strategies, practice,
reviewing, mastery,
verification and
enrichment
Mathematic
achievement
(study II)
Study investigates the
effects of IMPROVE on
mathematics processes
compared to a control
group
+; positive effect of
IMPROVE on math
achievement
Israel High school 247 Quasi-
experimental
ANOVA
Mevarech, Z. R.
Kramarski, B.
1997 Math IMPROVE: method
that combines learning
strategies, practice,
reviewing, mastery,
verification and
enrichment
Mathematics
achievement
(study I)
Study investigates the
effects of IMPROVE on
mathematics achievement
compared to a control
group
+; positive effect of
IMPROVE on math
achievement
Israel High school 265 Quasi-
experimental
ANOVA
187
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Mevarech, Z. R.
Kramarski, B.
2003 Math Metacognitive training
vs. worked-out
examples
Mathematics
achievement
Study investigates 2
approaches within
cooperative learning
settings: a) meta-cognitive
training, b) learning from
worked-out examples
+; positive effect of
metacognitive training
compared to worked-out
examples on math
achievement
Israel High school 122 Quasi-
experimental
ANOVA
Miller, S. D.
Meece, J. L.
1997 Reading Reading and language
arts assignments:
increasing opportunity
to learn
Motivation/inter
est, learning
strategies
Study examines how
different reading and
language art assignments
influenced 3rd-grade
students' motivational
goals, strategy use, and
anxiety.
+/o; In classes with high
implementation of
intervention students
showed less ego-
orientation in motivational
goals, but no differences in
their strategy use
USA Primary
school
187 Quasi-
experimental
ANOVA
Möller, K.
Jonen, A.
Hardy, I.
Stern, E.
2002 Science Constructivist teaching;
3 conditions: a) without
structure, b) with
structuring elements, c)
direct teaching
Science
achievement
Study investigates the
effect of structure within
constructivist teaching and
compares 3 conditions: a)
without structure, b) with
structuring elements, c)
direct teaching
+; group a outperforms
group b and c; group b
outperforms group c
Germany Elementary
school
190 Quasi-
experimental
ANOVA
Muijs, D.
Reynolds, D.
2000 Math Direct teaching, time
on task, constructivist
methods, math
language, effective
teaching
Mathematics
achievement
Study investigated the
effect of teacher behaviours
and classroom organisation
on student progress in
mathematics in years Math-
5 of primary schools
o/+; positive relationship
for direct teaching, time on
task, constructivist
methods and effective
teaching; no relationship
for math language
UK Primary
school
2128 Correlational Multi-
level
Muthukrishna, N.
Borkowski, J. G.
1995 Math Learning strategies
training; differences in
learning strategies
training for discovery
Mathematics
achievement,
motivation,
processes
Study investigates the role
of learning strategies
training on mathematics
understanding, learning
+; positive effect of
learning strategies training
compared to control group
for achievement,
USA Primary
school
106 Quasi-
experimental
ANOVA
188
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
learning, direct and
discovery combined;
direct teaching
processes and overall
interest and motivation
motivation, and learning
strategies; differences
between treatment groups
show positive effects for
discovery learning
compared to direct
instruction
Nolen, S. B. 2003 Science Narrow-focused
classwork and low
quality of teacher-
student interactions;
cognitive activation
Science
achievement;
science
motivation
Study investigation the
relationship between
student perceptions of
teaching quality and their
achievement and
motivation development in
science
-; negative effect of
narrow-focused class work
and low quality of teacher-
student interactions; +;
positive effect of cognitive
activation on student
motivation
USA High school 377 Correlational Multi-
level
189
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Pauli, C.
Reusser, K.
Waldis, M.
Grob, U.
2003 Math Constructivist teaching
approaches
Mathematics
processes
(learning
motivation,
engagement),
interest,
achievement
Video study that
investigates the practice of
so-called "enrichment of
learning settings and
teaching methods" (ELF) in
the sense of constructivist
teaching in Switzerland and
its effects on student
achievement, interest, and
learning processes
+/o; positive effect of ELF
on learning processes; no
differences compared to
direct teaching on
achievement and interest
Switzerlan
d
Secondary
school
1402 Quasi-
experimental
ANOVA
Perry, N. E. 1998 Other/All Self-regulated learning
strategies teaching
Motivation/attit
udes, learning
strategies
Study investigates writing
and portfolio activities as a
method to teach self-
regulated learning
strategies. Effects of the
intervention (high and low
implementers) with respect
to student attitudes and use
of learning strategies are
taken into account
+; positive effect of high
implementation of self-
regulated learning
strategies on attitudes and
learning strategies
USA Primary
school
94 Quasi-
experimental
ANOVA
Ramsden, J. M. 1997 Science Context-based
instruction
Chemistry
grades and
predicted
achievement
scores
Study compares the
context-based SALTERS
approach with a more
traditional approach
o; no difference between
groups
UK Comprehensi
ve school
216 Quasi-
experimental
ANOVA
190
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Reezigt, G.J.
Creemers, B.P.M.
de Jong, R.
2003 Other/All Quality of curriculum;
implementation
curriculum; use of
tests, grouping
procedures, homework,
clear goal setting,
evaluation, feedback,
corrective instruction,
time for learning,
opportunity to learn
Mathematics
and language
achievement
Study tests a model on
educational effectiveness;
investigates the effects of
teaching variables on math
and language achievement
-/o/+; +; positive effects
for quality of curriculum,
implementation of
curriculum, homework,
clear goal setting; o; no
effects for testing,
grouping, evaluation,
corrective instruction; -;
for feedback, time for
learning; opportunity to
learn
Netherland
s
Elementary
school
3762 Correlational Multi-
level
Seidel, T. 2003 Science Time on task, student
work/experiments;
active student
participation, cognitive
activation, relevance of
contents, quality of
teacher student
interactions, feedback
learning
processes:
motivation and
learning
strategies
Video study on teaching
factors and quality of
learning processes
+; positive effects of
variables on learning
motivation and learning
strategies
Germany Secondary
school
137 Quasi-
experimental
ANOVA
Seidel, T.
Prenzel, M.
et al.
2002 Science time on task, student
work/experiments;
individual work
learning
processes:
motivation and
learning
strategies
Video study on teaching
factors and quality of
learning processes
o/+; no effects for student
experiments; effects for
time on task, amount of
time for individual work
Germany Secondary
school
318 Quasi-
experimental
ANOVA
191
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Seidel, T.
Rimmele, R.
Prenzel, M.
2003 Science Quality of class work:
indicators of active
student participation,
cognitive activation,
relevance by using
examples, quality of
teacher-student
interactions, feedback
learning
processes,
science interest
Video study on teaching
factors and quality of
learning processes and
interest development
+; effect of quality of class
work on learning
processes and science
interest
Germany Secondary
school
344 Quasi-
experimental
Multi-
level
Seidel, T.
Rimmele, R.
Prenzel, M.
2005 Science Clear goal setting and
lesson coherence
learning
processes,
science
achievement,
science interest
Video study on teaching
factors and quality of
learning processes,
achievement and interest
development
+/o; effect of goal clarity
and coherence on learning
processes and achievement
development; no effect on
science interest
Germany Secondary
school
344 Quasi-
experimental
Multi-
level
She, H. C.
Fisher, D.
2002 Science Teacher support:
cognitive activation,
Encouragement &
praise, climate, control
enjoyment of
science lessons,
interest in
science
Study investigates teacher
communication behaviour
and its role for students'
enjoyment of science and
their interest in science
-/o/+; processes: positive r
for cognitive activation,
climate; no relationship for
support, feedback;
negative r for control;
interest: positive r for
cognitive activation, no
relationship for support,
feedback, climate, control
Taiwan High school 1138 Correlational Regression
/correlatio
n
192
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Staub, F. C.
Stern, E.
2002 Math Constructivist teacher
beliefs
Mathematics
achievement
gains
Study investigates the
effect of a cognitive
constructivist orientation on
students achievement gains
in word-problem solving
+; positive effect of
teacher constructivist
orientations on student
achievement gains from
grade 2 to 3
Germany Elementary
school
496 Correlational Multi-
level
Stevens, R.J.
Slavin, R.E.
1995 Reading Cooperative learning
program (CIRC)
Reading
achievement,
metacognitive
strategies,
attitudes
2-year study on long-term
effects of a cooperative
learning approach called
CIRC: student worked in
heterogeneous learning
teams, explicit instruction
on learning strategies,
writing process approach
+; positive effects over 2
years for CIRC approach
compared to matched
control schools on
achievement and
metacognitive strategies;
no effect on attitudes
USA Elementary
school
1299 Experimental Multi-
level
Stipek, D.
Salmon, J. M.
Givvin, K. B.
Kazemi, E.
Saxe, G.
MacGyvers, V. L.
1998 Math Positive affect/climate;
orientation towards
understanding
Mathematics
processes
(motivation,
positive
emotions,
engagement),
math
achievement
Study investigates
associations among
instructional practices,
motivation and learning of
fractions in mathematics
+; positive effect of
teachers showing positive
affect and teaching
towards understanding on
student learning processes;
no effect of affect on
achievement, but effect of
teaching towards
understanding on
achievement
USA Elementary
school
624 Correlational Regression
/correlatio
n
193
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Stolarchuk, E.
Fisher, D.
2001 Other/All Structure, teacher
support,
feedback/monitoring,
discipline
Attitudes
towards
computers;
achievement:
enquiry skills
Study is part of an
evaluation of the
effectiveness of laptop
computers in science
classrooms; thereby, effects
of teaching factors on
student attitudes and
enquiry skills are taken into
account
o/+; attitudes: positive r
for structure, support; no r
for feedback and
discipline; enquiry skills:
no r
Australia High school 433 Correlational Regression
/correlatio
n
Sumfleth, E.
Rumann, S.
Nicolai, N.
2004 Science Cooperative learning
vs. direct teaching
Science
understanding
Study compares effects of
cooperative learning with
effects of direct instruction
in an experimental setting
+; positive effect of
cooperative learning on
science understanding
Germany Secondary
school
210 Experimental ANOVA
Taylor, B. M.
Pearson, P. D.
Peterson, D. S.
Rodriguez, M. C.
2003 Reading Higher-level
questioning; time on
task, comprehension
reading strategies
Reading
comprehension
Study investigated the
effects of teaching on
students' reading
achievement; students were
chosen from high poverty
schools in the Unites States
+; positive effects of
higher-level questioning,
time on task; - for reading
strategies
USA Primary
school
792 Correlational Multi-
level
Thuen, E.
Bru, E.
2000 Other/All Teacher support,
monitoring, autonomy,
classroom climate,
competition, relevance
Engagement Study investigates on-task
orientation as an indicator
for student engagement and
its relationship to teaching
variables; representative
Norwegian sample
Positive relationships for
teacher support,
monitoring, autonomy,
climate, competition,
relevance
Norway 2006 Correlational Path
analysis
194
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Tiedemann, J.
Billmann-Mahecha,
E.
2004 Other/All Classroom climate Mathematics
achievement
Study investigates the role
of student and class
variables on the
development of
mathematics and reading
achievement; as teaching
variable classroom climate
is taken into consideration
+; positive effect of
classroom climate on
mathematics achievement
Germany Elementary 710 Correlational Multi-
level
Tomoff, Joan
Thompson, Marilyn
Behrens, John
2000 Math Drill & practice;
working with textbooks
Mathematics
achievement
Study uses TIMSS data to
investigate relationship
between teaching practices
and TIMSS achievement
scores
negative effect of drill &
practice on achievement
scores; positive effect of
working with textbooks
USA High school 3400 Correlational Multi-
level
Townsend, M. A. R.
Hicks, L.
1997 Math Cooperative learning Mathematics
value
Study compares
cooperative learning groups
with traditional teaching
with respect to subject
value and motivation to
learn
+; positive effect of
cooperative learning on
values and motivation to
learn; gender effects: girls
perceive higher values and
motivation within
cooperative learning
settings; boys show no
difference with
cooperative or traditional
settings
New
Zealand
Middle
school
162 Quasi-
experimental
ANOVA
195
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Turner, J. C. 1995 Reading Cognitive activation:
open tasks
Reading
processes:
learning
strategies,
learning
motivation
Study examines the effects
of instructional contexts on
children's reading
motivation. Observers
coded types of literacy
tasks (open vs. closed) and
the children's voluntary use
of motivated behaviours
(strategy use, persistence)
+; positive effect on
student strategy use,
persistence and attention
control
USA Primary
school
84 Quasi-
experimental
Other
Turner, J. C.
Midgley, C.
Meyer, D. K.
Gheen, M.
Anderman, E. M.
Kang, Y.
Patrick, H.
2002 Other/All Mastery goal
orientation in class,
performance goal
orientation in class
Avoidance
behaviour of
students
Study investigates the
relation between learning
environment and students'
reported use of avoidance
strategies
+; positive effect of
mastery goal orientation
on less avoidance
behaviour; no effect of
performance goal
orientation on avoidance
behaviour
USA Elementary
school
1197 Correlational Multi-
level
Udziela, T. 1996 Other/All Formal learning
strategy training
Reading
achievement
Study examined the effect
of teaching formal learning
strategy skills on reading
achievement
o; no effect of formal
learning strategy training
on reading achievement
USA Middle
school
181 Quasi-
experimental
ANOVA
196
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Van Horn, M. L.
Ramey, S. L.
2003 Reading Constructivist teaching:
project for
developmentally
appropriate practices
(DAP); variables:
integrated curriculum,
climate, student centred
methods
Reading
achievement
The educational ideology
of Developmentally
Appropriate Practices
(DAP) in the sense of
constructivist teaching is
influential, but without
little empirical studies;
study investigates long-
term effects of DAP on
reading achievement
-/o/+; year 1: negative
effects for integrated
curriculum and climate;
positive effect for student-
centred methods; year 2-3:
no effects of DAP on
achievement scores
USA Primary
school
1995:
4764;
1996:
2690;
1997:
1569;
Experimental Multi-
level
Wenglinsky, H. 2002 Math Hands-on learning;
higher order thinking;
assessment
NAEP test:
mathematics
Study explores the link
between classroom
practices and student
academic performance by
applying multilevel
modelling to 1996 NAEP
data in mathematics
-/+; positive relation for
hands-on activities and
higher order thinking;
negative relation for
assessment
USA High school 7146 Correlational Multi-path
White, B. Y.
Frederiksen, J. R.
1998 Science ThinkerTools Inquiry
Curriculum: training
scientific inquiry
processes
Science
achievement
and inquiry
performance
Study investigates the
effect of an intervention
called ThinkerTools
Inquiry Curriculum
compared to a matched
control group;
+; positive effects of
inquiry teaching on
physics achievement and
inquiry skills (processes);
especially positive effects
for low-achieving
students: reach
achievement of high
achieving control group
USA High school Experimental ANOVA
Wigfield, A.
Guthrie, J. T.
Tonks, S.
Perencevich, K. C.
2004 Reading Concept-oriented
reading instruction
(CORI) vs. multiple
Strategy Instruction
Reading
interest, reading
motivational
processes
Study compares two
approaches to foster
reading strategy skills: the
domain-specific approach
+; positive effects of
CORI on motivational
processes (curiosity,
challenge) as well as
USA Primary
school
350 Quasi-
experimental
Math
197
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
(SI) (formal learning
strategies)
CORI and the general
approach SI
affective outputs such as
self-efficacy
Willms, J. D.
Somers, M. A.
2001 Reading Testing, climate Reading
achievement
Study examines the
relationships between
schooling outputs
(mathematics and language
achievement, time to
complete primary
schooling) and student
background, school and
teaching factors
+; positive effect of testing
and climate
Latin
America
(13
countries)
Primary
school
52000 Correlational Multi-
level
Willms, J. D.
Somers, M. A.
2001 Math Testing, climate Mathematics
achievement
Study examines the
relationships between
schooling outputs
(mathematics and language
achievement, time to
complete primary
schooling) and student
background, school and
teaching factors
+; positive effect of testing
and climate
Latin
America
(13
countries)
Primary
school
52000 Correlational Multi-
level
198
Authors Year Domain Instruction variables Output
variables
Summary Direction of effects Country School type Nr of
students
Design Methods
Yair, G. 2000 Other/All Authentic experiences,
choice/autonomy, high
level skills/cognitive
activation
Student
learning
processes:
intrinsic
motivation
Study investigates the
relation between three
instructional variables and
student learning
experiences, measured by
the ESM (experience
sampling method)
+; positive r for authenic
experiences, choice,
cognitive activation
USA Elementary
to secondary
school
865 Correlational Regression
/correlatio
n
199
Appendix: Abstracts of research articles on instruction effectiveness
Differential effectiveness by gender of instructional methods on achievement in mathematics at tertiary level Educational Studies in Mathematics, 37(1), 83
Adedayo, O. A.
1998
Teaching as persuasion
Teaching and Teacher Education, 18(7), 795-813
Alexander, P. A., Fives, H., Buehl, M. M. & Mulhern, J.
2002
A design experiment was undertaken to explore the effects of science lessons, framed as
persuasion, on students' knowledge, beliefs, and interest. Sixth and seventh graders
participated in lessons about Galileo and his discoveries focusing on the personal costs and
public controversies surrounding those discoveries. In selected classrooms, lessons were
teacher led, while others were student led. Participants' knowledge, beliefs, and interest were
compared to peers in other science classes. There were significant differences between
persuasion and comparison classrooms on all variables. However, teacher-led lessons were
more effective at changing students' knowledge, whereas student-led lessons had more impact
on students' beliefs. (C) 2002 Elsevier Science Ltd. All rights reserved.
Using classwide peer tutoring to teach beginning algebra problem-solving skills in heterogeneous classrooms
Remedial and Special Education, 18(6), 367-379
Allsopp, D. H.
1997
This study examined the effectiveness of using classwide peer tutoring (cwpt) in heterogeneous
middle school math classrooms to teach students in beginning algebra problem-solving skills.
The literature on cwpt demonstrates its effectiveness with basic academic skills, but little
research addresses whether cwpt can be effective for teaching higher order thinking skills. This
study compared the effectiveness of cwpt with traditional independent student practice.
Additionally, the performance of students at risk of math failure (students whose grade in math
was a d or an f and/or who scored a stanine of 3 or lower in the math section on a standardized
assessment) was compared with the performance of students not at risk of math failure. an
analysis of the data indicates that both cwpt and independent student practice were effective
strategies for helping students to learn beginning algebra problem-solving skills. Neither
strategy was significantly more effective than the other. Students at risk of math failure
demonstrated slightly greater performance gains than did students not at risk of math failure.
Related findings indicate that cwpt was most effective with 14- and 15-year-old students.
implications for using cwpt for increasing students' higher order thinking skills is discussed as
well as its use in heterogeneous classrooms and with middle school students.
Learning to value mathematics and reading: Relations to mastery and performance-oriented instructional practices
Contemporary Educational Psychology, 26(1), 76-95
200
Anderman, E. M., Eccles, J. S., Yoon, K. S., Roeser, R., Wigfield, A. & Blumenfeld, P.
2001
Changes in students' achievement values in mathematics and reading were examined in a
sample of children and early adolescents. Hierarchical linear modeling techniques were used to
account for both classroom- and student-level effects. At the student level, positive changes in
students' achievement values were associated positively with self-concept of ability and the
previous year's achievement values in both reading and math. Measures of teachers' mastery-
and performance-oriented instructional practices were included in the full HLM model. Students
experienced decrements in achievement values, after controlling for other student and
classroom-level variables, in classrooms where performance-oriented instructional practices
were used. In the full model, sell-concept of ability was related positively to increases in
achievement values, whereas gender was unrelated to changes in achievement values. (C)
2001 Academic Press.
Discussion-based approaches to developing understanding: Classroom instruction and student performance in middle and high school English
American Educational Research Journal, 40(3), 685-730
Applebee, A. N., Langer, J. A., Nystrand, M. & Gamoran, A.
2003
This study examines the relationships between student literacy performance and discussion-
based approaches to the development of understanding in 64 middle and high school English
classrooms. A series of hierarchical linear models indicated that discussion-based approaches
were significantly related to spring performance, controlling for fall performance and other
background variables., These approaches were effective across a range of situations and for
low-achieving as well as high-achieving students, although interpretations are complicated
because instruction is unequally distributed across tracks. Overall, the results suggest that
students whose classroom literacy experiences emphasize discussion-based approaches in the
context of high academic demands internalize the knowledge and skills necessary to engage in
challenging literacy tasks on their own.
Accelerating math development in head start classrooms
Journal of Educational Psychology, 94(4), 762-770
Arnold, D. H., Fisher, P. H., Doctoroff, G. L. & Dobbs, J.
2002
Teachers implemented a 6-week classroom intervention designed to promote emergent math
skills and math interest in preschool-aged children. Teachers in experimental classrooms
incorporated math-relevant activities into their daily routine during circle time, transitions,
mealtime, and small-group activities. Control classrooms engaged in their typical activities. After
the program, experimental children scored significantly higher than control children on a
standardized test of math ability and enjoyed math activities more than the control children, as
measured by both teacher and self-report. Teachers rated the program as highly satisfactory
and reported that they increased their own enjoyment and skill in implementing math activities in
their classrooms. The intervention effects were largely accounted for by substantial gains by
boys, whereas girls showed much smaller program response.
Children's cooperative behavior and interactions in trained and untrained work groups in regular classrooms
201
Journal of School Psychology, 35(3), 261-279
Ashman, A. F. & Gillies, R. M.
1997
This study examined group and individual factors that facilitate changes in cooperation and
learning outcomes in trained and untrained work groups of elementary school-age children. The
study had two foci. The first was to determine if the cooperative behaviours and interactions of
children in classroom groups who were trained in cooperative learning skins were different from
those of children who were given no training, and the second was to investigate small group
interactions and achievement in these groups over time. The results showed that there were
observable differences between student interactions in the two conditions and these differences
were maintained over time. Compared with children in the untrained groups, those in the trained
groups were consistently more cooperative and helpful to each other; they actively tried to
involve each other in the learning task by using language which was more inclusive (e.g.,
frequent use of ''we''), and they gave more explanations to assist each other as they worked
together. It appeared that as the children worked together over time, they became more
responsive to the learning needs of each other. Furthermore, the children in the trained groups
performed significantly better on the learning outcomes questionnaire than those in the
untrained groups. (C) 1997 Society for the Study of School Psychology. Published by Elsevier
Science Ltd.
Vocabulary tricks: Effects of instruction in morphology and context on fifth-grade students' ability to derive and infer word meanings
American Educational Research Journal, 40(2), 447-494
Baumann, J. F., Edwards, E. C., Boland, E. M., Olejnik, S. & Kame'enui, E. J.
2003
This quasi-experimental study compared the effects of morphemic and contextual analysis
instruction WC) with the effects of textbook vocabulary instruction (TV) that was integrated into
social studies textbook lessons. The participants were 157 students in eight fifth-grade
classrooms. ne results indicated that (a) TV students were more successful at learning textbook
vocabulary; (b) MC students were more successful at inferring the meanings of novel affixed
words; (c) MC students were more successful at inferring the meanings of morphologically and
contextually decipherable words on a delayed test but not on an immediate test; and (d) the
groups did not differ on a comprehension measure or a social studies learning measure. The
results were interpreted as support for teaching specific vocabulary and morphemic analysis,
with some evidence for the efficacy of teaching contextual analysis.
202
Unterrichtsgestaltung, verständnisvolles Lernen und multiple Zielerreichung im Mathematik- und Physikunterricht der gymnasialen Oberstufe Baumert, J. & Köller, O.
Opladen: Leske + Budrich
2000
no abstract available
How the socioecological characteristics of the classroom affect academic achievement European Journal of Psychology of Education, 15(2), 173-189
Bennacer, H.
2000
The aim of this study was to grasp and understand the social-psychological mechanisms by
which the classroom socioecological environment determines academic achievement. We first
developed and rested a general theoretical model (on a sample of 51 classrooms containing
1123 junior high school students) which allows us to consider the classroom social climate as a
dependent and independent variable. We then attempted to accurately define the influence
mechanism and develop a structural model to account for the relationships and
interrelationships between variables. We also tried to reach two principal goals in the
psychology of learning environments: one is to determine what classroom social environments
would be advantageous to students, and the other is to identify the factors that determine that
climate. The results indicated a climate-dependent discrepancy between teachers' grades and
achievement test scores. They have some practical implications for optimizing the classroom
social climate.
Where knowledge construction, equity, and context intersect: Student learning of science in small groups
Journal of Research in Science Teaching, 34(10), 1039-1065
Bianchini, J. A.
1997
To provide insight into the strengths and limitations of using groupwork to promote excellence
and equity in science education, the researcher investigated how students using the Complex
Instruction model of groupwork constructed scientific knowledge. Eighty sixth-grade students
and one life science teacher at an urban middle school participated in this study. Qualitative
analysis of videotapes and interviews makes dear that group discussions rarely moved beyond
observational or procedural matters, that students of high status (perceived academic ability and
popularity) had greater access to their groups materials and discourse, and that students made
few connections among the contexts of school, science, and everyday life. Quantitative
analyses of participation during groupwork and performance on unit tests show that high-status
students had significantly higher rates of on-task talk than their middle-or low-status
counterparts, and that those students who talked more learned more as well. Thus, although
groupwork should not be summarily dismissed as an instructional strategy, group tasks and
implementation must be further refined to adequately address the dual goals of excellence and
equity: Students need greater guidance in how to talk and do science, and teachers, greater
assistance in eliminating differences in student participation and achievement. (C) 1997 John
Wiley & Sons, Inc.
203
Cooperative learning in elementary schools: Effects of the jigsaw method on student achievement in science
Psychologie in Erziehung Und Unterricht, 49(3), 172-183
Borsch, F., Jurgen-Lohmann, J. & Giesen, H.
2002
Jigsaw, a cooperative learning environment, was implemented in eight third and fourth grade
classes and compared with traditional science instruction in control classes. In jigsaw every
student becomes an "expert" for one section of the topic, which he or she then teaches to his or
her teammates. Children in the cooperative classes were expected to show greater increase in
knowledge than their traditionally learning counterparts independent of level of prior knowledge.
Within the cooperative classes the influence of being an expert or listener on achievement was
assessed. Results of the study show a substantially greater increase in declarative knowledge
in the cooperative classes; the difference between groups was stable after four months.
Children on all levels of prior knowledge profited from the cooperative learning method. Experts
in the cooperative teams achieved better results about their sections of the topic than listeners.
Listeners outperformed traditionally instructed children in most of the topics.
The effects on student achievement and attitudes when using integrated learning systems with cooperative pairs
Etr&D-Educational Technology Research and Development, 45(1), 51-64
Brush, T. A.
1997
The purpose of this study was to determine whether combining cooperative learning strategies
with instruction delivered using an Integrated Learning System (ILS) produced academic and
attitudinal gains in students. Sixty-five fifth-grade students were randomly divided into two
groups, cooperative and individual. Students in the cooperative group worked on ILS math
activities with a partner. Students in the individual group worked on the same activities by
themselves. Achievement and attitudinal data were collected for the students prior to the
experimental treatment and at the end of the treatment period. Results revealed that students
using an ILS for mathematics instruction performed better on standardized tests and were more
positive toward math and the computer math activities when they worked in cooperative groups
than when they worked on the same activities individually.
Gender and science learning early in high school: Subject matter and laboratory experiences
American Educational Research Journal, 34(2), 297-331
Burkam, D. T., Lee, V. E. & Smerdon, B. A.
1997
This study used a large and nationally representative, longitudinal database, NELS:88, to
identify important factors related to gender differences in 10th-grade science performance. It
built on an earlier study focusing on 8th-grade science performance, wherein gender differences
were found to be related to (a) subject matter (life versus physical science), (b) student ability
level, and (c) frequency of hands-on lab opportunities. The moderate unadjusted advantage for
8th-grade boys on the physical science test widened by the 10th grade. The gender differences
were smaller on the life science test and favored males among students of average and above-
average ability and females among the less able students. Hands-on lab activities-relatively
infrequent in high school science classes-continued to be related to all students' performance,
204
but especially to girls', These findings suggest the importance of the active involvement of
students in the science classroom as a means to promote gender equity. Implications for the
underrepresentation of women in physical science careers are discussed.
Evaluation of a program to teach phonemic awareness to young-children - a 2-year and 3-year follow-up and a new preschool trial Journal of Educational Psychology, 87(3), 488-503
Byrne, B. & Fieldingbarnsley, R.
1995
This article reports a follow-up study of children in Grades 1 and 2 who had been instructed in
phonemic awareness in preschool. Compared to a control condition, the trained children were
superior in nonword reading 2 and 3 years later and in reading comprehension at 3 years.
Control children furnished a disproportionate number of readers dependent on sight word
reading. The superiority of the experimental condition did not extend to measures of
automaticity in reading. W. A. Hoover and P. B. Gough's (1990) ''simple view'' of reading
(Reading Comprehension = Listening Comprehension X Decoding) was supported. In a
supplementary experiment, preschool children were trained with the program by their regular
teachers and showed greater progress in aspects of phonemic awareness than the control
condition from the main experiment However, they did not gain as much as those in the more
intensely trained experimental condition.
The effects of literacy instruction on primary students' reading and writing achievement Reading Research and Instruction, 39(1), 3-26
Cantrell, S. C.
1999
This study examined the reading and writing achievement of students in primary classrooms in
which teachers implemented recommended literacy instruction to varying degrees and
described teachers' practices within these varying classrooms. The reading and writing
achievement of 21 students from four classrooms in which teachers adhered to recommended
literacy practices to high degrees was compared with the achievement of 19 students from four
classrooms in which teachers implemented recommended practices to low degrees. Findings
revealed that the group of students taught by teachers who adhered to recommended practices
to high degrees outperformed the group of students taught by teachers who adhered to these
practices to low degrees on every measure of Literacy achievement with significant differences
in comprehension, fluency, quality of writing, and use of language mechanics. The teachers of
the higher achieving students implemented practices such as using high-quality children's
literature, engaging students in extensive reading and writing, and teaching skills in the context
of meaningful reading and writing experiences. The teachers of the lower achieving students
were less likely to employ these practices and were more apt to provide isolated skill instruction.
205
The impact of different forms of multimedia CAI on students' science achievement Innovations in Education and Teaching International, 39(4), 280-288
Chang, C. Y.
2002
The study described in this paper has explored the effects of teacher-centred vs. student-
centred multimedia computer-assisted instruction (CAI) on the science achievements of tenth-
grade students. A total of 244 tenth-grade senior high-school students (attending six science
classes) participated in this pre-test/post-test comparison-group experiment. During a one-week
period, one group of students (n = 123) were taught by a teacher-centred multimedia CAI
scheme (TMCAI) whereas the other group of students (n = 121) was subjected to a student-
centred multimedia CAI (SMCAI) effect. An analysis of covariance on the Earth Science
Achievement Test post-test scores with students' pre-test scores as the covariate revealed that
the teacher-centred teaching approach was more effective in promoting students' science
achievements than was the student-centred method - especially on the knowledge and
application levels of the cognitive domain. Some implications for the implementation of
multimedia CAI within secondary-science classrooms are discussed.
The use of a problem-solving-based instructional model in initiating change in students' achievement and alternative frameworks
International Journal of Science Education, 21(4), 373-388
Chang, C. Y. & Barufaldi, J. P.
1999
This study examined the effects of a problem-solving-based instructional model on Earth
science students' achievement and alternative frameworks. The investigations employed a pre-
test/post-test control group design to detect any significant change. The 172 participants
enrolled in four Earth science classes received six weeks of the problem-solving-based
instruction. Selected items from Taiwan Entrance Examinations for Senior High School mere
used to measure students' achievement in Earth science content. An open-ended question
instrument was developed by the researchers to examine students' conceptual change. Results
of an analysis of covariance on achievement post-test scores revealed that the problem-solving-
based instructional model did significantly improve the achievement of students (p < 0.05),
especially at the application level (p < 0.05). A chi-square analysis on students' alternative
frameworks measure indicated that students who were taught using the problem-solving-based
instructional model did experience significant conceptual changes than did students who
experienced the traditional-lecture type teaching method (p < 0.001).
Patterns of discourse in two kinds of literature discussion
Reading Research Quarterly, 36(4), 378-411
Chinn, C. A., Anderson, R. C. & Waggoner, M. A.
2001
This study examines the effects of two different instructional frames for discussion-traditional
Recitations and an alternative to Recitations called Collaborative Reasoning-on patterns of
discourse in fourth-grade literature discussions. These two instructional frames differ on four key
parameters that define an instructional frame: (a) the stance to be taken, (b) who holds
interpretive authority, (c) who controls turntaking, and (d) who controls the topic of the
discourse. Two research questions were addressed. The first question was whether it is
possible for fourth-grade teachers and their students to implement Collaborative Reasoning, an
206
instructional frame that transfers much of the control over discourse to students, during their
reading lessons. Quantitative analyses were conducted of features of discourse including
turntaking, teacher questions, and cognitive processes manifested in students' talk. The results
showed that the teachers and students were generally successful at implementing the new
instructional frame. it proved more difficult to shift control over topic and turntaking to students
than to shift interpretive authority to students. The second question addressed the effects of the
new instructional frame on patterns of discourse. In comparison to Recitations, Collaborative
Reasoning discussions produced greater engagement and more extensive use of several
higher level cognitive processes, The results provide support for the use of Collaborative
Reasoning in fourth-grade reading lessons.
The influence of concept mapping on achievement, self-regulation, and self-efficacy in students of English as a second language
Contemporary Educational Psychology, 29(3), 248-263
Chularut, P. & DeBacker, T. K.
2004
This study investigated the effectiveness of concept mapping used as a learning strategy with
students in English as a Second Language classrooms. Seventy-nine ESL students participated
in the study. Variables of interest were students' achievement when learning from English-
language text, students' reported use of self-regulation strategies (self-monitoring and
knowledge acquisition strategies), and students' self-efficacy for learning from English-language
text. A randomized pre-test-post-test control group design was employed. The findings showed
a statistically significant interaction of time, method of instruction, and level of English
proficiency for self-monitoring, self-efficacy, and achievement. For all four outcome variables,
the concept mapping group showed significantly greater gains from pre-test to post-test than the
individual study group. The findings have implications for both practice and research. (C) 2003
Elsevier Inc. All rights reserved.
Unterrichtsqualität: eine Frage der Perspektive? Empirische Analysen zur Übereinstimmung, Konstrukt- und Kriteriumsvalidität Clausen, M.
Münster: Waxmann
2001
no abstract available
The course and malleability of a reading achievement in elementary school: The application of growth curve modeling in the evaluation of a mastery learning intervention
Learning and Individual Differences, 10(2), 137
Crinjen, A. A. M. & Feehan, M.
1998
Instructional and school effects on students' longitudinal reading and mathematics achievements
School Effectiveness and School Improvement, 11(2), 197-235
D'Agostino, J. V.
2000
207
For this study, Prospects, a data set on schools and students in the United States collected
during the early 1990s, was used to examine the effects of instructional and school
organizational characteristics on the longitudinal mathematics and reading achievements of
students from either a first- or third-grade cohort. Three schooling models were tested using
hierarchical linear modeling (HLM) while controlling for parental socioeconomic (SES) status.
Factors and variables that represented instructional and school features were derived from
teacher and principal responses to survey items. These features had direct and interactive
effects on mathematics achievement, supporting both an environmental and interactive model of
schooling. Further, schools characterized by teacher collegiality, support for innovation, principal
leadership, goal agreement, and community support contained teachers who employed
important instructional strategies more effectively, and students who had the highest
mathematics gains over the observed period.
A preliminary study of the effect of computer-assissted practice on reading attainment Journal of Research in Reading, 19(102-110
Davidson, J., Elcock, J. & Noyes, P.
1996
The effect of schools and classes on language achievement British Educational Research Journal, 29(6), 841-859
De Fraine, B., Van Damme, J., Van Landeghem, G., Opdenakker, M. C. & Onghena, P.
2003
This study addresses the effects of secondary schools and classes on language achievement in
Flanders, Belgium. The results of a three-level analysis (students within classes within schools)
indicate that the group composition at the class level is very important. In classes with a high
average initial cognitive ability or a large proportion of girls, the language achievement is higher.
These compositional effects are discussed with reference to type 'A' and type 'B' effects. The
analyses show that group composition is more important than educational practices in
accounting for differences in language achievement. With whom one is taught has a larger
impact than how one is taught. Indications of differential effectiveness of classes related to prior
achievement were found, with greater variations in effectiveness between classes for pupils of
low prior achievement.
208
Empirical evidence of a comprehensive model of school effectiveness: A multilevel study in mathematics in the 1st year of junior general education in the Netherlands
School Effectiveness and School Improvement, 15(1), 3-31
de Jong, R., Westerhof, K. J. & Kruiter, J. H.
2004
In the field of school effectiveness and school improvement, scholars as well as practitioners
often complain about the absence of theory to guide their work. To fill this gap, Creemers (1994)
developed a comprehensive model of educational effectiveness. In order to gain empirical
evidence, we tested some of the main components of the model in lessons of mathematics in
the lst year of lower general education in The Netherlands. The results show that the main
factors in the model-time spent, opportunity to learn, and the quality of instruction-are most
important in predicting achievement. No evidence was found for the relationships between
levels.
The effectiveness of Direct Instruction on the reading achievement of sixth graders Dowdell, T.
1996
Consistency of teaching approach and student achievement: An empirical test School Effectiveness and School Improvement, 11(1), 57-79
Driessen, G. & Sleegers, P.
2000
This article describes the results of a study into the relations between school, teacher, class,
and student characteristics in Dutch elementary schools. Central to the study were the socio-
ethnic background of the students, socio-ethnic class composition, language and math test
results, teaching approach, and consistency of teaching approach within the school. The major
question was whether student achievement levels vary according to the consistency of the
teaching approach after controlling for socio-ethnic background at both the individual and class
levels. The sample consisted of 7,410 grade 8 students and in total 1,714 teachers from 567
schools. The results of multilevel analyses showed consistency of teaching approach to be of no
relevance to achievement levels. The most important factor appeared to he the socio-ethnic
background of the students. Ethnic minority students perform less well than native Dutch
working-class students, who in turn perform less well than the Ether students studied. In
addition, students in classes with a relatively high number of so-called disadvantaged students
perform less well independent of their individual socio-ethnic background.
Effects of teaching methods, class effects, and patterns of cognitive teacher-pupil interactions in an experimental study in primary school classes
School Effectiveness and School Improvement, 8(3), 327-353
Einsiedler, W. & Treinies, G.
1997
The investigation which is reported on tries to contribute to the explanation of school
effectiveness and instructional effectiveness by analyzing relations between instructional
methods and class contexts. The study consists of an experimental part and an observational
part. Within the experiment in fourth grades, two methods of structuring the subject matter which
are derived from theories of knowledge representation were compared to a control method. The
209
tendencies of the results emphasize the assumption that already in fourth grades it is possible
to acquire higher-order biological cause-and-effect knowledge by using a hierarchical or a
network-like structuring method. Further analysis yielded subgroups with homogeneous intra-
class regressions, which we classified as 'difference compensating' and as 'difference
increasing class contexts'. On the subgroup level we found aptitude-treatment interactions.
Different patterns within the verbal-cognitive teacher-pupil interaction turned out in the
observation analysis. They were located among the treatment groups and between the
subgroups 'difference compensating' and 'difference increasing'. Partly these patterns could be
used for the description of class contexts. Due to the fact that certain educational features of
classes were very important, micro characteristics of class contexts should be investigated more
intensively within the research on school and on instruction.
The role of instruction in learning to read: Preventing reading failure in at-risk children
Journal of Educational Psychology, 90(1), 37-55
Foorman, B. R., Francis, D. J., Fletcher, J. M., Schatschneider, C. & Mehta, P.
1998
First and 2nd graders (N = 285) receiving Title 1 services received 1 of 3 kinds of classroom
reading programs: direct instruction in letter-sound correspondences practiced in decodable text
(direct code); less direct instruction in systematic sound-spelling patterns embedded in
connected text (embedded code); and implicit instruction in the alphabetic code while reading
connected text (implicit code). Children receiving direct code instruction improved in word
reading at a faster rate and had higher word-recognition skills than those receiving implicit code
instruction. Effects of instructional group on word recognition were moderated by initial levels of
phonological processing and were mast apparent in children with poorer initial phonological
processing skills. Group differences in reading comprehension paralleled those for word
recognition but were less robust. Groups did not differ in spelling achievement or in vocabulary
growth. Results show advantages for reading instructional programs that emphasize explicit
instruction in the alphabetic principle for at-risk children.
Relationship among laboratory instruction, attitude toward science, and achievement in science knowledge
Journal of Research in Science Teaching, 34(4), 343-357
Freedman, M. P.
1997
This study investigated the use of a hands-on laboratory program as a means of improving
student attitude toward science and increasing student achievement levels in science
knowledge. Using a posttest-only control group design, curriculum referenced objective
examinations were used to measure student achievement in science knowledge, and a posttest
Q-sort survey was used to measure student attitude toward science. A one-way analysis of
variance compared the groups' differences in achievement and attitude toward science.
Analysis of covariance was used to determine the effect of the laboratory treatment on the
dependent achievement variable with attitude toward science as the covariable. The findings
showed that students who had regular laboratory instruction (a) scored significantly higher (p
<.01) on the objective examination of achievement in science knowledge than those who had no
laboratory experiences; (b) exhibited a moderate, positive correlation (r =.406) between their
attitude toward science and their achievement; and (c) scored significantly higher (p <.01) on
achievement in science knowledge after these scores were adjusted on the attitude toward
210
science covariable. There were no significant differences in achievement or attitude toward
science for the limited English proficiency groups. It was concluded that laboratory instruction
influenced, in a positive direction, the students' attitude toward science, and influenced their
achievement in science knowledge. It was recommended that science instruction include a
regular laboratory experience as a demonstrated viable and effective instructional method for
science teachers. This model of science instruction has been shown to be effective with
students of diverse backgrounds who live within large urban centers.
Peer-assisted learning strategies: Making classrooms more responsive to diversity
American Educational Research Journal, 34(1), 174-206
Fuchs, D., Fuchs, L. S., Mathes, P. G. & Simmons, D. C.
1997
The primary focus of this study was to determine the effectiveness of a classwide peer tutoring
program in reading for three learner types: low achievers with and without disabilities and
average achievers. Twelve schools, stratified on student achievement and family income, were
assigned randomly to experiment and control groups. Twenty teachers implemented the peer
tutoring program for 15 weeks; 20 did not implement it. In each of the 40 classrooms data were
collected systematically on three students representing the three learner types. Pre- and
posttreatment reading achievement data were collected on three measures of the the
Comprehensive Reading Assessment Battery. Findings indicated that, irrespective of type of
measure and type of learner, students in peer tutoring classrooms demonstrated greater
reading progress. Implications for policymaking are discussed.
Enhancing first-grade children's mathematical development with peer-assisted learning strategies
School Psychology Review, 31(4), 569-583
Fuchs, L. S., Fuchs, D., Yazdian, L. & Powell, S. R.
2002
The purpose of this study was to examine the effects of a dyadic peer-mediated treatment,
Peer-Assisted Learning Strategies (PALS), on first-grade children's mathematics development.
Within schools, 20 classrooms were assigned randomly to PALS or no-PALS groups. Teachers
implemented PALS three times each week for 16 weeks. Treatment fidelity was measured with
direct observation; teachers completed questionnaires about treatment effectiveness and
feasibility; and 323 students were pre- and posttested. Effects were separated for low-, average-
, and high-achievers, and effect sizes and the percentage of nonresponders for 18 students with
disabilities were calculated. Results indicated that treatment implementation was strong;
teachers judged PALS to be effective and generally feasible; and students with and without
disabilities, at all points along the achievement continuum, benefited from PALS. Implications for
research and practice are discussed.
A structural model of parent and teacher influences on science attitudes of eighth graders: Evidence from NELS: 88
Science Education, 82(1), 93-109
George, R. & Kaplan, D.
1998
Research on science attitudes has focused mostly on teacher variables and learning
environment variables. Furthermore, in the parent involvement literature, the outcome variable
211
of interest has been mostly science achievement rather than science attitudes. Limited research
is available on the joint influence of teacher and parent variables on science attitudes. This
article proposes a model of parent and teacher influences on the science attitudes of eighth
graders using data from the base year survey of the National Educational Longitudinal Study of
1988. The data were analyzed using structural equation modeling methodology for categorical
data. The results show that the availability of science facilities has a significant direct effect on
science experiments. Parental involvement has significant direct as well as indirect effects on
science attitudes mediated through science activities and library/museum visits. Science
activities have a significant direct effect on science attitudes. This study suggests that improving
the quality of science instruction and science activities in schools will have implications for
science education in schools and this will, in turn, indirectly affect the science attitudes of
students. More importantly, the findings of this study provide concrete empirical evidence that
patents play a very important role in the development of science attitudes of students. (C) 1998
John Wiley & Sons, Inc.
An evaluation of the relative effectiveness of NCTM standards-based interventions for low-achieving urban elementary students
Journal of Educational Psychology, 90(3), 560-569
Ginsburg-Block, M. D. & Fantuzzo, J. W.
1998
The effects of 2 instructional methods, problem solving and peer collaboration, were evaluated
for enhancing mathematics achievement, academic motivation, and self-concept of 104 low-
achieving 3rd and 4th graders. Students were assigned randomly to 1 of 4 conditions: control,
problem solving, peer collaboration, and problem solving + peer collaboration. Students in all
conditions met twice weekly for 30-min mathematics sessions over a 7-week period. Results
indicate that problem-solving students performed significantly higher than their counterparts who
did not receive problem solving on measures of computation and word problems and reported
higher levels of academic motivation, academic self-concept, and social competence. Students
who participated in peer collaboration scored higher on measures of computation and word
problems and reported higher levels of academic motivation and social competence than did
students who did not participate in peer collaboration. No significant interaction effect was
found.
212
The compatibility of cognitive and noncognitive objectives of instruction
Zeitschrift Für Padagogik, 41(4), 531-553
Gruehn, S.
1995
Today, cognitive and noncognitive objectives of school education are generally held to be of
equal importance. In spite of that, the empirical studies concerning direct vs. indirect instruction
show, the difficulty of devising teaching strategies that enhance both the pupils achievement
and their positive motivational and affective development. The present study aims at highlighting
the difference between school classes with both a positive cognitive and a positive noncognitive
development and classes in which only one or none of these goals is fulfilled. Based upon 137
seventh grade classes from different types of schools, these four groups are compared with
respect to the pupils' subjectively perceived teaching features. It can be shown that an efficient
way of teaching, a slow pace, and a rather teacher-centered style (achievement-enhancing
conditions) as well as a high pedagogical and socio-emotional competence or the teacher are
among the most important factors as regards the achievement of the multi-criterial objective
mentioned above.
Influences of concept-oriented reading instruction on strategy use and conceptual learning from text Elementary School Journal, 99(4), 343-366
Guthrie, J. T., Anderson, E., Alao, S. & Rinehart, J.
1999
We define reading engagement as the mutual support of motivations, strategies, and
conceptual knowledge during reading. To increase reading engagement, a collaborative team of
teachers, reading specialists, and university faculty implemented a year-long integration of
reading/language arts and science instruction known as Concept-Oriented Reading Instruction
(CORI). This instruction contained conceptual themes, real world science interactions, self-
directed learning, strategy instruction situated within conceptual contexts, peer collaborations,
self-expression of knowledge through portfolios and exhibits, and coherence of the curriculum.
Five teachers provided CORI to 53 grade 5 and 67 grade 3 students. 5 teachers provided
traditionally organized instruction aimed toward the same objectives to 53 grade 5 and 66 grade
3 students. Students were from 2 low-income schools. The CORI context increased strategy
use, conceptual learning, and text comprehension more than traditional instruction, when
background was controlled. Principles of contexts for engagement are discussed.
Benefits of opportunity to read and balanced instruction on the NAEP
Journal of Educational Research, 94(3), 145-162
Guthrie, J. T., Schafer, W. D. & Huang, C. W.
2001
The National Assessment of Educational Progress (NAEP) requires reading comprehension
processes that may be increased by students' amount of engaged reading, parental education,
and gender, along with balanced reading Instruction and opportunity to read. To examine the
effects of those variables on reading achievement and engagement, the authors analyzed the
1994 Grade 4 Maryland NAEP with hierarchical linear modeling to construct both between-
school and between-teacher models. Amount of engaged reading significantly predicted reading
achievement on the NAEP, after parental education was statistically controlled, Balanced
reading instruction significantly predicted reading achievement after accounting for students'
213
engaged reading and parental education. Findings confirmed expectations from the proposed
theoretical perspective on reading engagement. Policy implications included an emphasis on
some instructional variables in the reading engagement model.
Does concept-oriented reading instruction increase strategy use and conceptual learning from text?
Journal of Educational Psychology, 90(2), 261-278
Guthrie, J. T., Van Meter, P., Hancock, G. R., Alao, S., Anderson, E. & McCann, A.
1998
The authors define reading engagement as the mutual support of motivations, strategies, and
conceptual knowledge during reading. To increase reading engagement, a collaborative team
designed a year-long integration of reading/language arts and science instruction (Concept-
Oriented Reading Instruction, CORI). The authors compared students who received this
instruction to similar students who received traditionally organized instruction aimed toward the
same objectives. A path analysis showed that CORI had a positive effect on strategy use and
text comprehension for students at Grades 3 and 5 when accounting for past achievement and
prior knowledge. CORI also had a positive, indirect effect on conceptual knowledge mediated by
strategy use, and this instruction facilitated conceptual transfer indirectly through several paths
simultaneously The findings are discussed in relation to a growing literature on instructional
contexts for motivated strategy use and conceptual learning from text.
Growth of literacy engagement: Changes in motivations and strategies during concept-oriented reading instruction
Reading Research Quarterly, 31(3), 306-332
Guthrie, J. T., VanMeter, P., McCann, A. D., Wigfield, A., Bennett, L., Poundstone, C. C., Rice,
M. E., Faibisch, F. M., Hunt, B. & Mitchell, A. M.
1996
THIS STUDY describes changes in literacy engagement during 1 year of Concept-Oriented
Reading Instruction (CORI), a new approach to teaching reading, writing, and science. Literacy
engagement was defined as the integration of intrinsic motivations, cognitive strategies, and
conceptual learning from text. To promote literacy engagement in classrooms, our team
designed and implemented CORI in two third- and two fifth-grade classrooms in two schools.
One hundred and forty students participated in an integrated reading/language arts-science
program, which emphasized real-world science observations, student self-direction, strategy
instruction, collaborative learning, self-expression, and coherence of literacy learning
experiences. Trade books replaced basals and science textbooks. According to 1-week
performance assessments in the fall and spring, students gained in the following higher order
strategies: searching multiple texts, representing knowledge, transferring concepts,
comprehending informational text, and interpreting narrative. Children's intrinsic motivations for
literacy correlated with cognitive strategies at.8 for Grade 5 and.7 for Grade 3. All students who
increased in intrinsic motivation also increased in their use of higher order strategies. A sizeable
proportion (50%) of students who were stable or decreased in intrinsic motivation failed to
progress in higher order strategies. These findings were discussed in terms of a conceptual
framework that embraces motivational, strategic, and conceptual aspects of literacy
engagement.
Effects of integrated instruction on motivation and strategy use in reading
214
Journal of Educational Psychology, 92(2), 331-341
Guthrie, J. T., Wigfield, A. & Von Secker, C.
2000
Effects of instructional context on intrinsic and extrinsic motivation have been examined with a
variety of studies. This quasi experiment compared students receiving an instructional
intervention designed to increase intrinsic motivation with students receiving traditional
instruction. Concept-oriented reading instruction (CORI) integrated reading and language arts
with science inquiry. It emphasized learning goals, real-world interaction (hands-on science
activities), competence support (strategy instruction), autonomy support (self-directed learning),
and collaboration. Traditional classrooms had the same content objectives and comparable
teachers but different pedagogy. Children in CORI classrooms scored higher on motivation than
did children in traditional classrooms, with effect sizes of 1.94 for curiosity and 1.71 for strategy
use. Grade-level differences were found for recognition and competition. The results show that
classroom contexts can be constructed to influence motivational outcomes positively.
Enhancing the validity and usefulness of large-scale educational assessments.2. Nels-88 Science Achievement American Educational Research Journal, 32(3), 555-581
Hamilton, L. S., Nussbaum, E. M., Kupermintz, H., Kerkhoven, J. I. M. & Snow, R. E.
1995
This study is second in a series demonstrating that achievement tests are multidimensional and
that using psychologically meaningful subscores in national educational surveys can enhance
test validity and usefulness. NELS:88 8th- and 10th-grade science tests were subjected to full
information item factor analysis. Factors reflecting everyday knowledge, scientific reasoning,
chemistry knowledge, and reasoning with knowledge were obtained in 8th grade. Quantitative
science, spatial-mechanical, and basic knowledge and reasoning were distinguishable factors in
10th grade. Regression analyses showed that different patterns of prior math and science
achievement, and of course taking, were associated with each 10th-grade science factor.
Teacher emphasis on problem solving and understanding related more to quantitative science,
and basic knowledge and reasoning. Spatial-mechanical reasoning showed the strongest
gender and ethnicity effects; it related also to science museum visits but not to instructional
variables. It is recommended that multidimensional achievement scores be used to capture
student and teacher effects that total scores used alone miss.
215
A motivational model of rural students' intentions to persist in, versus drop out. of high school Journal of Educational Psychology, 95(2), 347-356
Hardre, P. L. & Reeve, J.
2003
Using self-determination theory, the authors tested a motivational model to explain the
conditions under which rural students formulate their intentions to persist in, versus drop out of,
high school. The model argues that motivational variables underlie students' intentions to drop
out and that students' motivation can be either supported in the classroom by autonomy-
supportive teachers or frustrated by controlling teachers. LISREL analyses of questionnaire data
from 483 rural high school students showed that the provision of autonomy support within
classrooms predicted students' self-determined motivation and perceived competence. These
motivational resources, in turn, predicted students' intentions to persist, versus drop out, and
they did so even after controlling for the effect of achievement.
Die Integration von Repräsentationsformen in den Sachunterricht der Grundschule Hardy, I., Jonen, A., Möller, K. & Stern, E.
Münster: Waxmann
2004
Unterrichtsqualität und Leistungsentwicklung: Ergebnisse aus dem SCHOLASTIK- Projekt Helmke, A. & Weinert, F. E.
1997
Effects of thematically integrated mathematics instruction on students of mexican descent Journal of Educational Research, 88(5), 290-300
Henderson, R. W. & Landesman, E. M.
1995
The effects of thematically integrated instruction in mathematics on achievement, attitudes, and
motivation in mathematics among middle school students of Mexican descent were
investigated. A school-university collaborative effort led to the development and testing of a
thematic approach undertaken as a means of contextualizing Instruction for students considered
to be at risk for school failure. Instruction relied heavily on small, collaborative learning groups
and hands-on activities designed to help students make real-world sense of mathematical
concepts. As hypothesized, experimental and control students made equivalent gains in
computational skills, but experimental students (thematic treatment) surpassed controls in
achievement on mathematical concepts and applications. The two programs did not have a
differential effect on attitudes toward mathematics or self-perceptions of motivation in
mathematics, but motivational variables did predict achievement outcomes for both groups.
Issues of ''opportunity to learn'' the full range of mathematics content of the curriculum within a
thematic approach are examined.
216
Modelling student progress in studies of educational effectiveness
School Effectiveness and School Improvement, 9(3), 310-333
Hill, P. W. & Rowe, K. J.
1998
It is argued that a crucial requirement in studies of educational effectiveness is the modelling of
change or growth in student learning. To illustrate one approach to achieving this end, results
are presented from multivariate multilevel analyses of three-waves of data for three Grade level
cohorts of students From a longitudinal study designed to explain variation in elementary school
students' progress in literacy achievement. The article provides estimates of the influence of
prior achievement and social background factors including 'critical events', on students'
progress, examines the extent to which progress can be accounted for by the grouping effects
of students within classes and schools over successive years, and provides estimates of the
effects of explanatory variables at the student- and class/teacher-levels.
Thinking aloud together: A test of an intervention to foster students' collaborative scientific reasoning
Journal of Research in Science Teaching, 36(10), 1085-1109
Hogan, K.
1999
This study addressed the question of how to increase students' competencies for regulating
their co-construction of knowledge when tackling complex collaborative learning tasks which are
increasingly emphasized as a dimension of educational reform. An intervention stressing the
metacognitive, regulatory, and strategic aspects of knowledge co-construction, called Thinking
Aloud Together, was embedded within a 12-week science unit on building mental models of the
nature of matter. Four classes of eighth graders received the intervention, and four served as
control groups for quantitative analyses, in addition, the interactions of 24 students in eight focal
groups were profiled qualitatively, and 12 of those students were interviewed twice. Students
who received the intervention gained in metacognitive knowledge about collaborative reasoning
and ability to articulate their collaborative reasoning processes in comparison to students in
control classrooms, as hypothesized. However, the treatment and control students did not differ
either in their abilities to apply their conceptual knowledge or in their on-line collaborative
reasoning behaviors in ways that were attributable to the intervention. Thus, there was a gap
between students' metacognitive knowledge about collaborative cognition and their use of
collaborative reasoning skills. Several reasons for this result are explored, as are patterns
relating students' outcomes to their perspectives on learning science. (C) 1999 John Wiley &
Sons, Inc.
Student gender and teaching methods as sources of variability in children's computational arithmetic performance
Journal of Genetic Psychology, 158(3), 333-345
Hopkins, K. B., McGillicuddyDeLisi, A. V. & DeLisi, R.
1997
An experimental study of the effects of a didactic teaching approach and a constructivist
teaching approach on 3rd- and 5th-grade boys' and girls' performance on arithmetic
computation problems was conducted. Two groups of children, matched on the basis of initial
computation performance as well as grade and gender, were taught how to solve arithmetic
problems using one of these two instructional approaches. Analysis of subsequent computation
217
test performance revealed that 5th graders scored higher than 3rd graders, and there was a
significant interaction between gender and instruction group. After instruction, girls in the
didactic group outperformed boys in both instruction groups and girls who had been taught
using constructivist approaches. Gender differences in computation performance can appear
relatively early, by 3rd grade, if didactic instruction strategies are used to leach computational
rules. The pattern of gender differences suggests that didactic teaching does not handicap
boys. Rather, it appears that didactic instruction enhances computational performance in girls.
Peer mediated intervention in reading instruction using pause, prompt and praise
Educational Studies, 21(3), 361-377
Houghton, S., Litwin, M. & Carroll, A.
1995
A multiple baseline research design across subjects (pairs) was used to examine the
effectiveness of peer tutoring in reading using the Pause, Prompt and Praise tutoring procedure.
Twelve II year-old, Year 6 students of varying reading ability were trained in the systematic use
of delayed attention, prompting and praise to tutor 12 Year 6 students of a similar age and
similar reading ability. Individual trends in tutor behaviours were examined using a statistical
programme specifically designed for analysing data; From interrupted time series research
designs. A one-tailed t-test was also computed to test for significant differences in pre- and
post-test means in reading achievement. Continuous data collection throughout the programme
indicated that all peers were effective in increasing their use of the tutoring behaviours; not all
tutors experienced statistically significant increases, however. Both peer tutors and tutees made
statistically significant gains in reading accuracy and comprehension over the course of the
intervention.
Adaptive instruction and pupil achievement School Effectiveness and School Improvement, 10(2), 172-192
Houtveen, A. A. M., Booij, N., de Jong, R. & van de Grift, W.
1999
In this article the results are reported of a quasi-experiment on effects of adaptive instruction on
reading results of children in the first year of reading instruction in Dutch primary schools. The
research involved 456 pupils from 23 schools (12 experimental and 11 control group schools).
Teachers in the experimental group show significantly higher adaptive instruction behaviour
consisting of: optimizing time on task, using the model for direct instruction, working according
to the principles of the 'phonics construction method' for initial reading and degree of diagnostic
teaching, than the teachers in the control group. The pupils in the experimental group were
significantly more successful on the attainment of higher reading results in comparison with the
pupils in the control group. This experimental effect does not only remain after correction for the
pretest, but also after correction for intelligence, social economical background and pleasure in
reading.
Teacher appropriation and student learning of geometry through design
Journal for Research in Mathematics Education, 31(1), 71-88
Jacobson, C. & Lehrer, R.
2000
In 4 Grade 2 classrooms, children learned about transformational geometry and symmetry by
designing quilts. All 4 teachers participated in professional development focused on
218
understanding children's thinking in arithmetic. Therefore, the teachers elicited student talk as a
window for understanding student thinking and adjusting instruction in mathematics to promote
the development of understanding and used the same tasks and materials. Two of the 4
teachers participated in additional workshops on students' thinking about space and geometry,
and they elicited more sustained and elaborate patterns of classroom conversations about
transformational geometry. These differences were mirrored by students' achievement
differences that were sustained over time. We attribute these differences in classroom discourse
and student achievement to differences in teachers' knowledge about typical milestones and
trajectories of children's reasoning about space and geometry.
Boys and girls in the performance-based science classroom: Who's doing the performing?
American Educational Research Journal, 35(3), 477-496
Jovanovic, J. & King, S. S.
1998
The aim of this study was to examine whether over the school year boys and girls equally share
in performing the behaviors required of hands-on activities (e.g., manipulating the equipment,
directing the activity, observing) in the performance-based science classroom. In addition, we
examined whether these performance behaviors accounted for changes in boys' and girls'
science attitudes (i.e., ability perceptions and task value beliefs) at the end of the school year.
The sample included 165 students (53% female, mean age = 12.21) in six Grade 5-Grade 8
performance-based science classrooms where the teachers associated with these classrooms
were identified not only as exemplary hands-on science instructors but also instructors sensitive
to increasing girls' participation in science. Our results indicated that being actively involved in
the performance-based science classroom predicted students' end-of-the year science
attitudes. However, boys and girls did not participate equally in these classrooms. Moreover, we
found that for girls, but not boys, there was a decrease in science ability perceptions over the
school year, suggesting that boys and girls experienced these classrooms differently.
Unterrichtsqualität aus Schülerperspektive: Kulturspezifische Profile, regionale Unterschiede und Zusammenhänge mit Effekten von Unterricht Klieme, E. & Rakoczy, K.
Opladen: Leske + Budrich
2003
no abstract available
219
Cognitive-metacognitive training within a problem-solving based Logo environment British Journal of Educational Psychology, 67(425-445
Kramarski, B. & Mevarech, Z. R.
1997
Background. The present study investigated the effects of metacognitive training implemented
within a problem-solving based Logo environment on students' ability to construct graphs and
reflect on their learning. Aims, (1) To compare achievement outcomes of students who learned
to construct graphs within a problem-solving based Logo environment that was either
embedded with or with no metacognitive training; and (2) To examine the differences in
students' cognitive-metacognitive behaviours under the different conditions. Samples.
Participants were 68 students who studied in four computer classrooms. Methods. Intact
classrooms were randomly assigned to one of two treatment groups: one was exposed to a
problem-solving based Logo environment with metacognitive training (N=34) and the other to
the same Logo environment with no metacognitive training (N=34). All students were examined
at the beginning and at the end of the study on the Graph Construction Examination. In addition,
at the end of the study students were interviewed to assess their cognitive-metacognitive.
behaviours. Furthermore, to examine possible differences (if any exist) between the two
conditions prior to the beginning of the study, students were administered the Raven Advanced
Matrices Examination, and the Graph Interpretation Test. Results, Although no significant
differences were found between groups prior to the beginning of the study on all pretreatment
measures, at the end of the study the students who were exposed to the metacognitive
treatment tended to construct graphs better than their counterparts who were not exposed to
such treatment. The metacognitive group was also better able to reflect on their learning than
their counterparts who were not exposed to such training. In addition, structured interviews
indicated the positive effects of the metacognitive training on students' information processing,
social-cognitive interaction, and error detection. Conclusions, Being exposed to metacognitive
training exerts positive effects on students' achievement outcomes and cognitive-metacognitive
behaviours.
Effects of multilevel versus unilevel metacognitive training on mathematical reasoning
Journal of Educational Research, 94(5), 292-300
Kramarski, B., Mevarech, Z. R. & Lieberman, A.
2001
The effects of 3 instructional methods on mathematical reasoning were investigated. The
methods are (a) cooperative learning embedded within multilevel metacognitive training (MMT),
(b) cooperative learning embedded within unilevel metacognitive training (UMT), and (c)
learning in the whole class with no metacognitive training. MMT was implemented in-
mathematics and English classrooms; UMT was used only in mathematics classrooms; and the
whole class with no metacognitive training served as a control group. Results indicated that
students who were exposed to MMT significantly outperformed their counterparts who were
exposed to UMT who, in turn, significantly outperformed the control group. Effects of MMT were
observed on students while they solved mathematical problems. Theoretical and practical
implications of the study are discussed.
Effectiveness of explicit and constructivist mathematics instruction for low-achieving students in the Netherlands
Elementary School Journal, 104(3), 233-251
220
Kroesbergen, E. H., Van Luit, J. E. H. & Maas, C. J. M.
2004
In this study we compared the effects of small-group constructivist and explicit mathematics
instruction in basic multiplication on low-achieving students' performance and motivation. A total
of 265 students (aged 8-11 years) from 13 general and 11 special elementary schools for
students with learning and/or behavior disorders participated in the study. The experimental
groups received 30 minutes of constructivist or explicit instruction in groups of 5 students twice
weekly for 5 months. Pre- and posttests were conducted to compare the effects on students'
automaticity, problem-solving, strategy use, and motivation to the performance of a control
group who followed the regular curriculum. Results showed that the math performance of
students in the explicit instruction condition improved significantly more than that of students in
the constructivist condition, and the performance of students in both experimental conditions
improved significantly more than that of students in the control condition. Only a few effects on
motivation were found. We therefore concluded that recent reforms in mathematics instruction
requiring students to construct their own knowledge may not be effective for low-achieving
students.
Classroom and developmental differences in a path model of teacher expectancy effects
Child Development, 72(5), 1554-1578
Kuklinski, M. R. & Weinstein, R. S.
2001
A path model of teacher expectancy effects was evaluated in a sample of 376 first- through fifth-
grade urban elementary school children. The roles of two moderators (classroom perceived
differential treatment environment and developmental differences) and one mediator (children's
self-expectations) of teacher expectancy effects on children's year-end achievement were
examined. Significant differences in effects and effect sizes are presented. Both classroom
environment (high versus low in differential treatment, as seen through children's eyes) and
developmental differences moderated the strength of teacher expectancy effects. Generally,
stronger effects were found in classrooms in which expectancy-related cues were more salient
to children, but developmental differences moderated which effect was most pronounced. A
significant age-related decline in direct effects on ending achievement was interpreted as
evidence that teacher expectations may tend to magnify achievement differences in the early
grades, but serve to sustain them in later grades. Support for indirect effects (teacher
expectations --> children's self-expectations --> ending achievement) was limited to upper
elementary grade classrooms perceived as high in differential treatment. In contrast to prior
research that emphasized small effect sizes, the present analyses document several instances
of moderate effects, primarily in classrooms in which expectancy-related messages were most
salient to children. These results underscore the importance of explicit attention to the inclusion
of moderators, mediators, and multiple outcomes in efforts to understand teacher expectancy
effects.
221
Multiple Ziele im Mathematikunterricht. Ph.D. Thesis Kunter, M.
Berlin: Freie Universität Berlin
2004
no abstract available
Enhancing the validity and usefulness of large-scale educational assessments.1. Nels-88 mathematics achievement American Educational Research Journal, 32(3), 525-554
Kupermintz, H., Ennis, M. M., Hamilton, L. S., Talbert, J. E. & Snow, R. E.
1995
This study demonstrates that the validity and usefulness of mathematics achievement tests can
be improved fry defining psychologically meaningful subscores that yield differential relations
with student, teacher, and school variables. The NELS:88 8th- and 10th-grade math tests were
subjected to full information item factor analysis. Math knowledge and math reasoning factors
were distinguished at both grade levels. Regression analyses showed that student attitudes,
instructional variables, course, and program experiences related move to knowledge, whereas
gender, SES, and some ethnic differences related more to reasoning. Teacher emphasis on
higher order thinking, student use of home computers, and early experience with advanced
math courses related to both dimensions. It is recommended that national educational surveys
use multidimensional achievement scores, not total scores alone.
Enhancing the validity and usefulness of large-scale educational assessments.3. NELS:88 mathematics achievement to 12th grade
American Educational Research Journal, 34(1), 124-150
Kupermintz, H. & Snow, R. E.
1997
The present study demonstrates the utility of a multidimensional representation of student
mathematics achievement. Third in a series of studies using the math test item level data from
the NELS:88 longitudinal study, it carries the analyses through the 12th grade. Full information
factor analysis was employed to investigate the structure of math achievement create
meaningful achievement scales, and explore their relationships with student background
academic program, and instructional variables, The findings support a basic distinction between
Mathematical Reasoning (MR) and Mathematical Knowledge (MK) for two of the three
administered test forms (low and middle levels), replicating previous findings for the 8th- and
10th-grade data. A more complex structure for the high test form reflected variation in problem-
solving strategies as well as in content. Regression analyses suggested differential patterns of
relationships between student, program, and instructional variables for math achievement
subscores. These patterns were also compared with results for the unidimensional total score.
Discussion emphasizes implications for score intepretations, both normative and criterion-
referenced, within the context of a validity argument.
The significance of the classroom effect in primary schools: An application of Creemers' comprehensive model of educational effectiveness
School Effectiveness and School Improvement, 11(4), 501-529
Kyriakides, L., Campbell, R. J. & Gagatsis, A.
2000
222
This article presents findings of an attempt to test Creemers' model of educational effectiveness
by using data derived from an evaluation study in Mathematics in which 30 schools, 56 classes
and 1,051 pupils of the last year of primary school of Cyprus participated. More specifically, we
examine whether the pupil, classroom and school variables show the expected effects on pupils'
achievement in Mathematics. Research data concerned with pupils' achievement in
Mathematics were collected by using two different forms of assessment (external assessment
and teacher's assessment). Questionnaires were administered to pupils and teachers in order to
collect data about most of the variables included in Creemers' model. The findings support the
main assumptions of the model. The influences on pupil achievement are multilevel and the net
effect of classrooms was higher than that of schools. Implications for the development of
research on school effectiveness are drawn.
Physics instruction of the upper secondary level: An empirical analysis of the learning-teaching-culture from a constructivist perspective
Zeitschrift für Didaktik der Naturwissenschaften, 2(Labudde, P. & Pfluger, D.
1999
In an empirical study, the dissemination and the efficiency of a constructivist approach was
analysed in physics instruction of the upper secondary level. Four dimensions were
distinguished in order to operationalise the constructivist approach: a dimension of the
individual, one of the contents, another one of communication and co-operation, and finally one
of teaching methods. Based on these dimensions, we developed a student questionnaire with
224 items. It was used as an additional instrument in the Third International Mathematics and
Science Study (TIMSS) in Switzerland. Our sample included 670 students of 152 classes at the
end of grade 13. The data of our student questionnaire and of the TIMSS physics test are
analysed to answer the three research questions: How far is the physics instruction at the upper
secondary level related to a constructivist approach? What is its efficiency as seen by the
students? What are the correlations between various educational variables of a constructivist
approach on the one side and motivational variables, physics performance, and epistemological
views on the other side?
Teaching biology in a group mastery learning mode: High school students' academic achievement and affective outcomes
International Journal of Science Education, 18(4), 447-462
Lazarowitz, R., Baird, J. H. & Bowlden, V.
1996
Two learning units, the cell and plant topics were taught for seven weeks in four 10th grade
classes. Two experimental classrooms (N = 52) were instructed in a Group Mastery Learning
(GML) method, and two comparison classrooms (N = 61) in Individualized Mastery Learning
(IML). For entry behaviour, data was collected on students' aptitudes (GATE), and their math
and science scores. Students' pre- and post-tests scores on academic achievement, self-
esteem perception of classroom learning environment (CLE) and altitudes toward biology were
tested using covariance analysis. Students in the GML classrooms had higher scores in self-
esteem, attitudes toward biology, and in the sub-scales of the CLE. In the IML classes, a decline
on these measures occurred. Students in the GML method, scored higher in the cell unit, while
students in the IML method scored higher in the plant unit. This result is discussed within the
framework of the complexity of the learning units.
223
When each one has one: The influences on teaching strategies and student achievement of using laptops in the classroom
Etr&D-Educational Technology Research and Development, 51(3), 23-44
Lowther, D. L., Ross, S. M. & Morrison, G. M.
2003
In this study, we examined the educational effects of providing fifth-, sixth-, and seventh-grade
students with 24-hour access to laptop computers. Specifically we examined the impact of
laptops on classroom activities, and on student use of technology and their writing and problem-
solving skills. Participating teachers received computer integration training using the iNtegrating
Technology for inQuiry (NTeQ) model to develop problem-based lessons that engage students
in critically examining authentic issues, and strengthen research and writing skills. A matched
treatment-control group design was employed, in which classes taught at the same grade levels
in five participating schools served as the laptop 0 computer per student) and control (5+
computers per class) contexts. Participants included students, teachers, and parents from the
two groups. Although systematic observations revealed relatively few differences in teaching
methods between laptop and control classrooms, laptop students used computers more
frequently, extensively, and independently. Writing assessment results showed substantial and
significant advantages for laptop over control students, with six of eight effect sizes exceeding
+0.80. Results also showed significant advantages for the laptop group on five of the seven
components of the problem-solving task.
Parallel classes: Differences and similarities. Teacher effects and school effects in secondary schools
School Effectiveness and School Improvement, 9(4), 437-473
Luyten, H. & de Jong, R.
1998
Results from school effectiveness studies that took into account differences among teachers or
departments within schools indicate that these teacher or department differences outweigh the
differences between schools. The present study shows that differences in student achievement
between parallel classes taught by different teachers may be very smell. Coordination efforts
among teachers relate primarily to the content and goals of instruction. With respect to
classroom instruction large differences between teachers may still be detected. School
effectiveness differences usually found across grades and subjects may be: due only in part to
loose internal coordination. The fact that it is extremely difficult for teachers to compare the
results of teaching across grades and subjects may be equally important.
The Baltimore Curriculum Project: Final report of the four-year evaluation study Mac Iver, M. A., Kemper, E. & Stringfield, S.
Baltimore, M.D.: Center for Research on the Education of Students Placed At Risk
2003
Student engagement in instructional activity: Patterns in the elementary, middle, and high school years
American Educational Research Journal, 37(1), 153-184
Marks, H. M.
2000
224
Although student engagement with the intellectual work of school is import taut to students'
achievement and to their social and cognitive development, studies over a span of two decades
have documented low levels of engagement, particular v in the classroom. Examining several
theoretical perspectives that attempt to explain engagement through comprehensive
frameworks, this study evaluates the effect on engagement of school reform initiatives that are
consistent with the theories. The study also investigates whether patterns exist in students'
engagement, whether the patterns arc, consistent across grade levels, and whether class
subject matter (mathematics or social studies) differentially affects engagement. The sample
includes 3,669 students representing 143 social studies and mathematics classrooms in a
nationally selected sample of 24 restructuring elementary, middle, and high schools. Because of
the nature of the nested data (students nested within classrooms nested within schools), the
analysis is conducted using hierarchical linear modeling in its three-level application (HLM3L).
The reform initiatives, which are consistent with the theories, eliminate personal background
effects. Together with classroom subject matter, they substantially influence engagement. The
results are generally consistent across grade levels.
Teacher effectiveness and learning for mastery
Journal of Educational Research, 92(5), 279-285
Martinez, J. G. R. & Martinez, N. C.
1999
The effect of an excellent or master teacher's use of learning for mastery (LFM) procedures on
student learning was examined. Although LFM research strongly supports a procedural effect,
close scrutiny suggests a possible teacher-effect/procedural-effect confounding. The authors
extended previous research reporting a main effect for mastery learning when the teacher was
rated average, but no main effect for an excellent or master teacher. Performance in 2 mastery
and 2 traditional classes of intermediate algebra, taught by a master teacher, was compared.
Findings from a 2 x 2 randomized factorial design that controlled for repeatable or mastery
testing, the LFM procedure, and pretesting indicated that student achievement on a final
examination did not differ across mastery and control classes; however, instructor time was
more than double in the mastery classes.
225
Peer-assisted learning strategies for first-grade readers: Responding to the needs of diverse learners
Reading Research Quarterly, 33(1), 62-94
Mathes, P. G., Howard, J. K., Allen, S. H. & Fuchs, D.
1998
THIS RESEARCH examines the effectiveness of Peer-Assisted Learning Strategies for First-
Grade Readers (First-Grade PALS) as a tool for enhancing the reading achievement of different
learner types, particularly low-achieving students, representing the range of academic diversity
typically present in primary grade classrooms. First-Grade PALS helps teachers to
accommodate this diversity (a) by decentering instruction through peer mediation so that
students become more actively involved in the learning process, (b) by including provisions for
integrating phonological and alphabetic skills into the decoding of words in connected text, and
(c) by providing extensive and repeated exposure to a variety of children's literature. In this
research, the efficacy and feasibility of First-Grade PALS were examined in naturally
constituted, academically heterogeneous first-grade classes, during rime normally allocated for
reading instruction. Twenty first-grade teachers and 96 first-grade students (46 low, 20 average-
, and 20 high-achieving) participated. Ten teachers incorporated First-Grade PALS into their
reading program; 10 continued to teach reading as usual. Data collected included (a) time-
series phonological awareness and reading fluency data and (b) pre-and post-measures of
concepts of print, decoding, fluency, and comprehension. Students and teachers also were
asked to rate their satisfaction with various aspects of First-Grade PALS. Results indicate that
all learner types were positively affected by participation in First-Grade PALS, with the greatest
gains indicated for low-achieving students. Likewise, both students and teachers implemented
First-Grade PALS with relative ease, demonstrated high fidelity, and reported high levels of
satisfaction.
Phonological training and the alphabet principle - Evidence for reciprocal causality
Reading Research Quarterly, 30(4), 830-852
McGuinness, D., McGuinness, C. & Donohue, J.
1995
NINETY-FOUR first-grade children were used to develop a predictive reading battery. The
Lindamood Auditory Conceptualization Test (LAC) of phonological awareness was the strongest
predictor of reading recognition and decoding skills. Following test development, 42 children
were trained using a structured phonological reading method emphasizing English phonology
and phoneme/grapheme correspondence These children were compared to a control group
who received a whole-language-plus-phonics approach. A battery of phonological processing
tasks was given to all children at the beginning and the end of the school year. Experimental
groups had significant gains in reading real and nonsense words compared to their previous
scores and to the controls. A test of phonological awareness (the LAC) was a strong predictor of
subsequent reading success regardless of training group and discriminated between good and
average readers 8 months later, as did tests of verbal short-term memory. However, children in
all groups improved by the same amount in phonological awareness. The authors conclude that
phonological awareness is a necessary but not sufficient condition for reading skill.
Effects of metacognitive training embedded in cooperative settings on mathematical problem solving
Journal of Educational Research, 92(4), 195-205
226
Mevarech, Z. R.
1999
Effects of 3 cooperative learning environments on mathematical problem solving were
compared: (a) metacognitive training in both constructing connections and strategy application,
(b) direct instruction regarding strategy application without training in constructing connections,
and (c) neither metacognitive nor strategy training. One hundred and seventy-four 7th-grade
Israeli students participated in the study. Those exposed to the metacognitive training
significantly outperformed their counterparts who were exposed to the strategy instruction, who,
in turn, significantly outperformed students who received neither kind of training (the
cooperative-control group). Theoretical and practical implications of the study are discussed.
IMPROVE: A multidimensional method for teaching mathematics in heterogeneous classrooms
American Educational Research Journal, 34(2), 365-394
Mevarech, Z. R. & Kramarski, B.
1997
The purpose of the present research was to design an innovative instructional method for
reaching mathematics in heterogeneous classrooms (with no tracking) and to investigate its
effects on students' mathematics achievement. The method is based on current theories in
social cognition and metacognition. It consists of three interdependent components:
metacognitive activities, peer interaction, and systematic provision of feedback-corrective-
enrichment. The method is called IMPROVE, the acronym of which represents all the teaching
steps that constitute the method: Introducing the new concepts, Metacognitive questioning,
Practicing, Reviewing and reducing difficulties, Obtaining mastery, verification, and Enrichment.
The research includes two studies, both implemented in seventh grades: One focused on in-
depth analyses of students' information processing under the different learning conditions (N =
247), and one investigated the development of students' mathematical reasoning over a full
academic year (N = 265). Results of both studies showed that IMPROVE students significantly
outperformed the nontreatment control groups on various measures of mathematics
achievement.
The effects of metacognitive training versus worked-out examples on students' mathematical reasoning
British Journal of Educational Psychology, 73(449-471
Mevarech, Z. R. & Kramarski, B.
2003
Background. The present study is rooted in a cognitive-metacognitive approach. The study
examines two ways to structure group interaction: one is based on worked-out examples (WE)
and the other on metacognitive training (MT). Both methods were implemented in cooperative
settings, and both guided students to focus on the problem's essential parts and on appropriate
problem-solving strategies. Aims. The aim of the present study is twofold: (a) to investigate the
effects of metacognitive training versus worked-out examples on students' mathematical
reasoning and mathematical communication; and (b) to compare the long-term effects of the
two methods on students' mathematical achievement. Sample. The study was conducted in two
academic years. Participants for the first year of the study were 122 eighth-grade Israeli
students who studied algebra in five heterogeneous classrooms with no tracking. In addition,
problem-solving behaviours of eight groups (N = 32) were videotaped and analysed. A year
227
later, when these participants were ninth graders, they were re-examined using the same test
as the one administered in eighth grade. Method. Three measures were used to assess
students' mathematical achievement: a pretest, an immediate post-test, and a delayed post-test.
ANOVA was carried out on the post-test scores with respect to the following criteria: verbal
explanations, algebraic representations and algebraic solution. In addition, chi-square and
Mann-Whitney procedures were used to analyse cooperative, cognitive, and metacognitive
behaviours. Results. Within cooperative settings, students who were exposed to metacognitive
training outperformed students who were exposed to worked-out examples on both the
immediate and delayed post-tests. In particular, the differences between the two conditions
were observed on students' ability to explain their mathematical reasoning during the discourse
and in writing. Lower achievers gained more under the MT than under WE condition.
Conclusions. The findings indicate that the kind of task and the way group interaction is
structured are two important variables in implementing cooperative learning, each of which is
likely to have different effects on mathematical communication and achievement outcomes.
Enhancing elementary students' motivation to read and write: A classroom intervention study
Journal of Educational Research, 90(5), 286-299
Miller, S. D. & Meece, J. L.
1997
The authors examined how different reading and language arts assignments influenced 3rd-
grade students' motivational goals, strategy use, and achievement affect, that is, anxiety and
self-concept. Participants included 8 teachers and 187 students from 1 suburban school. During
weekly planning sessions, the teachers modified their reading and language arts assignments
to increase the number of opportunities students had (a) to write multiple paragraphs, (b) to
collaborate with peers, and (c) to monitor their progress over extended periods of time. The
students' motivational patterns varied according to how well the teachers implemented these
changes in their assignments. In classrooms where the teachers implemented the desired
modifications, the students were less focused on teacher approval or normative standards of
evaluation. Contrary to expectations, the intervention did not influence the students' patterns of
strategy use or achievement affect. The results highlight ways in which teachers can structure
reading and language arts assignments to positively influence the students' motivational
profiles.
Die Förderung von naturwissenschaftlichem Verständnis bei Grundschulkindern durch Strukturierung der Lernumgebung
Zeitschrift Für Padagogik, 45. Beiheft(176-191
Möller, K., Jonen, A., Hardy, I. & Stern, E.
2002
School effectiveness and teacher effectiveness in mathematics: Some preliminary findings from the evaluation of the mathematics enhancement programme (primary) School Effectiveness and School Improvement, 11(3), 273-303
Muijs, D. & Reynolds, D.
2000
228
In this study the effect of teacher behaviours and classroom organisation on pupils' progress in
mathematics was studied in years 1, 3 and 5 of primary schools in the UK participating in a
mathematics intervention programme. Data on a total of 78 teachers and 2,128 pupils was
collected. Teacher behaviours were measured using a classroom observation instrument
developed for the project, and pupils were tested in March and July of 1998 using a curriculum-
appropriate Numeracy test developed by the National Foundation for Educational Research.
Background data on pupils was also collected at both testing occasions. Using multilevel
modelling techniques it was found that teacher behaviours were able to explain between 60%
and 100% of pupils' progress on the Numeracy tests. Amount of time spent teaching the whole
class was not related directly to pupils' progress, but structural equation models were tested in
which time spent teaching the whole class was found to be related to effective teaching
behaviours and thus indirectly to pupil progress. The implications of the study for British
educational policies and for educational research more generally are discussed.
How learning contexts facilitate strategy transfer
Applied Cognitive Psychology, 9(5), 425-446
Muthukrishna, N. & Borkowski, J. G.
1995
The present study assessed the role of context in the acquisition and transfer of a mathematical
strategy. One hundred and six children were assigned to four conditions: direct strategy
instruction, guided discovery, direct teaching plus discovery, or a control condition. The
intervention consisted of fourteen sessions during which the number-family strategy, useful for
addition and subtraction, was taught. Third grade students in the guided discovery condition
performed better than those in the direct instruction condition on far transfer problems that
measured deep conceptual understanding. Students who had total or partial exposure to guided
discovery held stronger beliefs and adopted more positive goals about the importance of
mathematical understanding and peer collaboration, attributed less importance to task extrinsic
reasons for success, and reported greater use of deep processing strategies than students
exposed to direct, explicit instructions. Finally, students in the discovery conditions were able to
communicate more effectively during problem solving than students in the direct instructions
condition.
229
Learning environment, motivation, and achievement in high school science
Journal of Research in Science Teaching, 40(4), 347-368
Nolen, S. B.
2003
In a study of the relationship between high school students' perceptions of their science learning
environments and their motivation, learning strategies, and achievement, 377 students in 22
introductory science classrooms completed surveys in the fall and spring of their ninth-grade
year. Hierarchical linear regression was used to model the effects of variables at both the
classroom and individual level simultaneously. High intraclass agreement (indicated by high
parameter reliability) on all classroom environment measures indicated that students shared
perceptions of the classroom learning environment. Controlling for other factors, shared
perceptions that only the most able could succeed in science classrooms and that instruction
was fast-paced and focused on correct answers negatively predicted science achievement, as
measured on a districtwide curriculum-linked test. Shared perceptions that classrooms focused
on understanding and independent thinking positively predicted students' self-reported
satisfaction with learning. Implications of these results for both teaching and research into
classroom environments are discussed. (C) 2003 Wiley Periodicals, Inc.
"Enrichment of learning settings and teaching methods" in mathematics lessons in the German-speaking part of Switzerland
Unterrichtswissenschaft, 31(4), 291-320
Pauli, C., Reusser, K., Waldis, M. & Grob, U.
2003
In this article, results of a video-based study on the practice of so-called "enrichment of learning
settings and teaching methods" (Erweiterte Lehr- und Lernformen - ELF) in lower secondary-
level mathematics lessons in the German-speaking part of Switzerland are presented. Using
different data sources (video recordings, teacher and student questionnaires, performance
tests), the study examines (1) the prevalence of this teaching reform, which can be compared to
the concept of "open education", (2) characteristics of the methodological structure and the
quality of ELF teaching, and (3) possible links to subject performance and motivational
characteristics of the students. The results show that ELF teaching in mathematics lessons in
the 8th school year in German-speaking Switzerland is relatively widespread. It differs from the
methodological lesson structure of "traditional lessons" in the sense of an enrichment of the
teaching and learning forms employed, although teacher-directed elements are still present in
ELF lessons. The quality of ELF teaching was assessed by both pupils and external experts,
who in contrast to the learners, based their assessment solely on one video-recorded lesson.
The ELF teaching was generally judged more positively than traditional teaching. In addition,
pupils reported a greater sense of well-being in ELF lessons. In spite of the positive perception
of the lessons, no relationship was found between ELF teaching and subject performance,
interest and self-confidence.
230
Young children's self-regulated learning and contexts that support it Journal of Educational Psychology, 90(4), 715-729
Perry, N. E.
1998
Writing and portfolio activities provided a context for examining relations between classroom
contexts and young children's self-regulated learning (SRL). Data collection spanned 6 months
and included weekly visits to Grade 2 and 3 classrooms during regularly scheduled writing and
portfolio activities. Data included teacher questionnaires and observations and student
questionnaires, observations, and interviews. Young children deliberated about how to regulate
writing and demonstrated either mastery or performance orientations as a function of
classroom-specific tasks, authority structures, and evaluation practices. Findings support
sociocognitive models of learning regarding how classroom contexts affect students' beliefs,
values, expectations, and actions. Also they challenge assumptions that young children lack the
cognitive sophistication required for SRL and do not adopt motivational orientations that
undermine it.
How does a context-based approach influence understanding of key chemical ideas at 16+?
International Journal of Science Education, 19(6), 697-710
Ramsden, J. M.
1997
One of the characteristics of curriculum development in recent years has been an increased
emphasis on using contexts and applications as a means of developing scientific
understanding. Yet relatively little has been done to compare the effects of learning on context-
based approaches to science teaching with more traditional approaches. This study compares
the performance on a range of diagnostic questions of pupils following both a context-based
approach and a more traditional approach to high school chemistry. The study shows that there
is little difference in levels of understanding, but that there appear to be some benefits
associated with a context-based approach in terms of stimulating pupils' interest in science.
Empirical validity for a comprehensive model on educational effectiveness
School Effectiveness and School Improvement, 10(2), 24
Reezigt, G. J., Guldemond, H. & Creemers, B. P. M.
1999
Lehr-Lernskripts im Unterricht Seidel, T.
Münster: Waxmann
2003
Die vorgestellte Arbeit stellt eine Pilotuntersuchung des DFG-Projekts "Lehr-Lernprozesse im
Physikunterricht - eine Videostudie" dar. Es wird das Ziel verfolgt, anhand einer ausgewählten
Stichprobe differenziert Lehr-Lernskripts im Hinblick auf ihre Funktion für individuelle
Lernprozesse zu untersuchen. Lehr-Lernskripts werden dann als erfolgreich betrachtet, wenn
den Schülerinnen und Schülern im Unterricht Gelegenheiten eröffnet werden, sich
verstehensorientiert mit den behandelten Lerninhalten auseinanderzusetzen. Neben zentralen
231
Befunden der Unterrichtsforschung bilden Ansätze aus dem Bereich des Wissenserwerbs und
der Lernmotivation den theoretischen Hintergrund für die Konzeption der vorliegenden Arbeit.
Im Rahmen einer Videostudie wurde Physikunterricht (Sekundarstufe I) in sechs Schulklassen
aufgezeichnet und die Schülerinnen und Schüler im Anschluß daran zu verstehensorientierten
Lernprozessen und zu unterrichtlichen Lehr-Lernbedingungen befragt. Für die Kombination von
Videoanalysen und Schülerselbsteinschätzungen wurden differenzierte Beobachtungs- und
Fragebogeninstrumente entwickelt, erprobt und angewandt.
Die empirischen Befunde der vorgestellten Arbeit zeigen explorativ auf, wie auf der Basis der
angewandten Verfahren differenziert unterrichtliche "Gelegenheitsstrukturen" für individuelle
Prozesse beim Lernen beschrieben werden können. Zusammenfassend liegen Freiräume für
das geistiges Engagement und die Lernmotivation der Schülerinnen und Schüler weniger in der
Organisation unterrichtlicher Aktivitäten (z.B. im Ausmaß von Gruppenarbeiten,
Schülerexperimenten, etc.), sondern vielmehr in der Qualität der zugrunde liegenden Lehr-
Lernbedingungen. Charakteristische Lehr-Lernbedingungen in den untersuchten Klassen
stellen Aspekte der Klassengesprächsführung, der Zielorientierung sowie des
prozessorientierten Lehrens und Lernens dar.
Neben der inhaltlichen und theoretischen Diskussion der Befunde erfolgen methodische
Überlegungen zur Untersuchung von Lehr-Lernskripts. Abschließend werden auf der Basis der
Befunde dieser Arbeit Nutzungsmöglichkeiten videobasierter Lehr-Lernforschung für die
Reflexion routinisierter Unterrichtshandlungen sowie für die Entwicklung problemorientierter
Lehr-Lernumgebungen aufgezeigt.
"Can everybody look to the front of the classroom please?" - Patterns of instruction in elementary physics classrooms and its implications for students` learning
Unterrichtswissenschaft, 30(1), 52-77
Seidel, T., Prenzel, M., Duit, R., Euler, M., Geiser, H., Hoffmann, L., Lehrke, M., Müller, C. &
Rimmele, R.
2002
The paper reports findings about instructional patterns in elementary physics instruction. The
investigation of teaching and learning processes bases on the combination of video analysis
and student questionnaires. With regard to a well-defined sub-sample patterns of classroom
activities within the first three months of elementary physics instruction are investigated.
Indicators for successful physics instruction refer to the students` perceptions of cognitive
engagement, self-determined learning motivation and supportive teaching and learning
conditions within the lessons investigated. The findings of the video analysis indicate that
already within the first three months of elementary physics instruction patterns of classroom
organization can be described. Due to the organization of classroom activities limits for
students` individual learning processes are settled.
232
Opportunities for learning motivation in classroom discourse - Combination of video analysis and student questionnaires
Unterrichtswissenschaft, 31(2), 142-165
Seidel, T., Rimmele, R. & Prenzel, M.
2003
The paper presents findings of the DFG-project "Teaching and learning processes in physics
instruction - a videotape classroom study" and refers to opportunities for learning motivation in
classroom discourse. It is assumed that a strongly narrow-focused classwork has a negative
effect on self-determined learning motivation and on the students interest in physics. Thereby
video analysis is combined with student questionnaires and results of HLM-analysis are
reported. The findings indicate systematic differences between school classes in the quality of
classwork. School classes with a strongly narrow-focused classwork show a negative effect on
the students quality of self-determined learning motivation. Over the course of one school year a
negative effect of narrow-focused classwork on the students interest in physics is shown.
Clarity and Coherence of Lesson Goals as a Scaffold for Student Learning
Learning and Instruction, 4 (forthcoming)(Seidel, T., Rimmele, R. & Prenzel, M.
2005
The article addresses clarity and coherence of lesson goals as a scaffold for student learning. In
13 physics classes video recordings of the introductory lesson of two topics were rated with
respect to clarity and coherence of the lesson structure. HLM analyses showed a positive effect
of classes with high goal clarity and coherence on the students' reports on supportive learning
conditions, self-determined learning motivation and organising learning activities. Over the
course of a school year high goal clarity and coherence resulted in a positive competence
development. No effect was found for individual interest development.
Teacher communication behavior and its association with students' cognitive and attitudinal outcomes in science in Taiwan
Journal of Research in Science Teaching, 39(1), 63-78
She, H. C. & Fisher, D.
2002
In the study described in this article a questionnaire was employed that can be used to assess
students' and teachers' perceptions of science teachers' interpersonal communication behaviors
in their classroom learning environments. The Teacher Communication Behavior Questionnaire
(TCBQ) has five scales: Challenging, Encouragement and Praise, Non-Verbal Support,
Understanding and Friendly, and Controlling. The TCBQ was used with a large sample of
secondary science students in Taiwan, which provided additional validation data for the TCBQ
for use in Taiwan and cross-validation data for its use in English-speaking countries. Girls
perceived their teachers as more understanding and friendly than did boys, and teachers in
biological science classrooms exhibited more favorable behavior toward their students than did
those in physical science classrooms. Differences were also noted between the perceptions of
the students and their teachers. Positive relationships were found between students'
perceptions of their teachers' communication behaviors and their attitudes toward science.
Students' cognitive achievement scores were higher when students perceived their teacher as
using more challenging questions, as giving more nonverbal support, and as being more
understanding and friendly. The development of both teacher and student versions of the TCBQ
233
enhances the possibility of the use of the instrument by teachers. (C) 2002 John Wiley Sons,
Inc.
The nature of teachers' pedagogical content beliefs matters for students' achievement gains: Quasi-experimental evidence from elementary mathematics
Journal of Educational Psychology, 94(2), 344-355
Staub, F. C. & Stern, E.
2002
In a longitudinal study of 496 students in 27 self-contained German elementary school
classrooms, performance in mathematical word problems and arithmetic tasks was measured at
the end of Grades 2 and 3. A questionnaire was used to assess the degree to which teachers'
pedagogical content beliefs in elementary mathematics reflect a cognitive constructivist
orientation, rather than an associationist or direct-transmission view of learning and teaching.
Our findings show that a cognitive constructivist orientation was associated with larger
achievement gains in mathematical word problems. Moreover, teachers with a direct
transmission view were not more successful than teachers with a cognitive constructivist
orientation in fostering students' computational proficiency.
Effects of a cooperative learning approach in reading and writing on academically handicapped and non-handicapped students
Elementary School Journal, 95(3), 241-262
Stevens, R. J. & Slavin, R. E.
1995
The value (and convergence) of practices suggested by motivation research and promoted by mathematics education reformers
Journal for Research in Mathematics Education, 29(4), 465-488
Stipek, D., Salmon, J. M., Givvin, K. B., Kazemi, E., Saxe, G. & MacGyvers, V. L.
1998
Zn this study we discuss convergence between instructional practices suggested by research
on achievement motivation and practices promoted in the mathematics instruction reform
literature, and we assess associations among instructional practices, motivation and learning of
fractions. Participants included 624, fourth- through sixth-grade students and their 24 teachers.
Results indicated that the instructional practices suggested in literature in both research areas
positively affected students' motivation (e.g., focus on learning and understanding; positive
emotions, such as pride in accomplishments; enjoyment) and conceptual learning related to
fractions. Positive student motivation was associated with increased skills related to fractions.
234
An investigation of teacher-student interpersonal behavior in science classrooms using laptop computers
Journal of Educational Computing Research, 24(1), 41-55
Stolarchuk, E. & Fisher, D.
2001
This study was part of a larger evaluation of the effectiveness of laptop computers in grades 8
and 9 science classrooms, in a sample of Australian Independent Schools. In the study
described in this article, students' perceptions of their teacher's interpersonal behavior were
assessed using the Questionnaire on Teacher Interaction (QTI). As this was the first time this
questionnaire had been used in science classrooms where laptop computers were being used,
important validation data are provided. Associations between teacher-strident interpersonal
behavior and students' attitudes to science and their enquiry skill achievement were also
investigated. Students' attitudes to science were assessed using a scale adapted from the Test
of Science-Related Attitudes (TOSRA) and achievement was measured using scales from the
Test of Enquiry Skills (TOES). The QTI, attitude scale, and enquiry skills scales were
administered to 433 laptop students in grades 8 and 9 science classes, in nine Independent
schools across four Australian states. Descriptive statistics confirmed the reliability and validity
of the QTI for science laptop classroom research. Generally, laptop students' perceptions of
teacher-student interpersonal relationships were found to be positively associated with students'
attitudinal and cognitive achievement outcomes.
Schulische und häusliche Kooperation im Chemieanfangsunterricht Sumfleth, E., Rumann, S. & Nicolai, N.
Münster: Waxmann
2004
Reading growth in high-poverty classrooms: The influence of teacher practices that encourage cognitive engagement in literacy learning
Elementary School Journal, 104(1), 3-28
Taylor, B. M., Pearson, P. D., Peterson, D. S. & Rodriguez, M. C.
2003
In this study we investigated the effects of teaching on students' reading achievement. More
specifically, based on a framework of reading instruction maximizing students' cognitive
engagement in literacy learning, curricular and teaching variables, including aspects of word-
recognition and comprehension instruction, approaches to teaching such as telling versus
coaching, and the enabling of students' active versus passive responding to literacy activities,
were investigated to explain growth on reading comprehension, fluency, and writing measures
over a school year in grade 1-5 classrooms. Participants included 88 teachers and 9 randomly
selected students per classroom in 9 high-poverty schools across the United States that were
engaged in a literacy instruction reform project. Teachers were observed 3 times across the
school year during a reading lesson. Hierarchical linear modeling showed that a number of
teaching variables explained substantial variation in student growth on several measures of
reading achievement. Looking across all of the data, the most consistent finding was that
teachers who emphasized higher-order thinking, either through the questions they asked or the
tasks they assigned, promoted greater reading growth among the 9 target students in their
235
classrooms. We examine the results of our work in relation to a framework of teacher
effectiveness maximizing students' cognitive engagement in literacy learning.
Learning environment, meaningfulness of schoolwork and on-task-orientation among Norwegian 9th grade students
School Psychology International, 21(4), 393-413
Thuen, E. & Bru, E.
2000
The main aim of this study was to explore how learning environment dimensions are related to
on-task-orientation, and how these relations are mediated by students' perceptions of the
meaningfulness of schoolwork. The study was conducted as a survey among a representative
sample of 2006 Norwegian 9th graders. Results showed that a majority of the students reported
to be on-task-oriented during school lessons, however, between 10 and 20% reported to have
problems with their on-task-orientation At the same time, slightly more than half of the students
reported their schoolwork to be useful, whereas one in five students reported their schoolwork to
be rather uninteresting. On-task-orientation was also found to be related to students'
perceptions of several learning environment dimensions, primarily to students' perceptions of
teacher support (academic and emotional), the possibility for influencing ones' schoolwork and
the meaningfulness of schoolwork. Associations of perceptions of teachers' support and student
influence with on-task-orientation were partly mediated via perceptions of the meaningfulness of
schoolwork. Finally, results indicate that there is likely to be considerable room for improvement,
within Norwegian schools at least, in the areas of teachers' emotional support of students,
students' influence on schoolwork and the meaningfulness of school subjects. For practising
school psychologists results seem to imply an increased focus on the learning environment, and
particularly on the importance of the social emotional dimensions for students' motivation and
on-task-orientation In order to bring about changes in the learning environment, counselling
teachers on how to create a supportive learning environment with student influence could be
one important way of approaching this held.
Kontextfaktoren der Schulleistung im Grundschulalter. Ergebnisse aus der Hannoverschen Grundschulstudie
Zeitschrift Fur Pädagogische Psychologie, 18(2), 113-124
Tiedemann, J. & Billmann-Mahecha, E.
2004
Within the scope of the Hanover Primary School Study, the significance of context factors of
school performance were established for 710 third-grade students. Amongst others, the
following context factors were ascertained: Aggregated cognitive ability level, age of the
students, and classroom climate. The data were analyzed using hierarchical linear modeling.
The results substantiate the dual function of cognitive abilities within the framework of the
genesis of school performance: On the one hand, cognitive abilities directly control individual
performance in mathematics, on the other hand, however, they also indirectly control success
over effects at the aggregated level. In addition, classroom climate and student age proved to
be significant. At the school level it was shown that the performance differences between
classes in schools with either a high or a low proportion of immigrant children reach a
magnitude that corresponds to the provision of more than one year of schooling.
Measuring NCTM-recommended practices and student achievement with TIMSS
236
Tomoff, J., Thompson, M. & Behrens, J.
New Orleans, LA.: 2000
Classroom goal structures, social satisfaction and the perceived value of academic tasks
British Journal of Educational Psychology, 67(1-12
Townsend, M. A. R. & Hicks, L.
1997
Background, Perceptions of academic task value are affected by positive factors such as the
intrinsic value of the task, and negative factors such as the loss of opportunities to engage in
other desired activities. Previous research has shown that students' pursuit of social activities
may be a negative factor that reduces the task value given to mathematics and language
activities. However, the desire to meet social goals may be influenced by the degree to which
classroom instruction provides opportunities for peer involvement and affiliation. Aims, This
study examined the relationship between academic task values (for mathematics and language)
and perceptions of personal social satisfaction for children in classrooms using a cooperative,
interactive learning structure or in regular classrooms. Samples, Participants were 162 children
(aged approximately 12-13 years) in six Form 2 classes at two intermediate schools. Three
classes were taught by teachers using cooperative learning strategies. Methods. Children in
each of the six classrooms were administered a Social Satisfaction Scale (to assess satisfaction
with peer social relationships), the Academic Task Values Scale for mathematics and language,
and the Competition and Task Orientation subscales of the Classroom Environment Scale.
Results, Task values for engagement in mathematics and language activities were higher, and
perceived costs lower, in classrooms using a cooperative goal structure. Higher task values and
lower perceived costs were also associated with higher social satisfaction in students. These
effects were mediated by gender. Conclusions, Research in achievement motivation needs to
examine the coordination of multiple goals which extend beyond the academic domain.
The influence of classroom contexts on young childrens motivation for literacy
Reading Research Quarterly, 30(3), 410-441
Turner, J. C.
1995
THIS STUDY examined the effects of instructional contexts on children's motivation for literacy.
Eighty-four first-grade children in six basal classrooms and in six whole language classrooms
were observed during daily literacy instruction. Literacy tasks were classified as open (child-
specified processes/goals, higher order thinking required) or closed (other-designated
processes/goals, recognition/memory skills required). Observers recorded the types of literacy
tasks children completed and children's voluntary use of motivated behaviors (strategy use,
persistence, attention control) during task completion. Children were interviewed after
observation. The strongest predictor of motivation was the literacy task. During open tasks,
children used more reading strategies, persisted longer, and controlled their attention better
regardless of instructional condition. Factors in open tasks that appeared to influence children's
motivation were opportunities for challenge, for student control, for satisfying interests, and for
collaboration. These results suggest that researchers should go beyond philosophical and
curricular assumptions to examine how students actually interact with literacy contexts.
Therefore, studies of motivation for literacy should focus on how literacy contexts, including
tasks, influence children's actions and understandings.
237
The classroom environment and students' reports of avoidance strategies in mathematics: A multimethod study
Journal of Educational Psychology, 94(1), 88-106
Turner, J. C., Midgley, C., Meyer, D. K., Gheen, M., Anderman, E. M., Kang, Y. & Patrick, H.
2002
The relation between the learning environment (e.g., students' perceptions of the classroom
goal structure and teachers' instructional discourse) and students' reported use of avoidance
strategies (self-handicapping, avoidance of help seeking) and preference to avoid novelty in
mathematics was examined. Quantitative analyses indicated that students' reports of avoidance
behaviors varied significantly among classrooms. A perceived emphasis on mastery goals in the
classroom was positively related to lower reports of avoidance. Qualitative analyses revealed
that teachers in high-mastery/low-avoidance and low-mastery/high-avoidance classrooms used
distinctively different patterns of instructional and motivational discourse. High incidence of
motivational support was uniquely characteristic of high-mastery/low-avoidance classrooms,
suggesting that mastery goals may include an affective component. Implications of the results
for both theory and practice are discussed.
Effect of formal study skills training on sixth grade reading achievement Udziela, T.
1996
The effects of Developmentally Appropriate Practices on academic outcomes among former head start students and classmates, Grades 1-3
American Educational Research Journal, 40(4), 961-990
Van Horn, M. L. & Ramey, S. L.
2003
The educational ideology of Developmentally Appropriate Practices (DAP) in childhood
education is influential, despite remarkably little empirical study. This article relates DAP to
changes in achievement and receptive language among former Head Start children and
classmates in Grades 1-3 (including between 1,564 and 4,764 children in 869 to 1,53 7
classrooms). The authors applied multilevel growth curve modeling techniques to estimate
overall DAP effects and to examine possible interactions with sex, ethnicity, grade, and poverty.
ne results were consistent across years, with only a few significant effects of DAP, some
positive and others negative. Collectively, the results indicate that DAP as observed in
classrooms accounts for little or no variation in children's academic performance. The article
details the methodological and theoretical implications for future inquiry.
The link between teacher classroom practices and student academic performance
Educational Evaluation and Policy Analysis, 10(12), Wenglinsky, H.
2002
Inquiry, modeling, and metacognition: Making science accessible to all students
Cognition and Instruction, 16(1), 3-118
White, B. Y. & Frederiksen, J. R.
1998
238
Our objective has been to develop an instructional theory and corresponding curricular materials
that make scientific inquiry accessible to a wide range of students, including younger and lower
achieving students. We hypothesized that this could be achieved by recognizing the importance
of metacognition and creating an instructional approach that develops students' metacognitive
knowledge and skills through a process of scaffolded inquiry, reflection, and generalization.
Toward this end, we collaborated with teachers to create a computer enhanced, middle school
science curriculum that engages students in learning about and reflecting on the processes of
scientific inquiry as they construct increasingly complex models of force and motion
phenomena. The resulting ThinkerTools Inquiry Curriculum centers around a metacognitive
model of research, called the Inquiry Cycle, and a metacognitive process, called Reflective
Assessment, in which students reflect on their own and each other's inquiry. In this article, we
report on instructional trials of the curriculum by teachers in urban classrooms, including a
controlled comparison to determine the impact of including or not including the Reflective
Assessment Process. Overall, the curriculum proved successful and students' performance
improved significantly on both physics and inquiry assessments. The controlled comparison
revealed that students' learning was greatly facilitated by Reflective Assessment. Furthermore,
adding this metacognitive process to the curriculum was particularly beneficial for low-achieving
students: Performance on their research projects and inquiry tests was significantly closer to
that of high-achieving students than was the case in the control classes. Thus, this approach
has the valuable effect of reducing the educational disadvantage of low-achieving students
while also being beneficial for high-achieving students. We argue that these findings have
strong implications for what such metacognitively focused, inquiry-oriented curricula can
accomplish, particularly in urban school settings in which there are many disadvantaged
students.
239
Children's motivation for reading: Domain specificity and instructional influences
Journal of Educational Research, 97(6), 299-309
Wigfield, A., Guthrie, J. T., Tonks, S. & Perencevich, K. C.
2004
The authors discuss the nature and domain specificity of reading motivation and present initial
results that examined how 2 reading instructional programs, Concept Oriented Reading
Instruction (CORI) and multiple Strategy Instruction (SI), influenced 3rd-grade children's intrinsic
motivation to read and reading self-efficacy. Each reading program occurred during the fall of
the school year and lasted 12 weeks. Approximately 150 3rd-grade children participated in
CORI; 200 3rd-grade children participated in SI. Results of pre- and posttest analyses of
children's responses to a reading motivation questionnaire showed that children's intrinsic
motivation to read and reading self-efficacy increased only in the CORI group.
Family, classroom, and school effects on children's educational outcomes in Latin America
School Effectiveness and School Improvement, 12(4), 409-445
Willms, J. D. & Somers, M. A.
2001
This study, which is based on a longer report by Willms and Somers (2000), employs
hierarchical linear regression models to examine the relationships between 3 schooling
outcomes (language and mathematics achievement, and time to complete primary schooling)
and family background, as well as various school policies and practices. The analyses employ
data from the Primer Estudio Intenacional Comparativo (PEIC; UNESCO, 1998), which includes
data for 13 Latin American countries, with samples of approximately 100 schools in each
country, and 40 grades 3 and 4 pupils sampled in each school. The study finds that the
relationship between schooling outcomes and family background varies among countries. The
most successful country, Cuba, has uniformly effective schools, and relatively small inequities
along social class lines and between the sexes. Across all countries, the most effective schools
tend to be those with: high levels of school resources; classrooms which are not multigrade, and
where students are not grouped by ability; classrooms where children are tested frequently;
classrooms and schools with a high level of parental involvement and classrooms that have a
positive classroom climate, especially with respect to classroom discipline. The article
concludes with a discussion about how we might improve capability to monitor school
performance in low-income countries.
Reforming motivation: how the structure of instruction affects students' learning experiences
British Educational Research Journal, 26(2), 191-210
Yair, G.
2000
Over the past decade, several US school and instructional reforms have sought ways to counter
trends of mediocrity in education. These reforms are grounded in structural motivation theories
which postulate that students' learning experiences are optimised when instruction is authentic,
challenging, demands skills, and allows for student autonomy. This study set out to investigate
empirically the effects of these four structural characteristics of instruction on students' learning
experiences. Using a unique methodological design, the current investigation measures
students' learning experiences with a confirmatory factor analysis. The four factors which
240
emerged are next predicted with a series of structural variables, The results show that high
quality learning experiences are indeed authentic, allow choice, and demand student skills.
Boring and alienated experiences are produced when these instructional characteristics are
absent. The findings suggest that the structures of instruction that disaffect students are
overwhelmingly represented in students' daily school life; those that spark their hearts are not
frequent enough to motivate students. They also imply that students do not have a general
tendency to be emotionally depressed in school; rather, they perceive their experiences to be
highly influenced by specific structural characteristics of instruction.
PART III
PROPORTIONAL SUPERVISION
243
Chapter 6
Criteria for assessing school self-evaluation approaches
Introduction
In the Netherlands, two quality aspects in the inspection framework refer to quality care
procedures and conditions for quality care at school, respectively. In this chapter the
quality indicators that are in the Inspection framework will be compared to state of the
art perspectives on school self-evaluation.
Various frames of reference will be used to put these quality indicators into perspective:
- Sets of criteria that have been used in the Netherlands to assess school self-
evaluation instruments and procedures for quality care; i.e. those of Q*Primair,
Q5 and the Inspectorate;
- Sets of criteria that have been used by foreign inspectorates and groups of
inspectorates, i.e. SICI;
- More general evaluation standards (i.e. those of the Joint Committee of
Standards in the USA, chaired by Daniel Stufflebeam);
- A closer look at the utilization of school self-evaluation instruments and
application of quality care;
- A closer look at accuracy standards applied to school self-evaluation, the issue
of objectivity.
The chapter provides descriptive material on all these contributions in order to assess
the current state of affairs with respect to proportional supervision, and the respective
quality indicators in the Inspection Framework. In the concluding section some of the
sustaining difficulties with internal quality care and school self-evaluation are
addressed, and implications for proportional supervision are discussed.
Terminology
Educational quality
First, a short overview of the basic concepts of quality care, school evaluation and
school self-evaluation will be given.
In Chapter 2, the concept of educational quality was clarified on the basis of the Context
Input Process Output model. In addition, the Supervision framework of the Inspectorate
was compared to conceptions and frameworks of educational quality, e.g. the Quinn and
Rohrbaugh model.
244
It was concluded that, as a school evaluation framework, the Supervision framework
could be classified as being associated with the effectiveness perspective on educational
quality.
In the effectiveness perspective, the process indicators of the Inspection Framework can
be seen as effectiveness enhancing conditions that are inspired by empirical school
effectiveness research and by an international consensus on good teaching practice.
Indicators that represent school self-evaluation and quality care have a double
significance in the framework: as one of the effectiveness enhancing school conditions
but also as a part of the overall supervision framework in the sense of the so called
proportional supervision. (This comes down to the strategy of making the frequency and
intensity of direct supervision conditional on schools’ having well-developed school
self-evaluation procedures; the better developed the school self-evaluation procedures,
the less frequent and intensive the direct supervision).
Quality care
Quality care in schools involves all the measures taken at school- and classroom level to
maintain and improve the quality of their education. It relates to all activities aimed at
formulating quality goals and achieving, maintaining and systematically improving the
quality of education. It involves determining the goals of the school and the way the
goals can be reached, the execution and evaluation of quality policy and the deciding on
actions that might be necessary for maintaining and improving the quality of the school
(Hendriks, 2001; Hendriks, Doolaard & Bosker, 2002).
School self-evaluation
School self-evaluation forms a crucial aspect of quality care. School self-evaluation is
an internal evaluation of the school as a whole or of sub-units of the school. It could be
‘completely internal’, but in a self-evaluation also extensive use of external capacity
could be made. The decisive point is that the school is the initiator and the prime
audience of the self-evaluation (Scheerens, Glas & Thomas, 2003). School self-
evaluation can be defined as a procedure started by the school for gaining information
on the design and goals of education in order to take policy decisions for maintaining
and improving of the quality of the school (according to Voogt, 1995; Van Petegem,
2001).
Self-evaluation could be carried out at the outset of applying the well-known quality
care cycle, in order to map the strong and weak aspects of the school. In this case the
self-evaluation is aimed at diagnosing needs and settling goals for improvement. But
self-evaluation can also be used to determine whether the intended goals of quality
improvement have been achieved. The aim is then to determine, if the school, given the
intended goals, is on the right track.
245
Criteria for assessing quality care proposed in the Netherlands
Criteria used by the Education Inspectorate
In the Netherlands, the framework of quality care in education is determined by
legislation, general orders in council, and notes of the Education Council, the parliament
and the Ministry of Education, Culture and Sciences (Kamphof, 2001). In the 'Quality
Act' (Primary Education Act, 1998), the responsibility for the quality of education and
the pursuance of a quality policy to ensure its improvement is explicitly mentioned as a
duty of the school itself. It is decreed that all schools develop a system of quality
assurance. Moreover, it is set out that the school is required to develop three policy
documents: a school plan, a school prospectus and an arrangement for complaints
(Ministry of Education, Culture and Sciences, 1995; Karstanje, 1997; Scheerens &
Hendriks, 2002).
In keeping with these developments, external evaluation by the Inspectorate was
adapted. In the law on the Supervision of Education (WOT, 2002), validation of the
quality assurance policy of the school is seen as a major part of the inspection process
and the degree of external school inspection is proportional to the extent in which
schools succeed in appropriate forms of quality care and school self-evaluation. Article
12 of the Law on the Supervision of Education states that the results of the self-
evaluation will be adopted by the Inspectorate if:
- the judgements of the school relate to the key characteristics of quality as laid down
in the Supervision Act,
- the data of the school’s self-evaluation are of more recent date than those the
Inspectorate already has at it’s disposal;
- the school self-evaluation gives a founded and reliable picture of the quality of the
school:
- the school uses sound evaluation methods;
- pupils, parents and external experts have been involved in the evaluation;
- the quality goals the school is aiming for are ambitious enough and,
- the quality care system of the school is robust enough and the results show a rather
stable situation.
The Supervision of Education Act also states that the Inspectorate is authorised to
promote the quality of the school. Supervision should stimulate the implementation of
excellent quality care as well as a permanent improvement of the quality. In case serious
problems exist and remain with regard to the quality of education, the Inspectorate can
advise the Minister of Education to offer the school and the school board some kind of
‘provision’. This provision may be financial but may also take the form of extra staff or
246
specific advice (Inspectorate of Education, 2002a and b; Ministry of Education, Culture
and Sciences, 2002; Renkema, 2002).
In 2005 an updating and overhaul of the Frameworks took place. From then onwards,
there are two indicators that refer to quality care (see text box 1).
Quality Aspect 1:
The school takes care of the assurance and improvement of the quality of education.
1.9 The school knows its entrance situation, including the specific needs of the student
body.
1.10The school systematically evaluates the quality of its performance in terms of
learning results
1.11The school systematically evaluates the quality of learning, teaching and
counselling
1.12The school has formulated measurable improvement targets.
1.13The school carries out improvement activities in a systematic way
1.14The school guarantees the quality of learning and teaching
1.15The school guarantees the quality of the school examination and of other
evaluation instruments.
1.16The school reports about the realized quality of education to interested parties
(parents, students, competent authorities, funding agencies and sponsors).
Quality Aspect 2:
The conditions for quality care are in place
2.1 School management initiates and steers the quality care
2.2 Quality care is connected to the school’s vision with respect to learning and
teaching as stated in the school plan.
2.3 The school management takes care of a professional school culture
2.4 The school takes care of an effective communication about the quality of
education.
2.5 Staff, school management, pupils, parents and competent authorities are all of
them being involved in the school’s quality care.
Since 1997, the Inspectorate evaluates the state of affairs with regard to the internal
quality assurance of schools and presents the findings in the yearly Education Report.
The state of affairs with respect to quality care in primary schools
In 2003, 13 per cent of the primary schools had a quality care system that satisfied the
full set of indicators on quality care. Twenty-six per cent of schools satisfied the
247
minimum requirements for quality assurance, which means that they, in each case,
fulfill the first two indicators (see Table 6.1).
In the past six years, the percentage of schools that satisfied the minimum requirements
for quality assurance was highest (38 per cent) in the school year 2000. In this year,
schools had to prepare their first school plan and school guide. In the years after, when
there was no pressure for a school plan, the percentage of schools receiving a positive
judgement of the Inspectorate dropped back to around 26 per cent in 2001 and the years
after (Inspectorate of Education, 2004).
Table 6.1: Primary schools with a positive judgment about their quality assurance
system (%) (Inspectorate of education, 2003)
1998
N=379
1999
N=576
2000
N=467
2001
N=421
2002
N=255
2003
N=163
Quality assurance on the whole 28.8 31.6 36.6 26.4 24.9 26.3
Indicators:
1. The school has formulated targets
with regard to learning and teaching
and performance in terms of learning
results
21.2
43.8
45.0
35.3
33.1
42.6
2. The school systematically and
periodically evaluates the quality of
education
26.3
42.4
53.5
38.2
39.8
31.1
3. On the basis of (integral) quality
assessment the school has planned
improvement activities for the longer
term
44.3
51.3
46.2
46.5
52.1
4. On the basis of recent quality
analyses the school has planned
improvement activities for the present
school year
73.0
79.4
70.0
73.3
66.0
5. The school has taken measures to
maintain and improve the quality
51.8
47.2
43.8
47.2
38.9
Secondary schools
In Table 6.2 it is shown that in 2004, 36 per cent of the schools for secondary education
schools satisfied the minimum requirements for quality assurance. These schools have
formulated targets with regard to the maintenance and improvement of education, and
have taken measures to achieve these targets (indicator 1 in Table 6.2). Next to this
these schools systematically and periodically evaluate the quality of their education and
learning results (indicator 2). The percentage of schools that meet the full set of
248
indicators for quality care is considerable lower. In 2003, 11 per cent of secondary
schools met all indicators for quality care; in 2002 this was the case for 9 per cent of the
schools.
Table 6.2: Secondary schools with a positive judgment about the quality assurance
system (%) (Inspectorate of education, 2004)
2001/2002
N=436
2002/2003
N=163
Quality assurance on the whole 34 36
Indicators:
1. The school has formulated targets with regard to the
maintenance and improvement of the quality of education
73
74
2. The school systematically and periodically evaluates the
quality of education
37
36
3. The school involves educational innovations in their policy-
development
94
90
4. The school involves school development in their policy-
development
95
94
5. The school has taken measures to maintain and improve the
quality
63
68
6. The school examines the effects of quality care and adjusts the
school policy accordingly
12
16
The Inspectorate noticed that the systematic and periodic evaluation of the quality of
education is inadequate in the majority of schools due to an unsatisfactory quality care
system. Although a wide choice of school self-evaluation instruments is available many
of these instruments offer insufficient possibilities in their present form 1) to evaluate
the educational processes in an objective way, and 2) to relate the processes to the
outcomes of education (i.e. the learning results).
At the same time availability of reliable performance data has been increasing and
gained in acceptance. By an increased use of standardized tests the Inspectorate was
able to determine the learning results at 90 per cent of the primary schools, while in
secondary schools the Inspectorate could make use of the exam results.
The principle of proportional inspection was introduced in 2003, and up to now, limited
experiences are available (see also in the next paragraphs the pilot projects of Q5 and
Q*Primair). Yet, it has become clear that the generally limited quality of data obtained
by school self-evaluation, in particular with regard to school processes, turned out to be
an important restriction with regard to the implementation of proportional supervision.
Nevertheless the Inspectorate noticed, just like in 2003, that schools appreciate the
249
reciprocal relationship between quality assurance at school and supervision by the
Inspectorate. They experience the supervision as complementary to both the self-
evaluation and school development (Inspectorate of Education, 2003, 2004).
Criteria used by Q5 and Q*Primair
In order to support schools and their governing boards with their quality assurance and
to give a strong impulse to the development and implementation of systematic quality
assurance, with subsidy of the Ministry of Education, the organizations of governing
bodies of the schools, the organization of school managers and the teachers’ unions
(these latter organizations are only involved in primary education) set up the national
project groups Q5 for secondary education and Q*Primair for primary education.
Q5
Q5 was initiated in 1999, and it expires in December 2005. The project attempted to
stimulate systematic and integral quality care in schools for secondary education, i.e. it
was aimed at stimulating schools to:
- Develop a system of quality management as an integral part of school development;
- Involve all relevant groups inside and outside schools;
- Present results of self-evaluation to third parties (other schools, experts, 'critical
friends');
- Participate in networks, in order to exchange information and expertise;
- Publish information about their quality (Beelaerts, Bousché, De Goeij, De Graaff,
Horsman & Klifman, 1999).
Q5 defines sound quality care as deliberately, systematic, cyclic and integral, with a
focus on the domains of teaching and learning, vision and leadership, school culture,
results and appreciation. Q5 judges quality policy as acceptable if it satisfies both the
requirements of good quality care as stated above and the legal requirements, if the
stakeholders accept it and, if it is practical and transparent (Horsman, z.j.). The
practicality encompasses the balance of efforts and output. The quality policy is
acceptable if the required endeavours end in sufficient points of departure for feasible
improvement and maintenance of quality at three levels of accountability of the school;
i.e. the school as a whole, the teams and the individual teacher.
Q5 operationalized the objectives of the project into statements, however without setting
concrete targets. Due to the expected differences in nature and implementation of
quality care between schools, Q5 remained somewhat undecided about the position to
take with respect to the state of quality care in schools for secondary education to be
reached by the end of the project.
250
The last written down assumptions about quality care in secondary education date from
end 2003 (Q5, 2003). In the annex “Achieved aims” it is stated that by the end of the Q5
project the situation probably should be that:
- 25 to 30 per cent of the schools applies quality care as an integral and systematic
part of school development;
- 50 per cent of the schools applies simple forms of quality care for school
development;
- 80 per cent of the schools has experience with school self-evaluation;
- 60 per cent of the schools presents the results of the self-evaluation to third parties;
- 90 per cent of the schools involves in each case parents and pupils in the self-
evaluation, but preferable also other parties;
- 75 per cent of the schools has made well argued choices for setting standards;
- 50 per cent of the schools exchange information with other schools, preferably also
in an international setting.
During the Q5 project, the progress of the implementation of quality care in secondary
education was monitored three times, i.e. at the beginning of the project, after the first
two project years and at the end of the project (Haaijer & van Velzen, 2002; Van
Beekveld & Terpstra, 2005).
Q5 participated in several national and international pilot projects. The most important
one for this study is the so-called ABC project. The ABC project has introduced a 3-step
quality system. The five participating schools first carried out their own self-evaluation
and prepared a report (phase A). In the B-part third parties (e.g. other schools, experts,
'critical friends') then visited the school and assessed the results of the school self-
evaluation and the self-evaluation report of the school, and presented their findings and
feedback in a report. Lastly, in the C part, the Education Inspectorate carried out a
Periodic Quality Inspection (PKO), based on the results of the A and B parts and in
accordance with the principles of proportional inspection, and drew a report (Horsman,
Q-vijver, april 2001).
The ABC project started in 2001 and ended in 2003. The project was proved to be a
success for the participating schools and resulted in a set of models and instruments
interesting for all secondary schools that want to implement or continue such a quality
system. The most important products concern recommendations for the planning and
implementation of self-evaluation activities, a manual for writing a self-evaluation
report, recommendations for the composition of a commission of third parties, a
working procedure for the commission of third parties as well as a format for their
251
report and recommendations for the preparation and execution of a periodic quality
inspection (PKO).
Q*Primair
Q*Primair started in 2001 and will expire towards the end of 2006. At the outset of the
project Q*Primair formulated the aims they are striving for in the project. In addition
Q*Primair asked the consultancy and research center Oberon to make an inquiry of the
situation with regard to quality care at the beginning of the project (a baseline
measurement).
Below the goals Q*Primair is pursuing in the project are presented. The starting
situation in 2001 as measured by Oberon (2002) is shown between brackets.
The main aim of Q*Primair is that in 2006, 80 per cent of the schools will have a well
functioning quality care system. This implies that in 2006 the state of affairs with regard
to quality care in primary education is, that:
80 per cent of the schools has formulated their quality care policy in such a way that
it is clear how the school satisfies on the one hand the legal requirements with
regard to quality care as well as on the other hand the quality goals as set and aimed
for by the school itself (starting situation: applies to 20 per cent of the schools);
80 per cent of the schools systematically monitors the quality of their education and
takes action to maintain and improve the quality (starting situation: applies to 31 per
cent of the schools);
In their school plan, 80 per cent of the schools has formulated targets for the quality
of teaching and learning as well as its performance in terms of learning results
(starting situation: applies to 40 per cent of the schools);
80 per cent of the schools systematically and regularly evaluates the quality of its
teaching and learning and performance (starting situation: applies to 47 per cent of
the schools);
Based on their evaluation of the quality, 80 per cent of the schools has planned an
improvement route for the long term (starting situation: applies to 49 per cent of the
schools);
Based on their evaluation of the quality, 90 per cent of the schools has planned
improvement actions for the current school year (starting situation: applies to 77 per
cent of the schools);
80 per cent of the schools uses a system of instruments and procedures for
monitoring progress of the pupils and takes care of guidance an counseling in case
of problems (starting situation: applies to 54 per cent of the schools);
252
80 per cent of schools involves parents and pupils in the school’s quality care but
preferably also other stakeholders (starting situation: applies to 12 per cent of the
schools);
80 per cent of the schools reports about the realized quality of education to
interested parties (in each case parents, school board and personnel (starting
situation unknown);
In order to objectify the self-evaluation, 30 per cent of the schools have put the
results of the self-evaluation at least once to independent third parties (starting
situation unknown) (Hofman, Dijkstra, Hofman & De Boom, 2004).
The state of affairs with regard to quality care in the schools and the progress that has
been made during the implementation of the project was measured for the first time by
the end of 2003; the second measurement is planned for 2005 or 2006 (Hofman,
Dijkstra, Hofman & De Boom, 2004).
Next to this, in 2004 Q*Primair made an assessment of the available instruments for
school self-evaluation, in order to help with the selection of a school self-evaluation
instrument that fits with the school-specific situation. For this activity, by order of both
Q5 and Q*Primair, Hofman & Hofman (2003) developed an assessment framework,
and Q*Primair assessed the available instruments (Dijkstra, van der Linde & Majoor,
2004).
Two projects were started focusing on the balance of internal quality care and external
school inspection.
Similar to the ABC project of Q5, in January 2004 Q*Primair started the project
“Ziezo” (Self-evaluation: Investing in a Visible Educational Quality) (Q*Primair,
2003). In this project, for a period of two years, 27 schools are working on a
methodology for self-evaluation and external review, connected with proportional
inspection by the Inspectorate. The basic idea is threefold:
- The process of school development and quality improvement will be stimulated if
schools are able to portray their own quality and these subsequently make validate
by other schools;
- With the self-evaluation and external review also processes of accountability will be
started; and
- Also, the foundation will be laid for proportional inspection (Q*Primair, 2003).
The participating schools are enthusiastic. In January 2005, 23 schools have finished the
phase of self-evaluation. For most of the schools the writing of a self-evaluation report
was quite an intensive learning experience. The schools are still looking for a working
253
method that fits with their situation. Also, the commitment for the project within the
schools is not always clear: the teachers are not always involved, and in a stronger
degree this is also the case for parents and students (Q*Primair, 2005).
Second, recently the project “Balans” has been launched. In this project 8 governing
boards with more than 15 schools within their board will work on the design and try out
of a system of proportional inspection, based on a well-developed quality care policy
plan that fits with the own organization. The crux of the project lies in the exploration
and design of sound methods for school self-evaluation in school boards with more than
one school and several management levels of responsibility and decision-making (e.g.
the school level, the above-school level and the level of the school board), this in
connection with an external review and proportional inspection. The idea is that if
governors, multi-school managers and school leaders are able to evaluate their own
quality as well as to make their quality judged by other governing boards, in dialogue
with the Inspectorate, there will be a basis to draw up a covenant for proportional
inspection of the schools. Next to this the project is aimed at further stimulating the
process of school self-evaluation and external review in primary education (Wolters,
2005).
Criteria to assess school self-evaluation instruments
For the purpose of educational monitoring and school self-evaluation more and more
tools and instruments have become available. During the past years several evaluation
studies of instruments for quality assurance have been carried out (Cremers-van Wees
e.a., 1996a and b; Hendriks, 2000; Duurkoop et al., 2002; Dijkstra, van der Linde &
Majoor, 2004). In these studies instruments for quality assurance, among others, have
been classified on content (the areas covered), the methodology, the respondents, the
information the instrument produces and the utilization for school development (i.e the
help for quality improvement and quality control).
Content
Ideally school self-evaluation should look at all aspects of school functioning
(Scheerens, Glas & Thomas, 2003). However, if only for practical purposes, one needs
to make choices and set priorities. Models of school quality and educational
effectiveness, like the Dutch Quality Model of the Institute for Dutch Quality (cf.
Duurkoop, Majoor & Pinth, 2002) and the competing values model of Quinn and
Rorhbaugh (cf. Scheerens, 1989, 1990; Van de Venne, 1999; Van de Venne et al., 2000)
offer a first helping hand to judge choices. Moreover, designers, authors and users of
254
school self evaluation instruments aim more and more at correspondence of the
instrument with the framework of the Inspectorate of Education. A third rationale to
select variables can be found in the results of school effectiveness and instructional
effectiveness. The rationale in choosing the process variables is to select those
malleable variables that, in research, have been shown to be associated with relatively
high ‘value-added’ achievement, either in the cognitive or the affective domains.
Methodology
With regard to methodology, a distinction can be made between quantitative and
qualitative approaches. Qualitative approaches usually make use of “open” research
formats, like unstructured interviews, have a strong dependence of the view of the
persons that are part of the evaluandum, and could provide often more elaborated and
narrative descriptions of the object situation (Scheerens, Glas & Thomas, 2003). With
quantitative procedures, the approaches for data collection must conform to
requirements concerning objectivity and reliability. With qualitative approaches, less
use is made of the formal requirements of sets of instruments. However, some authors
(e.g. Yin, 1981, Russ-Eft & Preskill, 2001) argue that, also in the case of qualitative
approaches, evaluators and researchers need to examine very well the trustworthiness of
the evaluation procedures and results.
Respondents
Paramount is which groups of stakeholders, among others pupils, teachers, head
teachers, other staff, the school board, schools for secondary and further education, as
well as the labour market and wider community, will be involved in the data collection.
Former research (Cremers-van Wees et al., 1996a and b) showed that in most cases only
teachers and school leaders were involved as data providers. Nowadays, satisfaction of
clients is an important aspect of quality care and it is stimulated to involve parents and
pupils more explicitly.
The technical adequacy of school self-evaluation instruments
The use of school self-evaluation instruments should lead to sound information, on the
basis of which well-thought judgements to quality maintenance and improvement can
be made. To ensure that the evaluation instrument will reveal and convey technically
adequate information, the evaluation instrument should be reliable and valid. Moreover
the instrumentation should have a sound theoretical basis and the procedures of data
collection, analysis, and judgements should be “objective” and standardised as much as
possible.
Reliability refers to the extent of accuracy of the results, i.e. the degree in which the
results are internally consistent, can be reproduced and in which the instrument is free
255
of errors of measurement (Lam, 1996, Voeten, 2000). The reliability is a necessary but
not sufficient condition for the validity. When designing a new instrument, the validity
is possibly even more important and needs to be examined very thoroughly. Construct
validity refers to the degree in which the instrument measures what it intends to
measure, and discriminates when compared to operationalizations of other concepts.
Content validity reflects the degree to which the set of items or observation categories is
representative for a specific domain of contents or skills. Criterion validity assesses the
degree to which an instrument predicts a certain external criterion.
The above-mentioned concepts of reliability and validity are more readily applicable to
quantitative procedures of data collection than to qualitative descriptions. Diverse
authors (Russ-Eft & Preskill, 2001; Scheerens, Glas & Thomas; 2003) mention a great
number of measures that could be taken to ensure that all has been done to achieve a
high degree validity in case of both qualitative and quantitative procedures; i.e.:
- To examine very carefully the accuracy of which data have been stored and coded as
well as that all data have been included in the analysis;
- To repeat measurements;
- To feed back the results to the respondents and to gain the confidence of
respondents so that the information they provide will be more likely reliable and
valid;
- To check the results as well as content of the draft report with both the stakeholders
and respondents;
- To apply diverse methods of triangulation, including among others the use of
several data sources, several data collection methods, several evaluators and several
theoretical perspectives;
- To consult colleagues both formal and informal about all what has been seen, heard,
experienced and interpreted. This could be helpful to explore alternative
explanations;
- To ask for an audit by competent third parties outside the school with the aim to
verify the reliability of the evaluation and results, and;
- To pilot the instruments in a part in order to verify if the instruments yield the
information that has been sought.
Utility and use
Elements of use concern the analysis and interpretation of the information received as
well as the utilisation of the information for improving their functioning (Visscher,
2002). The utility refers to practical aspects like the choice of evaluation procedures, i.e.
the effort and time required, the compatibility with the skill level of respondents as well
as the stakeholders’ conceptions with regard the degree to which the evaluation
purposes are and will be achieved by means of the chosen procedures. The latter refers
256
to the responsiveness of the information collected to the needs and interests of the
clients and other stakeholders. It includes both the relevance of the content of the
evaluation (instruments) (i.e. are the concepts measured seen as the essential
characteristics of the quality education, is nothing missing, is the trade-off between
comprehensiveness and selectivity of all possibly relevant topics well considered and
recognised), as well the content and usefulness of the feedback (reports) (i.e. the
clearness and relevance of the feedback as well as the extent to which the results are
recognizable) (Bosker & Hendriks, 1997).
In the evaluation literature three types of utilisation are distinguished:
- Direct or instrumental utilisation: the decision and actions are based on the
information;
- Conceptual utilisation: the evaluative information is not used directly, but influences
the thinking of decision-makers and as such may have an impact on their thinking;
- Convincing (symbolic) utilisation: this concerns using the information selectively to
legitimise someone’s own viewpoint in discussion with others (Rossi, Freeman &
Lipsey, 1999).
The use of evaluative information, like feedback, is not guaranteed, even not the direct,
instrumental utilization. Weiss (1998, in Visscher & Coe, 2002, p. 58) states that “that
evaluation information can be used partially, in fragments intermittently,
inappropriately, or not all”. She stresses that all efforts at dissemination assume
implicitly that the evaluator knows some truth that all practitioners should know, and
that the new knowledge will lead to behavioural change. This assumption has not held
up under many conditions.
According to Gray (2002) direct utilization of feedback is difficult, because the trust in
the data first has to grow within schools, the right interpretation is not always clear, the
data often do not show what has to be done, it is difficult to prepare an action plan, the
utilization of the requires a lot of time and energy, not all schools meet the required
skills, and the external support for schools in interpreting and using the information
often is lacking. Applying performance feedback, problem finding, developing of
remedies and the implementing of remedies all require skills and a level of
organizational functioning that many schools probably do not meet (Visscher & Coe,
2003).
Similarly Scheerens, Glas and Thomas (2003) mention that in school self-evaluation the
setting of evaluation priorities and the collection of the data are likely to get most of the
attention. According to them there is a strong risk that evaluation results will be
underutilized whenever frameworks to interpret the data are missing and
communications between practitioners and evaluators is complicated.
257
Assessment (judgement) of school self-evaluation instruments by Q*Primair
As mentioned above during the past years several evaluation studies of instruments for
quality assurance have been carried out and evaluation framework have been developed.
In this paragraph the evaluation framework of Hofman and Hofman (2003) as well as
the assessment of school self-evaluation instruments based on it (Dijkstra, Van der
Linde & Majoor, 2004), will be described.
The first evaluation framework is based on both the accountability perspective and the
improvement perspective.
Accountability refers to the responsibility of schools to provide information about
school performance and functioning to relevant outside audiences (Scheerens, Glas &
Thomas; 2003). In this perspective the emphasis is on quality assessment and on the
various quality dimensions. The criteria used are united in the Context-Input-Processes-
Output model. Because the ‘processes’ are subdivided into ‘Processes at school level’
and ‘Processes at school department, team, classroom and pupil level’ the authors speak
of the CIPPO model.
Stimulating learning and self-improvement of schools is the main target in the
improvement perspective (Chelimsky & Shadish, 1997). Here the quality assessment is
used as a basis for improvement and development, and the evaluation-feedback-action
sequence functions as central mechanism. The criteria used in the framework are based
on the theories on ‘the school as a learning organization’, the school as a high reliability
organization’ and ‘the school as professional culture’. The criteria are ordered according
to the PDCA-cycle of Deming (Hofman & Hofman, 2003).
The fact that both perspectives are included in the Q5 and Q*Primair evaluation
framework implies that the criteria on the one hand (i.e. the accountability perspective)
should be based on the relevant dimensions to evaluate the quality of the school. On the
other hand (i.e. the improvement perspective) the framework should include ways in
which school improvement and development methods and procedures can be mapped
(Hofman, Dijkstra, Hofman & De Boom, 2004).
The second evaluation framework described is the last version as used in the Q*Primair
assessment of school self-evaluation instruments (Dijkstra, Van der Linde & Majoor,
2004, p. 11-13) and consists of eight aspects (sub-aspects within brackets):
1. General descriptive characteristics of the instrument
(E.g. name of the instrument, publisher, theoretical basis and rationale according
to author(s), characterisation of instrument, format, price, evaluation method(s),
258
respondents, frequency of data-collection, time required, possibilities for support
during implementation and data collection)
2. Reliability of the instrument
(Information on reliability included in the instrument or manual, number of data
sources used, number of types of respondents included, guarantee of anonymity
and abilities to check the data)
3. Validity and standards
(Information on validity (e.g. content validity, construct validity, criterion-
related validity) included in the instrument and manual, type of standards (e.g.
national standards, standards specific for various target groups or areas (e.g.
denomination, school size), involvement of external stakeholders, accreditation
of instrument)
4. Utility quality
(Quality of the material, easy use of software, possibility for integration of other
data already available, possibility for inclusion of other indicators/standards,
time required for data collection, method of data analysis, feedback mechanisms,
necessity of external support, possibilities to collect longitudinal data)
5. Coverage of quality dimensions (Context-Input-Processes- Output)
a. Context (Contacts with supply schools for and schools for further education,
involvement of parents, involvement of the local community/neighbourhood,
involvement of higher administrative levels (e.g. school board, multi-school
management, municipality), involvement in networks, consortium of
mainstream and special schools (WSNS), external experts and agencies)
b. Input (Pupil population, characteristics of the environment, contacts with
other schools, financial and material resources, school building and learning
environment
c. Processes at school level (vision and mission, educational leadership,
success-oriented ethos/high expectations, school management/personnel
policy, professional culture/staff development policy, internal and external
communication, degree of tuning in teaching, school climate, coverage of
curriculum and evaluative potential)
d. Processes at school department, team, classroom and pupil level (Quality of
the teaching-learning process (e.g. continuous learning route, effectiveness),
effective learning time/opportunity to learn, instruction and learning
strategies, testing and monitoring, differentiation and grouping procedures,
didactic approach, pupil care, guidance and counselling, out-of-school
activities, study skills/homework)
e. Output (results with regard to cognitive functioning of pupils, results with
regard to other competencies of pupils, success rates (upward mobility,
259
throughput, downward mobility) and drop-out rates), satisfaction of the
school team, satisfaction of the pupils and satisfaction of the parents)
6. Support and suggestions for improvement and school development
a. Plan – dimension (Description of starting situation, development of
vision/mission, accomplishing of commitment, setting of targets and
priorities, formulating of measurable goals, creation of a functional
communication structure, planning of resources and personnel, planning of
evaluation moments)
b. Do-dimension (Stimulating educational leadership, attention for policy and
strategy, stimulation of a professional culture, attention for professional
development, application of functional communication, implementation of
activity plan, actual implementation, internal and external communication)
c. Check-dimension (Implementation of evaluation, Internal and external
involvement in evaluation, actual monitoring and application of procedures,
analysis of data from evaluation, application of standards, analysis of
satisfaction and improvement, reporting to internal and external
stakeholders)
d. Act/Adapt – dimension (Integration into monitoring system, enlargement of
areas, standardization of resources (personnel and material), standardization
of routines in procedures, implementation of checks in case of failure,
implementation of measures for quality maintenance, attention for
accreditation, guidelines for the start of a new cycle)
7. Function of the instrument (accountability, improvement or both)
a. Just accountability
b. Mainly focused on accountability with possibilities for school improvement
c. Mainly focused on school improvement with possibilities for accountability
d. Just school improvement
8. Final judgement
Q*Primair asked two independent assessors to judge the most used self-evaluation
instruments. These instruments are WMK PO (Werken Met Kwaliteitskaarten, Primair
Onderwijs), KMPO (KwaliteitsMeter Primair Onderwijs), KIK (Kwaliteit in Kaart),
ZEBO (ZelfEvaluatie in het BasisOnderwijs), Kansschool, Q-INIS-NL (International
Network of Innovative School Systems), LTP/OTP/PTP (Leerling-, Ouder-Personeels-
TevredenheidPeiling), WerKlim, DIS (DiagnoseInstrument Schoolontwikkeling), SON
Kwaliteitscyclus, KWIN (KwaliteitsINdruk).
Table 6.3 shows the final judgement of each instrument per aspect.
260
Table 6.3: Judgement of the 11 most used school self-evaluation instruments for
primary education (Hofman, Dijkstra, Hofman & De Boom, 2004, p. 15)
Instrument Reliability Validity and standards
Utility Coverage of quality dimensions (CIPPO)
Suggestions for school development (PDCA-cycle)
C I P P O P D C A
1. WMK PO 3 1 4 5 4 5 5 4 3 3 3 2
2. KMPO 3 1 4 4 4 5 5 5 4 3 3 2
3. KIK 3 1 4 5 5 5 5 5 5 3 4 3
4. ZEBO 4 2 4 1 1 4 4 2 2 1 1 1
5. Kansschool 3 2 4 3 4 5 4 1 5 4 4 3
6. INIS PO 2 1 3 2 1 5 4 5 1 1 1 1
7. LTP 3 2 4 2 4 4 3 4 3 3 1 1
8. WerKlim 4 1 4 1 1 3 1 1 1 1 1 1
9. DIS 4 2 4 2 2 4 4 1 3 4 2 1
10. SON 2 1 4 5 4 5 5 3 3 2 2 2
11. KWIN 3 1 2 3 3 4 5 5 2 1 1 1
Colophon: 1 = weak, 2 = moderate, 3 = reasonable, 4 = good, 5 = excellent.
From Table 6.3 it appears that the reliability of the instruments appears to be good or
reasonable for most of the instruments. The validity and the standard setting on the
other hand are moderate for 4 of the instruments and weak for the remaining 7
instruments. The coverage of quality dimensions differs per instrument. Five of the
instruments cover more or less all the quality dimensions; other instruments are mainly
focused on process indicators. With regard to school development and improvement,
four instruments can be used during all the phases of the school development cycle;
other instruments are more appropriate for one or two phases or are more specific
designed for quality evaluation (in the evaluation framework of Q*Primair this is taken
up as the accountability function).
Satisfaction of school leaders with the instrument they themselves use most frequently
In the monitor ‘Quality Care’ of Q*Primair (Hofman, Dijkstra, Hofman & De Boom,
2004) school leaders were asked for their opinion about the school self-evaluation
instrument they themselves use most frequently. In the Tables 6.4 and 6.5 the results are
presented for respectively the suitability of the instruments on aspects of quality care
and the suitability on aspects of school development. In Table 6.4 the satisfaction of the
261
school leader with the reliability, validity and utility of the instrument they themselves
use most frequently is presented.
The instruments in the tables refer to six out of the ten most frequently used
instruments, and all include process measures5. Moreover, these instruments are also
judged by Q*Primair (see Table 6.3), and as will be shown later, as well by the
Education Inspectorate.
The instruments measuring just outcome measures (pupil monitoring systems (CITO,
DLE), the CITO Primary School Leavers Attainment Test and the CITO entry test for
pupils of grade 5, 6 and 7) are left out.
Suitability for quality care
From Table 6.4 it appears that school leaders on average rate six school self-evaluation
instruments as good or excellent on suitability for quality care. Only DIS received a
moderate judgment. The instruments that were rated highest (KIK, SON and WMK) are
the instruments that in earlier inventory of Q*Primair (Duurkoop et al., 2002) were
characterized as instruments that portray the school ‘fully and thoroughly’.
Furthermore, as could be expected because instruments primarily measuring outputs are
left out, for all instruments the scores on the aspects ‘The school has a sound judgement
in the quality of its performance’ and ‘Gaining insight into the school environment’ are
relatively low.
5 The instruments are WMK-PO (Werken Met Kwaliteitskaarten, Primair Onderwijs), KMPO (KwaliteitsMeter Primair Onderwijs), SON Kwaliteitscyclus, KIK (Kwaliteit in Kaart), DIS (DiagnoseInstrument Schoolontwikkeling) and ZEBO (ZelfEvaluatie in het BasisOnderwijs).
262
Table 6.4: Suitability of six frequently used school self-evaluation instruments on
aspects of quality care (Hofman, Dijkstra, Hofman & De Boom, 2004, p. 44)
Self-evaluation instrument
WMK PO
KMPO
SON
KIK
DIS
ZEBO
Quality care in general 3.2 2.9 3.1 3.2 2.6 3.1 School development in general 3.1 2.8 3.0 3.2 2.6 3.1 Systematic evaluation of the quality of learning, teaching and results 3.0 2.8 3.1 2.9 2.3 2.8
The school has a sound judgement in the quality of its performance
2.5 2.5 2.2 2.8 2.0 2.4
Accounting for the quality of education to stakeholders 2.9 3.0 2.8 3.1 2.2 3.0
Gaining insight into the entrance situation
3.2 3.1 3.0 3.1 2.6 3.0
Working deliberately on improvement of the educational quality
3.2 2.8 3.2 3.2 2.6 3.0
Evaluation of improvement actions 2.9 2.7 2.8 3.1 2.5 2.9
Formulating a plan with targets for improvement 3.1 2.7 3.1 3.0 2.9 2.9
Carrying out of improvement plan 2.9 2.8 2.9 3.1 2.6 2.8
Development of a vision on quality care
3.0 2.7 3.0 3.1 2.3 2.5
Gaining insight into the school environment
2.4 2.5 2.6 2.6 1.9 1.9
Average score 2.9 2.8 2.9 3.0 2.4 2.8
Colophon: 1.0-1.9 = not, 2.0-2.5 = moderate, 2.6-3.0 = good, 3.1-4.0 is excellent
Suitability for school improvement
In Table 6.5 the opinion of the school leaders is shown with regard to the suitability of
the six school self-evaluation instruments for school development and improvement.
Again, school leaders rated five of the six instruments on average as good on suitability
for school improvement, and once more DIS got a moderate score. Further, the
suitability of the instruments was rated high on the areas ‘Guidance and care of pupils’,
‘Satisfaction of the school team’ and ‘School climate’.
263
Table 6.5: Suitability of six frequently used school self-evaluation instruments on
aspects of school improvement (Hofman, Dijkstra, Hofman & De Boom,
2004, p. 45)
Instrument
Aspects:
WMK
PO
KMPO
SON
KIK
DIS
ZEBO
Improvement of guidance and care of
pupils
2.9 2.8 2.8 3.0 2.5 2.9
Improvement of the quality of testing
pupils
2.7 2.7 2.4 2.9 2.1 2.7
Improvement of performance 2.6 2.5 2.5 2.9 2.3 2.7
Improvement of teaching and learning
process
2.9 2.6 2.9 3.1 2.3 3.0
Improvement of the didactic
approaches of teachers
2.8 2.4 2.7 3.0 2.2 3.1
Improvement of satisfaction school
team
2.7 3.0 2.8 3.1 2.4 3.0
Improvement of effective use
available learning time
2.7 2.2 2.7 2.9 1.9 2.7
Improvement of school climate 2.8 2.7 2.8 3.1 2.4 2.9
Average score 2.8 2.6 2.7 3.0 2.3 2.9
Colophon: 1.0-1.9 = not, 2.0-2.5 = moderate, 2.6-3.0 = good, 3.1-4.0 is excellent
School leaders satisfaction with the reliability, validity and utility of the instruments
The satisfaction of schools is inquired on the basis of questions on reliability, validity
and utility. As is shown in Table 6.6, for most aspects school leaders are satisfied with
the reliability, validity and utility of the instrument they most frequently use. School
leaders are less satisfied with the possibilities of the instruments to compare their own
school with other schools, as well as the degree of external support that could be offered
by the publishers of the instruments.
264
Table 6.6: Satisfaction with reliability, validity and utility of six frequently used school
self-evaluation instruments (Hofman, Dijkstra, Hofman & De Boom, 2004,
p. 47)
Instrument
WMK
PO
KMPO SON KIK DIS ZEBO
Reliability and validity
Reliability 3.0 2.9 2.8 3.0 2.6 2.9
Guarantee of anonymity 2.9 2.9 2.8 2.9 2.9 3.1
Comparison of schools 2.3 2.4 1.9 2.3 1.9 2.9
Involvement of teachers 2.8 3.0 2.9 3.0 2.6 3.0
(External) support 2.6 2.5 2.5 3.0 2.4 2.5
Average score 2.7 2.7 2.6 2.8 2.5 2.9
Utility quality
Durability/ quality 3.0 2.9 3.0 2.9 2.5 2.9
Clarity of guidelines 2.9 2.9 3.1 2.9 2.9 3.0
Practical applicability 3.0 2.9 3.2 2.9 2.6 3.0
Quality of report 3.0 3.0 2.8 2.9 2.8 2.6
Factual delivery 3.0 2.8 3.0 2.9 2.5 3.0
Friendliness of use 2.8 3.0 3.0 3.0 2.9 2.9
Methods used 3.1 3.0 3.2 2.8 2.9 3.1
Time investment 2.7 2.8 3.0 2.7 2.6 3.0
Time required for analysis and
reporting
2.8 2.9 2.8 2.9 2.6 3.1
Time required for data collection 2.7 2.8 2.8 2.7 2.6 3.1
Quality of software 2.8 2.9 2.0 2.7 2.1 3.0
Utility of feedback / tips 2.6 2.6 3.0 2.6 2.1 2.6
Pleasure of team 2.6 2.7 2.7 3.0 2.2 2.8
Use of available material 2.3 2.5 2.2 2.7 1.9 2.5
Average score 2.8 2.8 2.8 2.8 2.5 2.9
Colophon: 1.0-1.9 = not, 2.0-2.5 = moderate, 2.6-3.0 = good, 3.1-4.0 is excellent
Assessment (judgement) of school self-evaluation instruments by the Education
Inspectorate
In 2003, the Education Inspectorate evaluated 10 instruments for school self-evaluation
in primary education from the point of view of the usefulness of the results of the school
self-evaluation for Inspection as well as from the potential value of school self-
evaluation instruments for quality care at school level. The aims were:
265
- To examine whether the instrument could deliver information on one or more
indicators in the Inspection Framework, in such a way that the results could be used
to form judgements about school quality;
- To examine if the instrument could deliver information outside of the indicators in
the Inspection Framework, in such a way that it informs the Inspectorate on
additional quality aspects the school wants to profile itself;
- To examine if the instrument, in case of judicious use, in potency could be a good
instrument for quality care according to the requirements of the Inspection
Framework, or contribute to specific phases of quality care (Working group self-
evaluation Primary education, 2002).
The following instruments were evaluated: WMK PO* (Werken Met Kwaliteitskaarten,
Primair Onderwijs), KMPO* (KwaliteitsMeter Primair Onderwijs), KIK* (Kwaliteit in
Kaart), DIS* (DiagnoseInstrument Schoolontwikkeling), ZEBO* (ZelfEvaluatie in het
BasisOnderwijs), IJKI (IJsselgroep Kwaliteitsinstrument), KOM (Kwaliteit op Maat),
KWIN (KwaliteitsINdruk), TKPO (Totale Kwaliteit Primair Onderwijs) and
Kwaliteitsbeleid in het onderwijs begint bij een persoonlijke keuze (KPK). Five of these
instruments (marked with an asterisk) were also judged by both Q*Primair and the
school leaders.
Table 6.7 shows the degree to which the indicator areas of the in the instruments are
covered by the instruments. From the table it appears that coverage of quality aspects in
self-evaluation instruments is highest on the areas on quality care, teaching and learning
process, school climate and special care and lowest for the areas on testing and learning
results. Considering the table form the perspective of the instruments, the instruments
WMK PO, IJKI and KIK cover almost all aspects fully or partly, while in DIS, KOM
and KWIN many aspects are not covered at all.
For each instrument additional areas with respect to the categories of the Inspection
Framework differ per instrument. Most often, school conditions are included like
personnel policy, internal communication, leadership, decision-making and allocation of
resources. Besides, ZEBO includes aspects of the working climate of teachers and
WMK PO and KOM include the area of the religious or ideological basis of the school.
266
Table 6.7: Coverage of Quality Aspects of 10 instruments for school self-evaluation on
Inspection Framework (2003) (Working group self-evaluation Primary
education, 2002)
Coverage of Quality
Aspects in Inspection
Framework (2003) W
MK
PO
KM
PO
KIK
DIS
ZE
BO
IJK
I
KO
M
KW
IN
TK
PO
KP
K
1. Quality care + + + +/- + + + + + +/-
2. Testing + - +/- - +/- + - - - -
3. Subject matter
coverage
+/- + + - - + - +/- - -
4. Time +/- + + - +/- +/- - - - +/-
5. Teaching and
learning process
+/- + + +/- + + +/- +/- - +/-
6. School climate + + +/- +/- +/- +/- + +/- +/- +/-
7. Special care + + + - + + +/- + - +/-
8. Learning results + - +/- - - +/- - - - -
Colophon: + = aspect is covered in instrument, +/- = aspect is partly covered, - = aspect is not
covered
The relation to key characteristics of quality (i.e. coverage of the areas) is just one
aspect for adoption of the results of the self-evaluation by the Inspectorate. Next to this,
the data of the school self-evaluation should be of recent date, the data should provide a
reliable picture of the quality of the school and the quality goals the school is aiming for
should be sufficiently ambitious.
The reliability is examined according to the following rules:
- Test instruments: The school makes use of externally standardized tests according to
the standards of the COTAN (Commission on Test Affairs in the Netherlands);
- Subject matter coverage: An audit or external review should demonstrate that the
school satisfies the legal requirements;
- Time: the instrument should explicitly ask for the legal requirement on spending of
time. Next to this, by means of documents, the school should provide information
about the amount of lessons not given;
- Teaching and learning process: a judgment of at least one of the following sources is
necessary: a judgment by pupils or parents, external review or a judgment system
within the school;
267
- School climate: Depending on the indicator, a judgment of parents, pupils, staff or
an external review is required;
- Guidance and counseling: At least an evaluative report should be available on the
different arrangements of guidance and counseling the school has taken;
- Results: At least the presence of demonstrable results acquired with standardized
instruments is required. With regard to the self-evaluation instrument it will be
examined if the package either includes standardized instruments or provides
relevant analytic results.
For an overall positive judgment of the reliability of the instrument, the reliability of all
quality aspects should be rated as ‘good’.
Finally, if the instruments met the requirements of coverage of quality aspects,
reliability, and the school has sufficiently ambitious goals, the setting of standards is
examined. Here, the requirement is that at least the standards of the Inspectorate are
met.
In general, the Inspectorate judged the reliability of the instruments as not sufficiently
and therefore a priori not suitable for proportional inspection (Working group self-
evaluation Primary education, 2002). However, on two aspects, ‘School climate’ and
‘Guidance and counseling’, some of the instruments proved to be reliable. Next to this,
one instrument got the judgment “reliable” on almost all aspects.
Above all the Inspectorate concluded that, with the exception of ZEBO, all instruments
leave much scope for own choices of the school in the school self-evaluation process,
i.e. with regard to the selection of quality aspects to be included as well as the addition
of other aspects, such as the selection of stakeholder groups involved, the determination
of standards and the assurance and actual application of anonymity. Therefore, the
actual use of the instrument could lead to a lower judgment of the reliability as
determined by the Inspectorate. This implies that the employees of the Inspectorate
should check the way the school self-evaluation instrument is used for each school
separately.
In Table 6.8 the degree is shown in which the available self-evaluation instruments
could potentially contribute to the quality care indicators of the Inspection Framework,
or to specific phases of quality care at the school level.
From Table 6.8, it appears that the instruments potentially could be useful to determine
the starting situation, to formulate targets and to carry out improvement activities. To a
certain extent judicious use of the instruments could also lead to a systematic evaluation
of the quality of the school as well as to systematic quality care. However, for almost all
268
instruments this implies that after the evaluation of the quality adequate measures
should still be invented and also actually be taken to guarantee and improve the quality
(Working group self-evaluation Primary education, 2002).
Table 6.8: Overview if the degree in which, judicious use of the instruments could
contribute to the realization of the quality care indicators of the Inspection
Framework, version 2003 (Working group self-evaluation Primary
education, 2002).
Instrument
WM
K P
O
KM
PO
KIK
DIS
ZE
BO
IJK
I
KO
M
KW
IN
TK
PO
KP
K
Quality aspects
QA 1. The school knows its entrance situation
+ + + +/ +/- +/- +/- +/- + +/-
QA 2. The school has formulated targets
+/- + + +/ +/- + + + +/- +/-
QA 3. The school systematically evaluates the quality of teaching, learning, and learning results
+/- + + - +/- + +/- +/- +/- +/-
QA 4. The school carries out improvement activities
+ + + +/ + + + + +/- +/-
QA 5. The school reports about realised quality to third parties
+/- + +/- - +/- +/- +/- + +/- -
QA 6. The quality care activities of the school could be characterised as systematic.
+/- + + +/ - +/- +/- + +/- +/-
Colophon: + = judicious use of the instrument gives sufficient cause, +/- = judicious use of
instrument gives limited cause, - = judicious use of instrument does not give sufficient cause
269
Sets of criteria that have been used by foreign inspectorates and groups of
inspectorates, i.e. SICI and OFSTED
SICI: The Effective School Self-Evaluation (ESSE) project
Early 2001, the European Commission (EC) approved funding for a proposal made by
The Standing International Conference of Central and General Inspectorates of Europe
(SICI) to carry out the Effective School Self-Evaluation (ESSE) project. The project ran
from April 2001 to March 2003. Fourteen SICI inspectorates participated, the Dutch
Inspectorate among them (SICI, 2003).
The aims of the project were:
To identify key indicators for the evaluation of the quality and effectiveness of self-
evaluation procedures in schools;
To develop a methodology for inspecting school self-evaluation across SICI
members;
To identify key strengths and weaknesses in school self-evaluation in different
countries;
To produce an analysis of how self-evaluation and external evaluation in different
countries relate to each other and what the most effective relationship between them
might be, and;
To produce case studies of effective self-evaluation (SICI, 2003, p. 5).
In the project, a framework for evaluating the effectiveness of school self-evaluation
and a methodology for managing an evaluation visit were developed and tested in visits
to first twenty-eight and in a next phase (case studies) nine schools across fourteen
countries/regions. On the basis of the experiences, the school self-evaluation framework
(the ESSE framework) was improved and common themes emerged in schools with
very effective school self-evaluation were brought together.
The ESSE framework
The ESSE framework has the aim to enable the collection of evidence and the formation
of judgments about the effectiveness of the process of self-evaluation within schools
and of the effectiveness of the external support provided by countries/regions to school
self-evaluation. The ESSE indicators are meant to be used at the level of meta-
evaluation. They are focused on evaluating 1) the quality of the external support to
school self-evaluation and 2) the effectiveness of school self-evaluation.
270
Each indicator can be judged against four levels of evaluation (very good, good, fair and
unsatisfactory). In the ESSE report (SICI, 2003) for each indicator an illustration of
both level 4 and level 2 is presented.
The quality of the external support to school self-evaluation
Within this area one single indicator (ES1) has been developed, consisting of five
themes (see box 2). As the project evolved, it became clear that the external support
provided to promote self-evaluation turned out to be crucial in establishing the best
culture and climate to allow self-evaluation to become effective.
The indicator could be used for a variety of purposes. Within the framework of self-
evaluation it is of help to identify strengths and weaknesses and inform the future
development of quality assurance and school improvement systems, and, in a more
general evaluation context, to evaluate the quality of support to self-evaluation across
countries/regions.
ES1 External support to school self-evaluation
How well does the external support provided by countries/regions contribute to
effective school self-evaluation?
Themes included:
Provision of statistical data for comparison and benchmarking
Provision of a set of quality standards/indicators
Provision of training in self-evaluation methods
- Provision of regular independent external inspection or moderation of self-
evaluation
- Provision of legislation
271
The effectiveness of school self-evaluation
Within this area four indicators have been developed, each indicator consisting of
different areas and themes (see the boxes below). The indicators could be of help in
building up a systematic picture of a range of key aspects of the quality assurance and
improvement arrangements in schools as well as the extent of their impact in practice.
KEY AREA 1: Vision and strategy
1.1 To what extent is there evidence of strong leadership that has generated a clear and shared
sense of common aims and purpose amongst staff in the school and a commitment to
continuous improvement?
1.2 Does the school have a clear and appropriate strategy for systematically reviewing the
quality of its work and putting in place the necessary improvements?
Quality indicator Themes included:
1.1 Aims and values Appropriateness and clarity of aims and values
Links with school self-evaluation and improvement
procedures
1.2 Strategy and policy for
self-evaluation and
improvement
Strategy for carrying out self-evaluation of key aspects
of provision
Strategy for planning school improvements
Consultation and communication arrangements
KEY AREA 2: Evaluation and improvement of key processes
2.1 How well does the school evaluate and improve the effectiveness of its staff and the way in
which they are managed and deployed?
2.2 How well does the school evaluate and improve its use of finance and physical resources?
Quality indicator Themes included:
2.1 Staff/human resources Staff review procedures
Staff development
Links between staff development and review and school
self-evaluation and planning
2.2 Finance and physical
resources Arrangements for monitoring and reviewing use of the
school’s budget and resources
Targeted use of finance in support of school
improvement
272
KEY AREA 3: Evaluation and improvement of key processes
3.1 How effectively does the school communicate a clear view of effective practice and
standards to be achieved in implementing its key processes?
3.2 How well does the school evaluate the effectiveness of its key processes as they are
delivered in practice?
3.3 How well does the school turn the results of self-evaluation into planned programmes of
action that have a clear impact on improving the processes concerned?
Quality indicator Themes included:
3.1 Policies, guidelines and
standards Policies and guidelines for key processes and
procedures
Use of policies and guidelines in the process of self-
evaluation
3.2 Planning and
implementation of self-
evaluation activities
Self-evaluation activities
Evaluation of stakeholder views
Rigor and reliability of self-evaluation activities
Use of an external perspective to support the self-
evaluation process
3.3 Planning and
implementation of action
for improvement
Selection of priorities for development
Planning of action
Implementation of planned improvements
KEY AREA 4: Evaluation and impact on key outcomes
4.1 How well does the school monitor and evaluate the quality of it key outcomes?
4.2 To what extent is there evidence that self-evaluation has led to measurable improvements?
Quality indicator Themes included:
4.1 Evaluation and
improvement of key
outcomes
Involvement of staff in evaluation of key outcomes
Use of statistical information and benchmarking data
Direct monitoring of pupil progress and achievement
Systematic use of a set of quality indicators or quality
standards
Use of feedback on the views of key stakeholders
4.2 Impact of self-evaluation
on improving key
outcomes
Progress in improving key statistical indicators of
outcomes targeted for improvement through self-
evaluation
Progress in improving stakeholder satisfaction related to
issues targeted for improvement through self-evaluation
Progress in improving aspects identified for audit trails
273
Themes emerged in schools with very effective self-evaluation
In the ESSE project nine case studies were undertaken, based on examples of good
practice in self-evaluation as identified through the twenty-eight school visits in the
second phase in the project. Common themes that emerged in schools with a very
effective self-evaluation, included:
Strong leadership;
Shared aims that are clearly understood by all key stakeholders;
Engagement of key stakeholders in self-evaluation and improvement activities;
Well set out and clearly communicated policies and guidelines;
Self-evaluation activities that focused on learning, teaching and improving
outcomes;
Strong staff commitment to self-evaluation;
Monitoring and evaluation processes that were systematic, rigorous and robust;
Well-planned action to develop and improve provision;
A beneficial balance between external support and challenge from local authorities
and/or national Inspectorates and internal quality assurance, and;
A generally strong infrastructure of national or local support for self-evaluation as a
process (SICI, 2003, p. 125).
School self-evaluation and school inspection in England: The office for standards in
education (OFSTED)
Also in England, school inspection puts a relative great emphasis on school self-
evaluation (Inspectorate of Education, 2002a and b; Ofsted, 2003, Scheerens, Glas &
Thomas, 2003). Like in the Netherlands, validating school self-evaluation is seen as a
major part of the inspection process and extent of inspections are proportional to need,
the most successful schools having the longest intervals between inspection.
In England, up to September 2005, a brief self-evaluation report should be completed by
the school before inspection and is used to focus inspection effort where it matters most.
From September 2005 onwards important changes will be introduced to the way schools
are inspected. The new arrangements will involve more frequent inspections that
involve less bureaucracy, place a greater emphasis on the school’s self-evaluation
procedures and outcomes and reduce the notice given before the inspection starts
(Ofsted, 2003; 2005).
274
The main features for inspecting schools in England from September 2005 are:
Short, focused inspections that take no more than two days in a school and
concentrate on close interaction with senior managers in the school, taking self-
evaluation evidence as the starting point;
Short notice of inspections to avoid schools carrying out unnecessary pre-inspection
preparation often associated with an inspection. Short notice should help inspectors
to see schools as they really are;
Small inspection teams with many inspections led by one of Her Majesty’s
Inspectors;
Three years as the usual period between inspections, though occurring more
frequently for schools causing concern;
Strong emphasis on school improvement through the use of the own self-evaluation
of the school, including regular input from pupils, parents and other stakeholders, as
the starting point for inspection and for the school’s internal planning and
development. To facilitate this, schools are strongly encouraged to update their self-
evaluation form on an annual basis;
A common set of characteristics to inspection in schools and other post-16 provision
of education from early childhood to the age of 19, and;
There will be two categories of schools causing concern, those deemed to require
special measures and those requiring a notice to improve.
Also in the new system, schools will have a range of internal processes for monitoring
their own performance and evaluating the effectiveness of their work in raising
achievement. Moreover the school self-evaluation should lead to a periodic updating of
the school improvement plan, which maps the priorities for action and sets out
programmes for implementing them (Ofsted, March 2005, the Framework for inspecting
schools from September 2005).
“Inspection will take account of or contributes to these processes in several ways:
A summary of the findings of self-evaluation undertaken by the school is
recorded in the self-evaluation form (SEF) and is used by the lead inspector to
focus inspection where it matters most. The school’s summary of its self-
evaluation is used as the basis for discussion between the lead inspector and
the senior team and, where possible, governors of the school;
The quality and use made of school self-evaluation are a good indication of the
caliber of management. Evidence of how effectively schools undertake self-
evaluation and the use they make of it helps inspectors to evaluate the quality
of management in the school and the capacity of the school to improve;
275
In order to promote the use of self-evaluation, the self-evaluation form (SEF),
which is completed by the school and updated at least annually, is designed to
match the common inspection schedule for schools and other post-16 provision
used by inspectors” (Ofsted, March 2005, the Framework for inspecting
schools from September 2005, p. 5 and 6).
The Self-evaluation form (SEF) described below is the one for primary and middle
schools and is derived from http://forms.ofsted.gov.uk/edc2003/sefpri.pdf).
The form consists of three parts:
Part A for recording the outcomes of the self-evaluation
Part B asking for factual information about the school
Part C asking for information about compliance with statutory requirements.
Part A, the evaluative section, is laid out in sections that correspond to the headings of
the evaluation schedule in the framework for the inspection of schools. At the end of
each section it is asked to grade the aspects on a four-point scale (i.e. grade 1:
outstanding, grade 2: good, grade 3: satisfactory and grade 4: inadequate). The sections
are (aspects within brackets):
Characteristics of the school;
Views of learners, parents/carers and other stakeholders;
Learners’ achievements and standards in their work;
Learners’ personal development and well-being;
The quality of provision
(Quality of teaching and learning, quality of the curriculum and other activities; quality
of care, guidance and support for learners);
Leadership and management;
Overall effectiveness and efficiency
(Overall effectiveness, capacity to make further improvements, improvement since the
last inspection, effectiveness and efficiency in the Foundation Stage).
General standards for evaluation
The Joint Committee on Standards for Educational Evaluation has published three sets
of standards for educational evaluations that are widely recognized. The Personnel
Evaluation Standards was published in 1988, The Program Evaluation Standards (2nd
edition) was published in 1994, and The Student Evaluations Standards was published
in 2003 (Joint Committee on Standards for Educational Evaluation, 1988, 1994, 2003;
see also http://www.wmich.edu/evalctr/jc). The Program Evaluation Standards
276
encompass 30 standards and are subdivided in four main categories, i.e. utility (to
ensure that the evaluation will serve the information needs of intended users), feasibility
(to ensure that an evaluation will be realistic, prudent, diplomatic and frugal), propriety
(to ensure that an evaluation will be conducted legally, ethically, and with due regard
for the welfare of those involved in the evaluation, as well those affected by its results)
and accuracy (the evaluation will reveal and convey technically adequate information
about the features that determine worth or merit of the program being evaluated).
A summary of the standards is shown in Table 6.9.
Table 6.9: The program evaluation standards: Summary of the standards (Derived from
http://www.wmich.edu/evalctr/jc/)
THE PROGRAM EVALUATION STANDARDS
Summary of the Standards
Utility Standards
U1 Stakeholder Identification Persons involved in or affected by the evaluation should be
identified, so that their needs can be addressed.
U2 Evaluator Credibility The persons conducting the evaluation should be both trustworthy
and competent to perform the evaluation, so that the evaluation findings achieve maximum
credibility and acceptance.
U3 Information Scope and Selection Information collected should be broadly selected to
address pertinent questions about the program and be responsive to the needs and interests of
clients and other specified stakeholders
U4 Values Identification The perspectives, procedures, and rationale used to interpret the
findings should be carefully described, so that the bases for value judgments are clear.
U5 Report Clarity Evaluation reports should clearly describe the program being evaluated,
including its context, and the purposes, procedures, and findings of the evaluation, so that
essential information is provided and easily understood.
U6 Report Timeliness and Dissemination Significant interim findings and evaluation reports
should be disseminated to intended users, so that they can be used in a timely fashion.
U7 Evaluation Impact Evaluations should be planned, conducted, and reported in ways that
encourage follow-through by stakeholders, so that the likelihood that the evaluation will be
used is increased.
Feasibility Standards
F1 Practical Procedures The evaluation procedures should be practical, to keep disruption to
a minimum while needed information is obtained.
F2 Political Viability The evaluation should be planned and conducted with anticipation of
the different positions of various interest groups, so that their cooperation may be obtained,
and so that possible attempts by any of these groups to curtail evaluation operations or to bias
277
or misapply the results can be averted or counteracted.
F3 Cost Effectiveness The evaluation should be efficient and produce information of
sufficient value, so that the resources expended can be justified
Propriety Standards
P1 Service Orientation Evaluations should be designed to assist organizations to address and
effectively serve the needs of the full range of targeted participants.
P2 Formal Agreements Obligations of the formal parties to an evaluation (what is to be done,
how, by whom, when) should be agreed to in writing, so that these parties are obligated to
adhere to all conditions of the agreement or formally to renegotiate it
P3 Rights of Human Subjects Evaluations should be designed and conducted to respect and
protect the rights and welfare of human subjects.
P4 Human Interactions Evaluators should respect human dignity and worth in their
interactions with other persons associated with an evaluation, so that participants are not
threatened or harmed.
P5 Complete and Fair Assessment The evaluation should be complete and fair in its
examination and recording of strengths and weaknesses of the program being evaluated, so that
strengths can be built upon and problem areas addressed.
P6 Disclosure of Findings The formal parties to an evaluation should ensure that the full set
of evaluation findings along with pertinent limitations are made accessible to the persons
affected by the evaluation and any others with expressed legal rights to receive the results.
P7 Conflict of Interest Conflict of interest should be dealt with openly and honestly, so that it
does not compromise the evaluation processes and results.
P8 Fiscal Responsibility The evaluator's allocation and expenditure of resources should
reflect sound accountability procedures and otherwise be prudent and ethically responsible, so
that expenditures are accounted for and appropriate
Accuracy Standards
A1 Program Documentation The program being evaluated should be described and
documented clearly and accurately, so that the program is clearly identified.
A2 Context Analysis The context in which the program exists should be examined in enough
detail, so that its likely influences on the program can be identified.
A3 Described Purposes and Procedures The purposes and procedures of the evaluation
should be monitored and described in enough detail, so that they can be identified and
assessed.
A4 Defensible Information Sources The sources of information used in a program evaluation
should be described in enough detail, so that the adequacy of the information can be assessed.
A5 Valid Information The information-gathering procedures should be chosen or developed
and then implemented so that they will assure that the interpretation arrived at is valid for the
intended use.
A6 Reliable Information The information-gathering procedures should be chosen or
developed and then implemented so that they will assure that the information obtained is
278
sufficiently reliable for the intended use.
A7 Systematic Information The information collected, processed, and reported in an
evaluation should be systematically reviewed, and any errors found should be corrected.
A8 Analysis of Quantitative Information Quantitative information in an evaluation should
be appropriately and systematically analyzed so that evaluation questions are effectively
answered.
A9 Analysis of Qualitative Information Qualitative information in an evaluation should be
appropriately and systematically analyzed so that evaluation questions are effectively
answered.
A10 Justified Conclusions The conclusions reached in an evaluation should be explicitly
justified, so that stakeholders can assess them.
A11 Impartial Reporting Reporting procedures should guard against distortion caused by
personal feelings and biases of any party to the evaluation, so that evaluation reports fairly
reflect the evaluation findings.
A12 Meta evaluation The evaluation itself should be formatively and summatively evaluated
against these and other pertinent standards, so that its conduct is appropriately guided and, on
completion, stakeholders can closely examine its strengths and weaknesses.
Cremers-van Wees et al. (1996a and b) applied the program evaluation standards on
self-evaluation instruments. According to these authors, an instrument is useful if it is
easy in use (i.e. it takes relatively a little amount time to administer the procedures, the
instrument can be used without extensive training or support and the procedures for the
data collection, analysis, reporting and interpretation of the data are clear). Moreover,
the goals of the evaluation and the usefulness and accessibility should be to the
information needs and interests of those involved (stakeholders). An instrument is
feasible if it is cost effectiveness, contains practical procedures (i.e. it is easy to use and
disruption should be kept to a minimum), and concentrates and anticipates on the
different positions of the various interest groups. It is important that the cooperation of
these groups may be obtained so that possible attempts of any of these groups to curtail
evaluation operations or to bias or misapply the results can be averted or counteracted.
The carefulness of the instrument will be determined by the ethical acceptability (i.e. the
evaluation instrument should be designed and conducted to respect and protect the
rights and welfare of human subjects). Finally, the accuracy refers to the reliability,
validity, and theoretical basis of the instrument as well as the clarity of purposes and
procedures.
279
A closer look at the application of quality care and experiences with the proportionality
of inspection
In this paragraph the attention will be focused on the results of monitoring and quasi-
experimental studies on school self-evaluation. Next to this, the experiences with the
new supervision system will be reported on.
Progress of quality care in schools for secondary education
By order of Q5, Van Beekveldt and Terpstra monitored the progress of quality care in
schools for secondary education three times, i.e. at the beginning of the project, after the
first two project years and at the end of the project (Haaijer & Van Velzen, 2002; Van
Beekveld & Terpstra, 2005). Each time, a short questionnaire was sent to all secondary
schools, and each time around 32 per cent of the schools returned the questionnaire.
Without elaborating all aspects included in the monitoring, below the results are
presented for both the state of the art with regard to the development of quality care and
the satisfaction of schools with conditions for quality care.
Table 6.10 presents the progress in the development of quality care in secondary
schools, according to the perceptions of the schools.
Table 6.10: Development of quality care in secondary schools (%) (Van Beekveld &
Terpstra, 2005, p. 3, 4, 5)
2004 2001 2000
Availability of long-term implementation plan for
quality care
Yes
No
50.0
50.0
23.8
75.7
13.3
86.7
Characterization of quality care system:
Systematically
Partly systematically, partly occasionally
Occasionally
20.6
60.6
11.1
15.5
55.3
28.2
8.2
60.6
31.2
Range of quality care areas:
A single area
Several areas
All areas (integral quality care)
10.6
65.6
23.9
31.1
56.3
9.7
35.8
23.9
10.0
Quality care is connected to the school policy
Yes
No
87.2
12.8
73.7
26.2
66.7
31.9
280
2004 2001 2000
The school has formulated measurable targets
Yes
No
42.8
56.7
29.6
67.0
18.9
80.6
The school has set standards in advance
Never or seldom
Sometimes
Most times or always
23.9
48.9
26.7
36.9
46.1
15.0
60.3
33.3
5.8
From Table 6.10, it appears that in 2004, 50 per cent of the schools pointed out to have
a long-term implementation plan for quality care. By then, 20.6 per cent of the schools
indicated that they carry out quality care in a systematic way; the majority however
characterized their quality system as partly systematically and partly occasionally. In
2004, the quality care system covered several areas in 65.6 per cent of the schools;
integral quality care arose on 23.9 per cent of the schools.
Forty-three per cent of the schools had formulated measurable targets. However, for the
time being standards are not often set preliminary to the measurements, just half of the
schools (48.9 per cent) sometimes set standards beforehand.
With regard to the conditions for quality care, each time, Q5 asked schools for their
perception on e.g. sound school self-evaluation instruments, support, internal facilities
and the knowledge of school management about quality care principles. The results are
shown in Table 6.11.
From this table it could be noticed, that quality care has become more integrated in
school policy. In 2004, in the majority of schools it is seen as an explicit responsibility
of school management and in half of the schools, school management felt sufficiently
competent to carry out quality care. However, the percentage of schools that is satisfied
with the available instruments decreased between 2001 and 2004. Where in 2001 63.6
per cent of the schools were satisfied with the available self-evaluation, in 2004 this was
just 45 per cent.
281
Table 6.11: Conditions for quality care (%) (Van Beekveld & Terpstra, 2005, p. 10,
11, 12)
2004 2001 2000
Availability of sufficient self-evaluation instruments
Yes
No
45.0
52.8
63.6
23.3
32.3
45.1
Need for (external) support in development of quality
care6
Yes
No
57.8
38.9
-
-
-
-
Quality care is an explicit task of management
Task hours are available for quality care
Earmarked in-service training budget is available for
quality care
85.6
51.6
23.3
77.2
20.0
16.0
76.0
17.9
9.3
School management has sufficient knowledge with
regard to quality care
Yes
No
55.0
44.4
37.4
59.2
29.7
68.1
Progress of quality care in schools for primary education
By the end of 2003, the state of affairs with regard to quality care in primary schools
and the progress that has been made during the implementation of the Q*Primair project
was measured for the first time. Via the Internet 3500 primary schools were asked to
participate, of which 1211 school leaders (partly) completed the questionnaire. The
results are presented in two reports: the Q*Primair report (Hofman, Hofman, Dijkstra &
De Boom, 2004) and the report of the first part of the so-called BOPO study on quality
care in primary education (Hofman, Dijkstra, Hofman & De Boom, 2004).
In Table 6.12 the results with regard to the state of affairs on quality care are presented
for the aims that Q*Primair stated at the beginning of the project. In the table the aims
are arranged in order of decreasing degree of occurrence in primary schools. The total
number of valid responses for this question was 935.
6 Not measured in 2000 and 2001
282
Table 6.12: State of affairs with regard to quality care in primary education
Our school: No
%
No, but
implementing
%
Yes
%
Average
.. Uses a system of instruments and procedures for
monitoring progress of the pupils and takes care of
guidance an counseling in case of problems
0.4 10.3 89.3 2.9
.. Has planned improvement actions for the current
school year (based on the evaluation of the quality)
4.3 12.1 83.6 2.8
.. Systematically monitors the quality of their
education and takes action to maintain and improve
the quality
0.5 28.8 70.7 2.7
.. Involves parents in the quality care 3.2 21.2 75.6 2.7
.. Reports about the realized quality of education to
parents, school board and personnel
3.3 21.1 75.6 2.7
.. Systematically and regularly evaluates the quality
of its teaching and learning and performance
0.5 32.9 66.5 2.7
.. Has formulated in the school plan targets for the
quality of teaching and learning as well as its
performance in terms of learning results
6.0 24.1 69.9 2.6
.. Has planned an improvement route for the long
term (Based on a quality evaluation)
5.5 28.2 66.4 2.6
.. Has formulated their quality care policy in such a
way that it is clear how the school satisfies the legal
requirements
5.6 34.4 60.0 2.5
.. Has formulated their quality care policy in such a
way that it is clear how the school satisfies the
quality goals as set and aimed for by the school
itself
5.6 38.7 55.7 2.5
.. Asks for pupils’ opinions 22.7 40.9 36.4 2.1
.. Has put the results of the self-evaluation at least
once to independent third parties
40.2 19.1 40.7 2.0
Baseline quality care system 2001 (Oberon, 2002)
Q*Primair (N = 935)
2.6
From the results Hofman, Hofman, Dijkstra and De Boom (2004) conclude that since
the start of the project, according to the perception of the school leaders, 20 per cent of
the schools has made a development in the direction of a more systematic quality care
system. Next to this, when considering the aims of Q*Primair for 2006, it becomes clear
that two of the aims already have been met. These concern 1) the schools uses a system
283
of instruments and procedures for monitoring progress of the pupils and takes care of
guidance and counseling in case of problems (the aim here was 80 per cent), and 2) the
school has put the results of the self-evaluation at least once to independent third parties
(the aim was 30 per cent). However, in which capacity the latter has been done by the
40 per cent of schools and which role e.g. the stimulating supervision of the Inspectorate
therein seemed to play, is unknown (Hofman, Dijkstra, Hofman & De Boom, 2004).
A first glance on the experiences of schools with the system of proportional supervision
Anticipating on the evaluation of the Law of the Supervision of Education (WOT) in
2007, the Minister of Education, Culture and Sciences asked the SCO-Kohnstamm
Institute to provide a first insight into the experiences of schools for primary, secondary
and secondary vocational education as well as of schools for special education with the
new working method of the Inspectorate.
In the research five questions were at the center:
1. How do schools experience the new supervision method of the Inspectorate (i.e.
the principle of proportionality as well as the new inspection frameworks) since
the implementation of the Law on the Supervision of Education?
2. Does the working method of the Inspectorate leave schools sufficient scope for
making their own choices and for taking their own responsibility?
3. How is the balance between the self-evaluation of the school and the quality
judgment of the Inspectorate? Is the inspection stimulating and does it lead to
improvement?
4. Does the Inspectorate sufficiently take into consideration the specific
circumstances and environment of the school?
5. How do schools experience the publishing of inspection reports (Emmelot,
Karsten, Ledoux & Vermeulen, 2004)?
To answer these questions firstly two group conversations were organized, one with
experts in the area of school improvement and quality care and another with
representatives of umbrella organizations, school boards and unions. At the same time,
telephonic interviews took place with organizations involved in quality care or the
development of instruments for quality care as well as with representatives from
schools.
Next, in a second phase, eleven panel conversations with school heads and above school
level managers were organized. Finally, a questionnaire was sent to all primary and
secondary schools that had experienced a periodic quality inspection (PKO). In the
survey 310 primary schools and 55 schools for secondary education took part. Due to
the low number of periodic quality inspections already carried out in special education
284
and in secondary vocational education, it was decided not to include these types of
schools in the survey.
Findings from the group conversations and telephonic interviews
Generally, the academics and representatives of the school boards showed considerable
hesitation with regard to the stimulating role of the Inspectorate and the feasibility of the
principle of proportionality. An important reason for this is the still limited knowledge
about the characteristics of a sound system of inspection as well as about the necessary
instruments and interventions. Moreover, some academics were of the opinion that the
situation of meta-evaluation and proportional inspection could never be reached.
Schools for primary and perhaps also secondary education still are not ready for self-
evaluation. Also, as long as the Inspectorate judges the education processes, schools
will not feel stimulated to take their responsibility and “is the report from the
Inspectorate mainly seen as a pleasant and free advice for quality improvement”
(Emmelot, Karsten, Ledoux & Vermeulen, 2004, p. 104).
The telephonic interviews with experts confirm the findings. The impression of the
experts was that the implementation of a quality care system usually happens on the
initiative of the school board or above school management. Schools make use of the
Inspectorater report, however, in most cases just for a one-off analysis of strong and
weak points when drawing up the school plan. Next to this, the experts were of the
opinion that the close correspondence between most school self-evaluation instruments
and the framework of the Inspectorate does not foster the premises of the Law on the
Supervision of Education. Therefore a more autonomous disposition of the schools is
needed. On the contrary, some other experts just argue for the development of a format
for school self-evaluation. At this moment, inspectors and schools often conceive self-
evaluation in different ways, and these conceptions do not always meet the
requirements.
According to some experts the intensification of supervision has led to an improvement
of the quality. The indicators of the Inspection framework are well specified; the way to
reach them is more diffuse (Emmelot, Karsten, Ledoux & Vermeulen, 2004).
285
Findings from the surveys and the panel consultations of school heads and above school
management
From the surveys it was again confirmed that schools strongly differ with regard to the
implementation of quality care. Although almost all schools started the process, many of
them are still in the phase of occasional and limited quality care. And, as appearing
from the surveys (i.e. the school self reports), primary schools seem to have made more
progress with the implementation of quality care than secondary schools. Where in
primary education 18.8 per cent of the schools reported that they have an integral
quality care system, in secondary education only 10.8 per cent of the schools reported
this. At secondary level, 33.3 per cent of the schools reported that quality care is
incidental and partial, in primary education 20.6 per cent of the schools reported this.
In special education, school self-evaluation and quality are not very much developed.
Therefore the experiences with proportional inspection are very limited yet. Next to this,
in special education, schools find the standardized working method of the Inspectorate
more difficult, especially because they feel that the Inspectorate does not sufficiently
take into account the pupil composition of the school (Emmelot, Karsten, Ledoux &
Vermeulen, 2004).
Primary and secondary schools reported that they appreciate the Inspection report for
quality improvement somewhat, but not to a large extent. They perceive the report as
most useful when reflecting on their own quality care and when updating the school
plan. Secondary schools more often make use of the Inspection report than primary
schools.
In the panel discussions with primary school heads and above-school management it
was stressed that the Inspectorate should be lenient in holding schools accountable for
insufficient quality. Many schools find themselves more in the beginning phases of
quality care and still need time for choosing a method or quality care system as well as
for implementing it properly. Moreover, like the experts, schools are also of the opinion
that the Inspection framework should not dominate the self-evaluation too much.
All groups involved in the SCO-Kohnstamm study (schools, experts and representatives
of umbrella organizations, school boards and unions) endorsed the principle of
proportionality. Two aspects of proportionality were distinguished, i.e. 1)
proportionality according to the quality of the school, and 2) proportionality depending
on the degree of implementation of quality care.
With regard to the first aspect of proportionality, at this moment most stakeholders find
it too early for an evaluation. Since the Law on the Supervision of schools went into
effect, most schools received just one periodic quality inspection (PKO), and additional
286
inspections (NO) and inspections into the quality improvement of the school (OKV)
hardly have been taken place up to now.
With regard to the latter aspect, the experience of panel members and schools is that
proportionality, to the extent that it depends on the degree of quality care, still not gets
properly off the ground. According to the schools, the Inspectorate uses the first
periodic quality inspections (PKO) above all as a zero measurement, starting from their
own standardized working method.
The authors of the report conclude that the balance between internal quality care and
external supervision has not been sufficiently materialized, so far. The basic idea of the
Law on the Supervision of education (WOT), i.e. including both the judgment and
stimulation of the quality of schools, is endorsed by the schools. Experts on the contrary
are more skeptical and posed the question if stimulating the quality of schools should be
part of the package of tasks of the Inspectorate (Emmelot, Karsten, Ledoux &
Vermeulen, 2004).
A closer look at accuracy standards applied to school self-evaluation, the issue of
objectivity
Systematic evaluation is full of inherent tensions and contradictions. At the same time
evaluations are expected to be “objective” and “engaging”, they are about “facts” and
“judgments”, they often have an “external” element and are expected to be used
“internally”. Actors are sometimes expected to play the rather passive role of
information providers, but then they are also expected to be active partners in the
shaping of evaluation questions and the interpretations of results.
In the case of school self-evaluation these tensions are partially avoided because it
seems to be evident that school self-evaluation takes a clear position on what side of
these pairs of opposites it stands:
it is internal rather than external;
it is improvement rather than accountability oriented
it uses methods that are transparent to the practitioners
all actors in the school are expected to play an active rather than a passive role.
However, in actual practice school self-evaluation has also external, objectifying and
judgmental aspects (Scheerens, Glas & Thomas, 2003).
Three principles appear to be of particular importance:
287
- to resolve the issue of objectivity by providing schools with well-developed
instruments that meet standards of reliability and validity;
- to assure that school self-evaluation happens in a context that is not experienced as
threatening;
- to systematically address issues of implementation of procedures and the use of
results
On all three aspects the situation in the Netherlands appears to be favorable, when
making a comparison with other countries. There is a good starting position with respect
to the availability of standardized instruments, e.g. the CITO pupil monitoring system
for primary schools. Next, schools in the Netherlands are not penalized for evaluation
results, and third, the Netherlands has an elaborate education support structure that
could assist schools with implementation and use of evaluation procedures.
Conclusion
The idea of proportional supervision is a creative application of decentralization and
“subsidiarity” in education, meaning that all that can possibly and reasonably be carried
out at a lower administrative level, should not be carried out at a lower level (Scheerens,
1997). But perhaps this formulation also indicates the Achilles heel of this strategy. Can
school self evaluation really meet the demands of external evaluation? What the chapter
has shown is that a lot of developmental effort has been and is being invested in
enabling, facilitating and stimulating school self evaluation and school quality care in
the Netherlands. A relatively large set of school self evaluation systems and instruments
has been developed, and organizations like Q5 and Q Primair have invested in the
development of criteria to asses these instruments and procedures and in experimental
good practice applications. It is interesting to note that the quality standards developed
and applied by Q5 and Q Primair differ from those applied in studies of the Education
Inspectorate in the sense of the importance given to what Stufflebeam et al. (1971)
subsume under ‘accuracy standards’. The Inspectorate is more consistent in seeing
criteria such as reliability and validity as essential for the value of school self evaluation
and is generally more severe and critical in assessing the available set of instruments.
What is being practiced under headings such as school self-evaluation and quality care
is a mixture of elements that have a somewhat different orientation and strongly
different traditions. The main orientations are: a) a scientific evaluation orientation,
emphasizing accuracy, reliability and validity of procedures; b) an administrative and
client oriented branch of quality management systems, and finally, c) a more qualitative,
opinion based school improvement orientation. From the user perspective of schools
each have their strong and weak points:
288
- schools may appreciate the objectivity and rigor of the use of structured instruments
that have been tested for reliability and validity, but most of all find themselves at
odds with the standardization implications, strictness of procedures, and the targeted
scope that is easily condemned as narrow and reductionist;
- quality care systems may have the fashionable appeal of the fast moving business
world and the more real asset of being responsive to clients; yet these systems
usually fail to address the primary process of teaching and learning, and they may
paint caricatures in their efforts to “proceduralize” educational processes;
- most suspect among these three, at least from an evaluation perspective, is the
school improvement perspective that forgoes rigor in establishing fact, and may
favour superficial consensus about change over sound diagnosis; in fact there is a
perpetual dilemma in educational analysis and action that is caused by a discrepancy
between the urge to adapt and change, and knowledge that is at best partial and
contested.
In our view the scientific perspective should prevail in the idea of proportional
supervision. According to proportional supervision, a part of external evaluation is in
fact delegated to schools. This could only work if schools applied the same rigor in
method that would be required of external evaluation.
The evidence presented in this chapter with regards to the actual state of play of school
self-evaluation indicates that only a minority of schools starts to meet the current
standards of the Inspection. Given the need for rigorous internal evaluation this state of
affairs could be improved by the Inspectorate making a stronger effort to make well-
developed, scientifically founded instruments for school self-evaluation available to
schools. In actual practice this should be done by stimulating instrument development
and facilitating integral systems of school self-evaluation that use state of the art
practice in areas like pupil monitoring systems based on Item Response Models,
computerized school administration systems, and empirically validated process
indicators.
289
Instruments
WMK PO (Werken Met Kwaliteitskaarten, Primair Onderwijs; Bos, 2003)
KMPO (KwaliteitsMeter Primair Onderwijs; Van Beekveldt & Terpstra/AVS, 2001)
KIK (Kwaliteit in Kaart; Edux onderwijsadviseurs, 2001)
ZEBO (ZelfEvaluatie in het BasisOnderwijs; Hendriks & Bosker, 2003)
Kansschool (Beek & Dijkhuizen, 2002)
Q-INIS-NL (International Network of Innovative School Systems, Bertelsmann
Foundation, 2003)
LTP/OTP/PTP (Leerling-, Ouder-, PersoneelsTevredenheidPeiling; Dulmers
Organisatieontwikkeling, 1999/2004)
WerKlim (Boersma, Van Tiezen, Huistra & De Vos, 2002)
DIS (DiagnoseInstrument Schoolontwikkeling; Boersma, Osinga & De Vos, 1997)
SON Kwaliteitscyclus (SON/Ijsselgroep, 2000)
KWIN (KwaliteitsINdruk; Boogaarts, Scholten & De Jong, 1998/2001)
290
PART IV
THE CHANGING CONTEXT OF SUPERVISION
293
Chapter 7
Rethinking societal functions of education; implications for
outcome and process indicators
Introduction
Modernization of the society has implications for education. Trends like globalization,
the knowledge society, shifts in the dominant orientation of the economy, immigration
and demographic developments evoke a rethinking of the basic functions of educations.
Ultimately this kind of thinking is bound to address the core goals and means of
education, and as such the basic subject matter areas in quality evaluation systems, as
the one of the Dutch Inspectorate. In this chapter a relatively eclectic approach will be
chosen in illustrating some of the current modes of thinking about re-orientation of the
societal function of education. The three classic functions distinguished by sociologists
will be the starting point, and these will be complimented with related economic
theories and concepts. The concept of competency is a focalization point in rethinking
the connectivity between schooling and societal functioning. Some of these new
competencies will be illustrated and placed among a continuum of education outcome
indicators. The supposition that new end terms may also have implications for studying
processes and defining process indicators is illustrated by the case of informal learning
for citizenship. Some tentative considerations for the current Inspection frameworks are
formulated at the end of the chapter.
Sociological categories with respect to the core societal functions of education
Classic functions that have been distinguished by sociologists are the qualification, the
selection and the legitimizing function of education (cf. Fend, 1981; Van Kemenade,
1981).
Qualification refers to the function of the education system that is related to preparing
youngsters to participate in society; both in terms of labour and in terms of a broader
spectrum of social participation. The most important practical education issue of the
qualification function is the relevance and connectivity of what is learned at school and
the demands of society at large, the labour market in particular.
294
The selection function takes the perspective of the education system as a regulator of
entrance to different positions, power and prestige in society. According to this
perspective the degree of differentiation of an educational system should roughly be
matched to important differentiations in society. The most important educational issue
connected with the selection function is the issue of equity in education in its broadest
sense; particularly with respect to the dependence of educational achievement on ethnic
and socio economic background of the students.
The legitimizing function sees education as an initiation in the central normative
frameworks of society. Education has a role in putting across norms and values that are
of great importance for individuals to function well in a given society. Legitimizing is
also described in terms of integration in society. The biggest current challenge in this
domain is the multi-cultural society, with its confrontation of different value systems,
and aspirations to define a common core of “citizenship”.
These sociological concepts, as a matter of course, look at education from a societal
perspective, while pedagogical theories emphasize personal development, or “Bildung”
(cf. Baethge et al, 2003). These authors see “individual regulation capacity” as a central
goal dimension of education, which refers to ”the competency of the individual to give
shape to his or her own biography, in relationship with the environment” .
These authors (Baethge et al., ibid) also distinguish current socio-economic mega
trends, which put the central functions of education in a particular context. These mega-
trends are:
- tertiarization, by which they refer to the growing importance of the service sector in
western economies;
- aging of the population;
- internationalization and globalization;
- individualization and a corresponding change in values with respect to autonomy,
self-manifestation and implications for integration;
- informatization and knowledge intensity.
In subsequent sections the implications of modern phenomena related to the core
societal functions of education for evaluation frameworks like those in use with the
Dutch Inspectorate of Education will be discussed. This discussion will focus on
implications for outcome indicators that should reflect possibly new emphases in
educational goals; and implications for process indicators, as reflecting a re-orientation
in the means and methods of education. But first, a more extended look will be taken at
economic theories that are related to the qualification and selection function.
295
The economic perspective: human capital and related theories
The economist approach to education generally analyses the rational choice behaviour
of actors in education. Particularly the question about the so-called utility of a person
who follows education is studied. This means that all the costs a person has to make to
follow a particular education program are counted on the one hand. On the other hand
the benefits in terms of increased earnings (i.e. the difference in earnings between
having obtained a particular qualification and not having obtained that qualification) are
calculated. This is the basis of the best-known economic theory on education: human
capital theory. Other theories also consider the utility of employers to engage persons
with certain qualification levels. The research that economic theories have generated
gets particularly interesting for the topic at hand, when the question is raised about the
competencies that make qualified individuals more attractive to companies than
unqualified individuals. In their turn these competencies could be taken as key targets
for schools and training institutes to maximise the external relevance of their programs.
Human capital theory treats education as a trade-off between costs and benefits. “In this
theory, individuals incur costs, including tuition and foregone earnings while in school,
to acquire skills that increase their productivity and result in subsequent wages higher
than they otherwise would have earned. A long tradition of research has focussed on the
magnitude of the return to educational investment, showing that human capital
investments tend to pay off at least as well as investment in physical capital” (Levy &
Murnane, 2001, p. 154). One of the research supported characteristics that was found in
explaining this general result was that employees with more formal education have
better ability to deal with unanticipated events (like non-routine problems at work or
forced job changes), than do workers with little formal education (ibid, p. 154).
Signalling theory concentrates more on the perspective of the employers. Employers use
the information on formal levels of schooling as a “signal” that potential employees
possess certain skills that are valued for the work in question. Recent research, the
International Adult Literacy Study has shown that it is not just the formal qualification
level that counts, but some competencies, in this case literacy, has an effect over and
above the formal level of schooling.
Principal agent theory presents an economic perspective on the utility functions of
employees on the one hand and managers on the other. The theory assumes that the
agent (the employee) has a utility function that may differ from what is valued by the
employer. For example, a certain degree of leisure, that employees could permit
themselves may be pleasant for the former but not so much for the principal (employer).
296
In theory the employer would need to closely monitor the behaviour of the employee to
minimize such non task-related priorities or activities. Since monitoring may be difficult
and costly, relying on employees having acquired certain basic skills and attitudes that
give some guarantee of reliability, education functions as a substitute for monitoring
and supervision. It is interesting to note that the competencies in question are not just
technical skills but also more attitudinal characteristics like perseverance and honesty.
Implications for educational goals
Levy and Murnane (2001) summarize research findings that are the result of various
kinds of economic research that are driven by these theories and have generally been
targeted to the question what competencies, related to formal schooling, could explain
the attractiveness of qualified individuals to employers. They mention five “key
competencies”:
1) Basic reading and mathematical skills are important in determining long-run labour
market outcomes, including the ability to adjust to changing circumstances.
2) The ability to communicate effectively, both orally and in writing, is important in
determining long-run labour market outcomes.
3) In modern firms it is increasingly important to possess the ability to work
productively in groups.
4) The latter condition also emphasises elements of “emotional intelligence” including
the ability to relate well to other people.
5) Familiarity with computers is of growing importance in the labour market.
They present their findings as tentative, however. The fact that employers are ready to
pay wage premiums to employees with higher levels of formal schooling may be
explained by the hypothesis that these employees have better communication and
teamwork skills and/or are more apt to adaptation and continuous learning, but the
evidence is inconclusive. These authors also point at a number of institutional
arrangements that blur the picture of a straightforward relationship between companies
hiring and promotion policies and acquired key-competencies. They mention “the
organization of work, the links between firms and educational institutions, and the
nation’s distribution of income as modified by the welfare state” (ibid, p. 167).
Identifying key competencies as a general approach to closing the gap between
schooling and the demands of society
297
The term “competencies” is used, as the magic word to express what is needed to
function effectively in a given social and cultural context. Clearly this question goes
beyond that of education responding to the demands of the labour market. Similarly the
concept of “competency” is meant to have a broader meaning than the transmission of
knowledge and skills in institutionalised education settings, like schools. “Key
competencies are considered as structured around meeting demands of a high degree of
complexity and are comprised of cognitive as well as motivational, ethical, social and
behavioural components” (Weinert, 2001, cited by Rychen, 2001, p. 11). It is
recognized that acquiring these competencies also happens in other settings than formal
schooling, e.g. in the home situation of children and in the work-place (learning on the
job).
As the economic approach already illustrates, competencies are identified by analyzing
valued “real life” situations, such as the functioning of highly schooled individuals in
work situations. Taking a broader outlook this valued situation could be described as the
“effective” or “competent functioning” of an individual in a given society, i.e. a given
cultural, social and linguistic context (Haste, 2001). This author identifies five
“competency domains” which she feels are essential in the immediate future and
possibly universally relevant. These are: “technological competency; the ability to deal
with ambiguity and diversity, the ability to find and sustain community links; the
management of motivation, emotion and desire; and finally, the sense of agency and
responsibility”. (Haste, 2001).
Apart from these “comprehensive” characteristics (cognitive, motivational, social and
ethical components), Rychen (2001, p. 11, 12) mentions two other core elements in
current, multi-disciplinary based, definitions of competencies:
- “Key competencies are seen as transversal or generic in the sense that they enable
individuals or groups to participate effectively in all relevant social fields with their
specific power and social relations, challenges and capital at stake.
- Key competencies call for a higher order of mental complexity involving an active
and reflective approach, which includes the capacity to distance oneself from one’s
own socialising process and even one’s own values”.
Outcome indicators
When approached from the context of formal schooling, the idea of competencies can
be placed on a continuum of types of educational outcomes, already introduced in a
preceding section, that runs from specifically content oriented to “content free”
personality traits. Discrete positions on this continuum are presented in Figure 7.1.
298
- outcomes as measured by tests included in textbooks
- outcomes as measured by implemented school curricula (teacher developed)
- outcomes as measured by tests based on the intended national curriculum
- outcomes as measured by international tests covering the common core of a range
of national curricula, e.g. TIMSS
- “literacy” tests, aimed at measuring basic skills in reading, mathematical and
scientific reasoning, e.g. PISA
- competencies as multi-facetted dispositions of individuals, including cognitive,
motivational and possibly other components
- personality traits, like internally or externally determined locus of control,
independence, general intelligence
Figure 7.1: A continuum of educational outcomes, running from highly content bound
to personality dependent
It remains to be seen whether all “ambitions” inherent in the idea of key-competencies
can be met. A core question is whether the multiple component idea of competencies
survives empirical measurement. For example in a current dissertation study an
instrument was developed to measure “career developmental competencies”, involving
cognitive, behavioural and motivational components. In analyzing the instrument it
appeared that these three components were relatively independent, and thus could
hardly be seen as forming one integrated construct (Kuijpers, 2003).
Perhaps the heuristic value of the term will be that attention is drawn to other than
purely cognitive educational outcomes, whether these can be shown as an integral part
of a particular competency or as independent dimensions.
The practical meaning of the competency issue is that the answer to an effective linking
of education to societal functioning should partially be sought in the teaching of more
general skills and in paying more attention to motivational and attitudinal aspects. Also
the idea of meta-skills, such as general problem solving skills and “learning to learn”
could stimulate a more reflective and self-steering attitude of future citizens. In this way
“competencies” are considered as useful tools in serving the “employability” of future
citizens.
Overview of the most important categories of outcome indicators
299
In the table below, Table 7.1, an overview is given of the different categories of
outcome indicators that could be used to monitor quality aspects related to the
productivity, the effectiveness and equity of education.
Table 7.1: Overview of educational outcome indicators
Main categories of outcome indicators
Sub-categories Technical issues
Output indicators
Achievement measures - subject matter based - literacy (reading, mathematical,
scientific) - competencies (e.g. learning to
learn)
- Value-added effect measures;
growth curves - Assessment methodology
(ranging from multiple choice tests to authentic assessment)
- Criterion versus norm-referenced testing
Outcome/ attainment indicators
Attainment measures - graduation rates - proportion of students
graduated without delay - drop-out rates - class repetition rates
Controlling for selection oriented school policies
Impact indicators
Social participation rates - (for each attainment level) % of
employed at a certain job level - % of unemployed - (for lower school levels) %
enrolled in follow-up education - degree of social participation
(social capital) - adult literacy rates - average income, for each
attainment level; earning differentials
- skill shortages and surplus
Availability of national educational and labour market statistics Appropriate measures of social capital and adult literacy
Implications for process indicators: the case of education for citizenship
A rationale to identify process indicators
Educational goals, and their operational definitions in terms of behavioural objectives,
and testable criteria and standards, have always served as bridges between education
and the societal functions that education should serve. The concept of “competency” has
a central place in current re-thinking of the connection between the world of schooling
300
and societal demands. From the above discussion about key competencies it appeared
that the competency trend implies several emphases:
- more emphasis on somewhat general cognitive skills, for example in terms of the
diverse “literacies” (reading literacy, mathematical literacy etc.) emphasized in the
OECD PISA project, next to the mastery of subject matter based knowledge;
- emphasis on a broader scala of personality dimensions than just the cognitive one;
social skills, motivation to learn and even “emotional intelligence”;
- new contents and skills; for example knowledge about the civic society and
computer skills;
- a heightened attention for moral issues and different value systems
When it comes to process indicators and assessing the quality of educational processes,
the interesting question is, to what extent these “new goals” also require “new means
and methods” of education. A large portion of this report uses the school and
instructional effectiveness literature to look for a scientific foundation of process
indicators. The basic reasoning behind this approach is that processes of schooling, at
the school organizational and the classroom level, should be analyzed for their
instrumental potential with respect to the desired outcomes of schooling. In other words,
a process indicator is relevant to the extent that it reveals something about the “what
works” question. As will become clear in subsequent chapters there is a considerable
body of empirical research that has addressed this question with respect to basic subject
matter oriented educational objectives, particularly for mathematics, science and
language/reading; and a number of effectiveness enhancing factors have been identified.
These factors are mostly associated with the formal curriculum and with overt, direct
teaching approaches, although there are some climate factors as well. The question is
whether some of the newly identified competencies are also covered by the knowledge
base on school and instructional effectiveness. For some of the aspects mentioned in the
above, particularly the cognitive skills and new cognitive content areas, this is likely to
be the case. For others a “second way” in which the school functions as a learning
context is being considered:
“What seems to be more important, though, is the second way in which schools
contribute to students’ willingness to participate in a civil society: by opening the
classroom to a vital exchange of differing opinions, encouraging students to engage in
politically controversial discussions, creating a culture of mutual respect and
appreciation, and by giving students the chance to participate in classroom and school
decisions and to work cooperatively (Reinhardt & Tillmann, 2002; Henkenborg, 1995,
1997). As has been demonstrated in a number of empirical studies, these factors predict
several aspects of active citizenship (see Buhl, 2003; Diedrich, 2005; Hahn, 1999;
Oesterreich, 2003; Torney-Purta, 2002; Watermann, 2003). Informal learning for
301
citizenship will be presented as an exemplary case that illustrates this possibly new area
of process indicators.
Education for citizenship
In a context of increased complexity, with changing patterns of work, family life and
community engagement, the multi-cultural society, and internationalization, social
cohesion is increasingly seen as a concern for societies. Social cohesion refers to the
“quality of the trust and responsibility relationships existing in a society, both among its
members and between them and their institutions” (EU, 2004, p. 6). This concept is
closely related to social capital, which “refers to the norms and networks facilitating co-
operation either within or between groups” (OECD, 2001, p. 12). The concept of
citizenship refers to “The individual members’ commitment to the well-being of fellow
members of the society and their commitment to the functioning of the institutions of
society” (EU, 2004, p. 6)
International co-operation among European countries and a growing heterogeneity of
the (school) populations of most European countries have led to an increased interest in
education for citizenship. This increased interest is based on a concern for a common set
of norms and values and for an active participation of all in living up to these norms.
The core question is concerned with the role that schools can play in developing
citizenship, not just by means of formal, but particularly also by means of informal
learning.
Goals and methods in education for citizenship
The goals of education for citizenship have been described as having a cognitive
dimension, for example with respect to knowledge about democratic institutions, a
pragmatic dimension, in the sense of taking action and gaining experience, and an
affective dimension, in terms of an attachment to the societies and communities to
which one belongs. Social and communication competencies are considered of central
importance (cf. EU, 2004, Torney-Purta et al., 2001).
When it comes to methods in educating for citizenship at school, two broad categories
can be distinguished:
- approaches that see the school context as a micro-cosmos to exercise “school
citizenship” as a bridge to societal citizenship and state citizenship; this will be
indicated as the “context embedded approach”;
- approaches in which school citizenship is approached in the sense of specific goal
directed teaching and learning activities; this will be indicated as the “explicit
teaching approach”
Informal learning at school
302
In order to define informal learning at school, it is relevant to consider the following
definitions of formal, informal and non formal education:
Formal education: the hierarchically structured, chronologically graded 'education
system', running from primary school through the university and including, in addition
to general academic studies, a variety of specialised programmes and institutions for
full-time technical and professional training.
Informal education: the truly lifelong process whereby every individual acquires
attitudes, values, skills and knowledge from daily experience and the educative
influences and resources in his or her environment - from family and neighbors, from
work and play, from the market place, the library and the mass media.
Non-formal education: any organized educational activity outside the established formal
system - whether operating separately or as an important feature of some broader
activity - that is intended to serve identifiable learning clienteles and learning objectives
(cf. Coombs & Ahmed, 1974; Fordham, 1993).
Dimensions of the micro-cosmos of school life as a context of informal learning for
citizenship
According to the embedded perspective on citizenship oriented education the values and
norms of school life provide an exercise ground for important dimensions of civic
behaviour that also exist in the society at large. Three “media” of the values and norms
of school life are distinguished, the institutional rules and norms of the school as an
organization, the leadership style of the school head and the school culture. Related
concepts are the “hidden curriculum” and “school ethos”. All these components can be
seen as shaping the school’s identity.
McMeekin (2003), applies the concept of institutions at the micro level of
organizational functioning, when he writes about “the concept of institutions inside
school organizations” (ibid p. 3). Components of this concept are: formal rules, informal
rules, enforcement mechanisms, clarity of objectives and the “institutional climate”.
School culture can be defined as “the basic assumptions, norms and values, and cultural
artifacts that are shared by school members, and which influence their functioning at
school” (Maslowski, 1997, p. 5). Deal and Petterson (1999) refer to the “school’s own
informal rules, norms and expectations”.
Ethos is defined as the “feeling that results from the school culture” (Solvason, 2005).
School leadership is important as the way in which the school head embodies and
enforces school norms and values, particularly those that refer to discipline, respect for
others, co-operation, and autocratic vs. democratic decision-making.
Earlier research has shown that by experiencing a culture where the exploration and
expressing of opinions are supported by teachers (e.g., controversial discussions, an
303
open classroom climate), students form a positive attitude towards active citizenship and
democratic values (Diedrich, 2005, in press).
Learning by doing, explicit teaching and reflective teaching
Reflective teaching could be defined as an approach in which teachers refer to instances
of behavior by students, teachers and school heads that is relevant for the way the norms
and values of school life are lived up to, either in a positive or negative sense. In such a
reflective approach teachers could stimulate learning by experience by means of making
explicit relevant events.
The key issue is the question whether it can be demonstrated that students learn from
such instances like the school culture, institutional norms, and leadership emphasis (see
the list of areas in the table below). In the table below a tentative list of aspects of the
school’s identity is related to outcome dimensions of citizenship. School identity is seen
as the union of formal institutional rules, the school culture and the leadership style of
the school head. As such identity is seen as a learning context for citizenship:
School identity Outcome dimensions of citizenship Regulation of decision making at school Regulation of the voice of students at school Open classroom climate
Democracy and its defining characteristics
Visibility of the way decisions are made at school Instances of student participation in decision making on school matters Subjective experience of teachers and students about the school being hierarchical/autocratic or participative in decision making Whether the school head is perceived as autocratic or democratic by teachers and students Perceived opportunities for student participation in decision-making
Institutions and practices in democracy Political interests
Clarity of disciplinary rules at school Degree of strictness and consistency with which disciplinary rules are applied Whether students’ rights are formally stated The way praise and blame are used by the school head and teachers The consistency between staff and between staff and school head in maintaining discipline and enforcing rights and rules. The degree to which rights and duties are maintained in an equitable way for students
Citizenship, rights and duties Self concept of one’s own political competencies Engagement in political activities Seeking political information
304
with different backgrounds The way possible clashes between students’ cultural backgrounds and school rules are dealt with Tension in the school staff Cohesion in the school staff Students’ feeling of efficacy in discourse Students’ perception of violence Are there any symbols of national identity visible in the school (e.g. the raising of the flag, celebration of a king’s/queen’s birthday? How are special days of cultural minorities dealt with at school? How is the composition of the teaching staff in terms of cultural majority and minorities? Is assuring equal participation of parents from all cultural backgrounds a school goal? Is the management style of the school head supportive irrespective of cultural background of teachers?
National identity
Is the way students with different cultural backgrounds interact used as a basis for teaching/ pedagogical activities? Are staff members from cultural minorities well integrated in the school team? Are controversial aspects of a heterogeneous school population openly discussed in classes? Is background information on cultural identities provided in those instances?
International relationships
The degree to which the school staff support the mission and major goals of the school. The degree to which the school head is focussed on human relations and cohesion among staff. Heterogeneity and homogeneity of the school staff and the student population. The degree to which contacts between students from different cultural backgrounds are actively supported. Aspects where cultural diversity, next to common values, among students is explicitly supported. Students’ perception of the integration of different groups
Social cohesion and diversity Integration of foreigners
The case of informal learning for citizenship illustrate possibly new functions of
education and their possible implications for studying and evaluating school processes.
305
This is a relatively new area of educational research, without an established knowledge
base. Nevertheless the hypothetical function of school institutional norms, the school
culture and dominant leadership style could be compared to the Inspection Standards.
Conclusion
Re-orientation with respect to the qualification and integration function of education
emphasize new kinds of school outcomes, indicated as competencies or even key-
competencies. These competencies have a more general cognitive orientation as
compared to subject matter based objectives, and, in addition to the cognitive domain,
also address moral and social development. In the current framework the Inspectorate
concentrates on outcomes and learning results in basic subject matter areas like
language, reading, mathematics and arithmetic. To the extent that views on these more
general competencies are supported the inspectorate might reconsider the range of
outcomes to be included in the Inspection Framework. Measurement issues should be
taken into consideration, as the measurement of competencies has specific technical
difficulties (cf. Straetmans & Sanders, 2001).
As the case of informal learning for citizenship indicates, these new competencies also
draw the attention to specific process dimensions; like explicit institutional norms of
good conduct at school, and more implicit normative regularities embedded in the
school culture. This realization might stimulate a re-thinking of the current indicators in
the Inspection Framework with respect to the school climate.
306
307
Chapter 8
Priorities in the Inspection Framework and the discussion
about educational governance
Introduction
The pillars of “new public management”, seen from a steering perspective, are
increased autonomy on input and process dimensions, and control on outcomes.
Likewise systemic reform in education largely consists of two domains: deregulation
and decentralization on the one hand and establishing evaluation mechanisms on the
other. The concept of “functional decentralization” encompasses these two core
domains of systemic reform; as it explicitly considers patterns in which freedom on
process dimensions go together with strict supervision on performance. In this chapter
core concepts in the governance discussion, such as decentralization and accountability
will be explained. Next, the improvement potential of performance based reforms is
discussed; in which the question of the innovative potential of evaluation, supervision
and assessment mechanisms is a key question. Educational policy in the Netherlands,
for the last decade has closely followed the principals of new public management, with
a particular emphasis on school autonomy. At the same time the role of supervision is
being stressed. This happens both in the sense of “horizontal supervision”, which is
targeted at local stakeholders, and in the sense of “vertical supervision”, associated
with external evaluation and accountability (Koers, VO, 2004). Placing the Inspection
Framework against the background of the principals of new public management, and
considering empirical evidence on results of applications, allows for informed
conjecture about its effectiveness enhancing potential. A crucial question in a context of
increased school autonomy is the place of school process indicators in vertical
supervision. The chapter will be concluded by considering different scenarios for the
place of school process indicators in the Inspection Framework.
Systemic reform
Terms like “systemic reform” and “restructuring” are used to refer to changes in the
institutional and organizational infrastructure of a country’s education system.
New institutional economics, cf. North (1990) emphasizes the importance of
institutions; where institutions are “the rules of the game” or basic legislative
arrangements. “Marriage” and property rights, are frequently mentioned examples of
institutions. In developing countries quality assurance or quality improvement might
308
start with an analysis of the institutionalization of basic arrangements in the education
system, like: fixed teacher salaries; responsibilities of head teachers, official working
time, time tables etc.
Organizational infrastructure might have to be scrutinized as well. The “organizational
capacity” of the country’s Ministry of Education, for example, might be analysed and
found to be in need of improvement (Orbach, 1998).
Questions about organizational capacity of an educational system first of all regard the
issue of whether core functions have an “organizational home” in the system. For
example, initiating a national assessment is the more of a heavy task when there exists
no organization that has specialized in the development of educational achievement
tests in the country. The same applies when external supervision of schools is
considered at a fairly large scale and the country has no educational inspectorate.
Further criteria in determining the organizational capacity concern the well-functioning
of organizations in terms of effective leadership, ability to mobilize financial, material
and human resources and appropriate work practices (ibid).
The division of decision-making authority across the administrative levels of the
educational system has both institutional and organizational significance.
In Western countries “restructuring” and systemic reform are usually focused at
decentralization of decision-making authority and creating arrangements for
accountability. Sometimes these two major dimensions are combined in certain patterns
or arrangements that gear decentralization to accountability arrangements. So called
“Performance-based approaches to large-scale reform” (Leithwood et al., 2000) form a
case in point.
Decentralization and functional decentralization
The issue of decentralization acquires considerable more nuance and practicality when
it is being recognized that it is possible to decentralize in particular domains of decision-
making, while doing rather the opposite in other domains. This notion comes close to
what others have called “functional decentralization” (Bray, 1994). The concept of
functional decentralization will be explained in more detail by referring to an instrument
and data collection procedure that has been developed in the context of OECD.
The OECD-INES procedure to measure “locus of decision making” distinguishes three
facets of the rather crude distinction between centralisation and decentralisation:
- the tier or administrative level where a decision is taken; this dimension was referred to
as the locus of decision-making;
309
- the amount of discretion, or the degree of autonomy of decision-making at a particular
administrative level; this facet was called the mode of decision-making;
- the particular element of educational administration a decision belonged to; this facet
was referred to as the domain of decision-making.
These three facets can be related to existing categorisations in the relevant literature,
although the use of central concepts is by no means consistent among authors and
publications. Our three-dimensional conceptualisation is compared to the terminology as
clarified by Bray (1994, p. 819) in an analysis of alternative meanings of centralisation and
decentralisation.
The distinction between levels confirms to the concept of territorial decentralisation,
defined as "the distribution of powers between different tiers of government". In the
operationalization of this dimension we distinguished four tiers, to be further described in
the section on methods.
Degrees of autonomy in decision making at a particular level are reflected in terms that
refer to an increase in discretion. Again following Bray, deconcentration, delegation and
devolution are modes of decision making in which an increased amount of decision-
making authority resides at a lower level.
"Deconcentration is the process through which a central authority establishes field units,
staffing them with its own officers".
"Delegation implies a stronger degree of decision making at the lower level. However,
powers in a delegated system still basically rest with the central authority, which has
chosen to "lend" them to a local one".
"Devolution is the most extreme form of decentralization. Powers are formally held by
local bodies, which do not need to seek approval for their actions" (ibid, p. 819).
In the operationalization of this continuum of increasing autonomy, these abstract
definitions were avoided and respondents were asked to indicate whether decisions could
be taken within the framework determined by a higher level, in consultation with a higher
level or in full autonomy.
In order to determine elements or domains of educational administration, many
categorization schemes are available in the literature (e.g. James, 1994; Winkler, 1989;
Bacharach et al., 1990; Rideout and Ural, 1993). The common core of these
categorizations are three main areas:
a) an educational domain (goals, methods, curricula, evaluation procedures);
b) an organizational, managerial and administrative domain (including human resource
management, groupings and assignment and foundational regulations);
c) a dimension concerning finance and the way financial resources are applied.
310
In the operational classification that we chose four main categories were used, by splitting
up area b (organisational) into two domains "planning structures" and "human resources",
and including areas a and c.
The distinction between domains of decision-making in educational systems bears some
resemblance to Bray's use of the term "functional decentralisation" as cited from
Rondinelli: "Functional decentralisation refers to the dispersal of control over particular
activities" (Bray, 1994, p. 819). From the examples that he provides, however, it is not
clear whether in functional decentralisation an exhaustive set of domains of educational
decision-making is referred to, as is the purpose of the categorisation schemes cited above.
The common denominator is the recognition that educational systems may be centralised
in some domains of decision-making but not in others. The conclusion is therefore that in a
somewhat liberal use of the term our distinction between domains of educational decision-
making can be considered as a form of functional decentralisation.
To learn more about educational decision-making in OECD countries and to
systematically compare decision-making processes across countries, an instrument was
developed that examined the locus of decision-making in four important domains. As
stated above, these domains were: (1) the organization of instruction; (2) personnel
management; (3) planning and structures; and (4) resource allocation and use. Within
each of these four domains, between seven and fifteen decisions were examined. In the
domain entitled, “organization of instruction,” for example, the instrument focused on
decisions about such matters as textbook selection, grouping of pupils for instruction,
and assessment of pupils’ regular work. In “personnel management,” questions were
asked about hiring and dismissal of teachers and other school staff, duties and
conditions of service, and the setting of salary schedules. In “planning and structures,”
the focus was on creation and abolition of schools and grade levels, the design and
selection of programs of study, course content, and policies regarding credentials.
Finally, in the area of “resource allocation and use,” the instrument focused on decisions
about the allocation of resources for staff and materials, and the use of financial
resources for these purposes.
Each of the questions in the instrument was designed to identify the level at which
decisions are made in the governmental system (the “level” of decision making) and the
way decisions are made (the “mode” of decision making). Six “levels” of decision-
making were set out in the instrument. These include the following: (1) central
government; (2) state governments; (3) provincial/regional authorities or governments;
(4) sub-regional or inter-municipal authorities or governments; (5) local authorities or
governments; and (6) schools. Three “modes” of decision-making were examined in the
311
instrument. Decision could be made by an authority (1) autonomously, (2) within a
framework established by another level within the system, or (3) in consultation with
other levels in the system. Based on the instrument, it was possible to determine how
centralized or decentralized decision was overall, in each of the four domains, and for
individual education decisions.
Accountability; evaluative capacity and incentive based policies
In general terms, accountability refers to holding public institutions and services
responsible for the quality and output of their performance. Glass (1972) states that
accountability involves several loosely connected strands: “disclosure concerning the
product or service being provided; product or performance testing; and redress for poor
performance” (Glass, 1972). The third element implies that accountability is not just a
matter of providing and judging information but at least also “foreshadows” actions by
competent authorities in the sense of sanctions or rewards.
The first element – disclosure- requires that educational units, schools, in particular,
provide information on their service provision, and make themselves “open” for
external inspection and review. The second element distinguished by Glass stipulates
that output and product information should be part of the disclosure on service provision
and functioning. The third element emphasises that testing and review have implications
in the sense of rewards and punishments for organisations. This relates accountability to
incentive-based policies, like merit pay of teachers and output related financing of
schools.
Types of accountability are distinguished on the basis of who, or rather which kind of
unit or stakeholder, is supposed to use the information that is disclosed by schools and
teachers, and also who is supposed to apply the sanctions.
Elmore and Associates (1990) differentiate three “theories” of accountability on the
basis of this question: who uses the information. They distinguish three types:
- technical accountability, in which administrative units are supposed to take
decisions on the basis of scientifically sound achievement measurements;
- the client perspective, in which the clients of education, like the parents of the
pupils, “vote with their feet” in context of free choice of schools;
- the professional perspective; in which feedback on performance is basically used for
professional development. “Accountability is, therefore, to be accomplished by
deconstructing and reconstructing the meaning of schooling, collaborative planning,
and co-operative teaching and learning” (Elmore and Associates, 1990, cited by
MacPherson, 1990, p. 7).
312
In my opinion only the two first forms can be seen as types of accountability. The
“professional perspective” lacks the third element in Glass’ basic definition, namely the
application of rewards and sanctions. Moreover, what Elmore and Associates refer to as
the professional perspective on accountability comes closer to the notion of
“organisational learning” and the teacher as a reflective practitioner, as distinguished in
the classical work of Argyris and Schön (1974). When specifying the professional
perspective further, MacPherson also uses the term “empowerment” of teachers, which
is more closely associated with school-based and school initiated approaches to school
improvement (see the text on school improvement approaches further on).
In order to make the two “real” forms of accountability work, systems should have
evaluative capacity, that is structural and technical facilities to realise the kinds of
empirical disclosure and performance testing that accountability requires. Scheerens
(2000, informal paper for the OECD) mentions the following issues for assessing the
evaluative capacity of education systems:
a) Availability of a legal framework that enforces types of educational evaluation?
* If yes, specify the legal or semi-official (e.g. policy-plans or brochures from the Ministry of Education) requirements for:
- external evaluation - internal evaluation b) Does the system have an inspectorate? Tendencies in inspection. * If yes, describe - how the inspectorate is anchored in the decision-making structure - tendencies in policies towards and within the system of inspection * If not, how is the inspection function executed? c) Is there a national (or above the school) curriculum, national standards, national assessment
programme? In the case of Spain: how is the implementation in the regions? d) From the idea that evaluation is in a sense technology-driven, give a short indication of the
state of development of Educational Evaluation. * Number of Faculties of Educational Science; * Number of Research Institutes; * Number of Researches * Budget e) Existence of specialized departments or institutes for educational testing and evaluation? * Give a brief description of the infra-structure of institutes on evaluation. f) Evaluation culture? * Is there a special policy with respect to internal or external evaluation? * Characterize the general attitude of schools towards external and internal evaluation in
terms of, for instance, cooperative, resistant, initiating, ..
313
g) Which other regulatory mechanisms, as compared to evaluation, are of particular relevance in the country?
e.g. - mechanisms to select teachers and pupils - professionalization of teachers - financial input control h) What is the state of affairs concerning debates on possibly undesired consequences of
evaluation procedures for external or internal evaluation? E.g. political bias, resistance, 'red tape', undesired side-effects of hard competition between schools
With respect to the third defining element, the application of rewards and sanctions,
which can be brought under the heading of incentive based policies, research shows that
there are often considerable limitations. When it comes to technical or administrative
accountability reviewers usually have to conclude that few examples of straightforward
decision-making seem to exist. Cibulka and Derlin (1995), in their review of systems of
school performance reporting, for example, say that “school performance reporting
(SPR) is not considered very important by policy-makers or the general public”. They
conclude that it has not been demonstrated at all that “SPR can become a potent,
effective policy-lever”. Similar reservations have been based on empirical studies of the
use that parents make of school performance information in choosing a school for their
children (Bosker & Scheerens, 1999). Nevertheless there is evidence that accountability
raises actual student achievement. The question might even be raised whether the
evaluation and feedback mechanism should not be considered as the key lever or
“magic bullet” in systemic reform according to the principals of new public
management.
Does the evaluation-feedback mechanism provide the “silver bullet”?
The theoretical idea of retro-active, or “evaluation centered”, planning in “learning
organizations” has a lot of appeal. It coincides well with a pattern of centralization/
decentralization where processes are liberated and input and process control are being
replaced by output control.
In more practical terms one could observe that it has the advantage of rooting reflection
on the functioning of schools with an eye to school improvement, in empirical facts. As
such it differs from pro-active approaches like school development planning that have
often resulted in idealistic sounding documents that lead a quiet existence in various
drawers of cupboards, and have little impact on what actual happens in schools.
In more practical terms, at the school level, an evaluation and monitoring centered
approach to school improvement, appeals to related factors that have been shown of
relevance in educational effectiveness research. Assuming that the evaluative
314
information contains achievement data, a reflective use of this information stimulates an
achievement oriented school policy and climate. Collectively discussing evaluation
results requires collaboration and thus stimulates coordination within schools. Finally,
reflecting on evaluative information can be seen as stimulating a learning attitude
among school staff, including the identification of areas for professional development.
Retroactive planning could also be seen as a cornerstone for educational leadership.
Apart from theoretical considerations, as presented above there is also some empirical
evidence, particularly at the level of national educational systems that supports the
effectiveness enhancing impact of evaluation provisions.
Supporting evidence
Bishop, 1997: “Countries and Canadian provinces with standards and assessment based
reforms outperform other countries at a comparable level of development”
Rand News, July 25, 2000: “The most plausible explanation for the remarkable rate of
math. Gains by North Carolina and Texas is the integrated set of policies involving
standards, assessments and accountability that both states implemented in the late 1980s
and early 1990s
Wöβmann, 2000: “A micro-economic student-level estimation based on data [TIMSS]
from 39 countries reveals that positive effects on student performance stem from
centralized examinations and control mechanisms, school autonomy in personnel and
process decisions …”
François Modoux, Le Temps (French-Swiss daily newspaper, March 26, 2002) -
reporting on PISA-results
“Finland is the home of assessment. Schools self-assess, they evaluate themselves in
relation to other schools and seek to benefit from foreign experience”
Willms & Somers, 2000: (Primer Estudio Internacional Comparativo, UNESCO, 1998
- 13 Latin American countries)
“… testing students increase their scores by an average of 6 points …”
315
non-supportive evidence
Leithwood, Jantzi & Mascall, 2000: (Comparing Performance-Based Reform
programs in Kentucky, California, Chicago, Victoria (Aus) and New Zealand)
“… only Chicago has demonstrated significant increases in student achievement”
Sacks, 1999: (NAEP, 1995-1996 science and math. Proficiency)
“… two thirds of States with elaborate testing programs were below the national
average”.
“… 64% of States with “low” or “moderate” stakes assessment programs had above-
average achievement”
Can the bold conjecture about the evaluation feedback mechanism being the “silver
bullet” in school effectiveness and educational reform be maintained?
A frequent criticism of an evaluation-centered approach is that it usually leads to
reductionism. By focussing on measurable outcomes important educational objectives
that are more difficult to assess could become neglected. Despite of the fact that
evaluation technology limits the area of “unmeasurable” skills and competencies, and
has come up with new approaches like authentic testing and portfolio analysis, this is an
objection that should not be taken lightly.
Secondly, particularly when the stakes are high, evaluation and monitoring is vulnerable
to all kinds of political biases and distortions. This problem can only be partially
countered by striving for conditions in which the main actors feel confident, without
necessarily loosing the critical edge, that significant evaluations still need to have.
Thirdly, evaluation, particularly when “quality accreditation systems” from the business
world enter education, a lot of bureaucracy and red tape could be an undesired side-
effect.
Finally, evaluation and monitoring takes time, which ultimate will be subtracted from
the net-available teaching time.
Fourth and finally the use of evaluations, in an accountability context, but also in a
context of organizational learning, often falls short of its expectations. Cibulka and
Derlin (1995, 493-503) conclude, for example that “school performance reporting is not
considered very important by policy-makers or the general public”. Bosker and
Scheerens (1999) provide evidence that school performance reports do not actually
stimulate quality oriented decisions about school choice by parents. And also in school
self-evaluation contexts, actual use of the evaluation findings is often not actually
occurring (Scheerens & Hendriks, 2002).
316
These findings seem to suggest that evaluation results, in many cases, are not used
according to their “official” function of rational decision support, but may work in a
fuzzier, less direct way, for example by stimulating a result oriented attitude among
professionals in education.
Mixed patterns of decentralization and accountability provisions: the example of
performance-based approaches to large-scale reform
Letihwood, Jantzi, and Mascall (2000) state the following properties of the
“performance-based approach”:
1. A centrally determined, unifying vision, and explicit goals for student performance,
based on the vision.
2. Curriculum frameworks and related materials for use in accomplishing the goals set
for students.
3. Standards for judging the quality of degree of success of all students.
4. Coherent, well integrated policies that reinforce these ambitious standards.
5. Information about the organization’s (especially the students’) performance.
6. A system of finance and governance that devolves to the local school site
responsibility for producing improvements in system and student performance.
7. An agent that receives the information on organizational performance, judges the
extent to which standards have been met, and distributes rewards and sanctions, with
significant consequences to the organization for its success or failure in meeting
specified standards.
Leithwood and his co-authors evaluated the impact of five performance-based reform
projects (in Kentucky, California, New Zealand, Victoria (Australia), and Chicago) and
concluded that only Chicago had demonstrated significant increases in student
achievement. They also found that these achievement gains only occurred during the
last three of the ten years the program was analyzed. According to Fullan (2000, cited
by Hopkins, 2002) during the first six years of the program “the system operated in
decentralized fashion with little functional contact between schools and the district. In
other words too little structure characterized the operation”. During the latter years of
the program “five extra district-level functions were developed”, and these might
explain why students this better during the last years of the program that were
considered in the analyses:
- policy making increasingly supported decentralization
- there was a focus on local capacity building
- a system of rigorous accountability was introduced
317
- innovation was stimulated
- external support networks were established
(Hopkins, 2001, p. 3)
Combined arrangements of functional decentralization and accountability that appear to
be successful are characterized by centralization on the curriculum and assessment
dimension and increased autonomy in areas like personnel management and resource
management at school level. The example of the Chicago reform program points the
attention at two other dimensions that co-determine success:
- pronounced vertical coordination between higher administrative levels and the
school level;
- taking into consideration and stimulating local capacity.
Local capacity building has always been one of the main issues in school improvement.
School improvement being considered as a more school-based approach to educational
change and innovation as compared to systemic reform as discussed in this section.
Linking systemic reform and school improvement
School improvement as a field of academic study is seen as a specific branch of the study
on educational change. As will be pointed out further on, in some applications it is
explicitly related to the school effectiveness knowledge base, and, in still other
applications, its insights are also combined and integrated in perspectives on systemic
reform.
Matthew Miles’ overview of the development of this field of study, in the period between
the mid 1950’s and the mid 1990’s provides the flavour of what the field represents. Miles
discusses ten consecutive school change strategies:
1) Training for group skills; i.e. teaching school people fundamental skills of group
behaviour. Such skills were considered to be of key importance for developing co-
operation, “process analysis” and self-reflection for school teams and also as a subject to
be addressed in classroom teaching.
2) Innovation, diffusion and adoption; which came to the fore during the 60’s. This was the
period of programmed instruction and the idea of “teacher proof” curriculum materials; i.e.
curriculum materials and teaching methods that were specified to such a degree that bad
teaching could not spoil the deliverance to students. And even when it was recognised that
teaching material required an “interpretation” by teachers, for example in adapting to local
conditions, the criterion for successful implementation was coined in terms of the
“fidelity” to the externally determined “script”. Educational change experts like Miles, at
first, thought of temporary systems, like task forces, that were more actively involved in
318
interpretation, and later on developed concepts on more active adaptation and re-creation
of externally induced innovations.
3) Organisational self-renewal. Following developments in industry that came under the
heading of “organisational development”, during the late sixties the school as an
organisation was increasingly seen as the object and the agent of change. The aim was “to
induce organisational self-renewal through tactics of training, process consultation, data
feedback, problem-solving and structural change” (Miles, 1998, p. 48).
4) Knowledge transfer. In this area the more simplistic expectations of the Research
Development and Dissemination (RDD) strategies were challenged, again (as in point 2)
emphasising active reconstruction at school level of the knowledge that was offered from
outside. Capacity building was seen as a necessary prerequisite of good knowledge
transfer.
5) Creation of new schools. In this section Miles writes about the phenomenon that many
new and alternative school projects came into being in the 1960’s an 70’s in the USA. He
concludes that by analysing some of these, he learned that “good new schools can be
created, but that the task is very demanding, more complex than expected, and requires
assistance and political protection” (ibid, 50).
6) Supported implementation. In the late 70’s the “passive” idea of adoption of externally
induced change had been abandoned, and instead, implementation was being seen as a
longer term process of “adaptation”. Adaptation requiring that schools develop coherence
and meaning to external change initiatives. Based on his experiences of assisting some
large projects that recognised this implementation perspective Miles concluded that
“continued assistance” throughout the implementation process was of major importance.
7) Leading and managing local reform. According to Miles during the eighties there were
a lot of local initiates “many of them pushing hard on effective schools and effective
teaching programs”. On the basis of studying successful projects, Miles and his colleagues
identified characteristics of success local reform projects. He summarises as follows: “The
ideas of vision-building, pressure and initiative taking, and assistance have already been
outlined. The idea of empowerment is an extension of the concept of legitimacy for
planning and action, indicating in sharper terms that we found reform success closely
associated with the presence of a cross-role planning team with clear decision power over
change-related matters (such as project budgets, staff development, staffing patterns, and
related time)”. He goes on to say that he found three variables that were tied to successful
local reform. The notion that the planning style was “evolutionary” rather than
“architectural”. He describes evolutionary planning as “a journey in the service of an
evolving, increasingly shared vision”. Secondly he found that successful schools were
good at resourcing and problem coping (the slogan: “problems are our friends”).
8) Training of change agents. As concluded earlier, despite the importance of local
initiative, school change is usually in need of external support and facilitation, according to
319
Miles. Support, particularly with respect to the change process. In this context he identified
two major characteristics of successful support. “Developing trust and rapport. A great
deal seems to depend on a change agents’ ability to develop a strong, supportive,
contractually clear relationship with specific “clients” –groups and individuals involving in
change efforts”. The second characteristic of successful support is organisational diagnosis,
a data driven “understanding” of schools as organisations.
The last two strategies that Miles mentions are in fact about integrating school change in
system wide reform initiatives: “managing systemic reform” and “restructuring schools”.
Murphy (1993) states that “restructuring” in the USA usually has four main strategies for
reorganising education: providing choice and voice for parents, school-based management,
teacher empowerment, and teaching for understanding. The latter refers to a constructivist
orientation to teaching and learning.
Taking in consideration other seminal contributions to the conceptualisation of school
improvement, as those by Fullan and McLaughlin and Skillbeck, published in the
“International Handbook of Educational Change” (1998) edited by Hargreaves,
Lieberman, Fullan and Hopkins, the following can be seen as the key principles of this
orientation to educational change:
a) The school is the focus of educational change. This means that schools should be
analysed as organisations, seen in their local contexts and harbouring the major agents
of change, namely teachers.
b) A strong emphasis on the process dimension of educational change.
c) The importance of school based “implementation” in the sense of active adaptation or
“co-invention” of externally induced changes.
d) A human relations approach to educational change influenced by group dynamics and
the idea of teacher “empowerment”, capacity building and overcoming professional
isolation of teachers. The “counselling” approach of external change facilitators
perhaps also fits in this tradition.
e) An evolutionary “bottom up” view on educational planning and curriculum
development.
Within the scientific community active in this field quite a range of emphases can be
discerned. These vary from authors like Mitchell and Sackney (2000), who provide a post-
modernist view on school improvement and are strongly opposed to accountability and
other “mechanistic” approaches, to authors like Reynolds and Hopkins, who relate school
improvement to the school effectiveness research in emphasising learning and learning
outcomes. Still other contributions (e.g. Leithwood et al., 1999, and Hopkins, 2001)
integrate school improvement approaches and conceptualisations of systemic reform.
320
A major break-through in this field is the work of Slavin, who has proposed a “third” way,
in addition to the school improvement approach and systemic reform. (Slavin, 1996, 1998).
The characteristics of the school improvement approach as described in the above are
summarised by Slavin under the heading of “organisational development models”.
“Perhaps the dominant approach to school-by-school reform is models built around well-
established principles of organisation development, in which school staffs are engaged in
an extended process of formulating a vision, identifying resources (such as external
assistance, professional development, and instructional materials) to help the school
toward its vision, and often locating “critical friends” to help the school evaluate and
continually refine its approaches”. Of this approach Slavin says that it is time consuming
and expensive. Moreover, he claims that it is only effective for schools that already have a
strong capacity for change. “Such schools are ones in which staff is cohesive, excited
about teaching, led by a visionary leader willing to involve the entire staff in decisions, and
broadly aware of research trends and ideas being implemented elsewhere” (p. 1303). Such
schools he describes as “seed” schools. A second category of schools Slavin describes as
schools who would like to do a better job, but do not perceive the need of the capability to
develop new curricula. According to his categorisation these are schools with good
relations among staff and leadership, a positive orientation toward change, and some
degree of stability in the school and its district. Finally, as a third category, he refers to
schools “ in which even the most heroic attempts at reform are doomed to failure. Trying
to implement change in such schools is like trying to build a structure out of sand.” (ibid
1303). Accordingly he refers to these schools as “sand” schools.
School improvement of the organisational development kind (as we have seen the
predominant perspective on school improvement) is considered only feasible in “seed
schools”, which he estimates at 5% of all schools in the USA. Sand schools, also about 5%
of all schools would require fundamental changes before they can support any type of
school change. The overall majority of schools, according to Slavin, are the brick-schools
and they could most efficiently benefit from what he calls comprehensive reform models.
His own “Success for All” program is an example. Comprehensive reform models provide
schools with specific student materials, teachers’ manuals, focused professional
development, and relatively prescribed patterns of staffing, school governance, internal and
external assessment, and other features of the school organisation. It should be marked that
“Success for All” is one of the few improvement projects that has been thoroughly
empirically evaluated and has shown to be successful (Slavin, 1996, Scheerens & Bosker,
1997). Similar successes have been reported by Stringfield and others (1995) presenting
the idea of schools as “high reliability organisations”.
It is interesting to note that Slavin’s conception (and also its actual realisation in “Success
for All”) of Comprehensive Reform Models, seems to have returned full circle to the point
where, according to Miles, the school improvement movement started its human
321
relations/implementation approach in the 1950’s. Namely the discussion on the
applicability of externally developed pre-structured innovation programs and curriculum
material. The fact that there is clear evidence that this approach works is revolutionary, and
puts a question mark behind the efficiency of forty years of educational innovation based
on the less directive, bottom up, social psychological, organisational development
approach to school improvement. The question of efficiency seldom being raised from
within this tradition so eloquently described in Miles ten strategies for school change.
Conclusion: the position of process indicators in the Inspection Framework
To a considerable degree the principles of new public management can be subsumed under
the term “subsidiarity”, which has a long standing tradition in educational policy in the
Netherlands. In the history of education in the Netherlands the term subsidiarity was
used to refer to a specific way in which denominational pressure groups in education
liked to see the relationship between the state and corporations representing interest
groups in the educational field. According to the subsidiarity principle the state should
not interfere in matters that can be dealt with by organized units of professionals. In the
original case these organized units were the denominationally based corporations or
pressure groups of representatives in the education field, their umbrella organizations in
particular. “Subsidiarity” was the term preferred by the Roman-Catholic denomination,
while the Protestants spoke of “sovereignty in one’s own circle”. Leune (1987, 379-380)
points at the corporatistic nature of this kind of concepts. According to the subsidiarity
principle the state only acts subsidiary, that is, it only interferes as a replacement, when
needed. A simple example of subsidiarity is a driving-instructor, who takes over the
steering of a vehicle when the trainee makes a mistake, but in all other cases quietly
watches without interference. Within the context of the European Commission the term
subsidiarity is used to express the principle that what can be accomplished by the
member states should not be done by the central organs of the Union.
A straightforward application of the subsidiarity principle to the question who, at which
level, should evaluate what, could be that units or administrative levels should only take
responsibility for evaluating those areas over which they have direct operational control.
According to this reasoning one could maintain the position that the Inspectorate should
not supervise schools on process indicators, since schools are solely responsible for the
instruction process. Instead the Inspectorate, according to this position, would only
monitor schools on the basis of their outcomes, c.q. the learning results of the students.
In fact, several years ago, this point of view was taken in a report of the Onderwijsraad
(the most important advisory council in the field of education in the Netherlands), when
this council proposed a special set of achievement tests to assess school performance. In
322
terms of the current distinction between vertical and horizontal supervision, process
indicators would then become the sole domain of schools, and should be targeted in the
school’s own quality care and self-evaluation provisions. Three scenarios could be
distinguished:
a) the current situation in which the Inspectorate monitors on both outcome and
process indicators;
b) a scenario in which the schools receive assistance and guidelines, possibly in the
form of specific instruments and software to independently carry out school self-
evaluations that contain process and outcome indicators; while the inspectorate
refrains from monitoring process indicators;
c) a scenario of maximum school autonomy; in which schools are left free in choosing
the way they carry out self-evaluation; the inspectorate refrains from directly
monitoring school process indicators, but, within the context of “proportional
supervision” have an indirect influence on the school self-evaluation procedures of
schools.
In choosing between these alternatives it is not irrelevant to refer to one set of results in
the school improvement debate, namely that there are some incidental, but at the same
time striking results, indicating that externally pre-structured programs, sometimes have
very good results. The success of the recent numeracy and literacy reform programs in
the UK, can also be seen as evidence for this conclusion. Perhaps there are limits in the
degree to which process autonomy should be promoted in primary and secondary
education. Although this goes totally against the current dominant policy view in the
Netherlands, it should still be counted in an argument in favor of the monitoring of
process indicators by the Inspectorate.
323
Chapter 9
Summary and conclusions
Purpose of the study and central questions
The purpose of this study was described in the introduction to this report as verifying
the scientific basis of the indicators that are part of the Dutch Inspectorate’s Supervision
Frameworks for primary and secondary education and to relate the Supervision
Frameworks to central functions of education, such as qualification, integration and
personality development.
The way this could be done differed between these two main targets. With respect to the
first part, verifying the scientific basis of the indicators of the Inspection Framework, a
relatively straightforward connection can be made to the research literature on school
and instructional effectiveness. For this target area the term validating of the framework
can be used. For the second area, the one in which the inspection framework is
compared to current thinking about central societal functions of education, there is less
of a factual basis. “Grounding” the Supervision Frameworks for this part is more like
positioning the framework with respect to new developments and trends.
The following core questions were addressed in the study:
Are the process indicators on teaching and learning in the Inspection Frameworks
supported by the knowledge base on school and instructional effectiveness?
How feasible is the idea of proportional supervision, given the possibilities and state
of the art of school self-evaluation in the Netherlands?
Do the Inspection Frameworks manifest defensible choices with respect to outcome
and process indicators, and strategic applications, given current perspectives on
educational governance and modern interpretations of the core societal functions of
education?
The Inspection Framework
The central question of school supervision is: “What is the quality of education in a
particular school like?” (Inspectorate, 2002, p. 9) This question is differentiated
according to three sub questions:
1) What is the school’s quality care like?
2) What is the school’s quality in teaching and learning?
3) What is the quality of the learning results? (ibid, p.9)
324
The three domains are divided in the following quality aspects; as indicated in the table
below, based on the Inspection Framework for Primary Schools, formulated in 2002.
Quality domain Quality aspects per domain
Quality care 8. Systematic quality care by the school 9. Testing
Teaching and Learning 10. Subject matter coverage 11. Time 12. Stimulating and supportive teaching and learning
process 13. Safe, supportive and stimulating school climate 14. Special care for children with learning difficulties
Outcomes 8. Learning results in basic subjects
In 2005 this framework was adapted, as indicated in the table below.
Quality domain Quality aspects per domain
Quality care 1. The school takes care of the assurance and improvement of the quality of education
2. The conditions for quality care are in place
Teaching and Learning 3. The subject matter offered is aimed at a broad development of the pupils and on preparing them for further education and the labour market
4. The students get sufficient time to master the subject matter
5. The pedagogical approach of the teachers induces a safe and stimulating learning environment
6. The didactic approach of the teachers supports pupils’ learning
7. The students play an active and independent role during instructional activities
8. The school has a safe and stimulating climate 9. Guidance and counceling is aimed at a full
development of the students’ capacities 10. Students with special needs receive the care that they
require
Outcomes 11. The pupils attain the achievement results relative to their capacities
325
Clearly the 2005 version of the framework is more elaborate in distinguishing actual
quality care and conditions for quality care at school, and also in distinguishing between
pedagogical and didactic aspects of the teaching and learning situation. Moreover, broad
development, labour market orientation and development of students’ capacities reflect
an educational perspective that appears to be influenced by current conceptions about
education for citizenship, and the development of competencies.
The inspection Framework compared to quality concepts
In Chapter 2 of the report the Inspection Framework was related to interpretations and
concepts of educational quality. This lead to the following conclusions:
As a school evaluation framework the Supervision Framework of the Inspectorate can be
classified as being associated with the effectiveness perspective on educational quality. The
process indicators of the Inspection Framework can be seen as effectiveness enhancing
conditions that are inspired by empirical school effectiveness research and by an
international consensus on good teaching practice. From the latter perspective Vernooy,
2002, cited by Janssens, 2005, lists the following key variables:
The schools realise the goals set in advance;
The teachers have high expectations of the pupils' abilities;
A positive teaching-learning climate prevails;
The school is focused on getting all pupils to acquire the basic skills;
The school head and team feel responsible for the results and pupil wellbeing;
The school reserves sufficient time for language/reading;
The school uses good, co-ordinated teaching materials;
Professional development is a continual process;
There is a strong focus on prevention and the pupil's results are closely monitored;
Attention is given to the effectiveness of the language/reading lessons;
The school follows a strong team-oriented approach;
The school head is good at:
- Expressing a clear vision;
- Placing a strong focus on the results of pupils;
- Monitoring improvements;
- Being closely involved in what happens in the classroom.
When examining the actual quality indicators from the Inspection Framework the
following observations can be made:
- the reference point, in the sense of the effect criteria, or outcome indicators, are test
scores in basic school subjects;
326
- the majority of process indicators is defined at classroom level; there seems to be a
strong emphasis of the primary process of teaching and learning;
- indicators that represent school self-evaluation and quality care have a double
significance in the framework; they form a set of effectiveness enhancing factors in
their own right, but at the same time they are evaluated from a meta-perspective, with
the possible result that school quality care procedures actually replace direct inspection
by the Inspectorate (this latter idea is the so called proportional supervision);
- some of the classroom level indicators might be given a school level interpretation, as
in the case of programmed time, a supportive climate and content coverage (but the
quantitative use of the framework seems to be limited to judgements at classroom
level);
- organizational and managerial conditions that feature as more remote effectiveness
enhancing factors in multi-level school effectiveness models are treated as “possible
background and causes” of quality conditions at the level of the primary process;
examples are: school management, aspects of the school organization, school policies,
professionalization of staff and contacts with parents (Inspectorate, 2002, p. 9);
- there is little emphasis on resource input factors (like facilities, teaching equipment,
computers and the school budget, class size and pupil teacher ratio);
- issues of special care for pupils with learning difficulties and students that are
disadvantaged because of low socio economic status, cultural capital or of minority
background are brought together under the criteria that, broadly speaking, refer to
adaptive teaching;
- school composition, an important issue in recent school effectiveness studies, is not
treated;
- the broader view of organizational effectiveness, as represented in the Quinn and
Rohrbaugh framework, is not represented in the Inspection Framework. There is no
attention for externally oriented school policy, like providing information to parents
and local constituencies, connections with the local business world, nor is there much
attention for aspects of human resource management at school, and for a close
monitoring of formal procedures (this last aspect being strongly represented in “ISO
type” quality management systems);
- the softer areas of school culture and climate are represented in the inspection
framework, with attention for positive interactions, safety and discipline and feeling
supported and stimulated.
The overall strong association with the perspective of school effectiveness allows for a
investigation in the empirical support for the process indicators in the Framework, seen as
effectiveness enhancing conditions, instrumental to the attainment of cognitive outcomes
327
in basic subjects. In Part II of this report the research evidence on school and instructional
effectiveness was summarized, and updated by a recent review and research synthesis. The
analytic part of the quantitative research synthesis, or meta-analoysis, is reported in Annex
1.
A first comparison to the set of process indicators that are part of the Inspection
Framework to variables identified in school and instructional effectiveness research
In Chapter 2, when the Inspection Framework was compared to perspectives and
concepts of educational quality, it was concluded that it was closely in line with the
effectiveness perspective. This perspective implies that the choice of process indicators
is based on those malleable school and instructional variables that have been shown to
be associated with relatively high performance (expressed in students’ learning results
and attainment). This line of argumentation lies at the hart of the approach to assess the
validity of the quality indicators of the Inspection Framework by examining the
coverage of variables identified in educational effectiveness research. Chapter 3
provides a first overview of the research results in this field, while Chapters 4 and 5,
provide a further update, based on more recent research results.
The analyses in Chapter 3 indicated a fair correspondence between categories used by
the Inspectorate and main variables used in instructional effectiveness research. This
was expressed in the table below, in which major variables, addressed in instructional
effectiveness research were compared to the categories of the observation instrument
that is used by the Inspectorate.
There appears to be a fair correspondence between the indicators that represent the
quality aspect learning and instruction and main factors that have received empirical
support in school and instructional effectiveness research. This is reflected in the
summary table below.
328
Most important factors from review
Opportunity to learn
Learning time
Structured teaching
Stimulating engagement
Task-oriented climate
Mutual respect
Orderliness, safety
Monitoring and questioning
Feedback and reinforcement
Modelling learning/self-regulation
Categories in Observation check list
Learning time
Clear and structured teaching
Activating
Challenge
Support (mutual respect)
Orderly, functional learning environment
Evaluates whether objectives are reached
Feedback
Learning to use learning strategies
Classroom organisation
At the level of the school, i.e. the school level variables identified in school
effectiveness research, the correspondence is less. When considering the version of the
Supervision Framework that stems from 2002, there appeared to be an almost exclusive
concentration on the classroom level. In the 2005 version, school conditions are
recognized in the quality aspects that relate to conditions for school self-evaluation and
quality care. Next, parental involvement and “climate”, areas that have a place in the
2005 version, are also likely to have a school level interpretation. As a whole the focus
on the instruction level is still present in the 2005 version. This choice is quite
defensible from the perspective of the results of empirical effectiveness research, since
instructional variables are closer to learning results, and have also repeatedly been
shown to have stronger effects than the more “remote” school level factors. This general
conclusion is again supported by the results of our research syntheses as reported in
Chapters 4 and 5 and Annex 1. Nevertheless, from the position of school improvement,
and levers for change, it could be argued that school characteristics like achievement
orientation, and consensus and cooperation between staff might perhaps require a
stronger presence in the Inspection Framework.
The quality indicators of the Inspection Framework compared to the results of a
research synthesis on school and instructional effectiveness
The results presented in Chapters 4 and 5, as well as the computations of effect sizes for
the variables included in the quantitative meta-analysis, described in Annex 1, on school
and instructional effectiveness are summarized in the table below. The variables from
the research review and meta-analysis are matched to the indicators from the Inspection
329
Framework as much as possible. There is not a complete correspondence between the
variables addressed in the qualitative research review, expressed according to the so
called vote count procedures, and the variables that were included in the quantitative
meta-anlysis. Incidental iscrepancy is due to the fact that for the quantitative meta-
analysis the material had to be screened again for being amenable to the computation of
effect-sizes.
Table: Summary table linking the Supervision Framework to the research findings on
school and instructional effectiveness. The ratio’s presented are the number of positive
significant effects in terms of cognitive outcomes out of all studies that investigated the
factors and variables in question. When the ratio equals or is higher than 50% results
are presented in bold. The coefficients between brackets and printed in italics are the
effect sizes as computed as part of the quantitative meta-analyses, presented in Annex 1
330
Quality aspects and indicators Inspectorate
Factors and variables in Effectiveness Research
Quality Aspect 1: The school takes care of the assurance and improvement of the quality of education. 1.1 The school knows its entrance
situation, including the specific needs of the student body.
1.2 The school systematically evaluates the quality of its performance in terms of learning results
1.3 The school systematically evaluates the quality of learning, teaching and counceling
1.4 The school has formulated measurable improvement targets.
1.5 The school carries out improvement activities in a systematic way
1.6 The school guarantees the quality of learning and teaching
1.7 The school guarantees the quality of the school examination and of other evaluation instruments.
1.8 The school reports about the realized quality of education to interested parties (parents, students, competent authorities, funding agencies and sponsors).
Quality Aspect 2: The conditions for quality care are in place 2.1 School management initiates and
steers the quality care 2.2 Quality care is connected to the
school’s vision with respect to learning and teaching as stated in the school plan.
2.3 The school management takes care of a professional school culture
2.4 The school takes care of an effective communication about the quality of education.
2.5 Staff, school management, pupils, parents and competent authorities are all of them being involved in the school’s quality care
Evaluative potential at school level and goal directedness; 12/26 (.01 n.s.) Use of pupil monitoring system: 1/2 Job appraisal: 2/12 Clear focus on mastery of basic subjects: 9/14 Educational leadership and parental involvement in school policies: 20/74 The school leader acts as a councellor and controller of classroom teachers: 9/13 (.02 n.s.) The school leader as an initiator and facilitator of staff professionalization: 0/2 Emphasis on parental involvement in school policy: 0/13
331
Quality Aspect 3: The subject matter offered is aimed at a broad development of the pupils and on preparing them for further education and the labour market 3.1 The school has provided the
foundations of subject matter offering in the lower grades.
3.2 The school guarantees that the actual subject matter offering covers the examination program
3.3 The program (total of subject matter that is offered) is being connected to important societal and actual themes.
3.4 Subject matter that is offered in one grade connects to the previous and subsequent grades.
3.5 There is coherence between the subject matter offerings in the various subjects
3.6 Subject matter offerings are adapted to the educational needs of individual pupils
3.7 The school offers knowledge about the different cultures that are present in the Netherlands, on a regular basis, with reference to the corresponding norms and values.
3.8 Schools with over 20% of students that are weak in language, adapts Dutch language tuition to the needs of these students in all subjects
3.9 The schools offers content on civic education, social cohesion and norms and values
Quality aspect 4: The students get sufficient time to master the subject matter 4.1 The intended teaching time
corresponds to the legal norms 4.2 The structural (i.e. planned) amount
of lessons “not given” is minimal 4.3 The incidental amount of lessons “not
given” is limited 4.4 Non permitted absence of students is
Curriculum quality and opportunity to learn: 15/34 Opportunity to learn: 11/31 (.01 n.s.) Effective learning time: 33/108 School level (.10) Classroom level (.05)
332
limited 4.5 The teachers use the intended
teaching time in an efficient way 4.6 The school varies the amount of time
for teaching and learning relative to the educational needs of the students
Quality aspect 5: The pedagogical approach of the teachers induces a safe and stimulating learning environment 5.1 The teachers stimulate the pupils’ self
confidence 5.2 The teachers treat the pupils in a
respectful way 5.3 The teachers stimulate that pupils
treat one another with respect 5.4 The teachers realize a productive
atmosphere Quality aspect 6: The didactic approach of the teachers
supports pupils’ learning 6.1 The teachers provide insight in goal,
use and connectivity of the lesson activities
6.2 The teachers provide clear explanations
6.3 The teachers check whether the pupils have understood explanations and assignments
6.4 The teachers create a meaningful context for the learning process
6.5 The teachers stimulate the students to think (e.g by posing challenging questions)
6.6 The teachers provide substantive feedback to the pupils
6.7 The teachers provide insight into the pupils’ learning processes (simulating reflections on learning strategies)
6.8 The teachers take care of pupils being involved in the educational activities
6.9 The didactic approach is functional with respect to the pupils’ learning process
Homework: 20/44 Classroom climate: 21/84 Quality of interactions, teacher support 4/12 Structured instruction; clear an structured teaching: 46/115 (.07) Direct teaching: 17/57 Challenge: 46/92 (.08) Cognitive activation: 36/58 Feedback: 2/27 Learning to use learning strategies: 66/90 (.28) Meta-cognitive strategies: 21/24 Adaptive teaching and differentiation: 32/150 (.02 n.s.)
333
6.10 The teachers adapt their didactic approach to the differences between the pupils
6.11 The teachers use the analyses of pupils’ achievement for the way they shape their instruction
6.12 The teacher’s use of language is adapted to the language needs of the pupils
Quality aspect 7: The students play an active and independent role during instructional activities 7.1 Students are confronted with
stimulating activities and assignments 7.2 The students reflect on their own
learning processes 7.3 The pupils are given responsibility
for their own learning processes to a sufficient degree
7.4 The pupils learn to work together in an effective way
Quality aspect 8 The school has a safe and stimulating climate 8.1 The pupils manifest involvement in
school life 8.2 The staff manifest involvement in
school life 8.3 The parents feel involved in school
life 8.4 Staff and students interact in a
respectful way inside and outside classes
8.5 Staff an students feel safe at school 8.6 The pupils, staff and parents
experience the school leadership as supportive and stimulating with respect to the atmosphere at school
Activating: 48/154 ( .07) Cooperative learning: 17/23 School climate: 26/68 Parental involvement: 32/82 (.08) Orderly atmosphere: 16/30 (.15)
334
Quality aspect 9 Guidance and counceling is aimed at a full development of the students’ capacities 9.1 Guidance and counceling is aimed at
a good development and at the pupils’ well-being
9.2 The school uses a consistent system of instruments and procedures for monitoring progress and development of the pupils
9.3 The school uses information from primary school in the guidance and counceling of the pupils
9.4 The school supports pupils and parents with respect to career choices during the school programme
9.5 The school supports the pupils and parents with respect to choices regarding further education and the labour market
9.6 Special staff functions (like deans and mentors) see to it that pupils receive continuity in guidance throughout the career at school
Quality aspect 10: Students with special needs receive the care that they require 10.1 The school has early diagnosis of
pupils who need extra care 10.2 The school analyzes the kind of care
that selected pupils require 10.3 Special care is carried out in a
systematic, planned way 10.4 The school assesses the effects of
special care 10.5 Teachers and special staff for care co-
operate well 10.6 The school involves the parents of
pupils requiring extra care in the special activities
Monitoring: 3/7 (See above: adaptive teaching and differentiation)
335
The following observations can be made in interpreting these results:
- The amount of studies that address the evaluative potential of the school is fairly
limited (26 studies) and the proportion of significant positive effects is below the
50% threshold. The effect size computed in the meta-analysis is non significant and
very low (.01) The variable that stresses a clear focus on basic subjects, a “classic”
from the school effectiveness research literature, has a relatively high proportion of
significant positive effects, but the total number of studies that was used for this
conclusion is low (14 studies)
- When it comes to variables that can be associated with conditions for quality care at
school, the variables “educational leadership” and parental involvement in school
policy were considered. Only 20 out of 74 effects appeared to be statistically
significant in the expected positive direction. The variable “the school leader acts as
a counsellor and controller of classroom teacher” had a relatively high proportions
of positive effects, but the number of replications was limited (13). The quantitative
meta-analysis showed a very small non significatnt effect for educational leadership
(.02). When parental involvement was assessed in more general terms, namely in the
sense of parents being involved in school life in general (not just school policy) the
proportion of significant positive results 32/82, while the effect size computed in the
meta-analysis amounts to .08.
- In the domain of subject matter coverage, curriculum quality and opportunity to
learn showed a limited proportion of positive significant and a very low effect size
of .01. This finding does not correspond to earlier research reviews, and is lower
than expected.
- A similar conclusion can be drawn with respect to time related variables, effective
learning time and homework, with proportion of 33/108 and 20/44 both below the
50% threshold. Both variables received more support in earlier research reviews.
The effect sizes for learning time at school level and classroom level computed from
the meta-analyses were .10 and .05 respectively.
- Classroom climate and the quality of interaction between teachers and students had
a relatively low proportion of positive significant effects.
- A very striking finding from this research synthesis is the unexpectedly low
proportion of studies that showed a positive effect for variables like: structured
instruction, direct teaching, and feedback. These variables came out much stronger
336
in the research synthesis carried out by Walberg and Fraser in the eighties. The
coefficient of the effect size for structured teaching computed in the meta-analysis is
.07.
- Variables that are more in line with the ideas of “constructivist teaching”, like
challenge (46/92), cognitive activation, learning to use learning strategies (66/90)
and meta-cognitive strategies (21/24) came out more strongly. Again this is a
somewhat unexpected finding, as other recent review (e.g. Van der Werf, 2005)
place question marks behind the effectiveness of such approaches. The
corresponding effect sizes computed in the meta-analysis are .08 for challenging
learning conditions and .28 for learning to use learning strategies.
- Adaptive teaching and differentiation showed a relatively low proportion of positive
significant effects. The effect size in the meta-analysis was non-significant, .02. On
these variables earlier reviews paint a mixed picture.
- Among the climate variables an orderly atmosphere showed the largest proportion
of positive effects (16/30). The effect size for this variable was the highest of all
school variables with a coefficient of .15. Overall school climate did not reach the
50% threshold (26/68).
- Other “classics” from the school effectiveness research literature, like parental
involvement and consensus and cohesion among staff (not included in the above
summary table, because it is not a part of the inspection indicators) had relatively
low proportion of statistically significant positive effects (32/82 and 5/30
respectively). Although, as stated above, the coefficient for parental involvement
computed in the meta-anlysis (.08) was not that low as compared to other school
factors.
As a more general point, the results in the summary tables presented in Chapters 4 an 5
indicate that for variables that were supported in earlier studies, but which stayed below
the 50% threshold in this study, there was generally a large proportion of studies in
which results were in the expected positive direction, but were not statistically
significant; for example in the case of clear and structured teaching 62 out of 96 studies
had a non-significant effect, and only 3 out of 96 had a statistically significant negative
effect. This can be due to research technical considerations, like the size of the sample
that was used in the study. As matters stand the results should be taken with
considerable caution. The following implications can be drawn for the validity of the set
of quality indicators used in the Supervision Framework of the Dutch Inspectorate:
337
- There is an overall fairly good match between the set of variables that is studied in
educational effectiveness research and the process indicators (those concerning the
quality aspects school quality care and teaching and learning) of the Inspection
Framework.
- The fact that the current Frameworks concentrate more on effectiveness enhancing
conditions at classroom than on school level indicators is supported by our results,
in the sense that our results confirm the relatively greater impact of classroom level
variables, as compared to school level variables. In other words: our results do not
support a possible claim to add more school level indicators to the existing
framework.
- Our results do not support “the evaluative potential of schools” as an important
effectiveness enhancing condition. (As noted in Chapter 4, an earlier meta-anlysis
(Scheerens & Bosker, 1997) found a higher effect size for this variable). One should
therefore be careful in expecting much improvement in performance from
improving quality care at school. At the same time the relevance of the indicators on
quality care should be seen as undisputed, given the important strategic concept of
proportional supervision.
- As far as instructional strategies and didactic approaches are concerned the
Inspection Framework contains various elements that correspond to structured and
direct teaching as well as various elements that reflect a more constructivist
orientation to learning and instruction. Our results (unexpectedly, given the results
of earlier reviews) strongly support the inclusion of indicators related to activating
learning, providing challenging tasks, addressing learning strategies and meta-
cognition. At the same time the somewhat disappointing results with respect to
direct, structured teaching, should, in our view, not be used as a signal to abolish
some of the more “traditional” characteristics of structured teaching. Similar
conclusions should be drawn with respect to effective teaching time and opportunity
to learn. In a more general sense, therefore, our results support the “mix” of direct
teaching, and constructivist ideas, that is inherent in the current indicators on
teaching and learning.
- Climate aspects did not come out strongly in our study; with the clear exception of
an orderly atmosphere, and the corresponding indicator in the Framework is
therefore supported by our results.
338
Criteria for assessing school self-evaluation approaches
Two quality aspects in the Inspection Framework refer to quality care procedures and
conditions for quality care conditions at school, respectively. The quality indicators that
are in the Inspection framework were compared to state of the art perspectives on school
self-evaluation. In Chapter 6 various frames of reference were used to put these quality
indicators into perspective:
- Sets of criteria that have been used in the Netherlands to assess school self-
evaluation instruments and procedures for quality care; i.e. those of Q*Primair, Q5
and the Inspectorate;
- Sets of criteria that have been used by foreign inspectorates and groups of
inspectorates, i.e. SICI;
- More general evaluation standards (i.e. those of the Joint Committee of Standards in
the USA, chaired by Daniel Stufflebeam);
- A closer look at the utilization of school self-evaluation instruments and application
of quality care (among others, based on experiences from the ZEBO project);
- A closer look at accuracy standards applied to school self-evaluation, the issue of
objectivity.
The chapter provided descriptive material on all these contributions in order to assess
the current state of affairs with respect to proportional supervision, and the respective
quality indicators in the Inspection Framework.
The idea of proportional supervision is a creative application of decentralization and
“subsidiarity” in education, meaning that all that can possibly and reasonably be carried
out at a lower administrative level, should not be carried out at a lower level (Scheerens,
1997). But perhaps this formulation also indicates the Achilles heel of this strategy. Can
school self evaluation really meet the demands of external evaluation? What Chapter 6
has shown is that a lot of developmental effort has been and is being invested in
enabling, facilitating and stimulating school self evaluation and school quality care in
the Netherlands. A relatively large set of school self evaluation systems and instruments
has been developed, and organizations like Q5 and Q Primair have invested in the
development of criteria to asses these instruments and procedures and in experimental
good practice applications. It is interesting to note that the quality standards developed
and applied by Q5 and Q Primair differ from those applied in studies of the Education
Inspectorate in the sense of the importance given to ‘accuracy standards’. The
Inspectorate is more consistent in seeing criteria such as reliability and validity as
339
essential for the value of school self evaluation and is generally more severe and critical
in assessing the available set of instruments.
What is being practiced under the headings of school self-evaluation and quality care is
a mixture of elements that have a somewhat different orientation and strongly different
traditions. The main orientations are: a scientific evaluation orientation, emphasizing
accuracy, reliability and validity of procedures; an administrative and client oriented
branch of quality management systems, and finally, a more qualitative, opinion based
school improvement orientation. From the user perspective of schools each have their
strong and weak points:
- schools may appreciate the objectivity and rigor of the use of structured instruments
that have been tested for reliability and validity, but most of all find themselves at
odds with the standardization implications, strictness of procedures, and the targeted
scope that is easily condemned as narrow and reductionist;
- quality care systems may have the fashionable appeal of the fast moving business
world and the more real asset of being responsive to clients; yet these systems
usually fail to address the primary process of teaching and learning, and if they try,
they may paint caricatures in their efforts to “proceduralize” educational processes;
- most suspect among these three, at least from an evaluation perspective, is the
school improvement perspective that forgoes rigor in establishing fact, and may
favour superficial consensus about change over sound diagnosis; in fact there is a
perpetual dilemma in educational analysis and action that is caused by a discrepancy
between the urge to adapt and change, and knowledge that is at best partial and
contested.
In our view the scientific perspective should prevail in the idea of proportional
supervision. According to proportional supervision a part of external evaluation is in
fact delegated to schools. This could only work if schools applied the same rigor in
method that would be required of external evaluation.
The evidence presented in this chapter with regards to the actual state of play of school
self-evaluation indicates that only a minority of schools starts to meet the current
standards of the Inspection. Given the need for rigorous internal evaluation this state of
affairs could be improved by the Inspectorate by stimulating the development of
scientifically founded instruments for school self-evaluation and making these available
to schools. In actual practice this should be done by stimulating instrument development
and facilitating integral systems of school self-evaluation that use state of the art
practice in areas like pupil monitoring systems based on Item Response Models,
computerized school administration systems, and empirically validated process
indicators.
340
The Inspection Frameworks and rethinking the societal functions of education
Modernization of the society has implications for education. Trends like globalization,
the knowledge society, shifts in the dominant orientation of the economy, immigration
and demographic developments evoke a rethinking of the basic functions of education.
Ultimately this kind of thinking is bound to address the core goals and means of
education, and as such the basic subject matter areas in quality evaluation systems, as
the one of the Dutch Inspectorate.
Sociological and economic conceptualization and theory formation on the societal
effects of education tend to cumulate in underlining the competency concept as a basis
for assuring connectivity between education and the demands of society.
Levy and Murnane (2001) summarize research findings that are the result of various
kinds of economic research that are driven by these theories and have generally been
targeted to the question what competencies, related to formal schooling, could explain
the attractiveness of qualified individuals to employers. They mention five “key
competencies”:
6) Basic reading and mathematical skills are important in determining long-run labour
market outcomes, including the ability to adjust to changing circumstances.
7) The ability to communicate effectively, both orally and in writing, is important in
determining long-run labour market outcomes.
8) In modern firms it is increasingly important to possess the ability to work
productively in groups.
9) The latter condition also emphasises elements of “emotional intelligence” including
the ability to relate well to other people.
10) Familiarity with computers is of growing importance in the labour market.
When approached from the context of formal schooling, the idea of competencies can
be placed on a continuum of types of educational outcomes, already introduced in a
preceding section, that runs from specifically content oriented to “content free”
personality traits. Discrete positions on this continuum are presented in Figure 2.4.
341
- outcomes as measured by tests included in textbooks
- outcomes as measured by implemented school curricula (teacher developed)
- outcomes as measured by tests based on the intended national curriculum
- outcomes as measured by international tests covering the common core of a range
of national curricula, e.g. TIMSS
- “literacy” tests, aimed at measuring basic skills in reading, mathematical and
scientific reasoning, e.g. PISA
- competencies as multi-facetted dispositions of individuals, including cognitive,
motivational and possibly other components
- personality traits, like internally or externally determined locus of control,
independence, general intelligence
Figure 4 (Chapter 2): A continuum of educational outcomes, running from highly
content bound to personality dependent
The practical meaning of the competency issue is that the answer to an effective linking
of education to societal functioning should partially be sought in the teaching of more
general skills and in paying more attention to motivational and attitudinal aspects. Also
the idea of meta-skills, such as general problem solving skills and “learning to learn”
could stimulate a more reflective and self-steering attitude of future citizens. In this way
“competencies” are considered as useful tools in serving the “employability” of future
citizens.
Several main emphases could be distinguished:
- more emphasis on somewhat general cognitive skills, for example in terms of the
diverse “literacies” (reading literacy, mathematical literacy etc.) emphasized in the
OECD PISA project, next to the mastery of subject matter based knowledge;
- emphasis on a broader scala of personality dimensions than just the cognitive one;
social skills, motivation to learn and even “emotional intelligence”;
- new contents and skills; for example knowledge about the civic society and
computer skills;
- a heightened attention for moral issues and different value systems.
As it comes to drawing implications for the Inspection Framework of the current
underlining of competencies, this could, first of all, be seen in the domain of the quality
aspect of attainment and learning results. In this domain the Inspectorate appears to
depend on the use of tests and assessment instruments that are developed in other
contexts, such as the well-known CITO- end test for primary schools. As a general
strategy, the Inspectorate might exert their influence to stimulate the development of
342
valid measuring instruments that could assess certain competencies (for example, see
Diedrich, 2005). New kind of “end terms” might also imply considering new
approaches in education. In this respect the case of informal learning for citizenship
related competencies was discussed in Chapter 7. Such relatively new orientations in
schooling, teaching and learning are likely to give rise to re-consider the set of process
indicators, particularly those concerning the overall quality domain of teaching and
learning. In the case study on informal learning for citizenship a range of aspects of the
school’s identity, its institutional norms and culture, were seen as relevant for informal
learning. Research studies, such as those by Diedrich (2005), might be helpful to the
Inspectorate in a possible re-shaping and specification of those process indicators in the
framework that consider the learning environment, pedagogical approaches and the
school and classroom climate. When the version of the Supervision framework of 2002
is compared with the one of 2005, an adaptation that goes in the direction indicated can
already be discerned.
The application of the Inspection Framework in the context of educational governance
The pillars of “new public management”, seen from a steering perspective, are increased
autonomy on input and process dimensions, and control on outcomes. Likewise
systemic reform in education largely consists of two domains: deregulation and
decentralization on the one hand and establishing evaluation mechanisms on the other.
The concept of “functional decentralization” encompasses these two core domains of
systemic reform; as it explicitly considers patterns in which freedom on process
dimensions go together with strict supervision on performance. In Chapter 8, core
concepts in the governance discussion, such as decentralization and accountability were
explained. Next, the improvement potential of performance based reforms was
discussed; in which the question of the innovative potential of evaluation, supervision
and assessment mechanisms came out as a key question. Educational policy in the
Netherlands, for the last decade has closely followed the principals of new public
management, with a particular emphasis on school autonomy. At the same time the role
of supervision is being stressed. This happens both in the sense of “horizontal
supervision”, which is targeted at local stakeholders, and in the sense of “vertical
supervision”, associated with external evaluation and accountability (Koers, VO, 2004).
Placing the Inspection Framework against the background of the principals of new
public management, and considering empirical evidence on results of applications,
allows for informed conjecture about its effectiveness enhancing potential. A crucial
question in a context of increased school autonomy is the place of school process
indicators in vertical supervision.
343
It was argued that the principals of new public management could be subsumed under the
term “subsidiarity”, which has a long standing tradition in educational policy in the
Netherlands. In the history of education in the Netherlands the term subsidiarity was
used to refer to a specific way in which denominational pressure groups in education
liked to see the relationship between the state and corporations representing interest
groups in the educational field. According to the subsidiarity principle the state should
not interfere in matters that can be dealt with by organized units of professionals. In the
original case these organized units were the denominationally based corporations or
pressure groups of representatives in the education field, their umbrella organizations in
particular. “Subsidiarity” was the term preferred by the Roman-Catholic denomination,
while the Protestants spoke of “sovereignty in one’s own circle”. Leune (1987, 379-380)
points at the corporatistic nature of this kind of concepts. According to the subsidiarity
principle the state only acts subsidiary, that is, it only interferes as a replacement, when
needed. A simple example of subsidiarity is a driving-instructor, who takes over the
steering of a vehicle when the trainee makes a mistake, but in all other cases quietly
watches without interference. Within the context of the European Commission the term
subsidiarity is used to express the principle that what can be accomplished by the
member states should not be done by the central organs of the Union.
A straightforward application of the subsidiarity principle to the question who, at which
level, should evaluate what, could be that units or administrative levels should only take
responsibility for evaluating those areas over which they have direct operational control.
According to this reasoning one could maintain the position that the Inspectorate should
not supervise schools on process indicators, since schools are solely responsible for the
instruction process. In stead the Inspectorate, according to this position, would only
monitor schools on the basis of their outcomes, c.q. the learning results of the students.
In fact, several years ago, this point of view was taken in a report of the Onderwijsraad
(the most important advisory council in the field of education in the Netherlands), when
this council proposed a special set of achievement tests to assess school performance. In
terms of the current distinction between vertical and horizontal supervision, process
indicators would then become the sole domain of schools, and should be targeted in the
school’s own quality care and self-evaluation provisions. Three scenario’s could be
distinguished:
a) the current situation in which the Inspectorate monitors on both outcome and
process indicators;
b) a scenario in which the schools receive assistance and guidelines, possibly in the
form of specific instruments and software to independently carry out school self-
evaluations that contain process and outcome indicators; while the inspectorate
refrains from monitoring process indicators;
344
c) a scenario of maximum school autonomy; in which schools are left free in choosing
the way they carry out self-evaluation; the inspectorate refrains from directly
monitoring school process indicators, but, within the context of “proportional
supervision” have an indirect influence on the school self-evaluation procedures of
schools.
In choosing between these alternatives it is not irrelevant to refer to one set of results in
the school improvement debate, namely that there are some incidental, but at the same
time striking results, indicating that externally pre-structured programs, sometimes have
very good results. The success of the recent numeracy and literacy reform programs in
the UK, can also be seen as evidence for this conclusion. Perhaps there are limits in the
degree to which process autonomy should be promoted in primary and secondary
education. Although this seems to go against the current dominant policy view in the
Netherlands, it should still be counted in an argument in favor of the monitoring of
process indicators by the Inspectorate. The chapter on experiences with quality care and
school self-evaluation, to be summarized in a subsequent section, is also relevant to
weighing the pros and cons of these three scenarios.
Main conclusions of the study
- Our results leave the discussion concerning the instrumental effectiveness
orientation that is still predominant in the current Framework in comparison to a
broader outlook on organizational quality wide open. In our research synthesis we
find little support for expanding the current set on indicators that is primarily
concentrated at the classroom level with a lot of school level indicators, for example
on support and management functions. However, ongoing developments concerning
school management, task enlargement of teachers and human resources management
at school may be seen as arguments to give more room for indicators that are aimed
at measuring such school level phenomena.
- The research syntheses described in this report indicate a good match between the
set of indicators of the Inspection framework and the set of variables that is
frequently studied in educational effectiveness research. The research results are
relatively inconsistent for many variables, implying that the empirical basis for quite
a few variables is far from solid. This note of caution is definitely in place, and
discourages drawing strong conclusions from the research findings that were
presented. Striking outcomes were the relatively frequent support for variables like
cognitive stimulation and learning to use learning strategies, and a lesser support
(also in comparison to the results of earlier reviews) for variables associated with
345
structured teaching. These outcomes, in our view, should rather be seen as
supporting the mix of indicators in the Inspection Framework that include variables
from the direct instruction as well as from the cognitive activation paradigm.
- Given the state of play of quality care and school self-evaluation, there is every
reason for the Inspectorate to support the improvement of scientifically sound
instruments and methods of school self-evaluation.
- When observing the changes in the Framework between the version of 2002 and
2005 the additional emphases on broad development, labour market orientation and
development of students’ capacities, is in line with current thinking about enhancing
the societal relevance of education with respect to citizenship and competencies.
- Further implications of current thinking about the major societal functions in
education might stimulate the Inspectorate to enlarge the range of outcome measures
and assessments, and to stimulate research and development concerning the impact
of school institutional norms and school culture on formal and informal learning.
Such studies might lead to the inclusion of some additional process indicators
focussing at the learning potential of the school environment at large (i.e. including
climate and culture, behavioural norms apart from the explicit teaching
arrangements).
- Within the context of educational governance in the Netherlands proportional
inspection is a well-fitting concept, emphasizing school autonomy “wherever
possible”. Apart from this there is certain evidence about the effectiveness of reform
strategies in education that depend on externally structured programs. Therefore the
current mixture of direct supervision and indirect supervision (via school quality
care and proportional inspection) should perhaps not be abandoned for more radical
alternatives that decentralize supervision further.
346
347
Annex 1
Quantitative meta-analysis of school and instructional
effectiveness
Introduction
Apart from a vote-counting procedure, a quantitative meta-analysis was conducted. This
meta-analysis was carried out by means of the program Comprehensive Meta Analysis
Version 2. This program allows for conducting a meta-analysis based on random effects
models for effect size (Raudenbush, 1994). In this approach selected studies are
considered as a sample from the population of studies on school and teacher effects. The
main advantage of this kind of meta-analysis is that the information from each study is
weighted by the reliability of the information, in this case the sample size. Moreover,
differences in reported effect sizes can be modeled as a function of study characteristics,
which could be seen as moderators of the computed effect-sizes. For example, reported
effect sizes can be higher in one country than in another country, or, reported effect
sizes might differ between studies employing an experimental design as compared to
studies using a non-experimental design. The following moderators were used:
Cognitive outcome: subject (Do studies using mathematic achievement as
outcome measure report different effect sizes than studies using language
achievement as outcome measure?)
Adjustment for relevant student characteristics like prior knowledge, aptitude
and socio-economic status (Do studies adjusting for relevant student characteristics
report different effect sizes than studies that present uncorrected results?)
School type (Do studies carried out in primary schools report different effect sizes
than studies in secondary schools?)
Country (Do studies carried out in different countries (USA, UK, Netherlands;
other) show different effect sizes?)
Design (Do studies employing a (quasi-)experimental design show different effect
sizes than studies employing a correlational design?)
Method (Do studies using a multi-level analysis show different effect sizes than
studies analyzing (dis)aggregated data?
Fisher’s Z is used as an index for the effect of individual school and instructional
characteristics. This means that all statistics reported in the original research articles
348
(correlations, regression coefficients, t-values etc.) were transformed into this index,
using formulae presented by Rosenthal (1994). Unfortunately, it is common that many
articles do not contain information about the size of the relationship between dependent
and independent variables, in case this relationship is non-significant. For our meta-
analytical purposes the sizes of these relationships were set at zero. This could lead to
an underestimation of the effect sizes.
With regard to the interpretation of Fisher’s Z it must be noted that this coefficient is
very close to the conventional correlation coefficient (r), in particular when it concerns
small values of the correlation coefficient (as it is the case in this study).
The most well-know effect size coefficient, Cohen’s d, is related to the correlation
coefficient (and, hence, Fischer’s Z) in the following way:
d= 2r/√(1-r2)
Or, if one is happy with a close estimate:
d=2r.
For the interpretation of d, Cohen (1969) offers the following guidelines: a value of
d=.20 is small, a value of .50 is moderate, a value of .80 is large.
Finally, it must be noted that even small effect sizes are important. For example,
Rosenthal and Rubin (1982) showed that, when the impact of a particular drug on
survival equals r = .30 (d=0.60), this result translates into an improvement in survival
from 35% to 65%. So, the fact that effect sizes may appear small, does not necessarily
mean they are unimportant. This is also the case for education, in particular given the
large number of students in the school system. Think for example of the impact of one
school leader on student achievement. Although the impact may be small, in a typical
(Dutch) primary school it still affects hundreds of children.
Results for the school level factors
Mean effect sizes
The results of the quantitative concerning the school level factors are shown in Table
A.1.
In general, the effect sizes are small (ranging from -0.13 to 0.14) and only a few factors
have a significant relationship (i.e. an effect size that is larger than two times the
standard error) with the variable of interest.
349
Table A1: School level: mean effect size (expressed as Fischer's Z), standard error, p-value, number of "Studies". Random effects model for different outcome measures. Type of outcomes
Area Cognitive outcomes Non-cognitive outcomes ES SE p N ES SE p N
Achievement, orientation, high expectations 0.013 0.030 0.660 25 -0.134 0.038 0.000 12
Educational leadership 0.025 0.019 0.172 76 0.113 0.058 0.050 7
Consensus and cohesion among staff 0.011 0.026 0.685 38 0.036 0.067 0.590 4
Curriculum quality/ opportunity to learn 0.010 0.037 0.793 19
School climate: orderly atmosphere 0.146 0.027 0.000 34 -0.068 0.069 0.324 5
School climate: effect. orientation/good
internal relations 0.016 0.029 0.594 30 -0.042 0.059 0.472 6
Evaluative potential 0.015 0.039 0.700 17 -0.096 0.110 0.381 2
Parental involvement 0.075 0.015 0.000 97 0.000 0.110 1.000 2
Effective learning time 0.101 0.035 0.004 23 0.087 0.087 0.318 3
Differentiation 0.051 0.014 0.000 114 0.070 0.083 0.402 3
Cognitive outcomes: subject
Area Mathematics Language ES SE p N ES SE p N
Achievement, orientation, high expectations -0.030 0.031 0.332 18 0.056 0.086 0.513 3
Educational leadership -0.004 0.025 0.869 36 0.024 0.031 0.440 24
Consensus and cohesion among staff 0.014 0.032 0.665 21 0.007 0.041 0.870 14
Curriculum quality/ opportunity to learn 0.000 0.048 1.000 9 0.018 0.049 0.709 10
School climate: orderly atmosphere 0.039 0.034 0.247 18 0.064 0.050 0.203 10
School climate: effect. orientation/good
internal relations 0.020 0.045 0.656 10 0.010 0.043 0.824 13
Evaluative potential -0.027 0.044 0.545 11 0.092 0.063 0.147 6
Parental involvement 0.061 0.020 0.002 47 0.073 0.023 0.001 41
Effective learning time 0.035 0.046 0.454 11 0.100 0.049 0.042 11
Differentiation 0.059 0.018 0.001 55 0.046 0.019 0.019 57
With regard to the cognitive outcomes only four factors have a significant relationship.
The most important factor is School climate (or better, the existence of a safe and
orderly climate) (0.15), followed by Effective learning time (0.10), Parental
involvement (0.08) and Differentiation (0.05). All other factors do not have a significant
relationship.
The most important result with regard to the non-cognitive outcomes concerns the
relatively large and significant negative effect size of achievement orientation/high
350
expectations (-0.13) This could be interpreted as the negative feelings of students
experiencing a strong achievement pressure at school.
A similar argument might be used with respect to the factor school climate (a safe and
orderly climate). This factor has a negative relationship with non-cognitive outcomes,
although this effect is not statistically significant. Apart from achievement
orientation/high expectations, there is only one other school variable having a
significant relationship with non-cognitive outcomes; the factor Educational Leadership
does not only have a positive and significant impact on cognitive achievement, but also
non-cognitive outcomes. However, given the number of studies involved, this finding
should be treated with caution. It is based on only seven studies. In general, this is also
true for all findings concerning the relationship of school level variables and non-
cognitive outcomes. The number of studies involved is limited, making it difficult to
come up with ‘hard’ and definite conclusions about the relationship between school
level factors and non-cognitive outcomes.
The results regarding the differences between subject, namely mathematics and
language, are surprising in the sense that in most cases effect sizes are (slightly) larger
in the domain of language, cf. Scheerens and Bosker, 1997. In our case effect sizes are
consistently higher for language, except for the factors Differentiation and Consensus
and Cohesion among Staff. Yet, the differences in effect sizes between the two subject
matter areas are not very large. The largest difference concerns the factor Evaluative
Potential. For this factor the difference between studies using a mathematic test as
outcome variable and studies focusing on language achievement is 0.12 (-0.03 for
mathematics and 0.09 for language). A final important finding is that in both domains
(mathematics and language) only two factors have effect sizes which deviate
significantly from zero: Parental Involvement and Differentiation.
The influence of moderators
Effect sizes are different among studies. This raises the question which study
characteristics (or moderators) are responsible for these differences. Apart from
differentiating between type of outcomes and subject matter areas, the following
moderators were taken into account:
Adjustment for relevant student characteristics like prior knowledge, aptitude
and socio-economic status (Do studies adjusting for relevant student characteristics
report different effect sizes than studies reporting uncorrected results?)
School type (Do studies carried out in primary schools report different effect sizes
than studies in secondary schools?
351
Country (Do studies carried out in different countries (USA; UK, Netherlands;
other) come up with different effect sizes?)
Design (Do studies employing a (quasi-)experimental design show different effect
sizes than studies employing a correlational design?)
Method (Do studies using a multi-level analysis show different effect sizes than
studies analyzing (dis)aggregated data?)
The results of these (bivariate) analyses are shown in Tables A2 and A 3.
Table A2: School level: mean effect size (expressed as Fischer's Z), standard error, p-value, number of "Studies". Random effects model with adjustment or non-adjustment for student background conditions, school type and study design as moderators. Adjustments
Area Adjustments No adjustments ES SE p N ES SE p N
Achievement, orientation, high expectations -0.046 0.019 0.012 28 0.008 0.067 0.901 9
Educational leadership 0.041 0.017 0.019 54 0.011 0.038 0.780 29
Consensus and cohesion among staff 0.012 0.019 0.523 35 0.007 0.079 0.933 7
Curriculum quality/ opportunity to learn 0.009 0.026 0.739 19
School climate: orderly atmosphere 0.039 0.021 0.058 32 0.421 0.076 0.000 7
School climate: effect. orientation/good
internal relations 0.008 0.026 0.748 23 0.011 0.057 0.849 13
Evaluative potential 0.005 0.026 0.844 19
Parental involvement 0.026 0.011 0.024 70 0.192 0.039 0.000 29
Effective learning time 0.065 0.026 0.013 23 0.347 0.125 0.006 3
Differentiation 0.047 0.009 0.000 114 0.120 0.079 0.127 7
Schooltype
Area Primary schools Secondary schools ES SE p N ES SE p N
Achievement, orientation, high expectations -0.082 0.045 0.069 8 -0.027 0.026 0.298 29
Educational leadership 0.036 0.021 0.090 43 0.025 0.024 0.302 40
Consensus and cohesion among staff 0.010 0.023 0.659 31 0.020 0.047 0.676 11
Curriculum quality/ opportunity to learn 0.009 0.029 0.760 19
School climate: orderly atmosphere 0.093 0.027 0.000 23 0.179 0.039 0.000 16
School climate: effect. orientation/good
internal relations 0.015 0.034 0.664 16 0.000 0.033 0.991 20
Evaluative potential 0.013 0.031 0.681 18 -0.160 0.166 0.336 1
Parental involvement 0.056 0.013 0.000 79 0.104 0.031 0.001 20
Effective learning time 0.100 0.027 0.000 26
Differentiation 0.058 0.011 0.000 108 -0.009 0.039 0.810 13
352
Design
Area (quasi-) experimental correlational ES SE p N ES SE p N
Achievement, orientation, high expectations -0.037 0.025 0.139 37
Educational leadership 0.105 0.058 0.067 6 0.028 0.018 0.132 77
Consensus and cohesion among staff 0.013 0.025 0.587 42
Curriculum quality/ opportunity to learn 0.151 0.141 0.284 1 0.004 0.037 0.924 18
School climate: orderly atmosphere 0.122 0.026 0.000 39
School climate: effect. orientation/good
internal relations 0.006 0.027 0.820 36
Evaluative potential 0.005 0.037 0.897 19
Parental involvement 0.257 0.051 0.000 7 0.064 0.016 0.000 92
Effective learning time 0.369 0.069 0.000 3 0.099 0.033 0.003 26
Differentiation 0.053 0.021 0.010 28 0.048 0.015 0.002 93
A central issue in school and instructional effectiveness research is the adjustment for
student intake. It is a well-known fact that effect sizes of variables relating to school and
teacher characteristics become smaller (or even diminish) when student intake
characteristics are taken into account. This fact leads to the expectation that the mean
effect sizes of studies correcting for student intake characteristics are lower than the
mean effect size of studies not correcting for these intake characteristics. By and large
our results confirm this expectation. Most effect sizes are lower in studies taking into
account student intake characteristics than in studies not taking into account these
characteristics. The most important exception is the factor Educational Leadership. The
mean effect sizes in studies correcting for student intake characteristics is higher than in
studies not correcting for these characteristics. For most factors the difference between
studies with and without adjustment is rather dramatic. For example, the mean effect
size of the factor Parental Involvement is 0.03 for studies taking into account of student
intake characteristics, while this figure is 0.19 in studies not correcting for these
characteristics. With regard to the factor Effective Learning Time the decrease is even
more dramatic. The effect size of studies not correcting for relevant student
characteristics is 0.35, while for studies correcting for student characteristics this figure
is 0.07.
Previous results of meta-analytic studies on school and teacher effectiveness indicate
that, in case relevant effect sizes are reported, school type matters: effect sizes are
usually higher in studies conducted in primary schools than in studies carried out in
secondary schools (Scheerens & Bosker, 1997). Our present results (see Table A2),
however, show a mixed pattern. In some cases effect sizes are higher in ‘secondary
353
school’-studies than in ‘primary school’-studies, while in other cases the reverse
situation is true. For example, with regard to the factor School Climate (orderly climate)
the effect size is higher in studies focusing on secondary schools than in studies
conducted in primary schools. The same is true for the factor Parental Involvement.
However, the effect size concerning the factor Differentiation is larger in studies carried
out in primary schools than in studies focusing on secondary schools.
Another relevant moderator variable is the impact of research design. The results of this
analysis are shown in Table A2 as well. The results clearly indicate that the large
majority of school effectiveness studies are correlational and there are only relatively
few quasi-experimental studies. Therefore it is difficult to come up with definite
conclusions about the impact of this study characteristic. Yet, our results suggest that
research design matters a lot. In all cases, the effect sizes are (much) higher in studies
employing a (quasi-)experimental design than in studies using a correlational design.
These results underscore the pleas often expressed in the school effectiveness research
community for the application of experimental designs.
Differences in effect sizes between studies, carried out in different countries, has also
been addressed in previous meta-analyses. Witziers, Bosker and Krüger (2003,) for
example, report that Educational Leadership seems to be only an important factor in
USA-studies, while studies conducted in the Netherlands are characterized by no or
very small relationships between this factor and student achievement. This finding is
replicated in our analysis (see Table A3). If there are positive and significant
relationships between school effectiveness factors and student achievement, they can
mainly be found in studies carried out in the USA. Studies carried out in other countries
are characterized either by the absence of positive significant relationships between the
factors in question and student achievement or even by presence of negative significant
relationships. For example, in the Netherlands the effect size of the factor Achievement
Orientation is significantly negative, while only one factor shows a (significant) positive
result with student achievement. This is the factor Differentiation. These findings
indicate that either the results of school effectiveness are context bound or that in the
United States studies have more chance to be published in case their findings are
positive.
Finally, we compared differences in effect sizes between studies using different
statistical methods. More specifically the difference between studies employing multi-
level analyses and studies that did not apply multi-level techniques was analyzed. In the
latter case (studies not using multi-level techniques) the issue of data being defined at
different levels is either resolved by disaggregating school level or by aggregating
354
student level data. The results show that, in general, the studies that use multi-level
analyses report lower effect sizes than the latter studies.
Table A3: School level: mean effect size (expressed as Fischer's Z), standard error, p-value, number of "Studies". Random effects model with country and methods as moderators Countries
Area USA United Kingdom ES SE p N ES SE p N
Achievement, orientation, high expectations 0.010 0.043 0.815 18 0.000 0.014 1.000 6
Educational leadership 0.099 0.041 0.016 24 0.001 0.007 0.938 24
Consensus and cohesion among staff 0.014 0.050 0.777 16 0.007 0.014 0.605 6
Curriculum quality/ opportunity to learn
School climate: orderly atmosphere 0.199 0.046 0.000 17
School climate: effect. orientation/good
internal relations
0.023 0.067 0.727
9 -0.004 0.010 0.715 12
Evaluative potential
Parental involvement 0.215 0.036 0.000 30 0.000 0.010 1.000 12
Effective learning time 0.201 0.058 0.001 12
Differentiation 0.200 0.042 0.000 20 0.041 0.020 0.038 7
Countries (continued)
Area Netherlands Other countries ES SE p N ES SE p N
Achievement, orientation, high expectations
-
0.137 0.067 0.040 6 -0.074 0.028 0.010 7
Educational leadership
-
0.017 0.038 0.645 22 0.057 0.028 0.040 13
Consensus and cohesion among staff 0.044 0.054 0.414 10 -0.009 0.018 0.597 10
Curriculum quality/ opportunity to learn 0.004 0.038 0.924 18 0.151 0.119 0.203 1
School climate: orderly atmosphere 0.057 0.043 0.182 15 0.040 0.028 0.162 7
School climate: effect. orientation/good
internal relations 0.067 0.099 0.499 4 -0.012 0.025 0.627 11
Evaluative potential 0.005 0.037 0.898 19
Parental involvement 0.071 0.075 0.341 5 0.013 0.006 0.026 52
Effective learning time 0.009 0.045 0.850 14
Differentiation 0.101 0.032 0.001 31 -0.010 0.005 0.050 63
355
Method
Area Multi-level not Multi-level ES SE p N ES SE p N
Achievement, orientation, high
expectations -0.020 0.016 0.204 30 -0.059 0.087 0.497 7
Educational leadership 0.007 0.012 0.561 69 0.168 0.066 0.012 14
Consensus and cohesion among staff 0.011 0.016 0.486 39 0.000 0.143 1.000 3
Curriculum quality/ opportunity to
learn 0.008 0.024 0.729 19
School climate: orderly atmosphere 0.039 0.022 0.078 28 0.284 0.068 0.000 11
School climate: effect.
orientation/good internal relations 0.003 0.017 0.880 35 0.365 0.306 0.234 1
Evaluative potential 0.006 0.027 0.818 17 0.000 0.162 1.000 2
Parental involvement 0.015 0.010 0.126 74 0.269 0.048 0.000 25
Effective learning time 0.065 0.025 0.010 22 0.278 0.121 0.022 4
Differentiation 0.035 0.009 0.000 101 0.194 0.057 0.001 20
Results for the instructional variables
Mean effect sizes
The results of the quantitative analyses on the instructional variables are given in Table
A4. The effect sizes (expressed as Fischer’s Z) are given for the three student outcome
areas: cognitive outcomes, non-cognitive outcomes and learning processes. Overall,
most of the instructional variables have shown significant but rather low to medium
effect sizes. Furthermore, the largest effect sizes were found in the area of learning
processes (e.g. learning strategies ES=0.50), followed by effects on non-cognitive and
cognitive outcomes. This is in accordance with the theoretical assumption that learning
processes represents a more situative and proximal student throughput measure,
whereas cognitive and non-cognitive student outputs represent more stable and long-
term student measures.
356
Table A4: Teacher/ class level: mean effect size (expressed as Fischer's Z), standard error, p-value, number of "Studies". Random effects model for different outcomes. Type of outcomes
Area Cognitive outcomes Non-cognitive outcomes ES SE p N ES SE p N
Learning Time 0.047 0.010 0.000 141 0.226 0.041 0.000 12
Classroom organisation 0.028 0.014 0.040 65 0.064 0.028 0.020 29
Learning Environment 0.037 0.026 0.144 21 0.073 0.029 0.011 22
Clear and structured 0.073 0.012 0.000 90 0.082 0.035 0.020 17
Activating 0.056 0.011 0.000 130 0.137 0.026 0.000 31
Learning strategies 0.275 0.023 0.000 53 0.195 0.037 0.000 16
Challenge 0.077 0.012 0.000 102 0.194 0.019 0.000 53
Support 0.023 0.016 0.148 46 0.145 0.025 0.000 25
Feedback -0.026 0.019 0.165 48 0.093 0.033 0.005 20
Evaluation 0.028 0.014 0.046 61 0.076 0.073 0.302 4
Teacher characteristics 0.099 0.026 0.000 21 0.083 0.078 0.283 6
Adaptive Teaching 0.020 0.018 0.257 43 0.108 0.044 0.014 10
Practice -0.041 0.019 0.029 30 -0.017 0.055 0.754 8
Material 0.026 0.023 0.258 21 0.000 0.077 1.000 4
Integrative approaches 0.107 0.018 0.000 52 0.171 0.050 0.001 8
Type of outcomes (continued)
Area Learning Process
ES SE p N
Learning Time 0.090 0.032 0.005 10
Classroom organisation -0.064 0.056 0.252 10
Learning Environment 0.061 0.025 0.013 16
Clear and structured 0.053 0.015 0.000 42
Activating 0.141 0.028 0.000 17
Learning strategies 0.501 0.053 0.000 11
Challenge 0.120 0.011 0.000 67
Support 0.089 0.017 0.000 29
Feedback 0.120 0.028 0.000 17
Evaluation
Teacher characteristics
Adaptive Teaching 0.112 0.035 0.001 4
Practice -0.059 0.055 0.282 4
Material 0.175 0.043 0.000 9
Integrative approaches 0.086 0.028 0.002 12
357
Cognitive outcomes: subject
Area Mathematics Language ES SE p N ES SE p N
Learning Time 0.055 0.011 0.000 83 0.029 0.025 0.238 53
Classroom organisation 0.028 0.014 0.036 48 0.043 0.045 0.333 12
Learning Environment 0.035 0.029 0.224 14 0.046 0.124 0.709 2
Clear and structured 0.068 0.012 0.000 65 0.121 0.045 0.007 15
Activating -0.010 0.015 0.502 37 0.156 0.029 0.000 59
Learning strategies 0.182 0.051 0.000 9 0.260 0.054 0.000 24
Challenge 0.039 0.013 0.002 59 0.160 0.045 0.000 18
Support 0.041 0.021 0.057 18 0.024 0.041 0.553 15
Feedback -0.015 0.022 0.487 31 -0.047 0.051 0.353 12
Evaluation 0.033 0.016 0.036 34 0.017 0.031 0.592 26
Teacher characteristics 0.060 0.033 0.072 8 0.192 0.058 0.001 10
Adaptive Teaching 0.083 0.034 0.015 13 -0.005 0.030 0.863 29
Practice -0.038 0.016 0.015 29
Material 0.011 0.025 0.650 11 0.295 0.143 0.039 2
Integrative approaches 0.044 0.019 0.023 30 0.224 0.043 0.000 20
In the area of cognitive outcomes 10 out of the 15 instructional indicators have shown
significant and positive effects. The highest effect sizes were found for learning
strategies (.28), integrative approaches (.11), teacher characteristics (.10), challenge
(.08), clear and structured teaching (.07), activating (.06), and learning time (.05).
Despite the positive direction of effects the effect sizes have to be interpreted as small.
One negative effect has been found for the area of drill and practice (-.04). With regard
to non-cognitive outcomes 13 of 15 instructional variables have shown significant and
positive effect sizes. Highest effect sizes were found for learning time (.23), learning
strategies (.20), challenge (.19), integrative approaches (.17), support (.15), activating
(.14), adaptive teaching (.10), and feedback (.09). Thus, differences between effects of
instruction on cognitive and non-cognitive outcomes emerged. Instructional variables
such as support, feedback and adaptive teaching did not show significant effects on
cognitive outcomes. However, they have shown positive effects on non-cognitive
outcomes such as the students’ interest in a learning domain. In the area of learning
processes 13 of 15 variables have shown positive effect sizes. Highest effect sizes were
found for learning strategies (.50), material (.18), activating (.14), challenge (.12),
feedback (.12) and adaptive teaching (.11).
Next to the three outcomes measures, differences in effect sizes between the two
domains mathematics and language were taken into consideration. Thereby, effect sizes
358
of studies that have been conducted in the area of language have shown higher effect
sizes than mathematics studies. They were consistently higher for clear and structured
teaching, activating, learning strategies, challenge, teacher characteristics, material and
integrative approaches. These findings go along with the findings reported for school
effectiveness and have been reported in earlier meta-analyses (Scheerens & Bosker,
1997). Differences between the two domains were found for example in the area of
integrative approaches (language: .22; mathematics: .04). These differences are partly
due to different research foci and designs in the single domains: Integrative approaches
have been elaborated more profoundly in research on language instruction than in
mathematics instruction.
Overall, the findings show rather low effect sizes of instructional variables on student
outcomes. The effect sizes have been higher for instructional variables than for school
level variables. This pattern has been found in previous reviews and meta-analyses.
However, compared to earlier studies the effect sizes of the research of the last decade
show overall lower effect sizes than reported in earlier meta-analyses (Scheerens &
Bosker, 1997; Wang, Haertel & Walberg, 1993).
The influence of moderators
The influence of moderators has been investigated according to the five research
questions that have been outlined in the area of school effectiveness factors:
Adjustment for relevant student characteristics like prior knowledge, aptitude
and socio-economic status (Do studies adjusting for relevant student
characteristics report different effect sizes than studies reporting uncorrected
results?)
School type (Do studies carried out in primary schools report different effect
sizes than studies in secondary schools?
Country (Do studies carried out in different countries (USA; UK, Netherlands;
other) come up with different effect sizes?)
Design (Do studies employing a (quasi-)experimental design show different
effect sizes than studies employing a correlational design?)
Method (Do studies using a multi-level analysis show different effect sizes than
studies analyzing (dis)aggregated data?)
The results of the (bivariate) analyses are given in Tables A5, A6 and A7.
359
Table A5: Teacher/ class level: mean effect size (expressed as Fischer's Z), standard error, p-value, number of "Studies". Random effects model for different moderator variables Adjustments
Area Adjustments No adjustments ES SE p N ES SE p N
Learning Time 0.044 0.008 0.000 175 0.115 0.033 0.001 18
Classroom organisation 0.041 0.012 0.001 82 -0.010 0.034 0.765 22
Learning Environment 0.052 0.017 0.002 42 0.073 0.041 0.075 17
Clear and structured 0.063 0.010 0.000 132 0.091 0.031 0.003 23
Activating 0.067 0.010 0.000 162 0.146 0.037 0.000 16
Learning strategies 0.277 0.018 0.000 65 0.232 0.062 0.000 17
Challenge 0.093 0.011 0.000 127 0.145 0.014 0.000 95
Support 0.054 0.015 0.000 49 0.093 0.019 0.000 51
Feedback 0.011 0.015 0.444 67 0.050 0.027 0.064 30
Evaluation 0.023 0.011 0.045 76 0.174 0.151 0.248 1
Teacher characteristics 0.151 0.029 0.000 21 0.000 0.051 1.000 6
Adaptive Teaching 0.028 0.015 0.059 54 0.086 0.042 0.043 9
Practice -0.039 0.016 0.015 42
Material 0.079 0.018 0.000 42
Integrative approaches 0.108 0.015 0.000 72
Schooltype
Area Primary schools Secondary schools ES SE p N ES SE p N
Learning Time 0.049 0.014 0.001 116 0.042 0.009 0.000 77
Classroom organisation 0.040 0.027 0.133 26 0.022 0.011 0.044 78
Learning Environment 0.064 0.035 0.071 16 0.052 0.015 0.001 43
Clear and structured 0.087 0.019 0.000 62 0.053 0.010 0.000 93
Activating 0.168 0.022 0.000 76 0.042 0.009 0.000 102
Learning strategies 0.270 0.036 0.000 38 0.272 0.017 0.000 44
Challenge 0.172 0.031 0.000 28 0.116 0.007 0.000 194
Support 0.055 0.027 0.040 27 0.081 0.010 0.000 73
Feedback -0.016 0.035 0.636 23 0.028 0.011 0.013 74
Evaluation 0.022 0.022 0.314 41 0.020 0.013 0.130 36
Teacher characteristics 0.178 0.042 0.000 17 0.009 0.027 0.733 10
Adaptive Teaching 0.018 0.025 0.470 32 0.059 0.016 0.000 31
Practice -0.035 0.013 0.006 42
Material 0.295 0.132 0.025 2 0.059 0.015 0.000 40
Integrative approaches 0.122 0.021 0.000 56 0.089 0.030 0.003 16
360
Design
Area (quasi-) experimental correlational ES SE p N ES SE p N
Learning Time 0.056 0.023 0.017 30 0.050 0.009 0.000 163
Classroom organisation 0.551 0.126 0.000 1 0.028 0.011 0.013 103
Learning Environment 0.041 0.054 0.446 4 0.058 0.016 0.000 55
Clear and structured 0.100 0.016 0.000 55 0.047 0.011 0.000 100
Activating 0.172 0.015 0.000 100 0.012 0.012 0.326 78
Learning strategies 0.288 0.018 0.000 78 0.099 0.065 0.129 4
Challenge 0.147 0.018 0.000 47 0.111 0.009 0.000 175
Support 0.075 0.025 0.003 21 0.073 0.012 0.000 79
Feedback 0.080 0.033 0.016 16 0.011 0.014 0.424 81
Evaluation 0.013 0.073 0.861 4 0.024 0.012 0.042 73
Teacher characteristics 0.097 0.024 0.000 27
Adaptive Teaching 0.038 0.014 0.006 63
Practice -0.039 0.017 0.018 42
Material 0.363 0.044 0.000 14 0.021 0.020 0.305 28
Integrative approaches 0.147 0.019 0.000 52 0.045 0.026 0.081 20
First, studies have been differentiated according to the adjustments that had been taken
into account. In instruction research it has been shown that student pre-requisites such
as their pre-knowledge, aptitudes, SES play a central role in the amount of variance that
is explained by instructional variables. Thus, it has been investigated whether studies
differ in effect sizes according to the level of adjustment. In general, higher effects
would be expected for studies with no adjustments on student pre-requisites. The
findings of the quantitative meta-analyses support the assumptions. Overall, higher
effect sizes have been found for studies with no adjustments. For example, learning time
has shown an effect size of .12 for studies with no adjustment and .04 for studies with
adjustment. Studies with no adjustment on activating show an effect size of .15
compared to studies with adjustment (.09). The largest discrepancy was found for
evaluation, with an effect size of .17 for studies with no adjustment and .02 for studies
with adjustments. Despite the large number of instructional variables that have shown
discrepancies between adjustment and no adjustment, some variables emerged as stable.
Learning strategies, for example, has an effect size of .23 for studies with no adjustment
and .28 for studies with adjustment.
Second, differences between school types have been investigated. Thereby, effect sizes
are usually higher in studies that have been conducted in primary schools compared to
secondary level studies (Scheerens & Bosker, 1997). The results show a mixed pattern.
361
For the variables activating, teacher characteristics and material differences in favor of
primary schools emerged. No differences, for example, have been found in the area of
learning strategies and challenge, indicating that these instructional variables seem to
work effectively in primary and secondary schooling.
The research design represents the third moderator that has been taken into
consideration. Thereby, the studies have been classified as to whether a correlational or
a (quasi-) experimental research design had been applied. Overall, the differences
between studies in the strength of effect sizes are remarkable, showing that
experimental or quasi-experimental studies show larger effect sizes than correlational
survey-type studies. For classroom organization the effect size was .55 for experimental
and .03 for correlational studies; it was .10 for clear and structured teaching in
experimental studies compared to .05 in correlational studies. Experimental studies
investigating activating have shown an effect size of .17 whereas correlational studies
provided an effect size of .01. Learning strategies measured within an experimental
design had an effect size of .29 compared to .10 for correlational studies. The effect size
of material was .36 if investigated within an experimental design and .02 if investigated
within a correlational design.
These differences have important consequences for the field of instruction effectiveness
since a large number of studies (especially large scale studies) have been applying a
correlational design. The effect sizes of (quasi-) experimental studies are also rather
low, indicating an overall limited effect of instructional variables on learning outcomes.
But they are substantially higher than effect sizes that have been “produced” by
correlational studies. Thus, low effect sizes for instructional variables may partly be due
to the research design that has been applied in the studies. In consequence, low effect
sizes for instructional variables have to be interpreted carefully.
Furthermore, differences in effect sizes of studies that have been carried out in different
countries have been addressed. The findings are given in Table 6. Overall, lowest effect
sizes are found for the United Kingdom and for the Netherlands. Higher effect sizes are
shown for the United States of America and Germany. The highest effect sizes have
been found within a broader range of countries, summarized as “other countries”. In
U.S. studies highest effect sizes have been found for learning strategies (.29), teacher
characteristics (.18), integrative approaches (.16), support (.13) and adaptive teaching
(.11). In the United Kingdom the highest effect size is found for material (.31). In the
Netherlands, integrative approaches (.10) and clear and structured teaching (.07) have
shown the highest effect sizes, followed by learning time and evaluation (both .03). In
Germany, a broad range of instructional variables show significant and positive effect
sizes: learning time (.11), classroom organization (.10), clear and structured (.07),
362
activating (.11), challenge (.11), support (.10), feedback (.10), integrative approaches
(.24). A negative effect has been shown for practice (-.09). Studies that have been
conducted in other countries have also shown a broad range of relevant instructional
variables: learning time (.10), activating (.16), learning strategies (.25), challenge (.25),
support (.08), teacher characteristics (.30), and adaptive teaching (.14).
Table A6: Teacher/ class level: mean effect size (expressed as Fischer's Z), standard error, p-value, number of "Studies". Random effects model countries as moderator. Countries
Area USA United Kingdom ES SE p N ES SE p N
Learning Time 0.073 0.015 0.000 35 0.025 0.017 0.157 20
Classroom organisation -0.007 0.016 0.642 18
Learning Environment 0.056 0.017 0.001 17
Clear and structured 0.067 0.017 0.000 24 0.059 0.015 0.000 27
Activating 0.036 0.008 0.000 111
Learning strategies 0.294 0.016 0.000 52
Challenge 0.098 0.011 0.000 51 0.006 0.015 0.712 26
Support 0.125 0.026 0.000 10 0.003 0.018 0.864 18
Feedback 0.000 0.031 1.000 6
Evaluation -0.075 0.045 0.096 2
Teacher characteristics 0.179 0.047 0.000 12 0.000 0.031 1.000 6
Adaptive Teaching 0.112 0.036 0.002 4
Practice -0.022 0.013 0.086 25
Material 0.059 0.015 0.000 30 0.306 0.153 0.045 1
Integrative approaches 0.161 0.020 0.000 35 0.019 0.021 0.366 14
Countries (continued)
Area Netherlands Germany ES SE p N ES SE p N
Learning Time 0.027 0.009 0.002 100 0.114 0.032 0.000 19
Classroom organisation 0.000 0.085 1.000 2 0.102 0.023 0.000 49
Learning Environment 0.030 0.047 0.526 5 0.061 0.033 0.065 26
Clear and structured 0.068 0.018 0.000 21 0.065 0.017 0.000 68
Activating 0.008 0.058 0.889 2 0.109 0.026 0.000 34
Learning strategies 0.024 0.058 0.683 2
Challenge 0.000 0.085 1.000 2 0.106 0.014 0.000 111
Support -0.180 0.118 0.128 1 0.103 0.024 0.000 31
Feedback -0.025 0.016 0.113 29 0.101 0.024 0.000 44
Evaluation 0.026 0.013 0.040 42 0.088 0.056 0.115 6
Teacher characteristics 0.043 0.058 0.460 2 0.098 0.057 0.088 4
Adaptive Teaching -0.009 0.013 0.527 35 0.068 0.045 0.134 14
Practice -0.090 0.035 0.011 17
363
Material 0.063 0.075 0.401 1 0.038 0.046 0.407 10
Integrative approaches 0.098 0.037 0.009 6 0.237 0.058 0.000 6
Countries (continued)
Area Other countries ES SE p N
Learning Time 0.097 0.032 0.002 19
Classroom organisation 0.017 0.020 0.396 35
Learning Environment 0.061 0.036 0.095 11
Clear and structured 0.032 0.035 0.370 15
Activating 0.165 0.028 0.000 31
Learning strategies 0.252 0.043 0.000 28
Challenge 0.254 0.022 0.000 32
Support 0.081 0.020 0.000 40
Feedback 0.000 0.032 0.997 18
Evaluation 0.021 0.023 0.362 27
Teacher characteristics 0.304 0.095 0.001 3
Adaptive Teaching 0.137 0.038 0.000 10
Practice
Material
Integrative approaches 0.089 0.056 0.114 11
Finally, it has been investigated whether effect sizes differ between the methods that
have been applied in the studies. Thereby, the methodological development in the last
ten years has experienced a call and movement towards multi-level modeling, taking
into account the nested data structure of the field (students in classes, classes in school
types, school types in countries). Multi-level modeling is often applied in surveys and
correlational studies. Next to multi-level modeling a broader mix of methods is
summarized as not multi-level. The category not multi-level includes studies that have
been applying an experimental approach (in which a high control is given because
students and classes are randomly assigned to instructional treatments) as well as studies
that have been applying “traditional” correlational research designs (without controlling
for the nested structure of the data). The findings of the differentiation between multi-
level and non multi-level is given in Table 7.
364
Table A7: Teacher/ class level: mean effect size (expressed as Fischer's Z), standard error, p-value, number of "Studies". Random effects model. Analysis model as moderator. Method
Area Multi-level not Multi-level ES SE p N ES SE p N
Learning Time 0.030 0.009 0.001 162 0.163 0.021 0.000 31
Classroom organisation 0.034 0.018 0.063 36 0.031 0.014 0.029 68
Learning Environment 0.045 0.026 0.088 17 0.064 0.019 0.001 42
Clear and structured 0.052 0.011 0.000 113 0.114 0.019 0.000 42
Activating 0.052 0.023 0.025 27 0.080 0.010 0.000 151
Learning strategies 0.101 0.067 0.129 4 0.287 0.018 0.000 78
Challenge 0.058 0.014 0.000 78 0.148 0.009 0.000 144
Support 0.020 0.015 0.163 56 0.147 0.016 0.000 44
Feedback 0.001 0.016 0.943 56 0.055 0.020 0.006 41
Evaluation 0.011 0.013 0.387 68 0.157 0.039 0.000 9
Teacher characteristics 0.072 0.028 0.011 15 0.179 0.050 0.000 12
Adaptive Teaching 0.007 0.017 0.676 40 0.104 0.024 0.000 23
Practice -0.011 0.039 0.782 11 -0.045 0.018 0.011 31
Material 0.052 0.036 0.148 13 0.090 0.021 0.000 29
Integrative approaches 0.051 0.019 0.007 38 0.222 0.025 0.000 34
Overall, the findings show that multi-level modeling results in lower effect sizes
compared to non multi-level modeling. Whereas in studies with no multi-level modeling
all 15 indicators have shown significant effect sizes, only 6 of 15 variables have been
significant in multi-level studies. This is particularly striking for the variable learning
strategies that has shown a significant positive effect size of .29 in studies with no
multi-level modeling and a non significant effect size of .10 in multi-level studies.
However, these differences have to be interpreted carefully. Most of the studies on
learning strategies have been applying an experimental research design and have been
summarized as not multi-level. Furthermore, the number of studies differs remarkably
(not multi-level: N=78, multi-level: N=4).
To summarize, the findings of the meta-analyses differ in important ways from previous
findings on instructional effectiveness: the effect sizes are small, but most of the
indicators chosen for the Inspectorate Framework matter. The effect sizes “produced”
by studies in the last decade are smaller than the ones that have been reported in
previous reviews (cf. Scheerens & Bosker, 1997; Wang, Haertel & Walberg, 1993).
Several factors could be relevant for the even lower effect sizes of studies in the last
365
decade: The quality of research designs (e.g. the large number of correlational survey
studies), the possible “over-control” of influencing factors (e.g. within multi-level
modeling), as well as the quality of the operationalization of independent and dependent
variables.
366
367
References
Adedayo, O.A. (1998). Differential effectiveness by gender of insrtuctional methods on
achievement in mathematics at tertiary level. Educational Studies in Mathematics,
37(1), 83.
Alexander, P.A., Fives, H., Buehl, M.M., & Mulhern, J. (2002). Teaching as persuasion.
Teaching and Teacher Education, 18(7), 795-813.
Allsopp, D.H. (1997). Using classwide peer tutoring to teach beginning algebra
problem-solving skills in heterogeneous classrooms. Remedial and Special
Education, 18(6), 367-379.
Anderman, E.M., Eccles, J.S., Yoon, K.S., Roeser, R., Wigfield, A., & Blumenfeld, P.
(2001). Learning to value mathematics and reading: Relations to mastery and
performance-oriented instructional practices. Contemporary Educational
Psychology, 26(1), 76-95.
Anderson, C.S. (1982). The search for school climate: A review of the research. Review
of Educational Research, 52(3), 368-420.
Anderson, L.W. (1991). Increasing Teacher effectiveness. Paris: UNESCO, IIEP.
Anderson, L.W. (2004) Increasing Teacher Effectiveness. (Second Edition) Paris:
UNESCO, IIEP.
Anderson, L.W., & Pellicer, L.O. (1998). Towards an Understanding of Unusually
Successful Programs for Economically Disadvantaged Students. Journal of
Education for Students Placed at Risk, 3, 237- 263.
Annevelink, E. (2004). Class size: Linking teaching and learning. Enschede: s.n.
Applebee, A.N., Langer, J.A., Nystrand, M., & Gamoran, A. (2003). Discussion-based
approaches to developing understanding: Classroom instruction and student
performance in middle and high school English. American Educational Research
Journal, 40(3), 685-730.
Argyris, C., & Schön, D.A. (1974). Theory in practice: increasing professional
effectiveness. San Francisco: Jossey-Bass.
Arnold, D.H., Fisher, P.H., Doctoroff, G.L., & Dobbs, J. (2002). Accelerating math
development in head start classrooms. Journal of Educational Psychology, 94(4),
762-770.
Artelt, C., Baumert, J., McElvany, N.J., & Peschar, J. (2003). Learners for life. Student
approaches to learning. Results from PISA 2000. Paris: OECD.Anderson, C.S.
(1982). The search for school climate: A review of the research. Review of
Educational Research, 52(3), 368-420.
Ashman, A.F., & Gillies, R.M. (1997). Children's cooperative behavior and interactions
in trained and untrained work groups in regular classrooms. Journal of School
Psychology, 35(3), 261-279.
368
Bacharach, S.B, Bamberger, P., Conley, S.C., & Bauer, C. (1990). The Dimensionality
of Decision Participation in Educational Organizations: The Value of a Multi-
Domain Evaluative Approach. Educational Administration Quarterly, 26(2), 126-
167.
Baethge, M., Buss, K-P, & Lanfer, C. (2003). Konzeptionelle Grundlage fuer einen
Nationalen Bildungsbericht: Berufliche Bilding und Weiterbildung/Lebenslanges
Lernen. BMBF; Berlin.
Balli, S.J., Wedman, J.F., & Demo, D.H. (1997). Family involvement with middle-
grades homework: Effects of differential prompting. Journal of Experimental
Education, 66(1), 31-48.
Bangert, R.L., Kulik, J.A., & Kulik, C.C. (1983). Individualized systems of instruction
in secondary schools. Review of Educational Research, 53, 143-158.
Baumann, J.F., Edwards, E.C., Boland, E.M., Olejnik, S., & Kame'enui, E.J. (2003).
Vocabulary tricks: Effects of instruction in morphology and context on fifth-grade
students' ability to derive and infer word meanings. American Educational
Research Journal, 40(2), 447-494.
Baumert, J., & Köller, O. (2000). Unterrichtsgestaltung, verständnisvolles Lernen und
multiple Zielerreichung im Mathematik- und Physikunterricht der gymnasialen
Oberstufe. In J. Baumert & W. Bos & R. Lehmann (Eds.), TIMSS/III. Dritte
Internationale Mathematik- und Naturwissenschaftsstudie. Mathematisch und
naturwissenschaftliche Bildung am Ende der Schullaufbahn. Band 2 (pp. 271-
315). Opladen: Leske + Budrich.
Baumert, J., Blum, W., & Neubrand, M. (2001). Surveying the instructional conditions
and domain-specific individual prerequisites for the development of mathematical
competencies. Paper presented at the special meeting on the co-ordination of
national teacher components for PISA 2003, 28 May 2001, Munich, Germany.
Baumert, J., Blum, W., & Neubrand, J. (2002). Drawing lessons from PISA 2000 -
Long-term research implications: Gaining a better understanding of the
relationship between system inputs and learning outcomes by assessing
instructional and learning processes as mediating factors. Berlin: Max-Planck-
Institut für Bildungsforschung.
Beelaerts, M. Bousché, C. De Goeij, L., De Graaff, E., Horsman, K., & Klifman, H.
(1999). Projectplan Stimulering Kwaliteitszorg VO. (Available from [Q5, Burg.
Reigerstraat 74, Utrecht]).
Behets, D. (1997). Comparison of more and less effective teaching behaviors in
secondary physical education. Teaching And Teacher Education, 13(2), 215-224.
Bennacer, H. (2000). How the socioecological characteristics of the classroom affect
academic achievement. European Journal of Psychology of Education, 15(2),
173-189.
Berry, C. (2001). Achievement effects of multigrade and monograde primary schools in
369
the Turks and Caicos islands. International Journal of Educational Development,
21(6), 537-552.
Bianchini, J.A. (1997). Where knowledge construction, equity, and context intersect:
Student learning of science in small groups. Journal of Research in Science
Teaching, 34(10), 1039-1065.
Bishop, J. (1997). The effect of National Standards and Curriculum-Based Exams on
Achievement. The American Economic Review, 87(2), 260-264.
Bolhuis, S. (2003). Towards process-oriented teaching for self-directed lifelong
learning: a multidimensional perspective. Learning and Instruction, 13(3), 327-
347.
Borsch, F., Jurgen-Lohmann, J., & Giesen, H. (2002). Cooperative learning in
elementary schools: Effects of the jigsaw method on student achievement in
science. Psychologie in Erziehung Und Unterricht, 49(3), 172-183.
Bosker, R.J., & Hendriks, M.A. (1997). Betrouwbaarheid, validiteit en practische
bruikbaarheid van een instrumentarium ten behoeve van schoolzelfevaluatie in het
basisonderwijs. Enschede: OCTO.
Bosker, R.J., & Scheerens, J. (1999). Openbare prestatiegegevens van scholen;
nuttigheid en validiteit. Pedagogische Studiën, 76(1), 61-73.
Bransford, J. D., Brown, A. L., & Cocking, R. R. (2000). How people learn: Brain,
mind, experience, and school. Washington, DC: National Academy Press.
Bray, M. (1994). Centralization/decentralization and privatization/ publicization:
conceptual issues and the need for more research. In W.K. Cummings & A. Riddell
(eds.), Alternative Policies for the Finance, Control, and Delivery of Basic
Education [pp. 817-824]. Special issue of the International Journal of Educational
Research, 21(8).
Brophy, J. (2001). Generic Aspects of Effective Teaching. In Wang, M.C. & Walberg,
H.J. Tomorrow’s Teachers. McCutchan Publishing Company.
Brophy, J., & Good, T.L. (1986). Teacher behavior and student achievement. In M.
Wittrock (Ed.), Third Handbook of Research on Teaching [pp. 328-375]. New
York: MacMillan.
Brush, T.A. (1997). The effects on student achievement and attitudes when using
integrated learning systems with cooperative pairs. Etr&D-Educational
Technology Research and Development, 45(1), 51-64.
Buhl, M. (2003). Schulqualität und politische Sozialisation: Individuelle und
klassenbezogene Zusammenhänge des Schüler-Lehrer-Verhältnisses mit Aspekten
politischer Identität. Zeitschrift für Soziologie der Erziehung und Sozialisation,
23(4), 385-399.
Bund-Länder-Kommission für Bildungsplanung und Forschungsförderung. (1997).
Gutachten zur Vorbereitung des Programms "Steigerung der Effizienz des
mathematisch-naturwissenschaftlichen Unterrichts". Bonn: BLK.
370
Burkam, D.T., Lee, V.E., & Smerdon, B.A. (1997). Gender and science learning early
in high school: Subject matter and laboratory experiences. American Educational
Research Journal, 34(2), 297-331.
Burns, R.B., & Mason, D.A. (2002). Class composition and student achievement in
elementary schools. American Educational Research Journal, 39(1), 207-233.
Byrne, B., & Fieldingbarnsley, R. (1995). Evaluation of a program to teach phonemic
awareness to young-children - a 2-year and 3-year follow-up and a new preschool
trial. Journal of Educational Psychology, 87(3), 488-503.
Cameron, K.S., & Whetten, D.A. (1983) (eds.). Organizational Effectiveness. A
comparison of multiple models. New York: Academic Press.
Cameron, K.S., & Whetten, D.A. (1985). Administrative effectiveness in higher
education. The Review of Higher Education, 9, 35-49.
Cantrell, S. C. (1999). The effects of literacy instruction on primary students' reading
and writing achievement. Reading Research and Instruction, 39(1), 3-26.
Carroll, J.B. (1963). A model of school learning. Teachers College Record, 64, 722-733.
Carroll, J.B. (1989). The Carroll Model, a 25 year retrospective and prospective view.
Educational Researcher, 18, 26-31.
Chang, C.Y. (2002). The impact of different forms of multimedia CAI on students'
science achievement. Innovations in Education and Teaching International, 39(4),
280-288.
Chang, C.Y., & Barufaldi, J.P. (1999). The use of a problem-solving-based instructional
model in initiating change in students' achievement and alternative frameworks.
International Journal of Science Education, 21(4), 373-388.
Chelimsky, E., & Shadish, W.R. jr. (1997). Evaluation for the 21st century. A handbook.
London: Sage.
Chinn, C.A., Anderson, R.C., & Waggoner, M.A. (2001). Patterns of discourse in two
kinds of literature discussion. Reading Research Quarterly, 36(4), 378-411.
Chularut, P., & DeBacker, T.K. (2004). The influence of concept mapping on
achievement, self-regulation, and self-efficacy in students of English as a second
language. Contemporary Educational Psychology, 29(3), 248-263.
Cibulka, J.G., & Derlin, R.L. (1995). State educational performance reporting policies
in the U.S.: Accountability’s many faces. International Journal of Educational
Research, 23(6), 479-492.
Clausen, M. (2001). Unterrichtsqualität: eine Frage der Perspektive? Empirische
Analysen zur Übereinstimmung, Konstrukt- und Kriteriumsvalidität. Münster:
Waxmann.
Cohen, J. (1969). Statistical power analysis for the behavioral sciences. New York:
Academic Press.
Cohen, M. (1982). Effective schools: Accumulating research findings. American
Education, January-February, 13-16.
371
Collins, A., & Stevens, A. (1982). Goals and strategies of inquiry teachers. In: R. Glaser
(Ed.), Advances in Instructional Psychology, Vol. II. Hillsdale, N.J.: Lawrence
Erlbaum Associates.
Consolie, P.G. (1999). Achievement, attendance, and discipline in a year round
elementary school. Dissertation Abstracts International, 60-11, 5830.
Coombs, P.H., & Ahmed, M. (1974). Attacking rural poverty: how non-formal
education can help. London: Johns Hopkins Press. A research report for the
World Bank prepared by the International Council for Educational Development.
Cooper, H., Lindsay, J.J., Nye, B., & Greathouse, S. (1998). Relationships among
attitudes about homework, amount of homework assigned and completed, and
student achievement. Journal of Educational Psychology, 90(1), 70-83.
Cooper, H., Valentine, J.C., Charlton, K., & Melson, A. (2003). The effects of modified
school calendars on student achievement and on school and community attitudes.
Review of Educational Research, 73(1), 1-52.
Cotton, K. (1995). Effective schooling practices: A research synthesis. 1995 Update.
School Improvement Research Series. Northwest Regional Educational
Laboratory.
Creemers, B., & Werf, G. v. d. (2000). Economic viewpoints in educational
effectiveness: Cost-effectiveness analysis of an educational improvement project.
School Effectiveness and School Improvement, 11(3), 361-384.
Creemers, B.P.M. (1994). The Effective Classroom. London: Cassell.
Cremers-van Wees, L.M.C.M., Rekveld, I.J., Brandsma, H.P., & Bosker, R.J. (1996a).
Instrumenten voor kwaliteitszorg: inventarisatie en beschrijving. Enschede:
Universiteit Twente, Onderzoek Centrum Toegepaste Onderwijskunde.
Cremers-van Wees, L.M.C.M., Rekveld, I.J., Brandsma, H.P., & Bosker, R.J. (1996b).
Instrumenten voor zelfevaluatie. Beschrijving van 31 instrumenten. Enschede:
Universiteit Twente, Onderzoek Centrum Toegepaste Onderwijskunde.
Crinjen, A.A.M., & Feehan, M. (1998). The course and malleability of a reading
achievement in elementary school: The application of growth curve modeling in
the evaluation of a mastery learning intervention. Learning and Individual
Differences, 10(2), 137.
D'Agostino, J.V. (2000). Instructional and school effects on students' longitudinal
reading and mathematics achievements. School Effectiveness and School
Improvement, 11(2), 197-235.
D'Angelo, N., Reents, K., & Zomboracz, C. (1997). Improving reading achievement
through the use of parental involvement and paired reading. Chicago: Saint
Xavier University.
Darling-Hammond, L. (2000). Teacher Quality and Student Achievement: A Review of
State Policy Evidence. Education Policy Analysis Archives, 8(1). Available:
http://olam.ed.asu.edu/epaa/v8n1/.
372
Davidson, J., Elcock, J., & Noyes, P. (1996). A preliminary study of the effect of
computer-assissted practice on reading attainment. Journal of Research in
Reading, 19, 102-110.
Davies, J.K. (1972). Style and effectiveness in education and training: a model for
organizing, teaching and learning. Instructional Science, vol. 2.
De Fraine, B., Van Damme, J., Van Landeghem, G., Opdenakker, M.C., & Onghena, P.
(2003). The effect of schools and classes on language achievement. British
Educational Research Journal, 29(6), 841-859.
De Jong, R., Westerhof, K.J., & Kruiter, J.H. (2004). Empirical evidence of a
comprehensive model of school effectiveness: A multilevel study in mathematics
in the 1st year of junior general education in the Netherlands. School Effectiveness
and School Improvement, 15(1), 3-31.
De Klerk, L.F.W. (1985). ATI-onderzoek en differentiatie: een reactie. Pedagogische
Studiën, 62, 372-375.
Deal, T.E., & Peterson, K.D. (1999). Shaping School Culture. San Francisco: Jossey-
Bass.
Deinum, J. F. (2000). Schoolbeleid, instructie en leerresultaten. Groningen: GION.
Desimone, L. (1999). Linking parent involvement with student achievement: Do race
and income matter? Journal Of Educational Research, 93(1), 11-30.
Diedrich, M. (2005, in press). Connections between quality of school life and
democracy in German schools. To appear in Sliwka, A., & Diedrich, M. (Eds.),
International Perspective on Citizenship Education.
Dijkstra, N., Linde, A. van der, & Majoor, D. (2004). Kiezen voor kwaliteit.
Instrumenten de maat genomen. Den Haag: Q*Primair.
Doeleman, R., & Westerbeek, K. (2002). Evaluatie KEA; verslag van het kleinschalig
experiment achterstandsbetrijding in Rotterdam 1991-2000. Rotterdam: CED-
Groep.
Dougherty, K. (1981). After the fall: Research on school effects since the Coleman
report. Harvard Educational Review, 51, 301-308.
Dowdell, T. (1996). The effectiveness of Direct Instruction on the reading achievement
of sixth graders.
Doyle, W. (1985). Effective secondary classroom practices. In M.J. Kyle (Ed.),
Reaching for excellence. An effective schools sourcebook. Washington DC: US
Government Printing Office.
Doyle, W. (1986). Classroom organization and management. In M. C. Wittrock (Hrsg.),
Handbook of Research on Teaching [pp. 392-431]. New York: Macmillan.
Driessen, G., & Sleegers, P. (2000). Consistency of teaching approach and student
achievement: An empirical test. School Effectiveness and School Improvement,
11(1), 57-79.
Dunn, E. R. (1996). The effect of calendar configuration on elementary students'
373
achievement gains. Dissertation Abstracts International, 57-10, 4200.
Duurkoop, M., Majoor, D., & Pinth, M. de (2002). Kwaliteitszorg: Inventarisatie en
beschrijving van instrumenten. Retrieved from http://www.kwaliteitsring.nl
Edmonds, R.R. (1979). Effective schools for the urban poor. Educational Leadership,
37(1), 15-27.
Einsiedler, W., & Treinies, G. (1997). Effects of teaching methods, class effects, and
patterns of cognitive teacher-pupil interactions in an experimental study in
primary school classes. School Effectiveness and School Improvement, 8(3), 327-
353.
Elmore, R.F. and Associates (1990). Restructuring School; The Next Generation of
Educational Reform. San Francisco: Jossey Bass.
Elsäcker, W. v., & Verhoeven, L. (1997). Kleuters leren meer van voorlezen in kleine
groepen. Pedagogische studiën: tijdschrift voor onderwijskunde en opvoedkunde,
74(2), 117-129.
Emmelot, Y, Karsten, S, Ledoux, G., & Vermeulen, A. (2004). Ervaringen met het
vernieuwde onderwijstoezicht. Amsterdam: SCO-Kohnstamm Instituut.
Epstein, J.L., Herrick, S. C., & Coates, L. (1996). Effects of summmer home learning
packets on student achievement in language arts in the middle grades. Journal of
Early Adolescence, 15, 114-144.
European Union; Ministerie van Onderwijs, Cultuur en Wetenschap (2004).
Citizenship: made in Europe: living together starts at school. Den Haag:
Ministerie van OC&W.
Faerman, S.R., & Quinn, R.E. (1985). Effectiveness: the perspective from organization
theory. Review of Higher Education, 9, 83-100.
Fend, H. (1981). Theorie der Schule. München [etc.]: Urban & Schwarzenberg.
Foorman, B.R., Francis, D.J., Fletcher, J.M., Schatschneider, C., & Mehta, P. (1998).
The role of instruction in learning to read: Preventing reading failure in at-risk
children. Journal of Educational Psychology, 90(1), 37-55.
Fordham, P.E. (1993). Informal, non-formal and formal education programmes’ in
YMCA George Williams College ICE301 Lifelong Learning Unit 2, London:
YMCA George Williams College.
Fraser, B. J., Walberg, H. J., Welch, W. W., & Hattie, J. A. (1987). Syntheses of
educational productivity research. International Journal of Educational Research,
11, 145-252.
Freedman, M.P. (1997). Relationship among laboratory instruction, attitude toward
science, and achievement in science knowledge. Journal of Research in Science
Teaching, 34(4), 343-357.
Fuchs, D., Fuchs, L.S., Mathes, P.G., & Simmons, D.C. (1997). Peer-assisted learning
strategies: Making classrooms more responsive to diversity. American
Educational Research Journal, 34(1), 174-206.
374
Fuchs, L.S., Fuchs, D., Yazdian, L., & Powell, S.R. (2002). Enhancing first-grade
children's mathematical development with peer-assisted learning strategies.
School Psychology Review, 31(4), 569-583.
Fullan, M. (1998). The meaning of educational change: a quarter of a century of learning.
In A. Hargreaves, A. Lieberman, M. Fullan & D. Hopkins (Eds.), International
Handbook of Educational Change [pp. 214-228]. London: Kluwer.
Gage, N. (1965). Desirable behaviors of teachers. Urban Education, 1, 85-95.
Gagne, R.M (1972). Conditions of Learning. New York: Holt, Rinehart and Winston.
George, R., & Kaplan, D. (1998). A structural model of parent and teacher influences on
science attitudes of eighth graders: Evidence from NELS : 88. Science Education,
82(1), 93-109.
Ginsburg-Block, M.D., & Fantuzzo, J.W. (1998). An evaluation of the relative
effectiveness of NCTM standards-based interventions for low-achieving urban
elementary students. Journal of Educational Psychology, 90(3), 560-569.
Glass, G.V. (1972). The many faces of educational accountability. Phi Delta Kappan,
53, 636-639.
Good, Th.L., & Brophy, J. (1986). School effects. In: M.C. Wittrock (ed.), Handbook of
research on teaching [p. 328-375]. New York: McMillan Inc.
Gray, J. (2002). Jolts and reactions: Two decades of feeding back information on
schools’ performance. In A.J. Visscher & R. Coe (Eds.), School improvement
through performance feedback (pp. 143-162). Lisse/Abingdon/Exton/Tokyo:
Swets & Zeitlinger B.V.
Greenberg, E. (2004, April 12-16, 2004). Climates for learning. Paper presented at the
AERA, San Diego.
Griffith, J. (2003). Schools as organizational models: Implications for examining school
effectiveness. Elementary School Journal, 104(1), 29-47.
Grift, W. v.d., & Houtveen, A.A.M. (1999). Educational leadership and pupil
achievement in primary education. School Effectiveness and School Improvement,
10(4), 373-389.
Grift, W. v.d., Houtveen, T., & Vermeulen, C. (1997). Instructional climate in Dutch
secondary education. School Effectiveness and School Improvement, 8(4), 449-
462.
Gruehn, S. (1995). The compatibility of cognitive and noncognitive objectives of
instruction. Zeitschrift Für Padagogik, 41(4), 531-553.
Guthrie, J.T., Anderson, E., Alao, S., & Rinehart, J. (1999). Influences of concept-
oriented reading instruction on strategy use and conceptual learning from text.
Elementary School Journal, 99(4), 343-366.
Guthrie, J.T., Schafer, W.D., & Huang, C. W. (2001). Benefits of opportunity to read
and balanced instruction on the NAEP. Journal of Educational Research, 94(3),
145-162.
375
Guthrie, J.T., Van Meter, P., Hancock, G.R., Alao, S., Anderson, E., & McCann, A.
(1998). Does concept-oriented reading instruction increase strategy use and
conceptual learning from text? Journal of Educational Psychology, 90(2), 261-
278.
Guthrie, J.T., VanMeter, P., McCann, A.D., Wigfield, A., Bennett, L., Poundstone,
C.C., Rice, M.E., Faibisch, F.M., Hunt, B., & Mitchell, A.M. (1996). Growth of
literacy engagement: Changes in motivations and strategies during concept-
oriented reading instruction. Reading Research Quarterly, 31(3), 306-332.
Guthrie, J.T., Wigfield, A., & Von Secker, C. (2000). Effects of integrated instruction
on motivation and strategy use in reading. Journal of Educational Psychology,
92(2), 331-341.
Haaijer, R., & Velzen, J.D. van (2002). De stand van zaken van kwaliteitszorg bij
scholen voor voortgezet onderwijs. De resultaten van de eerste vervolgmeting.
Hoorn: Van Beekveld & Terpstra.
Haenn, J. F. (1996). Evaluating the promise of single-track year-round schools. ERS
Spectrum, Fall, 27-35.
Hahn,C.L. (1999). Citizenship Education: an empirical study or policy, practices and
outcomes. Oxford Review of Education, 25(1+2), 231-250.
Hallinger, P., Bickman, L., & Davis, K. (1996). School context, principal leadership,
and student reading achievement. Elementary School Journal, 96(5), 527-549.
Hamilton, L.S., Nussbaum, E.M., Kupermintz, H., Kerkhoven, J.I.M., & Snow, R.E.
(1995). Enhancing the validity and usefulness of large-scale educational
assessments .2. Nels-88 Science Achievement. American Educational Research
Journal, 32(3), 555-581.
Hampton, F.M., Mumford, D.A., & Bold, L. (1998). Parental involvement in inner-city
schools: The project fast extended family approach to success. Urban Education,
33, 410-427.
Hanushek, E.A. (1989). Expenditures, efficiency, and equity in education: the federal
Government’s role. The American Economic Review, 79(2), p. 46.
Hanushek, E.A. (1995). Interpreting Recent Research on Schooling in Developing
Countries. The World Bank Research Observer, 10(227-46).
Hanushek, E.A. (1997). Assessing the effects of school resources on student performance:
an update. Educational Evaluation and Policy Analysis, 19, 141-164.
Hardre, P.L., & Reeve, J. (2003). A motivational model of rural students' intentions to
persist in, versus drop out. of high school. Journal of Educational Psychology,
95(2), 347-356.
Hardy, I., Jonen, A., Möller, K., & Stern, E. (2004). Die Integration von
Repräsentationsformen in den Sachunterricht der Grundschule. In J. Doll & M.
Prenzel (Eds.), Bildungsqualität von Schule: Lehrerprofessionalisierung,
376
Unterrichtsentwicklung und Schülerförderung als Strategien der
Qualitätsverbesserung (pp. 267-283). Münster: Waxmann.
Haste, H. (2001). Ambiguity, autonomy, and agency: Psychological challenges to new
competence. In D.S. Rychen & L.H. Salganik (Eds.), Defining and selecting key
competencies [pp. 93-120]. Göttingen, Germany: Hogrefe & Huber.
Heck, R.H., & Marcoulides, G. A. (1996). School culture and performance: Testing the
invariance of an organizational model. School Effectiveness And School
Improvement, 7(1), 76-95.
Hedges, L.V., Laine, R.D., & Greenwald, R. (1994). Does money matter? A meta-analysis
of studies of the effects of differential school inputs on student outcomes.
Educational Researcher, 23, 3, 5-14.
Helmke, A., & Weinert, F.E. (1997). Unterrichtsqualität und Leistungsentwicklung:
Ergebnisse aus dem SCHOLASTIK- Projekt. In F.E. Weinert & A. Helmke
(Eds.), Entwicklung im Grundschlualter (pp. 241-251).
Henderson, R.W., & Landesman, E.M. (1995). Effects of thematically integrated
mathematics instruction on students of mexican descent. Journal of Educational
Research, 88(5), 290-300.
Hendriks, M. (2000). Kwaliteitszorg Voortgezet Onderwijs: Instrumenten en
Organisaties. Utrecht: Q5.
Hendriks, M. (2001). Systemen voor kwaliteitszorg in het primair en secundair
onderwijs. In R.J. Bosker (Red.), Kwaliteitszorg. Onderwijskundig Lexicon III
(pp. 47-72). Alphen aan den Rijn: Samsom.
Hendriks, M.A., Doolaard, S., & Bosker, R.J. (2002). Using school effectiveness as a
knowledge base for self-evaluation in Dutch schools: the ZEBO-project. In A.J.
Visscher & R. Coe (eds.). School improvement through performance feedback
(pp. 115-142). Lisse: Swets & Zeitlinger B.V.
Henkenborg, P. (1995). Politische Bildung durch Demokratie-Lernen im Schulalltag. In
W. Sander (Hrsg.), Handbuch politische Bildung [pp. 241-258]. Schwalbach/Ts.:
Wochenschau Verlag.
Henkenborg, P. (1997). Die Selbsterneuerung der Schule als Kultur der Anerkennung.
Politische Bildung, 30(3), 60-89.
Hill, P.W., & Rowe, K.J. (1998). Modelling student progress in studies of educational
effectiveness. School Effectiveness And School Improvement, 9(3), 310-333.
Hofman, R.H., & Hofman, W.H.A. (2003). Ontwerp van een beoordelingskader voor
zelfevaluatie-instrumenten voor scholen. Eindrapport.
Hofman, R.H., Dijkstra, N.J., Hofman, W.H.A., & Boom, J. de (2004). Kwaliteitszorg in
het primair onderwijs. Deelstudie 1 – Peiling 2003/2004. Groningen: GION.
Hofman, R.H., Hofman, W.H.A., & Guldemond, H. (1999). Social and cognitive
outcomes: A comparison of contexts of learning. School Effectiveness and School
Improvement, 10(3), 352-366.
377
Hofman, R.H., Hofman, W.H.A., & Guldemond, H. (2001). The effectiveness of
cohesive schools. International Journal of Leadership in Education, 4(2), 115-
135.
Hofman, R.H., Hofman, W.H.A., Dijkstra, N.J., & Boom, J. de (2004). Q*Primair Monitor
Kwaliteitszorg. Peildatum eind 2003. Groningen/Rotterdam: GION/RISBO.
Hogan, K. (1999). Thinking aloud together: A test of an intervention to foster students'
collaborative scientific reasoning. Journal of Research in Science Teaching,
36(10), 1085-1109.
Hopkins, D. (2001). School improvement for real. London: Routledge Falmer.
Hopkins, D. (2002). Educational Innovation: Generic Lessons Learned from (a)
Regional Practice. Keynote Paper Presentation at the DECIDE International
Seminar, University of Twente the Netherlands.
Hopkins, K.B., McGillicuddyDeLisi, A.V., & DeLisi, R. (1997). Student gender and
teaching methods as sources of variability in children's computational arithmetic
performance. Journal of Genetic Psychology, 158(3), 333-345.
Horsman, K. (2001). Het ABC van de kwaliteitszorg. Zelfevaluatie op alle niveaus.
Qvijver (2)2, pp. 3-5.
Horsman, K. (z.j.) De scholen centraal. Een goed en aanvaardbaar systeem van
kwaliteitszorg. De laatste Qvijver, een boekje van eigen deeg, (5)5. (CD-ROM,
available from [Q5, Burg. Reigerstraat 74, Utrecht]).
Houghton, S., Litwin, M., & Carroll, A. (1995). Peer mediated intervention in reading
instruction using pause, prompt and praise. Educational Studies, 21(3), 361-377.
Houtveen, A.A.M. (2002). Begrijpend leesonderwijs dat werkt: Evaluatie van het
adaptieve schoolverbeteringsproject "kwaliteitsverbetering begrijpend lezen".
Utrecht: ICO-ISOR.
Houtveen, A.A.M., Booij, N., de Jong, R., & van de Grift, W. (1999). Adaptive
instruction and pupil achievement. School Effectiveness and School Improvement,
10(2), 172-192.
Houtveen, A.A.M., Booij, N., Jong, R. d., & van de Grift, W. (1996). Adaptief
onderwijs en leerlingresultaten. Pedagogische studiën: tijdschrift voor
onderwijskunde en opvoedkunde, 73(6), 422-433.
Houtveen, A.A.M., Pijl, S.J., Pijl, Y.J., Reezigt, G.J., & Vermeulen, C.J. (1998).
Adaptief onderwijs: Stand van zaken in het WSNS-proces. De Lier: Academisch
Boekencentrum.
Inspectie van het Onderwijs/Inspectorate (2002). Waarderingskader voor het PO en VO.
Utrecht: Inspectie van het Onderwijs.
Inspectie van het Onderwijs (2002a). Veranderend toezicht. De inspectie van het
onderwijs en de wet op het onderwijstoezicht. Den Haag: SDU.
378
Inspectie van het Onderwijs (2002b). Toezichtkader Primair Onderwijs. Utrecht:
Inspectie van het onderwijs. Retrieved from
http://www.owinsp.nl/functie_en_taken/tkpo.pdf
Inspectie van het Onderwijs (2003). Onderwijsverslag over het jaar 2002. Den Haag:
SDU.
Inspectie van het Onderwijs (2004). Onderwijsverslag over het jaar 2002-2003. Den
Haag: SDU.
Inspectie van het Onderwijs/Inspectorate (2005). Waarderingskader voor het PO en VO.
Utrecht: Inspectie van het Onderwijs.
Jacob, B.A., & Lefgren, L. (2004). The impact of teacher training on student
achievement - quasi-experimental evidence from school reform efforts in
Chicago. Journal Of Human Resources, 39(1), 50-79.
Jacobson, C., & Lehrer, R. (2000). Teacher appropriation and student learning of
geometry through design. Journal for Research in Mathematics Education, 31(1),
71-88.
James, E. (1994). The public-private division of responsibility in education. In W.K.
Cummings & A. Riddell (eds.), Alternative policies for the finance, control, and
delivery of basic education. Special issue of the International Journal of
Educational Research, 21(8), 777-783.
Janssens, F.J.G. (2005). Toezicht op de kwaliteit van het onderwijs: In: B.P.M.
Creemers et al, Handboek Schoolorganisatie. Alphen a/d Rijn: Kluwer.
Joint Committee on Standards for Educational Evaluation (1988). The Personnel
Evaluation Standards: How to assess Evaluation of Educational Programs.
Thousand Oaks, CA (etc.): Sage.
Joint Committee on Standards for Educational Evaluation, James R. Sanders, Chair
(1994). The Program Evaluation Standards: How to assess Systems for
Evaluating Educators. Newbury Park, CA [etc.]: Sage.
Joint Committee on Standards for Educational Evaluation, Arlon R. Gullickson, Chair
(2003). The student evaluation standards: How to improve evaluation of students.
Thousand Oaks, CA (etc.): Corwin Press.
Jong, R. d., Westerhof, K.J., & Kruiter, J.H. (2004). Empirical evidence of a
comprehensive model of school effectiveness: A multilevel study in mathematics
in the 1st year of junior general education in The Netherlands. School
Effectiveness And School Improvement, 15(1), 3-31.
Jovanovic, J., & King, S.S. (1998). Boys and girls in the performance-based science
classroom: Who's doing the performing? American Educational Research
Journal, 35(3), 477-496.
Kamphof, G. (2001). Kwaliteitszorg in fasen. Aanbevelingen voor het ontwikkelen van
een meer integraal en systematisch kwaliteitsbeleid. (Concept van een brochure).
Retrieved from http://www.kwaliteitsring.nl
379
Karstanje, P.N. (1997). Deregulering en kwaliteitszorg: Stand van zaken in de
Nederlandse onderwijswetgeving. Nederlands Tijdschrift voor Onderwijsrecht en
Onderwijsbeleid, april, p. 3-29.
Klieme, E. & Rakoczy, K. (2003). Unterrichtsqualität aus Schülerperspektive:
Kulturspezifische Profile, regionale Unterschiede und Zusammenhänge mit
Effekten von Unterricht. In J. Baumert, C. Artelt, E. Klieme, M. Neubrand, M.
Prenzel, U. Schiefele, W. Schneider, K.-J. Tillmann (Eds.), PISA 2000. Ein
differenzierter Blick auf die Länder der Bundesrepublik Deutschland [pp. 334-
359] . Opladen: Leske + Budrich.
Klieme, E., & Reusser, K. (2003). Unterrichtsqualität und mathematisches Verständnis.
Unterrichtswissenschaft, 31(3), 194-205.
Kneese, C. (1996). Impact of the year-round calendar on student achievement in
alameda unified school district.
Kobarg, M. (2004). Die Bedeutung prozessorientierter Lernbegleitung für kognitive und
motivationale Prozesse im Physikunterricht – eine Videostudie. Unveröffentlichte
Diplomarbeit. Kiel: IPN / Fachbereich Psychologie der CAU Kiel.
Koers, VO (2004). De leerling geboeid, de school ontketend.
Kramarski, B., & Mevarech, Z.R. (1997). Cognitive-metacognitive training within a
problem-solving based Logo environment. British Journal of Educational
Psychology, 67, 425-445.
Kramarski, B., Mevarech, Z.R., & Lieberman, A. (2001). Effects of multilevel versus
unilevel metacognitive training on mathematical reasoning. Journal of
Educational Research, 94(5), 292-300.
Kroesbergen, E.H., Van Luit, J.E.H., & Maas, C.J.M. (2004). Effectiveness of explicit
and constructivist mathematics instruction for low-achieving students in the
Netherlands. Elementary School Journal, 104(3), 233-251.
Kuijpers, M.A.C.T. (2003). Loopbaanontwikkeling. Onderzoek naar ‘Competenties’.
Enschede: Twente University Press, dissertatie.
Kuklinski, M.R., & Weinstein, R.S. (2001). Classroom and developmental differences
in a path model of teacher expectancy effects. Child Development, 72(5), 1554-
1578.
Kulik, C.L.C., & Kulik, J.A. (1982). Effects of ability grouping on secondary school
students: a meta-analysis of research findings. American Educational Research
Journal, 19, 415-428.
Kunter, M. (2004). Multiple Ziele im Mathematikunterricht. Ph.D. Thesis. Berlin: Freie
Universität Berlin.
Kupermintz, H., Ennis, M.M., Hamilton, L.S., Talbert, J.E., & Snow, R.E. (1995).
Enhancing the validity and usefulness of large-scale educational assessments .1.
Nels-88 mathematics achievement. American Educational Research Journal,
32(3), 525-554.
380
Kupermintz, H., & Snow, R.E. (1997). Enhancing the validity and usefulness of large-
scale educational assessments .3. NELS:88 mathematics achievement to 12th
grade. American Educational Research Journal, 34(1), 124-150.
Kyle, M.J. (Ed.) (1985). Reaching for excellence. An effective schools sourcebook.
Washington D.C.: U.S. Government Printing Office.
Kyriakides, L., Campbell, R. J., & Gagatsis, A. (2000). The significance of the
classroom effect in primary schools: An application of Creemers' comprehensive
model of educational effectiveness. School Effectiveness And School
Improvement, 11(4), 501-529.
Labudde, P., & Pfluger, D. (1999). Physics instruction of the upper secondary level: An
empirical analysis of the learning-teaching-culture from a constructivist
perspective. Zeitschrift für Didaktik der Naturwissenschaften, 2.
Lam, J.F. (1996). Tijd en kwaliteit in het basisonderwijs. Ontwikkeling van een
onderzoeksinstrumentarium. [Academisch Proefschrift]. Enschede: Universiteit
Twente.
Lazarowitz, R., Baird, J.H., & Bowlden, V. (1996). Teaching biology in a group
mastery learning mode: High school students' academic achievement and affective
outcomes. International Journal of Science Education, 18(4), 447-462.
Lee, V.E., & Loeb, S. (2000). School size in Chicago elementary schools: Effects on
teachers' attitudes and students' achievement. American Educational Research
Journal, 37(1), 3-31.
Leithwood, K., & Jantzi, D. (2000). Principal and teacher leadership effects: A
replication. School Leadership and Management, 20(4), 415-434.
Leithwood, K., Jantzi, D., & Mascall, B. (2000). Large-scale reform: What works,
Unpublished manuscript. Toronto: OISE, University of Toronto.
Leune, J.M.G. (1987). Besluitvorming en machtsverhoudingen in het Nederlandse
onderwijsbestel. (Decision-making and the balance of power in Dutch education).
In: J.A. van Kemenade et al. (Red.), Onderwijs, bestel en beleid deel 2.
Groningen: Wolters-Noordhoff.
Levačić, R., Steele, F., Smees, R., & Malmberg, L. (2003, 11-13 September). The
relationship between school climate & head teacher leadership, and pupil
attainment: Evidence from a sample of English secondary schools. Paper
presented at the British Educational Research Association Annual Conference,
Edinburgh.
Levine, D.K., & Lezotte, L.W. (1990). Unusually Effective Schools: A Review and
Analysis of Research and Practice. Madison, Wise: Nat. Centre for Effective
Schools Research and Development.
Levy, F., & Murnane, R.J. (2001). Key competencies critical to economic success. In
D.S. Rychen & L.H. Salganik (Eds.), Defining and selecting key competencies
[pp. 151-173]. Göttingen, Germany: Hogrefe & Huber.
381
Lortie, D.C. (1973). Observations on teaching as work. In R.M.W. Travers (Ed.),
Second Handbook of Research on Teaching. Chicago: Rand McNally.
Lou, Y., Abrami, P. C., Spence, J. C., Poulsen, C., Chambers, B., & d'Apollonia, S.
(1996). Within-class grouping: A meta-analysis. Review of Educational Research,
66(4), 423-458.
Lou, Y.P., Abrami, P.C., & Spence, J.C. (2000). Effects of within-class grouping on
student achievement: An exploratory model. Journal Of Educational Research,
94(2), 101-112.
Lowther, D.L., Ross, S. M., & Morrison, G.M. (2003). When each one has one: The
influences on teaching strategies and student achievement of using laptops in the
classroom. Etr&D-Educational Technology Research and Development, 51(3),
23-44.
Luhmann, N. (1995). Social systems. Stanford, Ca.: Stanford University Press.
Luyten, H., & Jong, R. de (1998). Parallel classes: Differences and similarities. Teacher
effects and school effects in secondary schools. School Effectiveness and School
Improvement, 9(4), 437-473.
Mac Iver, M.A., Kemper, E., & Stringfield, S. (2003). The Baltimore Curriculum
Project: Final report of the four-year evaluation study. Baltimore, M.D.: Center
for Research on the Education of Students Placed At Risk.
MacPherson, R.J.S. (1990). Educative Accountability: Theory, Practice, Policy and
Research in Educational Administration. Oxford: Pergamon.
Marks, H.M. (2000). Student engagement in instructional activity: Patterns in the
elementary, middle, and high school years. American Educational Research
Journal, 37(1), 153-184.
Marks, H.M., & Printy, S.M. (2003). Principal leadership and school performance: An
integration of transformational and instructional leadership. Educational
Administration Quarterly, 39(3), 370-397.
Martinez, J.G.R., & Martinez, N.C. (1999). Teacher effectiveness and learning for
mastery. Journal of Educational Research, 92(5), 279-285.
Maslowski, R. (1997). Schoolcultuur: kenmerken en veranderingsmogelijkheden
[School culture: characteristics and levers for change]. In B. P. M. Creemers et al.
(Eds.), Handboek Schoolorganisatie en Onderwijsmanagement (pp. B1400/1-
B1400/25). Alphen aan den Rijn: Samsom Tjeenk Willink.
Mathes, P.G., Howard, J.K., Allen, S.H., & Fuchs, D. (1998). Peer-assisted learning
strategies for first-grade readers: Responding to the needs of diverse learners.
Reading Research Quarterly, 33(1), 62-94.
McEwan, P.J. (2003). Peer effects on student achievement: Evidence from Chile.
Economics Of Education Review, 22(2), 131-141.
382
McGuinness, D., McGuinness, C., & Donohue, J. (1995). Phonological training and the
alphabet principle - Evidence for reciprocal causality. Reading Research
Quarterly, 30(4), 830-852.
McMeekin, R.W. (2003). Networks of schools. Education Policy Archives, 11, 1-16.
Medley, D., & Mitzel, H. (1963). Measuring classroom behavior by systematic
observation. In: N.L. Gage (Ed.), Handbook of research on teaching. Chicago:
Rand McNally.
Meijnen, G.W., Lagerweij, N.W., & Jong, P.F. (2003). Instruction characteristics and
cognitive achievement of young children in elementary schools. School
Effectiveness and School Improvement, 14(2), 159-187.
Mevarech, Z.R. (1999). Effects of metacognitive training embedded in cooperative
settings on mathematical problem solving. Journal of Educational Research,
92(4), 195-205.
Mevarech, Z.R., & Kramarski, B. (1997). IMPROVE: A multidimensional method for
teaching mathematics in heterogeneous classrooms. American Educational
Research Journal, 34(2), 365-394.
Mevarech, Z.R., & Kramarski, B. (2003). The effects of metacognitive training versus
worked-out examples on students' mathematical reasoning. British Journal of
Educational Psychology, 73, 449-471.
Mijs, D., Houtveen, A.A.M., & Vernooy, C.G.T. (2001). Op weg naar adaptief
onderwijs bij beginnend lezen. Utrecht: ISOR.
Miles, M, (1998). Finding Keys to School Change: A 40-Yeat Odyssey. In A.
Hargreaves, A. Liebermann, M. Fullan & D. Hopkins (Eds.), International
Handbook of Educational Change. Dordrecht, Boston, London: Kluwer.
Miller, S.D., & Meece, J.L. (1997). Enhancing elementary students' motivation to read
and write: A classroom intervention study. Journal of Educational Research,
90(5), 286-299.
Ministerie van Onderwijs, Cultuur &Wetenschappen (1995). De school als lerende
organisatie. Kwaliteitsbeleid op scholen voor primair en voortgezet onderwijs.
[The school as a learning organization: Quality policy in schools for primary and
secondary education.] Den Haag: SDU, DOP.
Ministerie van Onderwijs, Cultuur & Wetenschappen; Ministerie van Landbouw,
Natuur en Visserij (2002). Wet op het Onderwijstoezicht. Den Haag: SDU.
Mitchell, C., & Sackney, L. (2000). Profound improvement: building capacity for a
learning community. Lisse: Swets & Zeitlinger.
Möller, K., Jonen, A., Hardy, I., & Stern, E. (2002). Die Förderung von
naturwissenschaftlichem Verständnis bei Grundschulkindern durch Strukturierung
der Lernumgebung. Zeitschrift für Pädagogik, 45. Beiheft, 176-191.
Muijs, D., & Reynolds, D. (2000). School effectiveness and teacher effectiveness in
mathematics: Some preliminary findings from the evaluation of the mathematics
383
enhancement programme (primary). School Effectiveness and School
Improvement, 11(3), 273-303.
Muijs, D., & Reynolds, D. (2001). Effective Teaching. London: Paul Chapman.
Muijs, D., & Reynolds, D. (2003). The effectiveness of the use of learning support
assistants in improving the mathematics achievement of low achieving pupils in
primary school. Educational Research, 45(3), 219-230.
Murnane, R.J. (1981). Interpreting the evidence on school effectiveness. Teachers
College Record, 83, 19-35.
Murphy, J. (1993). Restructuring Schooling: the equity infrastructure. School
Effectiveness and School Improvement, 4(2), 111-130.
Muthukrishna, N., & Borkowski, J.G. (1995). How learning contexts facilitate strategy
transfer. Applied Cognitive Psychology, 9(5), 425-446.
National Center for Education Statistics (NCS) (2002). Fundamental problems in the
measurement of instructional processes. Washington D.C.: NCS.
Neufeld, E., Farrar, E., & Miles, M.B. (1983). A review of effective schools research:
The message for secondary schools. Washington D.C.: National Commission on
Excellence in Education.
Nolen, S. B. (2003). Learning environment, motivation, and achievement in high school
science. Journal of Research in Science Teaching, 40(4), 347-368.
North, D.C., (1990). Institutions, Institutional Change and Economic Performance.
Cambridge, U.K.: Cambridge University Press.
Oberon (2002). Nulmeting kwaliteitszorg basisonderwijs. Eindrapportage. Utrecht:
Oberon.
OECD (2000). Literacy Skills for the World of Tomorrow: Further Results from PISA
2000. Paris: OECD.
OECD (2001). The well-being of nations. The role of human and social capital. Paris:
OECD.
OECD (2003). Education at a glance. Paris: OECD.
OECD (2003). Learners for Life. Student Approaches to Learning. Results from PISA
2000, Paris: OECD.
Oesterreich, D. (2003). Offenes Diskussionskklima im Unterricht und politische
Bildung von Jugenlichen. Zeitschrift für Pädagogik, 49(Heft 6), 817-836.
Ofsted (2003). Inspecting schools – the Framework for Inspecting Schools in England
from September 2003. London: Office for Standards in Education.
Ofsted (2005). Self-evaluation form for primary schools (with and without nursery
provision) middle schools (deemed primary). Retrieved from:
http://forms.ofsted.gov.uk/edc2003/sefpri.pdf
Ofsted (March 2005). The Framework for inspecting schools from September 2005.
Okpala, C.O., Okpala, A.O., & Smith, F.E. (2001). Parental involvement, instructional
expenditures, family socioeconomic attributes, and student achievement. Journal
384
Of Educational Research, 95(2), 110-115.
Olina, Z., & Sullivan, H.J. (2002). Effects of classroom evaluation strategies on student
achievement and attitudes. Etr&D-Educational Technology Research And
Development, 50(3), 61-75.
Opdenakker, M.C., & Damme, J. van (2001). Relationship between school composition
and characteristics of school process and their effect on mathematics achievement.
British Educational Research Journal, 27(4), 407-432.
Orbach, E. (1998). How to Assess the Capacity of Client Government to Implement an
Education Development Project. Washington: World Bank.
Overmaat, M., Ledoux, G., & Koopman, P. (1997). Adaptief onderwijs in Prima-i:
Stand van zaken en effecten. Amsterdam: SCO-Kohnstamm Instituut.Pauli, C.,
Reusser, K., Waldis, M., & Grob, U. (2003). "Enrichment of learning settings and
teaching methods" in mathematics lessons in the German-speaking part of
Switzerland. Unterrichtswissenschaft, 31(4), 291-320.
Paloczy, S.T. (1997). The impact of the elementary school year-round education pilot
program on student achievement and behavior in the Harlandale Independent
School District. Dissertations Abstracts International, 58-07, 2539.
Parcel, T.L., & Dufur, M.J. (2001). Capital at home and at school: Effects on student
achievement. Social Forces, 79(3), 881-911.
Pauli, C., Reusser, K., Waldis, M., & Grob, U. (2003). "Enrichment of learning settings
and teaching methods" in mathematics lessons in the German-speaking part of
Switzerland. Unterrichtswissenschaft, 31(4), 291-320.
Perry, N.E. (1998). Young children's self-regulated learning and contexts that support it.
Journal of Educational Psychology, 90(4), 715-729.
Phillips, M. (1997). What makes schools effective? A comparison of the relationships of
communitarian climate and academic climate to mathematics achievement and
attendance during middle school. American Educational Research Journal, 34(4),
633-662.
Postlethwaite, T.N., & Ross, K.N. (1992). Effective Schools in Reading. Implications for
Educational Planners. The Hague: IEA.
Prenzel, M. (2000). Steigerung der Effizienz des mathematisch-naturwissenschaftlichen
Unterrichts: Ein Modellversuchsprogramm von Bund und Ländern.
Unterrichtswissenschaft, 28(2), 103-126.
Prenzel, M., & Ostermeier, C. (in press). What can we learn from different forms of
evaluation: experiences of quality development program in mathematics and
science instruction. In J. Bennett, J. Holeman, & R. Millar (Hrsg.), Making the
difference: evaluation as a tool for improving science education. Münster:
Waxmann.
Pugh, G., & Telhaj, S. (2003, 17-20 September). Attainment effects of school
enmeshment with external communities: Community policy, church/religious
385
influence, and TIMSS-R mathematics scores in Flemish secondary schools. Paper
presented at the European Conference on Educational Research, Hamburg.
Purkey, S.C., & Smith, M.S. (1983). Effective schools: a review. The Elementary
School Journal, 83(4), 427-452.
Q*Primair (2003). Zelfevaluatie, visitatie en proportioneel toezicht. Projectplan.
Retrieved from:
http://www.qprimair.nl/ventura/engine.php?Cmd=seepicture&P_site=345&P_self
=1254&Random=825455206.
Q*Primair (2005). Verslag van de Ziezo-conferentie op 20 januari 2005. Retrieved
from:
http://www.qprimair.nl/ventura/engine.php?Cmd=seepicture&P_site=345&P_self
=1394&Random=2073455143.
Q5 (2003). Bijlage 1: Bereikte doelen. Internal publication. (Available from [Q5, Burg.
Reigerstraat 74, Utrecht]).
Quinn, R.E. & J. Rohrbaugh (1983). Spatial model of effectiveness criteria towards a
competing values approach to organizational analysis. Management Science, 29,
363-377.
Ralph, J.H., & Fennessey, J. (1983). Science or reform: some questions about the
effective schools model. Phi Delta Kappan, 64(10), 689-695.
Ramsden, J.M. (1997). How does a context-based approach influence understanding of
key chemical ideas at 16+? International Journal of Science Education, 19(6),
697-710.
Rand News Release (July 25, 2000) Explaining achievement gains in North Carolina
and Texas.
Raudenbush, S. (1994). Searching for balance between a priori and post hoc model
modification: Is a ‘general approach’ desirable? School effectiveness and school
improvement; 5(2), 196-198.
Reece, J.L., Myers, C.L., & Nofsinger, C.O. (2000). Retention of academic skills over
the summer months in alternative and traditional schools. Journal of Research and
Development in Education, 33, 166-174.
Reezigt, G.J. (1993). Effecten van differentiatie op de basisschool (Effects of grouping
in primary education). Groningen: RION.
Reezigt, G.J., Guldemond, H., & Creemers, B.P.M. (1999). Empirical validity for a
comprehensive model on educational effectiveness. School Effectiveness and
School Improvement, 10(2), 24.
Reezigt, G.J., Houtveen, A.A.M., Grift, W. v.d., & Guldemond, H. (2002).
Ontwikkelingen in en effecten van adaptief onderwijs in de klas en integrale
leerlingenzorg op schoolniveau. Groningen: GION.
Reinhardt, S., & Tillmann, F. (2002). Politische Orientierungen, Beteiligungsformen
und Wertorientierungen. In H.-H. Krüger, S. Reinhardt, C. Kötters-König, N.
386
Pfaff, R. Schmidt, A. Krappidel & F. Tillmann (Eds.), Jugend und Demokratie -
Politische Bildung auf dem Prüfstand [pp. 43-74]. Opladen: Leske + Budrich.
Reitsma, P., Wesseling, R., & Stiva, F. (1997). Het effect van computer-ondersteunde
oefeningen in klanksynthese bij kleuters. Pedagogische studiën: tijdschrift voor
onderwijskunde en opvoedkunde, 74(1), 1-20.
Renkema, G. (2002). Kwaliteitszorg, wat levert het op? Basisschoolmanagement, (15)8.
Alphen aan den Rijn: Samsom.
Reynolds, D., Hopkins, D., & Stoll, L. (1993). Linking school effectiveness knowledge
and school improvement practice: towards a synergy. School Effectiveness and
School Improvement, 4(1), 37-58.
Rideout, W.M., & Ural, P. (1993). Centralized and decentralized models of education:
comparative studies. Development Bank of Southern Africa. Centre for Policy
Analysis.
Roby, D.E. (2004). Research on school attendance and student achievement: A study of
Ohio schools. Educational Research Quarterly, 28(1), 3-14.
Rosenshine, B. (1979). Content, time, and direct instruction. In P. Peterson & H. J.
Walberg (Hrsg.), Research on teaching: Concepts, findings, and implications.
Berkeley: McCutchan.
Rosenshine, B., & Furst, N. (1973). The use of direct observation to study teaching. In:
R.M. Travers (Ed.), Second Handbook of Research on Teaching. Chicago: Rand
McNally.
Rosenthal, R. (1994). Parametric measures of effect size. In H. Cooper & L. V. Hedges
(Eds.), The handbook of research synthesis [pp. 231-244[. New York: Sage.
Rosenthal, R., & Rubin, D.B. (1982). Comparing effect sizes of independent studies.
Psychological Bulletin, 92(2).
Rossi, P.H., Freeman, H.E., & Lipsey, M.W. (1999). Evaluation: a systematic
approach. Thousand Oaks, Ca: Sage Publications.
Rowe, K.K.J., & Hill, P.W. (1998). Modeling educational effectiveness in classrooms:
The use of multi-level structural equations to model students' progress.
Educational Research and Evaluation, 4(4), 307-347.
Rust-Eft, D., & Preskill, H. (2001). Evaluation in organizations. A systematic approach
to enhancing learning, performance and change. Cambridge/Massachusetts:
Perseus publishing.
Rutter, M. (1983). School effects on pupil progress: research findings and policy
implications. Child Development, 54(1), 1-29.
Rychen, D.S. (2001). Introduction. In D.S. Rychen & L.H. Salganik (Eds.), Defining
and selecting key competencies [pp. 1-15]. Göttingen, Germany: Hogrefe &
Huber.
Sacks, P. (1999) Standardized Minds. Perseus Books. Cambridge, Massachusetts
387
Sammons, P., Hillman, J., & Mortimore, P. (1995). Key characteristics of effective
schools: A review of school effectiveness research. London: OFSTED.
Scheerens, J. (1987). Enhancing educational opportunities for disadvantaged learners.
Amsterdam: North-Holland Publishing Comp.
Scheerens, J. (1989). Wat maakt scholen effectief? Samenvatting en analyse van
onderzoeksresultaten. Lisse: Zwets & Zeitlinger, SVO-reeks.
Scheerens, J. (1990). School effectiveness research and the development of process
indicators of school functioning. School Effectiveness and School Improvement, 1,
61-80. Lisse: Zwets & Zeitlinger.
Scheerens, J. (1992). Effective Schooling, Research, Theory and Practice. London:
Cassell.
Scheerens, J. (1997). Conceptual models and theory-embedded principles on effective
schooling. School Effectiveness and School Improvement, 8(3), 269-310.
Scheerens, J. (2000). How to Measure the Evaluation Capacity of National Education
Systems. Internal document OECD, Paris.
Scheerens, J. (2000). Improving school effectiveness. Paris: UNESCO, IIEP,
Fundamentals of Educational Planning series no. 68.
Scheerens, J. (2003). Conceptual framework for the World Education Indicator survey on
schools and teachers. Paper for the OECD.
Scheerens, J. (2004). Critical issues in school effectiveness research. Jacobs Foundation
Conference on Educational Influences, Schloß Marbach, October. 21-23.
Scheerens, J., & Bosker, R.J. (1997). The Foundations of Educational Effectiveness.
Oxford: Elsevier Science Ltd.
Scheerens, J., & Creemers, B.P.M. (1996). School effectiveness in the Netherlands; the
modest influence of a research programme. School Effectiveness and School
Improvement, 7, 181-195.
Scheerens, J., & Hendriks, M. (2002). School self evaluation in the Netherlands. In D.
Nevo (Ed.), School-Based Evaluation: An International Perspective, vol. 8, pp.
113-143.
Scheerens, J., Glas, C., & Thomas, S.M. (2003). Educational evaluation, assessment
and monitoring. A systemic approach. Lisse: Swets & Zeitlinger.
Scheerens, J., Vermeulen, C.J.A.J., & Pelgrum, W.J. (1989). Generalizability of school
and instructional effectiveness indicators across nations. In: B.P.M. Creemers & J.
Scheerens (eds.), Development in school effectiveness research. Special issue of
the International Journal of Educational Research, 13(7), 789-800. Oxford:
Pergamon Press.
Schulmeiß, I. (2004). Vermischung von Lern- und Leistungssituationen und ihre
Bedeutung für die Qualität der Lernmotivation - eine Videostudie im
Physikunterricht. Kiel: IPN, Institut für Psychologie der CAU Kiel
(unveröffentlichte Diplomarbeit).
388
Segers, E., Verhoeven, L., Boot, I., Berkers, I., & Vermeer, A. (2001). ICT-
ondersteuning van de woordenschat van allochtone kleuters. Pedagogische
studiën: tijdschrift voor onderwijskunde en opvoedkunde, 78(5), 287-297.
Seidel, T. (2003). Lehr-Lernskripts im Unterricht. Münster: Waxmann.
Seidel, T., & Prenzel, M. (2003). Mit Fehlern umgehen - Zum Lernen motivieren.
Praxis der Naturwissenschaften - Physik in der Schule, 52(1), 30-34.
Seidel, T., & Prenzel, M. (2004). "What you see is what you get?" - Theoretical and
Methodological Challenges in Video Based Instruction Research. AERA
conference, San Diego.
Seidel, T., Prenzel, M., Duit, R., Euler, M., Geiser, H., Hoffmann, L., Lehrke, M.,
Müller, C., & Rimmele, R. (2002). "Can everybody look to the front of the
classroom please?" - Patterns of instruction in elementary physics classrooms and
its implications for students` learning. Unterrichtswissenschaft, 30(1), 52-77.
Seidel, T., Rimmele, R., & Prenzel, M. (2003). Opportunities for learning motivation in
classroom discourse - Combination of video analysis and student questionnaires.
Unterrichtswissenschaft, 31(2), 142-165.
Seidel, T., Rimmele, R., & Prenzel, M. (2005). Clarity and Coherence of Lesson Goals
as a Scaffold for Student Learning. Learning and Instruction, 4 (forthcoming).
Self-Brown, S.R., & Mathews, S. (2003). Effects of classroom structure on student
achievement goal orientation. Journal Of Educational Research, 97(2), 106-111.
She, H.C., & Fisher, D. (2002). Teacher communication behavior and its association
with students' cognitive and attitudinal outcomes in science in Taiwan. Journal of
Research in Science Teaching, 39(1), 63-78.
Silins, H., & Mulford, B. (2002). Schools as learning organisations: The case for
system, teacher and student learning. Journal of Educational Administration,
40(5), 425-446.
Slavin, R.E. (1987). A theory of school and classroom organization. Educational
Psychologist, 22(2), 89-108.
Slavin, R.E. (1995). Cooperative learning: Theory, research, and practice (2. Aufl.).
Boston: Allyn & Bacon.
Slavin, R.E. (1996). Success for all. Lisse: Swets & Zeitlinger.
Slavin, R.E. (1998). Every child, every school: success for all. Thousand Oaks, Ca.:
Corwin Press.
Smyth, E. (1999). Pupil performance, absenteeism and school drop-out: A multi-
dimensional analysis. School Effectiveness and School Improvement, 10(4), 480-
502.
Solvason, C. (2005). Investigating specialist school ethos, or do you mean culture?
Educational studies, 31(1), 85-94.
389
Stallings, J. (1985). Effective elementary classroom practices. In: M.J. Kyle (Ed.),
Reaching for excellence. An effective schools sourcebook. Washington, DC: US
Government Printing Office.
Stallings, J., & Mohlman, G. (1981). School policy, leadership style, teacher change
and student behavior in eight schools. Final report to the National Institute of
Education, Washington D.C.
Standing International Conference of Central and General Inspectorates of Europe
(SICI) (2003). Effective school self-evaluation. Retrieved from: www.sici.org.uk
Staub, F. C., & Stern, E. (2002). The nature of teachers' pedagogical content beliefs
matters for students' achievement gains: Quasi-experimental evidence from
elementary mathematics. Journal of Educational Psychology, 94(2), 344-355.
Stevens, R.J., & Slavin, R.E. (1995). Effects of a cooperative learning approach in
reading and writing on academically handicapped and nonhandicapped students.
Elementary School Journal, 95(3), 241-262.
Stipek, D.J., Feiler, R., Byler, P., Ryan, R., Milburn, S., & Salmon, J.M. (1998). Good
beginnings: What difference does the program make in preparing young children
for school? Journal Of Applied Developmental Psychology, 19(1), 41-66.
Stipek, D., Salmon, J.M., Givvin, K.B., Kazemi, E., Saxe, G., & MacGyvers, V.L.
(1998). The value (and convergence) of practices suggested by motivation
research and promoted by mathematics education reformers. Journal for Research
in Mathematics Education, 29(4), 465-488.
Stolarchuk, E., & Fisher, D. (2001). An investigation of teacher-student interpersonal
behavior in science classrooms using laptop computers. Journal of Educational
Computing Research, 24(1), 41-55.
Straetmans, G.J.J.M., & Sanders, P.F. (2001). Beoordelen van competenties van
docenten. Utrecht: Landelijk Programmamanagement EPS.
Stringfield, S.C., Bedinger, S. & R. Herman (1995). Implementing a private school
program in an inner-city public school: processes, effects, and implications from a
four year evaluation. Paper presented at the ICSEI Conference, Leeuwarden, the
Netherlands.
Stufflebeam, D.L., Foley, W.J., Gephart, W.J., Guba, E.G., Hammond, R.L., Merriman,
H.O., & Provus, M.M. (1971). Educational Evaluation and Decision Making in
Education. Itaca, Ill.: Peacock.
Sumfleth, E., Rumann, S., & Nicolai, N. (2004). Schulische und häusliche Kooperation
im Chemieanfangsunterricht. In J. Doll & M. Prenzel (Eds.), Bildungsqualität von
Schule: Lehrerprofessionalisierung, Unterrichtsentwicklung und
Schülerförderung als Strategien der Qualitätsverbesserung (pp. 284-302).
Münster: Waxmann.
Sweeney, J. (1982). Research synthesis on effective school leadership. Educational
Leadership, 39, 346-352.
390
Sweetland, S.R., & Hoy, W.K. (2000). School characteristics and educational outcomes:
Toward an organizational model of student achievement in middle schools.
Educational Administration Quarterly, 36(5), 703-729.
Taylor, B.M., Pearson, P.D., Peterson, D.S., & Rodriguez, M.C. (2003). Reading
growth in high-poverty classrooms: The influence of teacher practices that
encourage cognitive engagement in literacy learning. Elementary School Journal,
104(1), 3-28.
Thuen, E., & Bru, E. (2000). Learning environment, meaningfulness of schoolwork and
on-task-orientation among Norwegian 9th grade students. School Psychology
International, 21(4), 393-413.
Tiedemann, J., & Billmann-Mahecha, E. (2004). Kontextfaktoren der Schulleistung im
Grundschulalter. Ergebnisse aus der Hannoverschen Grundschulstudie. Zeitschrift
Fur Pädagogische Psychologie, 18(2), 113-124.
Tomoff, J., Thompson, M., & Behrens, J. (2000). Measuring NCTM-recommended
practices and student achievement with TIMSS. Paper presented at the Paper
presented at the Annual Meeting of the American Educational Research
Association, New Orleans, LA.
Torney-Purta, J. (2002). Challenges for Citizenship Education in the New Europe –
Patterns in the Civic Knowledge, Engagement, and Attitudes of European
Adolescents: The IEA Civic Education Study. European journal of education:
research, development, and policies, 37(2), 129-142.
Torney-Purta, J., Lehmann, R.H., Oswald, H. & Schulz, W. (2001). Citizenship and
education in twenty-eight countries: civic knowledge and engagement at age
fourteen. Amsterdam: IEA, the International Association for the Evaluation of
Achievement.
Townsend, M., & Choi, S.F. (2004, 16-18 September). Reading achievement in New
Zealand: Effects of parental self-efficacy and children's motivation. Paper
presented at the British Educational Research Association Annual Conference,
Manchester.
Townsend, M.A.R., & Hicks, L. (1997). Classroom goal structures, social satisfaction
and the perceived value of academic tasks. British Journal of Educational
Psychology, 67, 1-12.
Trautwein, U., Koller, O., Schmitz, B., & Baumert, J. (2002). Do homework
assignments enhance achievement? A multilevel analysis in 7th-grade
mathematics. Contemporary Educational Psychology, 27(1), 26-50.
Turner, J.C. (1995). The influence of classroom contexts on young children’s
motivation for literacy. Reading Research Quarterly, 30(3), 410-441.
Turner, J.C., Midgley, C., Meyer, D.K., Gheen, M., Anderman, E.M., Kang, Y., &
Patrick, H. (2002). The classroom environment and students' reports of avoidance
391
strategies in mathematics: A multi-method study. Journal of Educational
Psychology, 94(1), 88-106.
Udziela, T. (1996). Effect of formal study skills training on sixth grade reading
achievement (Reports - Research/Technical).
Uline, C.L., Miller, D.M., & Tschannen-Moran, M. (1998). School effectiveness: The
underlying dimensions. Educational Administration Quarterly, 34(4), 462-483.
Van Beekveld & Terpstra (2005). Rapportage Eindmeting Q5. Hoorn: Van Beekveld &
Terpstra.
Van de Gaer, E., Pustjens, H., Damme, J. van, & Munter, A. de (2004). Effects of
single-sex versus co-educational classes and schools on gender differences in
progress in language and mathematics achievement. British Journal Of Sociology
Of Education, 25(3), 307-322.
Van der Hoeven-Van Doornum, A.A. (1990). Effecten van leerlingbeelden en
streefniveaus op schoolloopbanen [The impact of teachers' expectations on
educational careers]. Nijmegen: ITS/OOMO.
Van der Werf, G. (1997). Differences in school and instruction characteristics between
high-, average-, and low-effective schools. School Effectiveness and School
Improvement, 8(4), 430-448.
Van der Werf, G., Creemers, B., Jong, R. de, & Klaver, E. (2000). Evaluation of school
improvement through an educational effectiveness model: The case of Indonesia's
PEQIP project. Comparative Education Review, 44(3), 329-355.
Van der Werf, M.P.C. (2005). Leren in het studiehuis: consumeren, construeren of
engageren? Groningen: GION.
Van Horn, M.L., & Ramey, S.L. (2003). The effects of Developmentally Appropriate
Practices on academic outcomes among former head start students and classmates,
Grades 1-3. American Educational Research Journal, 40(4), 961-990.
Van Kemenade, J.A. (red.) (1981). Onderwijs: Bestel en Beleid. Groningen: Wolters
Noordhoff.
Van Laarhoven, P., & Vries, A.M. de (1987). Effecten van de interklassikale
groeperingsvorm in het voortgezet onderwijs. Resultaten van een literatuurstudie.
(Effects of heterogeneous grouping in secondary schools). In: J. Scheerens &
W.G.R. Stoel (Red.), Effectiviteit van onderwijsorganisaties. Lisse: Swets &
Zeitlinger.
Van Petegem, P. (2001). Kwaliteitszorg of (zelf-)evaluatie. In R.J. Bosker (Red.),
Kwaliteitszorg. Onderwijskundig Lexicon III (pp. 27-46). Alphen aan den Rijn:
Samsom.
Vedder, P.H. (1985). Cooperative learning. A study on processes and effects of
cooperation between primary school children. The Hague: SVO.
Veenman, S., & Raemaekers, J. (1996). Retentie-effecten van een nascholings-
programma voor effectieve instructie en klassemanagement. Pedagogische
392
studiën: tijdschrift voor onderwijskunde en opvoedkunde, 73(5), 357-371.
Veenstra, D.R. (1999). Leerlingen, klassen, scholen. S.l.: s.n.
Venne, L. van de, Meijer, J. & Voorthuis, M., m.m.v. Van Wieringen, F. & Karsten, S.
(2000). Schaalgrootte en organisatie-effectiviteit in het basisonderwijs.
Amsterdam: SCO-Kohnstamm Instituut.
Venne, L.H.J. van de (1999). Een raamwerk voor het meten van de kwaliteit van
schoolorganisaties. Nederlands Tijdschrift voor Onderwijsrecht, 4, 259-271.
Visscher, A.J. (2002). A framework for studying School Performance Feedback
Systems. In A.J. Visscher & R. Coe (eds.), School improvement through
performance feedback (pp. 41-72). Lisse/Abingdon/Exton/Tokyo: Swets &
Zeitlinger B.V.
Visscher, A.J., & Coe, R. (2002). Drawing up the balance sheet for School Performance
Feedback Systems. In A.J. Visscher & R. Coe (eds.), School improvement through
performance feedback (221-254). Lisse/Abingdon/Exton/Tokyo: Swets &
Zeitlinger B.V.
Visscher, A.J., & R. Coe (Eds.) (2002). School improvement through performance
feedback. Lisse/Abingdon/Exton/Tokyo: Swets & Zeitlinger B.V.
Visscher, A.J., & Coe, R. (2003). School Performance Feedback Systems:
Conceptualisation, Analysis and Reflection. School Effectiveness and School
Improvement, 14(3), 321-349.
Voeten, (2000). Evaluatie van leervorderingen. In R.J.Bosker (Red.), Evalueren in het
onderwijs. Onderwijskundig lexicon. Editie III (pp. 15-40). Alphen aan den Rijn:
Samsom.
Voogt, J.C. (1995). Schooldiagnose. In H.P.M.Creemers (Red.), Onderwijskundig
lexicon. (pp. F3325- 1-38). Alphen aan den Rijn: Samsom.
Voorhis, F.L.V. (2003). Interactive homework in middle school: Effects on family
involvement and science achievement. Journal Of Educational Research, 96(6),
323-338.
Wang, M.C., Haertel, G.D., & Walberg, H.J. (1993). Toward a knowledge base for
school learning. Review of Educational Research, 63(3), 249-294.
Watermann, R. (2003). Gesellschaftsbilder im Jugendalter: Voorstellungen
Jugendlicher über die Ursachen sozialer Aufwärtsmobilität. Opladen: Leske +
Budrich.
Weeda, W.C. (1986). Effectiviteitsonderzoek van scholen. In: J.C. van der Wolf & J.J.
Hox (Red.), Kwaliteit van het onderwijs in het geding (About education quality).
Publicaties van het Amsterdams Pedogogische Centrum, nr. 2. Lisse: Swets &
Zeitlinger.
Weiss, C.H. (1998). Improving the use of evaluations: Whose job is it anyway? In A.J.
Reynolds & H.J. Walberg (Eds.), Advances in educational productivity (Vol. 7,
pp. 263-276). Greenwich/London: JAI Press.
393
Wenglinsky, H. (2002). The link between teacher classroom practices and student
academic performance. Educational Evaluation and Policy Analysis, 10(12).
Wet op het Primair onderwijs (Primary Education Act) (1998). Retrieved from
http://wetten.sdu.nl
White, B.Y., & Frederiksen, J.R. (1998). Inquiry, modeling, and metacognition: Making
science accessible to all students. Cognition and Instruction, 16(1), 3-118.
Wigfield, A., Guthrie, J.T., Tonks, S., & Perencevich, K.C. (2004). Children's
motivation for reading: Domain specificity and instructional influences. Journal
of Educational Research, 97(6), 299-309.
Willms, J.D., & Somers, M.-A. (2000). Schooling outcomes in Latin America. Report
prepared for UNESCO-OREALC and the Laboratorio Latinoamericano de la
Calidad de la Educación [The Latin American Laboratory for the Quality of
Education].
Willms, J.D., & Somers, M.A. (2001). Family, classroom, and school effects on
children's educational outcomes in Latin America. School Effectiveness and
School Improvement, 12(4), 409-445.
Winkler, D.R. (1989). Decentralisation in education: an economic perspective.
Washington:The World Bank, Population and Human Resource Development.
Witziers, B., Bosker, R.J., & Krüger, M. (2003). School leadership and student
outcomes. The elusive search for an association. .Educational Administration
Quarterly, 39 (3), 398-425.
Wolters, J. (2005). Projectplan Balans Interne kwaliteitszorg en extern toezicht.
Retrieved from:
http://www.qprimair.nl/ventura/engine.php?Cmd=seepicture&P_site=345&P_self
=1330&Random=2064385551.
Wong, K.C., Lam, Y.R., & Ho, L.M. (2002). The effects of schooling on gender
differences. British Educational Research Journal, 28(6), 827-843.
Working group self-evaluation Primary education, 13 december 2002. Handleiding bij
het beoordelingsformulier zelfevaluatie-instrumenten primair onderwijs. Interne
notitie Inspectie van het onderwijs.
Working group self-evaluation Primary education, 2 april 2003. Het analyseren van
instrumenten voor schoolzelfevaluatie: opzet, ervaring en reflectie. Interne notitie
Inspectie van het onderwijs.
Working group self-evaluation Primary education, 2002/2003. Beoordelingsformulier
zelfevaluatie-instrumenten primair onderwijs. Interne notitie Inspectie van het
onderwijs.
Wöβmann, L. (2000). Schooling resources, educational institutions, and student
performance: The international evidence. (Kiel Working paper No. 983). Kiel,
Germany: Kiel Institute of World Economics.
394
Yair, G. (2000). Reforming motivation: how the structure of instruction affects students'
learning experiences. British Educational Research Journal, 26(2), 191-210.
Yin, R.K. (1981). The case study as a serious research strategy. Knowledge, creation,
diffusion, utilization, vol. 3.
Young, D.Y. (1998). Rural and urban differences in student achievement in science and
mathematics: A multilevel analysis. School Effectiveness and School
Improvement, 9(4), 386-418.
Zellman, G., Stecher, B., Klein, S., & McCafrey, D. (1998). Findings from an
evaluation of the parent institute for quality education parent involvement
program. Santa Monica: RAND.
Zoelen, L. van, & Houtveen, A.A.M. (2000). Naar effectieve schoolverbetering:
Resultaten van het onderzoek naar de effecten op leerkracht- en leerlingniveau
van het adaptief onderwijsproject "kwaliteitsversterking rekenen en wiskunde".
Utrecht: ISOR.
395
Summary in Dutch/Nederlandstalige Samenvatting
Samenvatting en conclusies Doel van de studie en centrale vragen In de inleiding bij dit verslag is het doel van deze studie beschreven als het verifiëren van de wetenschappelijke basis van de indicatoren die deel uitmaken van de Toezichtkaders van de Nederlandse Onderwijsinspectie voor primair en voortgezet onderwijs en deze Toezichtkaders te relateren aan centrale onderwijsfuncties, zoals kwalificatie, integratie en persoonlijkheidsontwikkeling. De manier waarop dit kon worden gedaan, verschilde voor deze twee belangrijkste streefdoelen. Wat het eerste gedeelte betreft, het verifiëren van de wetenschappelijke basis van de indicatoren van het Inspectiekader, kan een relatief simpele koppeling worden gelegd met de onderzoeksliteratuur over school- en instructie-effectiviteit. Voor dit streefdoel kan de term valideren van het kader worden gebruikt. Voor het tweede gedeelte, de vergelijking van het Inspectiekader met de huidige ideeën over centrale maatschappelijke functies van het onderwijs, zijn er minder feiten voor handen. Het ‘onderbouwen’ van de Toezichtkaders voor dit gedeelte lijkt meer op het positioneren van het kader met betrekking tot de nieuwe ontwikkelingen en tendensen. De volgende kernvragen zijn in de studie aan de orde gekomen: Worden de procesindicatoren voor onderwijzen en leren in de Inspectiekaders ondersteund door het kennisbestand over school- en instructie-effectiviteit? Hoe haalbaar is het idee van proportioneel toezicht, gezien de mogelijkheden en de stand van zaken van schoolzelfevaluatie in Nederland? Laten de Inspectiekaders verdedigbare keuzes zien met betrekking tot resultaat- en procesindicatoren en strategische toepassingen, gezien de huidige standpunten ten aanzien van onderwijsbestuur en de moderne interpretaties van de belangrijkste maatschappelijke functies van het onderwijs?
Het Inspectiekader De centrale vraag met betrekking tot schooltoezicht is: ‘Hoe ziet de onderwijskwaliteit op een specifieke school eruit?’ (Inspectie, 2002, blz. 9). Deze vraag wordt onderverdeeld in drie subvragen:
4) Hoe ziet de kwaliteitszorg van de school eruit? 5) Wat is de kwaliteit van de school op het vlak van onderwijzen en leren?
396
6) Wat is de kwaliteit van de leerresultaten? (ibid, blz. 9) De drie domeinen worden verdeeld in de volgende kwaliteitsaspecten, zoals aangegeven in onderstaande tabel, gebaseerd op het Inspectiekader voor Primaire Scholen, geformuleerd in 2002.
Kwaliteitsdomein Kwaliteitsaspecten per domein
Kwaliteitszorg 15. Systematische kwaliteitszorg door de school
16. Toetsen
Onderwijzen en leren 17. Dekking leerstof
18. Tijd
19. Stimulerende en ondersteunende onderwijs- en leerprocessen
20. Veilig, ondersteunend en stimulerend schoolklimaat
21. Bijzondere zorg voor kinderen met leermoeilijkheden
Resultaten 8. Leerresultaten in basisvakken
In 2005 is dit kader aangepast, zoals aangegeven in onderstaande tabel.
Kwaliteitsdomein Kwaliteitsaspecten per domein
Kwaliteitszorg 11. De school draagt zorg voor het waarborgen en verbeteren van de kwaliteit van het onderwijs
12. De kwaliteitszorgcondities zijn aanwezig
Onderwijzen en leren 13. De aangeboden leerstof is erop gericht de leerlingen breed te ontwikkelen en voor te bereiden op verder onderwijs en de arbeidsmarkt
14. De leerlingen krijgen voldoende tijd om de leerstof te beheersen
15. De pedagogische benadering van de leerkrachten zorgt voor een veilige en stimulerende leeromgeving
16. De didactische benadering van de leerkrachten ondersteunt het leren van de leerlingen
17. De leerlingen spelen een actieve en onafhankelijke rol tijdens instructieactiviteiten
18. De school heeft een veilig en stimulerend klimaat
19. Begeleiding en advies zijn gericht op een volledige ontwikkeling van de capaciteiten van de leerlingen
20. Leerlingen met bijzondere behoeften krijgen de zorg die zij nodig hebben
Resultaten 11. De leerlingen bereiken de resultaten die overeenkomen met hun capaciteiten
397
De versie van 2005 is duidelijk gedetailleerder in het onderscheiden van feitelijke kwaliteitszorg en de kwaliteitszorgcondities op school, en ook in het onderscheiden van pedagogische en didactische aspecten voor onderwijs- en leersituaties. Bovendien weerspiegelen brede ontwikkeling, arbeidsmarktoriëntatie en ontwikkeling van de capaciteiten van leerlingen een onderwijsstandpunt dat beïnvloed lijkt te zijn door actuele opvattingen over onderwijs voor burgerschap, en de ontwikkeling van competenties.
Het Inspectiekader vergeleken met kwaliteitsconcepten In Hoofdstuk 2 van het verslag wordt het Inspectiekader gekoppeld aan interpretaties van en opvattingen over onderwijskwaliteit. Dit leidt tot de volgende conclusies: Als schoolevaluatiekader kan het Toezichtkader van de Inspectie worden geclassificeerd als gekoppeld aan het effectiviteitsperspectief ten aanzien van onderwijskwaliteit. De procesindicatoren van het Inspectiekader kunnen worden gezien als effectiviteits-verhogende condities die hun inspiratie vinden in empirisch onderzoek naar de effectiviteit van scholen en in een internationale consensus ten aanzien van goede onderwijspraktijken. Vanuit laatstgenoemd perspectief noemt Vernooy, 2002, aangehaald door Janssens, 2005, de volgende sleutelvariabelen: De scholen realiseren de vooraf gestelde doelen; De leerkrachten hebben hoge verwachtingen van de bekwaamheden van de leerlingen; Een positief onderwijs-leerklimaat heeft de overhand; De school is erop gericht dat alle leerlingen de basisvaardigheden verwerven; Schoolleider en team voelen zich verantwoordelijk voor de resultaten en het welzijn van de leerling; De school reserveert voldoende tijd voor taal/lezen; De school gebruikt goed, gecoördineerd lesmateriaal; Professionele ontwikkeling is een continu proces; Er is een sterke gerichtheid op preventie en de resultaten van de leerling worden nauwgezet gevolgd; Er wordt aandacht besteed aan de effectiviteit van de taal/leeslessen; De school volgt een sterk teamgeoriënteerde benadering; De schoolleider is er goed in: een heldere visie te formuleren; zich sterk te focussen op de resultaten van leerlingen; verbeteringen te bewaken; nauw betrokken te zijn bij wat er in de klas gebeurt.
Bij bestudering van de feitelijke kwaliteitsindicatoren uit het Inspectiekader, kan het volgende worden opgemerkt:
398
- het referentiepunt, in de zin van effectcriteria of resultaatindicatoren, wordt gevormd door toetsscores in basisschoolvakken;
- het merendeel van de procesindicatoren wordt gedefinieerd op klaslokaalniveau; er blijkt een sterke nadruk te liggen op het primaire proces van onderwijzen en leren;
- indicatoren die de schoolzelfevaluatie en kwaliteitszorg vertegenwoordigen, hebben een dubbele betekenis binnen het Kader; zij vormen een set van effectiviteits-verhogende factoren op zich, maar worden tegelijkertijd geëvalueerd uit een metaperspectief, met als mogelijk resultaat dat kwaliteitszorgprocedures van scholen feitelijk de directe inspectie door de Inspectie vervangen (dit laatste idee is het zogenoemde proportioneel toezicht);
- sommige indicatoren op klaslokaalniveau zouden een interpretatie op schoolniveau kunnen krijgen zoals het geval is bij geprogrammeerde tijd, een ondersteunend klimaat en dekking van de inhoud (maar het kwantitatieve gebruik van het kader lijkt beperkt te worden tot beoordelingen op klaslokaalniveau);
- organisatorische en managementcondities die zich voordoen als effectiviteits-verhogende factoren op afstand in modellen voor schooleffectiviteit op meerdere niveaus, worden beschouwd als ‘mogelijke achtergrond en oorzaken’ van kwaliteits-condities op het niveau van het primaire proces; voorbeelden zijn: schoolmanagement, aspecten van de schoolorganisatie, schoolbeleid; professionalisering van personeel en contacten met ouders (Inspectie, 2002, blz. 9);
- er ligt weinig nadruk op de resource-input factoren (zoals faciliteiten, onderwijs-apparatuur, computers en het schoolbudget, klassengrootte en de verhouding leerlingen/leerkrachten;
- speciale zorg voor leerlingen met leermoeilijkheden en leerlingen met een achterstand vanwege een lage sociaal-economische status of weinig cultureel kapitaal of een minderheidsachtergrond, wordt samengebracht onder de criteria die, grofweg, betrekking hebben op adaptief onderwijzen;
- schoolsamenstelling, een belangrijke kwestie in recente onderzoeken naar schooleffectiviteit, komt niet aan de orde;
- organisatorische effectiviteit in een ruimer perspectief, die aan de orde komt in het kader van Quinn en Rohrbaugh, valt niet onder het Inspectiekader. Er is geen aandacht voor het externe schoolbeleid, zoals het verschaffen van informatie aan ouders en de lokale achterban, relaties met de lokale zakenwereld, en er is evenmin veel aandacht voor aspecten van human resource management op school, en voor een strikte monitoring van formele procedures (dit laatste aspect komt uitgebreid aan de orde in kwaliteitsbeheersystemen van het ‘ISO-type’);
- de zachtere gebieden van schoolcultuur en -klimaat komen aan de orde in het Inspectiekader, met aandacht voor positieve interactie, veiligheid en discipline en het zich ondersteund en gestimuleerd voelen.
399
De totale sterke associatie met het perspectief van schooleffectiviteit maakt onderzoek naar de empirische ondersteuning van de procesindicatoren binnen het Kader mogelijk, gezien als effectiviteitsverhogende condities die van nut zijn voor het bereiken van cognitieve resultaten in basisvakken. In de volgende hoofdstukken zal een samenvatting worden gegeven van de onderzoeksevidentie voor school- en instructie-effectiviteit, geactualiseerd aan de hand van een recente review en onderzoekssynthese.
Een eerste vergelijking van de set van procesindicatoren die deel uitmaken van het Inspectiekader met de variabelen die naar voren zijn gekomen bij het onderzoek naar school- en instructie-effectiviteit In hoofdstuk 2, waarin het Inspectiekader wordt vergeleken met perspectieven en concepten op het vlak van onderwijskwaliteit, wordt geconcludeerd dat het Inspectiekader sterk in het verlengde ligt van het effectiviteitsperspectief. Dit perspectief houdt in dat de keuze van procesindicatoren gebaseerd is op die plooibare school- en instructievariabelen die gekoppeld blijken te zijn aan relatief hoge prestaties (uitgedrukt in leerresultaten en verworvenheden van leerlingen). Deze argumentatielijn vormt de kern van de benadering om de geldigheid van de kwaliteitsindicatoren van het Inspectiekader te beoordelen door de dekking te onderzoeken van de variabelen die bij het onderzoek naar onderwijseffectiviteit in kaart zijn gebracht. Hoofdstuk 3 geeft een eerste overzicht van de onderzoeksresultaten in dit veld, terwijl hoofdstuk 4 en 5 een verdere actualisatie verschaffen, op basis van recentere onderzoeksresultaten. De analyses in hoofdstuk 3 laten een redelijke overeenstemming zien tussen categorieën die worden gebruikt door de Inspectie en de belangrijkste variabelen die worden gebruikt bij onderzoek naar instructie-effectiviteit. Dit wordt duidelijk gemaakt in onderstaande tabel, waarin de belangrijkste variabelen, aan de orde gesteld in het onderzoek naar instructie-effectiviteit, zijn vergeleken met de categorieën van het observatie-instrument dat door de Inspectie wordt gebruikt. Er lijkt een redelijke overeenstemming te bestaan tussen de indicatoren die het kwaliteitsaspect leren en instructie vertegenwoordigen, en de belangrijkste factoren die empirisch ondersteund worden door het onderzoek naar school- en instructie-effectiviteit. Dit blijkt uit onderstaande samenvattende tabel.
400
Belangrijkste factoren uit de review
Gelegenheid om te leren Leertijd Gestructureerd onderwijs Stimulerende betrokkenheid Taakgericht klimaat Wederzijds respect Ordelijkheid, veiligheid Monitoring en vragen stellen Feedback en reinforcement Vormgeving leren/zelfregulatie
Categorieën in observatiechecklist
Leertijd Helder en gestructureerd onderwijs Activeren Uitdagen Ondersteunen (wederzijds respect) Ordelijke, functionele leeromgeving Evalueert of doelstellingen bereikt zijn Feedback Leerstrategieën leren gebruiken Organisatie in de klas
Op het niveau van de school, d.w.z. de variabelen op schoolniveau die in kaart zijn gebracht in het effectiviteitsonderzoek, bestaat er minder overeenstemming. Bij het beoordelen van het Toezichtkader uit 2002, blijkt er een bijna exclusieve concentratie op klaslokaalniveau te zijn. In de versie van 2005 worden schoolcondities erkend in de kwaliteitsaspecten die verband houden met de voorwaarden voor schoolzelfevaluatie en kwaliteitszorg. Vervolgens, betrokkenheid van ouders en ‘klimaat’, gebieden die voorkomen in de versie van 2005, of waarschijnlijk ook een interpretatie op schoolniveau hebben. De nadruk op het instructieniveau als geheel is nog steeds aanwezig in de versie van 2005. Deze keuze is heel goed verdedigbaar vanuit het perspectief van de resultaten van het empirisch effectiviteitsonderzoek, omdat instructievariabelen dichter bij de leerresultaten liggen, en omdat tevens herhaaldelijk is aangetoond dat zij sterkere effecten hebben dan de schoolniveaufactoren die meer ‘op afstand’ liggen. Niettemin kan vanuit het standpunt van schoolverbetering en veranderingsinstrumenten worden aangevoerd dat schooleigenschappen zoals prestatiegerichtheid, en consensus en samenwerking onder het personeel misschien beter in het Inspectiekader vertegenwoordigd moeten zijn.
De kwaliteitsindicatoren van het Inspectiekader in vergelijking met de resultaten van een onderzoekssynthese over school- en instructie-effectiviteit De in hoofdstuk 4 en 5 gepresenteerde resultaten, over respectievelijk school- en instructie-effectiviteit, zijn in onderstaande tabel samengevat. De variabelen die aan de orde komen zijn zoveel mogelijk gekoppeld aan de indicatoren uit het Inspectiekader.
401
Tabel: Samenvattende tabel waarin het Toezichtkader wordt gekoppeld aan de onderzoeksresultaten inzake school- en instructie-effectiviteit. De getoonde verhoudingen zijn het aantal positief significante effecten in termen van cognitieve uitkomsten van alle studies die de desbetreffende factoren en variabelen hebben onderzocht. Wanneer de verhouding 50% of hoger is, zijn de resultaten vetgedrukt.
Kwaliteitsaspecten en indicatoren Inspectie
Factoren en variabelen in effectiviteits-onderzoek
Kwaliteitsaspect 1: De school draagt zorg voor het waarborgen en verbeteren van de kwaliteit van het onderwijs. 1.1 De school kent haar beginsituatie, met
inbegrip van de specifieke behoeften van de leerlingen.
1.2 De school evalueert systematisch de kwaliteit van haar prestaties in termen van leerresultaten
1.3 De school evalueert systematisch de kwaliteit van leren, onderwijzen en adviseren
1.4 De school heeft meetbare verbeteringsdoelen geformuleerd.
1.5 De school voert verbeteringsactiviteiten op systematische wijze uit.
1.6 De school garandeert de kwaliteit van leren en onderwijzen
1.7 De school garandeert de kwaliteit van het schoolexamen en andere evaluatie-instrumenten.
1.8 De school brengt verslag uit over de gerealiseerde kwaliteit van onderwijs aan belanghebbenden (ouders, leerlingen, bevoegde autoriteiten, financierende instanties en sponsors).
Kwaliteitsaspect 2: De kwaliteitszorgcondities zijn aanwezig 2.1 Het schoolmanagement initieert en leidt
de kwaliteitszorg 2.2 De kwaliteitszorg is gekoppeld aan de
visie van de school over leren en onderwijzen, als verwoord in het schoolplan
Evaluerend vermogen op schoolniveau en doelgerichtheid; 12/26 Gebruik van leerlingvolgsysteem: 1/2 Functiebeoordeling: 2/12 Duidelijke gerichtheid op beheersing van basisvakken: 9/14 Onderwijskundig leiderschap en betrokkenheid van ouders bij schoolbeleid: 20/74 De schoolleider treedt op als adviseur en controleur van leerkrachten in de klas: 9/13
402
2.3 Het schoolmanagement draagt zorg voor een professionele schoolcultuur
2.4 De school draagt zorg voor een effectieve communicatie over de kwaliteit van het onderwijs.
2.5 Medewerkers, schoolmanagement, leerlingen, ouders en bevoegde autoriteiten zijn allen betrokken bij de kwaliteitszorg van de school
Kwaliteitsaspect 3: De aangeboden leerstof is erop gericht de leerlingen breed te ontwikkelen en voor te bereiden op verder onderwijs en de arbeidsmarkt 3.1 De school heeft de funderingen
geleverd voor het aanbod van leerstof in de lagere klassen.
3.2 De school garandeert dat de feitelijke aangeboden leerstof het examenprogramma dekt.
3.3 Het programma (totaal aangeboden leerstof) wordt gekoppeld aan belangrijke maatschappelijke en actuele thema's.
3.4 Leerstof die wordt aangeboden in het ene jaar sluit aan op de voorafgaande en volgende jaren.
3.5 Er is sprake van coherentie tussen het aanbod van leerstof in de verschillende vakken
3.6 Het aanbod van leerstof is aangepast aan de onderwijsbehoeften van individuele leerlingen.
3.7 De school biedt kennis over de verschillende in Nederland aanwezige culturen, op regelmatige basis, met verwijzing naar de corresponderende normen en waarden.
3.8 Scholen met meer dan 20% taalzwakke leerlingen passen het onderwijs Nederlands aan aan de behoeften van deze leerlingen in alle vakken.
3.9 De scholen bieden stof aan over burgerschapsonderwijs, sociale samenhang, en normen en waarden
De schoolleider als initiatiefnemer en facilitator van professionalisering van medewerkers: 0/2 Nadruk op betrokkenheid van ouders bij schoolbeleid: 0/13 Curriculumkwaliteit en gelegenheid om te leren: 15/34 Gelegenheid om te leren: 11/31
403
Kwaliteitsaspect 4: De leerlingen krijgen voldoende tijd om de leerstof te beheersen 4.1 De beoogde onderwijstijd komt
overeen met de wettelijke normen 4.2 Het structurele (d.w.z. geplande) aantal
‘niet-gegeven’ lessen is minimaal 4.3 Het incidentele aantal ‘niet-gegeven’
lessen is beperkt 4.4 De ongeoorloofde afwezigheid van
leerlingen is beperkt 4.5 De leerkrachten gebruiken de beoogde
onderwijstijd op efficiënte wijze 4.6 De school varieert de hoeveelheid tijd
voor onderwijzen en leren aan de hand van de onderwijsbehoeften van de leerlingen
Kwaliteitsaspect 5: De pedagogische benadering van de leerkrachten brengt een veilige en stimulerende leeromgeving met zich mee 5.1 De leerkrachten stimuleren het
zelfvertrouwen van de leerlingen 5.2 De leerkrachten behandelen de
leerlingen met respect 5.3 De leerkrachten stimuleren leerlingen
elkaar met respect te behandelen 5.4 De leerkrachten zorgen voor een
productieve sfeer Kwaliteitsaspect 6: De didactische benadering van de leerkrachten ondersteunt het leren van de leerlingen 6.1 De leerkrachten verschaffen inzicht in
doel, gebruik en onderlinge relaties van de lesactiviteiten
6.2 De leerkrachten geven duidelijke uitleg
6.3 De leerkrachten controleren of de leerlingen uitleg en opdrachten begrepen hebben
6.4 De leerkrachten scheppen een zinvol kader voor het leerproces
Effectieve leertijd: 33/108 Huiswerk: 20/44 Klimaat in de klas: 21/84 Kwaliteit van interacties, ondersteuning van leerkrachten 4/12 Gestructureerde instructie; duidelijk en gestructureerd onderwijs: 46/115 Direct onderwijs: 17/57 Uitdagen: 46/92 Cognitieve activering: 36/58
404
6.5 De leerkrachten stimuleren de leerlingen tot nadenken (bijv. door uitdagende vragen te stellen)
6.6 De leerkrachten leveren de leerlingen substantiële feedback
6.7 De leerkrachten bieden inzicht in de leerprocessen van de leerlingen (simuleren van beschouwingen over leerstrategieën)
6.8 De leerkrachten zorgen ervoor dat leerlingen betrokken worden bij onderwijsactiviteiten
6.9 De didactische benadering is functioneel met betrekking tot het leerproces van de leerlingen
6.10 De leerkrachten passen hun didactische benadering aan aan de verschillen tussen leerlingen
6.11 De leerkrachten gebruiken de analyses van de prestaties van leerlingen voor de manier waarop zij hun instructie vormgeven
6.12 Het taalgebruik van de leerkracht is aangepast aan de taalbehoeften van de leerlingen
Kwaliteitsaspect 7: De leerlingen spelen een actieve en onafhankelijke rol tijdens instructieactiviteiten 7.1 Leerlingen worden geconfronteerd met
stimulerende activiteiten en opdrachten 7.2 De leerlingen denken na over hun eigen
leerprocessen 7.3 De leerlingen krijgen in voldoende
mate verantwoordelijkheid voor hun eigen leerprocessen
7.4 De leerlingen leren doeltreffend samen te werken
Kwaliteitsaspect 8: De school heeft een veilig en stimulerend klimaat 8.1 De leerlingen tonen betrokkenheid bij
het schoolleven 8.2 Het personeel toont betrokkenheid bij
Feedback: 2/27 Leerstrategieën leren gebruiken: 66/90 Metacognitieve strategieën: 21/24 Adaptief onderwijs en differentiatie: 32/150 Activeren: 48/154 Samenwerkend leren: 17/23 Schoolklimaat: 26/68 Betrokkenheid van ouders: 32/82
405
het schoolleven 8.3 De ouders voelen zich betrokken bij het
schoolleven 8.4 Personeel en leerlingen gaan binnen en
buiten de klas met respect met elkaar om
8.5 Personeel en leerlingen voelen zich veilig op school
8.6 De leerlingen, het personeel en de ouders ervaren de schoolleiding als ondersteunend en stimulerend met betrekking tot de sfeer op school
Kwaliteitsaspect 9: Begeleiding en advies zijn gericht op een volledige ontwikkeling van de capaciteiten van de leerlingen 9.1 Begeleiding en advies zijn gericht op
een goede ontwikkeling en het welzijn van de leerlingen
9.2 De school hanteert een consistent systeem van instrumenten en procedures voor het bewaken van vooruitgang en ontwikkeling van de leerlingen
9.3 De school gebruikt informatie van de basisschool bij het begeleiden en adviseren van de leerlingen
9.4 De school ondersteunt leerlingen en ouders in hun loopbaankeuzes gedurende het schoolprogramma
9.5 De school ondersteunt leerlingen en ouders in hun keuzes met betrekking tot vervolgonderwijs en de arbeidsmarkt
9.6 Bijzondere staffuncties (zoals decanen en mentors) zien erop toe dat leerlingen een continue begeleiding ontvangen gedurende hun hele schoolcarrière
Kwaliteitsaspect 10: Leerlingen met bijzondere behoeften krijgen de zorg die zij nodig hebben 10.1 De school stelt vroege diagnoses bij
leerlingen die extra zorg nodig hebben
10.2 De school analyseert de soort zorg die
Ordelijke sfeer: 16/30 Monitoring: 3/7 (Zie boven: adaptief onderwijs en differentiatie)
406
specifieke leerlingen nodig hebben 10.3 Bijzondere zorg wordt gegeven op
een systematische, geplande manier 10.4 De school beoordeelt de effecten van
bijzondere zorg 10.5 Leerkrachten en bijzondere
zorgmedewerkers werken goed samen
10.6 De school betrekt ouders van leerlingen die extra zorg nodig hebben bij de bijzondere activiteiten
De volgende opmerkingen kunnen worden gemaakt bij de interpretatie van deze resultaten: - Het aantal studies waarin het evaluerend vermogen van de school wordt onderzocht,
is tamelijk beperkt (26 studies) en het aandeel van significante positieve effecten ligt onder de drempel van 50%. De variabele die wijst op een duidelijke gerichtheid op basisvakken, een ‘klassieker’ uit de onderzoeksliteratuur inzake schooleffectiviteit, kent een relatief hoog aandeel positief significante effecten, maar het totale aantal studies dat werd gebruikt voor deze conclusie is laag (14 studies).
- Wat betreft variabelen die gerelateerd kunnen worden aan kwaliteitszorgcondities
op school, is gekeken naar de variabelen ‘onderwijskundig leiderschap’ en betrokkenheid van de ouders bij het schoolbeleid. In totaal bleken slechts 20 van de 74 effecten statistisch significant te zijn in de verwachte positieve richting. De variabele: ‘de schoolleider treedt op als adviseur en controleur van leerkrachten in de klas’ had een relatief hoog aandeel positieve effecten, maar het aantal antwoorden was beperkt (13).
- Wat betreft de dekking van de leerstof, lieten de kwaliteit van het curriculum en de
gelegenheid om te leren een beperkt aandeel positief significant zien. Deze uitkomst stemt niet overeen met reviews van eerder onderzoek, en is lager dan verwacht.
- Een vergelijkbare conclusie kan worden getrokken met betrekking tot
tijdgerelateerde variabelen, effectieve leertijd en huiswerk, met aandelen van respectievelijk 33/108 en 20/44 beide onder de drempel van 50%. Beide variabelen kregen meer steun in reviews van eerder onderzoek.
- Het klimaat in de klas en de kwaliteit van de interactie tussen leerkrachten en
leerlingen hadden een relatief laag aandeel positief significante effecten.
407
- Een heel opvallende uitkomst van deze onderzoekssynthese is het onverwacht lage
aandeel studies dat een positief effect liet zien voor variabelen als: gestructureerde instructie, direct onderwijs, en feedback. Deze variabelen kwamen veel sterker naar voren in de onderzoekssynthese die in de jaren tachtig werd verricht door Walberg en Fraser.
- Variabelen die meer in het verlengde liggen van het idee van ‘constructivistisch
onderwijs’, zoals uitdagen (46/92), cognitieve activering, leerstrategieën leren gebruiken (66/90) en metacognitieve strategieën (21/24) kwamen sterker uit de bus. Dit is weer een tamelijk onverwachte uitkomst, aangezien door andere recente review (Van der Werf, 2005) vraagtekens worden geplaatst bij de effectiviteit van die benaderingen.
- Adaptief onderwijs en differentiatie lieten een relatief laag aandeel positief
significante effecten zien. Van deze variabelen werd in een eerdere review een gemengd beeld geschetst.
- Onder de klimaatvariabelen liet een ordelijke sfeer het grootste aandeel positieve
effecten zien (16/30). Het totale schoolklimaat bleef onder de drempel van 50% (26/68).
- Andere ‘klassiekers’ uit de onderzoeksliteratuur inzake schooleffectiviteit, zoals
betrokkenheid van de ouders en consensus en samenhang onder het personeel (niet opgenomen in bovenstaande samenvattende tabel omdat het geen deel uit maakt van de inspectie-indicatoren) hadden een relatief laag aandeel statistisch significant positieve effecten (respectievelijk 32/82 en 5/30).
Meer in het algemeen laten de resultaten in de samenvattende tabellen in hoofdstuk 4 en 5 zien dat voor variabelen die in eerdere studies werden ondersteund maar die in deze studie onder de drempel van 50% bleven, er over het algemeen een groot aandeel studies was waarin de resultaten zich in de verwachte positieve richting bevonden, maar niet statistisch significant waren; in het geval van helder en gestructureerd onderwijs hadden bijvoorbeeld 62 van de 96 studies een niet-significant effect, en slechts 3 van de 96 een statistisch significant negatief effect. Dit kan te wijten zijn aan onderzoekstechnische overwegingen, zoals de omvang van de steekproef die in de studie werd gebruikt. Zoals de zaken ervoor staan, dienen de resultaten met de nodige voorzichtigheid te worden geïnterpreteerd. De volgende gevolgtrekkingen kunnen worden gedaan met betrekking tot de geldigheid van de set van kwaliteitsindicatoren die in het Toezichtkader van de Nederlandse Inspectie worden gehanteerd.
408
- Er is over het algeheel een tamelijk goede match tussen de set van variabelen die worden onderzocht bij onderzoek naar onderwijseffectiviteit en de procesindicatoren (die betrekking hebben op de kwaliteitsaspecten kwaliteitszorg op school en onderwijzen en leren) van het Inspectiekader.
- Het feit dat de huidige Kaders zich meer richten op effectiviteitsverhogende
condities in de klas dan op indicatoren op schoolniveau wordt gestaafd door onze resultaten, in die zin dat onze resultaten de relatief grotere impact van variabelen op het niveau van de klas in vergelijking met variabelen op schoolniveau bevestigen. Met andere woorden: een mogelijke stelling dat er meer indicatoren op schoolniveau aan het bestaande kader moeten worden toegevoegd, wordt niet door onze resultaten gestaafd.
- ‘Het evaluerend vermogen van scholen’ blijkt volgens onze resultaten geen
belangrijke effectiviteitsverhogende conditie te zijn. We moeten daarom terughoudend zijn en niet te veel prestatieverbeteringen verwachten van een verbetering van de kwaliteitszorg op school. Tegelijkertijd dient de relevantie van de indicatoren inzake kwaliteitszorg te worden beschouwd als onbetwist, gezien het belangrijke strategische concept van proportioneel toezicht.
- Wat betreft instructiestrategieën en didactische benaderingen bevat het
Inspectiekader verschillende elementen die corresponderen met gestructureerd en direct onderwijs, alsmede verschillende elementen die een meer constructivistische benadering van leren en instructie weerspiegelen. Onze resultaten vormen een sterke verdediging (onverwacht, gezien de resultaten van eerdere reviews) voor het opnemen van indicatoren die betrekking hebben op het activeren van leren, het bieden van uitdagende taken en het aan de orde stellen van leerstrategieën en metacognitie. Tegelijkertijd dienen de ietwat teleurstellende resultaten met betrekking tot direct, gestructureerd onderwijs naar onze mening niet te worden gebruikt als signaal om enkele van de meer ‘traditionele’ eigenschappen van gestructureerd onderwijs af te schaffen. Vergelijkbare conclusies dienen te worden getrokken met betrekking tot effectieve leertijd en de gelegenheid om te leren. Meer in het algemeen steunen onze resultaten daarom de ‘mix’ van direct onderwijs en constructivistische ideeën, die inherent is aan de huidige indicatoren inzake onderwijzen en leren.
- Klimaataspecten kwamen in onze studie niet sterk naar voren, ordelijke atmosfeer
duidelijk uitgezonderd, en de corresponderende indicator in het Kader wordt daarom door onze resultaten gesteund.
409
De Inspectiekaders en het opnieuw bezien van de maatschappelijke functies van het onderwijs Modernisering van de maatschappij heeft gevolgen voor het onderwijs. Tendensen als globalisatie, de kennismaatschappij, verschuivingen van de dominante richting van de economie, immigratie en demografische ontwikkelingen leiden ertoe dat de basisfuncties van het onderwijs opnieuw worden bezien. Daarbij dienen uiteindelijk de kerndoelen en middelen van het onderwijs aan de orde te komen, en als zodanig de basisonderwerpen in kwaliteitsevaluatiesystemen, zoals dat van de Nederlandse Inspectie. De sociologische en economische concept- en theorievorming over de maatschappelijke effecten van het onderwijs lijkt ertoe te leiden dat de nadruk wordt gelegd op het competentieconcept, als basis die ervoor moet zorgen dat het onderwijs aansluit op de eisen van de maatschappij. Levy and Murnane (2001) vatten de resultaten samen van verschillende soorten economische onderzoeken waaraan deze theorieën ten grondslag liggen, en die over het algemeen gericht zijn op de vraag welke competenties, in relatie tot formele scholing, de aantrekkelijkheid van gekwalificeerde individuen voor werkgevers zouden kunnen verklaren. Zij noemen vijf ‘kerncompetenties’: 11) Basisvaardigheden op het vlak van lezen en wiskunde zijn belangrijk voor de
uiteindelijke resultaten op de arbeidsmarkt, alsmede het vermogen zich aan te passen aan veranderende omstandigheden.
12) Het vermogen om doeltreffend te communiceren, zowel mondeling als schriftelijk, is van belang voor de uiteindelijke resultaten op de arbeidsmarkt.
13) In moderne bedrijven wordt het steeds belangrijker in staat te zijn productief in groepen te werken.
14) Deze laatste conditie benadrukt ook het belang van elementen van ‘emotionele intelligentie’, zoals goed met andere mensen kunnen opschieten.
15) Bekendheid met computers wordt steeds belangrijker op de arbeidsmarkt. Benaderd vanuit de context van formele scholing kan het idee van competenties in een continuüm worden geplaatst van soorten onderwijsresultaten die hiervoor al zijn genoemd, van specifiek inhoudelijk georiënteerde resultaten tot ‘inhoudsvrije’ persoonlijkheidskenmerken. De afzonderlijke posities in dit continuüm zijn weergegeven in Figuur 7.1.
410
- resultaten als gemeten door toetsen in tekstboeken - resultaten als gemeten door (door leerkrachten ontwikkelde) geïmplementeerde
schoolcurricula - resultaten als gemeten door toetsen op basis van het beoogde nationale curriculum - resultaten als gemeten door internationale toetsen die de gemeenschappelijke kern
van een reeks nationale curricula omvatten, bijv. TIMSS - ‘literacy’-toetsen, gericht op het meten van basisvaardigheden op het vlak van
lezen, en wiskundig en wetenschappelijk redeneren, bijv. PISA - competenties als uit meerdere facetten bestaande karaktereigenschappen van
individuen, met inbegrip van cognitieve, de motivatie betreffende en mogelijk andere componenten
- persoonlijkheidskenmerken, zoals interne of externe locus of control, onafhankelijkheid, algemene intelligentie
Figuur 1 (hoofdstuk 7): Een continuüm van onderwijsresultaten, van sterk
inhoudgerelateerd tot persoonlijkheidsafhankelijk
De praktische betekenis van het competentievraagstuk is dat het antwoord op een doeltreffende koppeling van het onderwijs aan maatschappelijk functioneren deels gezocht dient te worden in het onderwijzen van meer algemene vaardigheden en in het besteden van meer aandacht aan motiverings- en houdingsaspecten. Ook het idee van metavaardigheden, zoals algemeen probleemoplossende vaardigheden en ‘leren leren’, zou een meer reflectieve en zelfsturende houding van toekomstige burgers kunnen stimuleren. Op die manier worden ‘competenties’ beschouwd als bruikbare instrumenten ten dienste van de ‘employability’ van toekomstige burgers. Er kunnen verscheidene belangrijke accenten worden onderscheiden: - meer nadruk op enigszins algemene cognitieve vaardigheden, bijvoorbeeld in
termen van de verschillende ‘literacies’ (leesvaardigheid, wiskundig inzicht, enz.) die benadrukt worden in het PISA-project van de OESO, naast het beschikken over kennis op basis van de leerstof;
- nadruk op een bredere reeks persoonlijkheidskenmerken dan alleen het cognitieve aspect; sociale vaardigheden, motivatie om te leren en zelfs ‘emotionele intelligentie’;
- nieuwe inhoud en vaardigheden, bijvoorbeeld kennis over de burgermaatschappij en computervaardigheden;
- toegenomen aandacht voor morele kwesties en verschillende waardesystemen. Wat betreft de gevolgen voor het Inspectiekader van het grote belang dat momenteel wordt gehecht aan competenties, deze zijn allereerst te zien op het gebied van het kwaliteitsaspect van verworvenheden en leerresultaten. Op dit gebied lijkt de Inspectie zich te verlaten op het gebruik van toetsen en beoordelingsinstrumenten die ontwikkeld worden in andere contexten, zoals de bekende CITO-eindtoets voor basisscholen. Als
411
algemene strategie zou de Inspectie haar invloed kunnen aanwenden om de ontwikkeling van geldige meetinstrumenten te stimuleren waarmee bepaalde competenties (bijvoorbeeld, zie Diedrich, 2005) beoordeeld zouden kunnen worden. Een nieuwe soort ‘eindtermen’ zou er ook toe kunnen leiden dat nieuwe benaderingen in het onderwijs worden overwogen. In dit opzicht is in hoofdstuk 7 de casus besproken van informeel leren voor aan burgerschap gerelateerde competenties. Dergelijke relatief nieuwe richtingen in scholing, onderwijzen en leren zullen er waarschijnlijk toe leiden dat de set van procesindicatoren, met name die welke betrekking hebben op het algemene kwaliteitsdomein van onderwijzen en leren, opnieuw worden bezien. In de casestudy over informeel leren voor burgerschap werd een reeks aspecten van de identiteit van de school, haar institutionele normen en cultuur, beschouwd als relevant voor informeel leren. Onderzoeksstudies zoals die van Diedrich (2005) zouden nuttig kunnen zijn voor de Inspectie als zij mogelijk een nieuwe vorm en specificatie wil toekennen aan die procesindicatoren in het Kader die betrekking hebben op de leeromgeving, pedagogische benaderingen en het klimaat op school en in de klas. Wanneer het Toezichtkader van 2002 wordt vergeleken met dat van 2005, kan al een aanpassing in genoemde richting worden waargenomen.
De toepassing van het Inspectiekader in de context van onderwijsbestuur
De pilaren van het ‘nieuwe overheidsmanagement’, gezien vanuit een sturingsperspectief, zijn toegenomen autonomie inzake input en proceseigenschappen, en controle over resultaten. Evenzo bestaat de systematische hervorming in het onderwijs grotendeels uit twee domeinen: deregulering en decentralisatie enerzijds, en het vaststellen van nieuwe evaluatiemechanismen anderzijds. Het concept van ‘functionele decentralisatie’ omvat deze twee kerngebieden van systematische hervorming; het neemt namelijk expliciet patronen in overweging waarin vrijheid ten aanzien van proceskenmerken gepaard gaat met strikt toezicht op prestaties. In hoofdstuk 8 zijn kernconcepten in de discussie over bestuur, zoals decentralisatie en verantwoording, toegelicht. Daarna is het verbeteringspotentieel van op prestaties gebaseerde hervormingen besproken, waarbij de kwestie van het innovatieve potentieel van evaluatie-, toezicht- en beoordelingsmechanismen als belangrijkste kwestie naar voren is gekomen. Het onderwijsbeleid in Nederland heeft het afgelopen decennium nauwgezet de principes van het nieuwe overheidsmanagement gevolgd, met een bijzondere nadruk op autonomie van de school. Tegelijkertijd wordt de rol van het toezicht benadrukt. Dit gebeurt zowel in de richting van ‘horizontaal toezicht’, gericht op lokale belanghebbenden, als in de richting van ‘verticaal toezicht’, gekoppeld aan externe evaluatie en verantwoording (Koers, VO, 2004). Als we het Inspectiekader tegen de achtergrond van de principes van het nieuwe overheidsmanagement plaatsen en de empirische evidentie voor de resultaten van toepassingen bestuderen, komen we tot een schatting op basis van informatie over het effectiviteitsverhogende potentieel
412
ervan. Een cruciale kwestie in een context van toegenomen autonomie van de school is de plaats van schoolprocesindicatoren bij verticaal toezicht. Gesteld werd dat de principes van het nieuwe overheidsmanagement ondergebracht zouden kunnen worden onder de term ‘subsidiariteit’, met een lange gevestigde traditie in het onderwijsbeleid in Nederland. In de onderwijsgeschiedenis van Nederland werd de term subsidiariteit gebruikt om te verwijzen naar een specifieke manier waarop confessionele pressiegroepen in het onderwijs graag de relatie zagen tussen de overheid en organisaties die de belangengroeperingen in het onderwijsveld vertegenwoordigen. Volgens het subsidiariteitsbeginsel zou de staat zich niet mogen mengen in zaken die door georganiseerde eenheden van professionals kunnen worden afgewikkeld. In eerste instantie waren deze georganiseerde eenheden de confessionele organisaties of pressiegroepen van vertegenwoordigers in het onderwijsveld, met name hun parapluorganisaties. ‘Subsidiariteit’ was de term die de rooms-katholieken hanteerden, terwijl de protestanten spraken van ‘soevereiniteit in eigen kring’. Leune (1987, 379-380) wijst op het corporatistische karakter van dit soort concepten. Volgens het subsidiariteitsbeginsel treedt de staat slechts subsidiair op, dat wil zeggen dat zij alleen ingrijpt als vervanging, wanneer nodig. Een eenvoudig voorbeeld van subsidiariteit is een rij-instructeur die het besturen van het voertuig overneemt wanneer de leerling een fout maakt, maar in alle andere gevallen rustig toekijkt zonder in te grijpen. Binnen de context van de Europese Commissie wordt de term subsidiariteit gebruikt voor het beginsel dat wat door de lidstaten tot stand kan worden gebracht niet door de centrale instellingen van de Unie dient te worden gedaan. Een rechtlijnige toepassing van het subsidiariteitsbeginsel op de vraag wie, op welk niveau, wat moet evalueren, kan zijn dat eenheden of beheersniveaus slechts verantwoordelijkheid moeten dragen voor het evalueren van die gebieden waarover zij rechtstreekse operationele controle hebben. Volgens deze redenering zou iemand van mening kunnen zijn dat de Inspectie geen toezicht op de procesindicatoren van scholen moet houden, omdat alleen de scholen verantwoordelijk zijn voor het instructieproces. De Inspectie zou daarentegen, volgens dit standpunt, scholen alleen moeten monitoren op basis van hun resultaten, c.q. de leerresultaten van de leerlingen. Enkele jaren geleden werd dit standpunt inderdaad ingenomen in een verslag van de Onderwijsraad (het belangrijkste adviesorgaan in het onderwijsveld in Nederland), toen deze raad een speciale set van prestatietoetsen voorstelde om de prestaties van scholen te meten. In termen van actueel onderscheid tussen verticaal en horizontaal toezicht, zouden de procesindicatoren dan het domein van uitsluitend de scholen worden, en gericht moeten zijn op de eigen voorzieningen voor kwaliteitszorg en zelfevaluatie van de school. Daarbij kunnen drie scenario's worden onderscheiden:
413
a) de huidige situatie waarin de Inspectie zowel de resultaten als de procesindicatoren bewaakt;
b) een scenario waarin de scholen assistentie en richtlijnen ontvangen, mogelijk in de vorm van specifieke instrumenten en software om zelfstandig schoolzelfevaluaties te verrichten die proces- en resultaatindicatoren bevatten; terwijl de Inspectie afziet van de monitoring van procesindicatoren;
c) een scenario van maximale autonomie van de school; waarin scholen vrijgelaten worden in de keuze van de manier waarop zij de schoolzelfevaluatie verrichten; de Inspectie ziet af van rechtstreekse monitoring van schoolprocesindicatoren, maar heeft binnen de context van ‘proportioneel toezicht’ een indirecte invloed op de zelfevaluatieprocedures van de scholen.
Bij de keuze tussen deze alternatieven is het van belang te verwijzen naar bepaalde conclusies in het schoolverbeteringsdebat, namelijk dat er enkele incidentele maar tegelijkertijd opvallende resultaten zijn waaruit blijkt dat van te voren extern gestructureerde programma's soms tot zeer goede resultaten leiden. Het succes van de recente hervormingsprogramma's voor wiskundig inzicht en leesvaardigheid in het VK kan ook als bewijs voor deze conclusie worden gezien. Misschien zijn er grenzen aan de mate waarin procesautonomie in het primaire en voortgezette onderwijs bevorderd zou moeten worden. Hoewel dit nog lijkt in te druisen tegen de huidige dominante beleidsvisie in Nederland, zou het nog steeds beschouwd moeten worden als argument voor de monitoring van procesindicatoren door de Inspectie. Het hoofdstuk over ervaringen met kwaliteitszorg en schoolzelfevaluatie, dat hieronder wordt samengevat, is eveneens relevant voor het afwegen van de voors en tegens van deze drie scenario's.
Criteria voor het beoordelen van schoolzelfevaluatiebenaderingen Twee kwaliteitsaspecten in het Inspectiekader verwijzen respectievelijk naar kwaliteitszorgprocedures en condities voor kwaliteitszorg op school. In dit hoofdstuk zullen de kwaliteitsindicatoren in het Inspectiekader worden vergeleken met de allernieuwste perspectieven op het vlak van schoolzelfevaluatie. In hoofdstuk 6 zijn verschillende referentiekaders gebruikt om deze kwaliteitsindicatoren in perspectief te plaatsen:
- Criteria die in Nederland gebruikt zijn om de schoolzelfevaluatie-instrumenten en procedures voor kwaliteitszorg van scholen te beoordelen; d.w.z. die van Q*Primair, Q5 en de Inspectie;
- Criteria die gebruikt zijn door buitenlandse inspecties en groepen inspecties, d.w.z. SICI;
- Meer algemene evaluatiestandaarden (d.w.z. die van het Joint Committee of Standards in de VS, onder voorzitterschap van Daniel Stufflebeam);
414
- Een nauwkeuriger blik op het gebruik van schoolzelfevaluatie-instrumenten en de toepassing van kwaliteitszorg (onder andere op basis van ervaringen met het ZEBO-project);
- Een nauwkeuriger blik op nauwkeurigheidsnormen die worden toegepast op schoolzelfevaluatie, de kwestie van objectiviteit.
Het hoofdstuk biedt beschrijvend materiaal over al deze bijdragen om de actuele stand van zaken met betrekking tot proportioneel toezicht te beoordelen, en de respectieve kwaliteitsindicatoren in het Inspectiekader.
Het idee van proportioneel toezicht is een creatieve toepassing van decentralisatie en ‘subsidiariteit’ in het onderwijs, en betekent dat alles wat mogelijk en redelijk op een lager beheersniveau kan worden gedaan, niet op een lager [hoger?] niveau dient te worden uitgevoerd (Scheerens, 1997). Maar misschien toont deze formulering ook de achilleshiel van deze strategie. Kan schoolzelfevaluatie werkelijk voldoen aan de eisen van externe evaluatie? Wat in hoofdstuk 6 is aangetoond, is dat er veel ontwikkelingsinspanningen zijn en worden verricht om schoolzelfevaluatie en kwaliteitszorg in Nederland mogelijk te maken, te faciliteren en te stimuleren. Er is een relatief grote set van schoolzelfevaluatiesystemen en -instrumenten ontwikkeld, en organisaties als Q5 en Q*Primair hebben geïnvesteerd in de ontwikkeling van criteria om deze instrumenten en procedures te beoordelen, en in experimentele goede praktijktoepassingen. Het is interessant op te merken dat de door Q5 en Q*Primair ontwikkelde en toegepaste kwaliteitsnormen afwijken van die welke zijn toegepast in studies van de Onderwijsinspectie in de zin van het belang dat wordt gehecht aan wat Stufflebeam e.a. (1971) verstaan onder ‘nauwkeurigheidsstandaarden’. De Inspectie beschouwt criteria zoals betrouwbaarheid en geldigheid meer consistent als essentieel voor de waarde van de schoolzelfevaluatie, en is over het algemeen strenger en kritischer bij de beoordeling van de beschikbare set van instrumenten. Wat onder de noemer van schoolzelfevaluatie en kwaliteitszorg wordt gedaan, is een mengsel van elementen met een iets andere richting en sterk afwijkende tradities. De belangrijkste richtingen zijn: een wetenschappelijke evaluatierichting, met de nadruk op nauwkeurigheid, betrouwbaarheid en geldigheid van de procedures; een administratieve en klantgerichte tak van kwaliteitsbeheerssystemen, en ten slotte een meer kwalitatieve, op meningen gebaseerde schoolverbetering. Vanuit het oogpunt van de scholen als gebruiker heeft elke richting haar sterke en zwakke punten: - scholen kunnen de objectiviteit en accuratesse van het gebruik van gestructureerde
instrumenten die getest zijn op betrouwbaarheid en geldigheid waarderen, maar de meeste blijken het niet eens te zijn met de implicaties van de standaardisatie, de starheid van procedures, en de doelgerichte reikwijdte die gemakkelijk wordt verworpen als bekrompen en reductionistisch;
415
- kwaliteitszorgsystemen kunnen de modieuze aantrekkingskracht hebben van de zich snel bewegende zakenwereld en het meer reële voordeel te reageren op klanten; toch lukt het gewoonlijk niet met deze systemen het primaire proces van onderwijzen en leren aan te pakken, en als dat getracht wordt, leidt dat vaak tot karikaturen in hun inspanningen om onderwijsprocessen in procedures te vatten;
- het meest verdachte van deze drie, ten minste uit het oogpunt van evaluatie, is het schoolverbeteringsperspectief, dat zich onthoudt van accuratesse bij het vaststellen van feiten, en kan leiden tot een oppervlakkige consensus over veranderen in plaats van een gedegen diagnose; in feite is er sprake van een eeuwig dilemma in onderwijsanalyse en actie, veroorzaakt door een discrepantie tussen de drang om aan te passen en te wijzigen, en de kennis die in het beste geval eenzijdig en omstreden is.
Naar onze mening dient het wetenschappelijke perspectief de overhand te hebben bij proportioneel toezicht. Bij proportioneel toezicht wordt een deel van de externe evaluatie in feite gedelegeerd aan de scholen. Dit kan slechts werken indien scholen dezelfde accuratesse in hun methodes hanteren als die welke wordt vereist van externe evaluatie. Uit het in dit hoofdstuk getoonde bewijs met betrekking tot de werkelijke stand van zaken op het vlak van schoolzelfevaluatie blijkt dat slechts een minderheid van scholen begint te voldoen aan de huidige normen van de Inspectie. Gezien de behoefte aan nauwgezette interne evaluatie, zou deze stand van zaken door de Inspectie verbeterd kunnen worden door de ontwikkeling van wetenschappelijk gebaseerde instrumenten voor schoolzelfevaluatie te stimuleren en deze aan scholen ter beschikking te stellen. In de praktijk zou dit gedaan moeten worden door de ontwikkeling van instrumenten te stimuleren en integrale systemen voor schoolzelfevaluatie te faciliteren die gebruik maken van geavanceerde praktijken op gebieden als leerlingvolgsystemen op basis van itemresponsmodellen, geautomatiseerde schooladministratiesystemen, en empirisch gevalideerde procesindicatoren.
Belangrijkste conclusies van de studie - Bij het observeren van de veranderingen in het Kader tussen de versie van 2002 en
die van 2005 blijkt meer nadruk te zijn gelegd op brede ontwikkeling, oriëntatie op de arbeidsmarkt en ontwikkeling van de capaciteiten van de leerlingen; dit stemt overeen met de huidige ideeën over vergroting van de maatschappelijke relevantie van het onderwijs met betrekking tot burgerschap en competenties.
- Verdere implicaties van de huidige ideeën over de belangrijkste maatschappelijke
functies in het onderwijs zouden de Inspectie kunnen stimuleren om de reeks
416
resultaatmetingen en –beoordelingen te vergroten, en R&D op het vlak van de impact van institutionele schoolnormen en schoolcultuur op formeel en informeel leren te stimuleren. Dergelijke studies zouden ertoe kunnen leiden dat bepaalde aanvullende procesindicatoren worden opgenomen die gericht zijn op het leerpotentieel van de schoolomgeving in totaal (d.w.z. met inbegrip van klimaat en cultuur, gedragsnormen los van de expliciete onderwijsregelingen).
- Onze resultaten laten volop ruimte voor de discussie over de gerichtheid op
instrumentele effectiviteit die in het huidige Kader nog steeds de overhand heeft in vergelijking met een bredere opvatting over organisatiekwaliteit. In onze onderzoekssynthese vinden we weinig steun voor uitbreiding van de huidige set van indicatoren die voornamelijk gericht is op het niveau van de klas, met een groot aantal indicatoren op schoolniveau, bijvoorbeeld over ondersteunende en managementfuncties. De aanhoudende ontwikkelingen met betrekking tot het schoolmanagement, de taakverruiming van leerkrachten en het human resources management op school kunnen echter worden beschouwd als argumenten om meer ruimte te geven voor indicatoren die gericht zijn op het meten van dergelijke verschijnselen op schoolniveau.
- Binnen de context van het onderwijsbestuur in Nederland is proportionele inspectie
een goed passend concept, dat de autonomie van de school ‘waar mogelijk’ benadrukt. Afgezien daarvan is er een zeker bewijs ten aanzien van de effectiviteit van hervormingsstrategieën in het onderwijs die afhankelijk zijn van extern gestructureerde programma's. Daarom dient de huidige mix van direct en indirect toezicht (via kwaliteitszorg van de school en proportionele inspectie) misschien niet te worden afgeschaft ten gunste van radicalere alternatieven die het toezicht verder decentraliseren.
- Gezien de stand van zaken met betrekking tot kwaliteitszorg en schoolzelfevaluatie
is er alle reden voor de Inspectie om de verbetering van wetenschappelijk solide instrumenten en methoden voor schoolzelfevaluatie te ondersteunen.
- De in dit verslag beschreven onderzoekssyntheses wijzen op een goede match tussen
de set van indicatoren van het Inspectiekader en de set variabelen die vaak bestudeerd worden in onderzoek naar onderwijseffectiviteit. De onderzoeks-resultaten zijn relatief inconsistent voor veel variabelen, wat erop wijst dat de empirische basis voor een groot aantal variabelen verre van solide is. Een waarschuwing is dus zeker op z'n plaats, en ontmoedigt het trekken van sterke conclusies uit de onderzoeksresultaten die gepresenteerd zijn. Opvallende uitkomsten waren de relatief veelvuldige steun voor variabelen als cognitieve
417
stimulering en leerstrategieën leren gebruiken, en een geringere steun (ook in vergelijking met de resultaten van eerdere reviews) voor variabelen die verband houden met gestructureerd onderwijs. Deze uitkomsten dienen naar onze mening eerder te worden gezien als ondersteuning voor de mix van indicatoren in het Inspectiekader die variabelen bevatten uit zowel het directe instructie- als het cognitieve activeringsmodel.
418