iea international civic and citizenship education study (iccs)
DESCRIPTION
IEA INTERNATIONAL CIVIC AND CITIZENSHIP EDUCATION STUDY (ICCS). A new comparative study of civic and citizenship education AERA Annual Meeting New York, 2008. Symposium Outline. Introduction Concept and Design Assessing Knowledge, Background & Perceptions - PowerPoint PPT PresentationTRANSCRIPT
IEA INTERNATIONAL CIVIC AND CITIZENSHIP EDUCATION STUDY
(ICCS)
A new comparative study of civic and citizenship education
AERA Annual Meeting
New York, 2008
AERAAnnual Meeting
March 2008
Symposium Outline
• Introduction• Concept and Design• Assessing Knowledge, Background &
Perceptions• Collecting School and Teacher Data• Studying CCE in the European Context• Citizenship Competencies in the Latin
American Region• Discussant• General Discussion
AERAAnnual Meeting
March 2008
Purposes of study
• International comparison of outcomes of civic and citizenship education at Grade 8
• Responses to challenges in civic and citizenship education in changed contexts since CIVED in 1999
AERAAnnual Meeting
March 2008
Instrumentation
• International cognitive test • International student questionnaire
– Background – Perceptions
• Teacher questionnaire• School questionnaire• National Contexts Survey• Regional student instruments
– European– Latin American– Asian (under development)
AERAAnnual Meeting
March 2008
Participating countries
AERAAnnual Meeting
March 2008
ICCS Current Status
• 38 countries participating
• Field trial– October 07 to January 08
• Field trial data– 19,800 students (30 per school)– 10,500 teachers (16 per school)– 668 schools– 29 countries
AERAAnnual Meeting
March 2008
Next steps
• Analysis of Field Trial data
• Revision of instruments
• Final forms
• Main Survey– Southern hemisphere (end 2008)– Northern hemisphere (early 2009)
• Data compilation and analysis (2009)
• Reporting (2010)
AERAAnnual Meeting
March 2008
Concept and Design of the International Civic and Citizenship
Study
Julian Fraillon and Wolfram Schulz
(ACER)
AERA Annual Meeting
New York, 2008
AERAAnnual Meeting
March 2008
Contents
• ICCS Overview
• The ICCS Assessment Framework Summary Introduction
• ICCS Assessment Framework links to CIVED
• ICCS Assessment Framework Structure
• ICCS Assessment Framework Domains - Detail
AERAAnnual Meeting
March 2008
ICCS Overview• The purpose of the International Civic and
Citizenship Education Study (ICCS) is to investigate, in a range of countries, the ways in which young people are prepared and consequently ready and able to undertake their roles as citizens.
• The study will report on:– student achievement in a test of knowledge and
conceptual understandings in civics and citizenship.
– data about student activities, dispositions, and attitudes related to civic and citizenship education.
– contextual data that will help to explain variation in the outcome variables.
AERAAnnual Meeting
March 2008
ICCS Overview• The study builds on the previous IEA studies
of civics and citizenship education.• The study is underpinned by six research
questions that address the degree to which Grade 8 (or equivalent)* students are ready and able to undertake their roles as citizens AND the contextual factors that play a role in this readiness.
*Mean age of student not less than 13.5 years
AERAAnnual Meeting
March 2008
ICCS Assessment Framework
• The Assessment Framework has been developed in consultation with experts and iterative review with ICCS country representatives.
• The Assessment Framework comprises two main components1. The civics and citizenship framework outlines
the aspects to be addressed when collecting the outcome measures through the cognitive test and the student perceptions questionnaire.
2. The contextual framework provides a mapping of context factors that might influence outcome variables and explain their variation.
AERAAnnual Meeting
March 2008
ICCS AF – Links to CIVED
• Key conceptualisation of student learning:
1. The student as the central agent in their civic world, with both an influence on and being influenced by their multiple connections with their civic communities.
2. Young people learn about civics and citizenship through their interactions with their multiple civic communities and not only through formal classroom instruction.
AERAAnnual Meeting
March 2008
ICCS AF – Links to CIVED
• Conceptually similar survey design matrices that link question types to civics and citizenship content.
• Some secure trend items from CIVED as a concrete scaling link between the two studies.
AERAAnnual Meeting
March 2008
ICCS AF - Structure
• The civics and citizenship framework consists of:– Four content domains– Four affective/behavioural domains– Two cognitive domains
AERAAnnual Meeting
March 2008
ICCS AF – Content Domains
• The four content domains in the ICCS Civics and Citizenship Framework are:
1. Civic society and systems
2. Civic principles
3. Civic participation
4. Civic identities.
AERAAnnual Meeting
March 2008
ICCS AF – Content Domains
AERAAnnual Meeting
March 2008
ICCS AF – Affective Behavioural Domains
• The four affective-behavioural domains in the ICCS Civics and Citizenship Framework are:
1. Value beliefs
2. Attitudes
3. Behavioural intentions
4. Behaviours.
AERAAnnual Meeting
March 2008
ICCS AF – Cognitive Domains
• The two cognitive domains in the ICCS framework are:
1. Knowing
2. Reasoning and analyzing.
AERAAnnual Meeting
March 2008
ICCS AF Design MatrixContent
Domain 1: Civic society and systems
Content Domain 2:
Civic principles
Content Domain 3:
Civic participation
Content Domain 4:
Civic identities
Cognitive Domains
Knowing I II III IV
Analysing and reasoning
V VI VII VIII
Affective-behavioural Domains
Value beliefs A B C D
Attitudes E F G H
Behavioural intentions
I
Behaviours J
AERAAnnual Meeting
March 2008
ICCS AF Contextual Domains
Wider communityEducational systemHistory and culture
School/classroom:CharacteristicsCompositionResources
Home environment:Family backgroundSocial group
Indicators related to:Civic society and systemsCivic principlesCivic participationCivic identities
Wider communityEducational policiesPolitical events
School/classroom:InstructionGovernance
Student:Socialization& learning
Home environment:CommunicationActivities
Antecedents Processes Outcomes
Student:Characteristics
AERAAnnual Meeting
March 2008
ICCS AF Contextual DomainsLevel of ... Antecedents Processes OutcomesNational and other communities
NCQ & other sources:Democratic historyStructure of education
NCQ & other sources:Intended curriculumPolitical developments
StT & StPQ & StBQ: Test results Student perceptions Student behaviours
School/classroom ScQ & TQ:School characteristicsResources
ScQ & TQ:Implemented curriculumPolicies and practices
Student StBQ:GenderAge
StBQ:Learning activitiesPracticed engagement
Home environment StBQ:Parent SESEthnicityLanguageCountry of birth
StBQ:CommunicationPeer-group activities
AERAAnnual Meeting
March 2008
Additional information
• The paper: [email protected] OR [email protected]
• The paper and/or the Assessment Framework: http://iccs.acer.edu.au
AERAAnnual Meeting
March 2008
Assessing Student Knowledge, Background and Perceptions in
the International Civic and Citizenship Study
AERA Annual Meeting
New York, March 2008
Wolfram Schulz (ACER)
Falk Brese (IEA DPC)
AERAAnnual Meeting
March 2008
Contents of presentation
• Overview of areas assessed with student instruments in ICCS
• General principle for scaling and analysis
• Review of measurement invariance
• Some preliminary results
AERAAnnual Meeting
March 2008
Student instruments
• Student cognitive test– One hour testing time– Six rotated booklets– Multiple-choice, true/false and open-ended items
• Student questionnaire– Questions on student background and items
measuring student perceptions – Different formats (with and without “don’t know”
category)– Three forms with different combinations of item
batteries
AERAAnnual Meeting
March 2008
Cognitive Test
• Four content domains– Civic society and systems– Civic principles – Civic participation– Civic identities
• Two cognitive domains– Knowing– Reasoning and analysing
• Cluster with CIVED link items– Multiple-choice items
AERAAnnual Meeting
March 2008
Student questionnaire(perceptions)
• Four content domains (as above)– Civic society and systems– Civic principles – Civic participation– Civic identities
• Affective-behavioural domains:– Value beliefs– Attitudes– Behavioural intentions– Behaviours
AERAAnnual Meeting
March 2008
Value beliefs
• Democratic values– Set of modified CIVED items
• Citizenship values– Items already used in CIVED
• Students’ acceptance of socially undesirable behaviour
AERAAnnual Meeting
March 2008
Attitudes (1)
• Students' self-cognitions related to Civics and Citizenship – Interest in political events and social
issues– Self-concept regarding political
participation (internal efficacy)– Citizenship self-efficacy – Perceptions of distinctiveness compared
to others in the country– Sense of belonging to communities
AERAAnnual Meeting
March 2008
Attitudes (2)
• Students' attitudes towards rights and responsibilities – Attitudes towards gender rights– Attitudes toward rights of ethnic/racial
groups– Attitudes toward immigrants– Attitudes towards policies to avert threats
to democracy
AERAAnnual Meeting
March 2008
Attitudes (3)
• Students' attitudes towards institutions – Trust in institutions – Responsiveness of the political system
(external efficacy) – Confidence in student participation at
school – Attitudes towards one’s nation – Satisfaction with country’s achievements
AERAAnnual Meeting
March 2008
Behavioural intentions
• Preparedness to participate in forms of civic protest – Legal– Illegal
• Behavioural intentions regarding future political participation as adult – Electoral participation– Active political participation
• Behavioural intentions regarding future participation as a young person
AERAAnnual Meeting
March 2008
Behaviours
• Involvement in civic-related participation in the community – Only minority of students is active in the
community!
• Involvement in civic-related activities at school
AERAAnnual Meeting
March 2008
Student questionnaire: background (1)
• Context of schools and classrooms– Classroom climate for civic and
citizenship education at school – Perceptions about students’ influence on
decision-making at school – Student perceptions of school climate
AERAAnnual Meeting
March 2008
Student questionnaire: background (2)
• Context of the home environment– Parental socio-economic status
• Parental occupation• Parental education• Household possessions
– Cultural/ethnic background • Country of birth (students and parents)• Language use at home• Self-reports on ethnicity (country-specific)
AERAAnnual Meeting
March 2008
Student questionnaire: background (3)
• Context of the home environment– Family composition– Indicators of social interaction
• Discussion of social/political issues• Media information• Social activities
AERAAnnual Meeting
March 2008
Student questionnaire: background (4)
• Context of the individual student– Age– Gender– Expected educational level
• with reference to ISCED classification
– Out-of-school activities
AERAAnnual Meeting
March 2008
Field trial data analysis
• All ICCS instruments were tested in international field trial (October to December 2007)
• Samples of about 600 students (from about 25 schools) per country
• Data from 29 countries in current (preliminary analysis)
AERAAnnual Meeting
March 2008
Purposes of field trial data analysis
• Inform on item selection for the main study
• Inform on most appropriate formats for questionnaire items
• Inform on selection of constructs– Predictive validity– Cross-country validity
AERAAnnual Meeting
March 2008
Test data analysis
• Item Response Theory– One-parameter Rasch model
• Review of item characteristics– Scalability of items– Dimensionality of items
• Review of differential item difficulty– Gender DIF– Item-by-country interaction
• Review of link items– Relative item difficulty
• Review of test length
AERAAnnual Meeting
March 2008
Analysis of Coder reliability
• 10 percent of cognitive items are open-ended and need to be coded
• Necessary to – inform the development of scoring rubrics– inform the translation verification of
scoring rubrics– examine the extent of between country
differences in coders
AERAAnnual Meeting
March 2008
Questionnaire data analysis
• Review percentages in each category and missing responses
• Analyse the dimensional structure of item batteries• Exploratory and confirmatory factor analyses
• Determine scaling properties of items and constructs• IRT scaling (Partial Credit Model)
• Review relationships between constructs and between constructs and test results
AERAAnnual Meeting
March 2008
Ways of Construct Validation in International Studies
• Assessing item dimensionality through confirmatory factor analysis and compare fit across countries
• Testing parameter invariance– by constraining item loadings (SEM)– by reviewing item-by-country interaction
(IRT)– by constraining item parameters (IRT)
• Analysing relationships with reference variables
AERAAnnual Meeting
March 2008
Some first results
• Field trial results are encouraging and indicate a high quality of most of the item material
• Cognitive test items have generally good scaling properties
• Most of the questionnaire items have good measurement properties
AERAAnnual Meeting
March 2008
Match of test and abilities
------------------------------------------------------------ | | | | | | 2 | | X|57 | XX| | XX| | XX|48 67 | XX|23 | XXX| | XXXX|78 | XXXX|31 45 | 1 XXXX|76 | XXXXXX|17 42 80 97 | XXXXX|12 41 72 | XXXXXX|59 68 73 77 83 | XXXXXXX|61 75 91 98 | XXXXXXXX|8 79 | XXXXXXXX|11 37 60 85 89 90 94 | XXXXXXXXX|14 29 47 | XXXXXXXX|6 53 65 70 84 92 | XXXXXX|22 27 63 81 | 0 XXXXXXX|44 64 88 | XXXXXXX|2 4 7 9 13 32 50 71 93 | XXXXXXXX|15 55 62 96 | XXXXXXXXXX|5 20 25 58 | XXXXXXXX|16 24 26 30 | XXXXXXX|18 43 46 49 51 52 54 82 86 | XXXX|1 21 28 | XXXXX|3 19 34 35 56 66 | XXXXX|40 74 | -1 XXX| | XXX|10 69 | XX|33 36 95 | XX| | X| | X| | X|38 87 | X|39 | | | | | -2 | | | | ============================================================ Each 'X' represents 123.5 cases ============================================================
AERAAnnual Meeting
March 2008
Questionnaire scale reliabilities (1)
AERAAnnual Meeting
March 2008
Questionnaire scale reliabilities (2)
AERAAnnual Meeting
March 2008
Multiple regression models for pooled sample
AERAAnnual Meeting
March 2008
The paper: [email protected]
The paper and/or the
Assessment Framework: http://iccs.acer.edu.au
AERAAnnual Meeting
March 2008
Collecting School and Teacher Data in International Civic and
Citizenship Study
Bruno Losito and Gabriella Agrusti(Universita Roma Tre)AERA Annual Meeting
New York, 2008
AERAAnnual Meeting
March 2008
• Civic and citizenship education in ICCS– Knowledge and understanding– Attitudes– Behaviors and behavioral intentions– Value beliefs
AERAAnnual Meeting
March 2008
ICCS Research QuestionWhat aspects of schools and education systems are
related to achievement in and attitudes to civics and citizenship including:
a) curriculum or program content structure and delivery
b) teaching practices such as those that encourage higher order thinking and analysis in relation to civics and citizenship
c) aspects of school organisation including opportunities to contribute to conflict resolution, participate in governance processes, and be involved in decision making.
ScQ and TcQ Rationale
AERAAnnual Meeting
March 2008
• Relevance of the school environment for CCE (school as a democratic learning environment)
• Whole school student experience
• Actual possibility to exercise citizen rights and responsibilities within the school
• School culture and classroom climate
ScQ and TcQ Rationale
AERAAnnual Meeting
March 2008
• School heads and teachers as “key factors” in CCE
• Teachers of all subjects (not only CCE reated subjects teachers)– international option
ScQ and TcQ Rationale
AERAAnnual Meeting
March 2008
ScQ Content
• School questionnaire content, constructs and variables
• Five research areas– School characteristics– School autonomy– School community relationships– School environment– CCE at school (implementation of CCE
at school level)
AERAAnnual Meeting
March 2008
ScQ Content
• Demographics
• School characteristics– Type of school– School enrolment– Number of teachers
• School autonomy (in management and in educational planning)
AERAAnnual Meeting
March 2008
ScQ Content
• School-community relationships– resources available to students in the
local community– issue of social tension within the local
community– issues of social tension within the school
• School environment– Teachers, parents participation– Sense of belonging to the school
(teachers, students, non teaching staff)
AERAAnnual Meeting
March 2008
ScQ Content
• CCE at school (aims of CCE, responsibilities in CCE
AERAAnnual Meeting
March 2008
TcQ Content
• Teacher questionnaire content, constructs and variables
• Teacher characteristics (demographics, work experience)
• Teachers confidence in teaching methods and approaches
AERAAnnual Meeting
March 2008
TcQ Content
• CCE at school- Aims of CCE- CCE teaching and learning activities- Teachers’ confidence in teaching CCE
topics- Improvement of CEE
AERAAnnual Meeting
March 2008
TcQ Content
• School environment– teachers’ participation in school
governance– importance given to students’ opinions– students’ opportunity to participate in
CCE activities– teachers’ participation in CCE related
activities– Students’ sense of belonging to the
school
AERAAnnual Meeting
March 2008
TcQ Content
• School community relationships
• Teaching strategies
• Classroom climate
AERAAnnual Meeting
March 2008
• Development process– Involvement of NRCs (country contexts)
• Uses of school and teacher data– reporting data at the country level– aggregated school level data
• Descriptive and predictive levels
AERAAnnual Meeting
March 2008
AERAAnnual Meeting
March 2008
Studying Civic and Citizenship Education
in the European context
David Kerr and Joana LopesNFER, England
AERA Annual MeetingNew York, 2008
[email protected]; [email protected]
ICCS – A new comparative study of civic and citizenship education
AERAAnnual Meeting
March 2008
European context
Growing interest in civic and citizenship education
AERAAnnual Meeting
March 2008
European context
Interest in civic and citizenship education:
AERAAnnual Meeting
March 2008
Purpose of European module
AERAAnnual Meeting
March 2008
European Context in ICCS
• International instrumentsEuropean items
• National Contexts Survey (NCS)European countries
• European module
AERAAnnual Meeting
March 2008
EM development
• Consultative, iterative process
• Influences:Individual European countriesCouncil of EuropeEuropean CommissionCentre for research in Lifelong Learning
(CRELL)
AERAAnnual Meeting
March 2008
Process of construction of EM
AERAAnnual Meeting
March 2008
Knowledge Test
Knowledge TestEuropean institutions and lawsparticularly European Union (EU)-related
Knowledge self-assessment
AERAAnnual Meeting
March 2008
Questionnaire
Affective-behavioural domain
Value Beliefs
Content Domain
Equal opportunities within European countries
Civic principles
Political and economical organisation of European countries
Civic society and systems
Enlargement of the European Union
Civic society and systems
AERAAnnual Meeting
March 2008
QuestionnaireAffective-behavioural domainAttitudes
Content Domain
European self-identity Civic identities
Openness to other European countries/cultures
Civic principles
Learning European languages
Civic principles
Freedom of travel, settlement and work within Europe
Civic principles
AERAAnnual Meeting
March 2008
Questionnaire
Affective-behavioural domainBehaviours
Content Domain
Participation in European-level groups or events
Civic participation
Interpersonal communication behaviours
Civic participation
AERAAnnual Meeting
March 2008
Questionnaire
Knowledge of European foreign languages
(self-assessed)
Confidence self-assessment
AERAAnnual Meeting
March 2008
Next steps
AERAAnnual Meeting
March 2008
Outcomes
International report
European regional report (likely)European module International instruments - European itemsNational contexts survey report – European
countries
AERAAnnual Meeting
March 2008
The Evaluation of Citizenship Competencies
in the Latin American Region
Fernando Reimers (Harvard University)
AERA Annual MeetingNew York, 2008
DISCUSSIONIEA International Civic and Citizenship
Education Study (ICCS)
Judith Torney-Purta
University of Maryland
AERA Annual Meeting
New York, 2008
ICCS:A new comparative study of civic and
citizenship education
Questions and Discussion
?? !! ** ##