Proposed student questionnaire for the main survey

National Research Coordinators Meeting Windsor, June 2008

NRCMeetingWindsor

June 2008

Contents of presentation

• Criteria used for selection of questionnaire material

• Overview of field trial questions and recommendations

• Length of proposed instrument

• Proposed changes

NRCMeetingWindsor

June 2008

Criteria for selection

• Field trial analysis results– Scaling properties (for scaled items)– Item dimensionality– Face validity– Review of relationship between related

variables/constructs

• Coverage of assessment framework• Feasibility of adaptations to national

contexts

NRCMeetingWindsor

June 2008

General proposals

• Only one (combined) student questionnaire (with both background and perceptions questions)

• Only one form

• No “don’t know” categories

NRCMeetingWindsor

June 2008

Review of field trial instruments

• Three categories of questions– To be retained without changes (in green)– To be modified (in yellow)– To be deleted (in red)

• Some questions are flagged as having “medium priority”

• Recent advice from the PAC meeting 7-8 June appears in orange

NRCMeetingWindsor

June 2008

Student characteristics

• Age (A01)

• Gender (A02)

• Ethnicity (optional) (A03)

• Expected educational level (A04)– without “don’t know” category

NRCMeetingWindsor

June 2008

Home environment

• Household composition (B01)– Medium priority only

• Country of birth (B02)

• Language use at home (B03)

NRCMeetingWindsor

June 2008

Parent background

• Parental occupation (B04, B06)– Good measure of socio-economic background– Somewhat larger percentages of missing values

in some countries

• Parental education (B05, B07)– Without “don’t know” category– Still considerable percentages of missing values

• Parental interest in political/social issues (B08)– Interesting results!

NRCMeetingWindsor

June 2008

Household possessions

• Household items (B09)– low reliabilities (especially in more

developed countries)– low correlations with other socio-

economic indicators)– PAC advice: Consider as an

international option

• Books in the home (B10)– Good indicator of socio-economic

background

NRCMeetingWindsor

June 2008

Measuring socio-economic background in the main survey

• It is proposed to have three indicators– Parental occupation (highest of both

parents)– Parental education (highest of both

parents)– Number of books at home

• Analysis will be undertaken to explore computation of composite index

NRCMeetingWindsor

June 2008

Student activities

• Out-of-school leisure time activities (C01)– Keep only items on reading and hanging out– Add: Homework and study time– PAC advice: Consider keeping also (at least)

computer and television use items• Activities out of school (C02)

– Discussion item form scale– Media information items do not scale but are of

interest as single items (wording change for TV news)

– Social participation items do not scale and should not be retained

– PAC advice: Consider combining items on organised social activities (boy scouts, youth clubs etc.)

NRCMeetingWindsor

June 2008

Students’ civic participation

• Civic participation in the community (C03)– Worked as scale and interesting associations

with other variables – PAC advice: An additional note to include

religious services was initially considered but should rather not be included

• Civic participation at school (C04)– Two items reflect different dimension (tutoring,

school newspaper) and are proposed to be deleted

– There were adaptation problems with the item on school protest, which is proposed as another deletion

– PAC suggestion: Consider same categories as for previous question

NRCMeetingWindsor

June 2008

School-related perceptions - 1

• Open climate for classroom discussion (D01)– Good scaling properties and interesting

correlations– PAC advice: Use rather same categories as in

CIVED• Student influence at school (D02)

– Good scaling characteristics but negative correlations with civic knowledge

– ISC sees this as question with medium priority– PAC advice: Additional item about

extracurricular activities

NRCMeetingWindsor

June 2008

School-related perceptions - 2

• School climate items (D03)– Good scaling properties for student-teacher

relation items (medium priority)– Students’ sense of belonging has not always

satisfactory scaling properties and should be deleted

• Students’ confidence in school participation (D04)– Negatively worded items did not load on the

same dimensions and should not be retained

NRCMeetingWindsor

June 2008

Perceptions of citizens and society - 1

• Democratic value beliefs (E01)– Agreement item format leads to highly skewed

responses– Alternative format proposed asked about

characteristics of “ideal society”– PAC advice: Keep original format

• Good citizenship beliefs (E02)– Two-dimensional structure confirmed– Proposed to delete three items that did not scale

well with either of the two scales– PAC advice: Consider keeping “obeying the

law” and “working hard” for single-item reporting

NRCMeetingWindsor

June 2008

Perceptions of citizens and society - 2

• General trust (E03)– Few interesting associations– Only item about trust in “people in

general” should be added to question about trust in institutions

• Acceptance of socially undesirable actions (E04)– Some problems with item scaling

properties and generally not too interesting results

NRCMeetingWindsor

June 2008

Self-perceptions - 1

• Interest in political and social issue (F01)– Generally good scaling properties– Five items to be retained and one modified

• Students’ self-concept (political internal efficacy) (F02)– Six items have good scaling properties– One (negatively phrased) item did not scale well

and should be deleted

NRCMeetingWindsor

June 2008

Self-perceptions - 2

• Feelings of distinctiveness (F03)– Doubts that items measure what is

intended

• Attachment to community levels (F04)– Results not very interesting– May be retained for regional instruments

NRCMeetingWindsor

June 2008

Rights and responsibilities - 1

• Gender rights items (G01)– One item did not scale well and should be

deleted

• Rights for ethnic/racial groups (G02)– Good scaling properties– Correct adaptation need to be ensured

(not minorities!)

NRCMeetingWindsor

June 2008

Rights and responsibilities - 2

• Rights for immigrants (G03)– Negatively phrased items reflect different

dimension and should not be retained– PAC suggestion: Keep item h (“When there

are not many [instead of enough]many [instead of enough] jobs available the number of <immigrants> should be limited“) for single-item reporting

• Reaction to threats to democracy (G04)– Items did not scale and only single-item reporting

might be considered– Proposed to delete this question– PAC advice: Include three items (b, d and f)

in democracy belief question (E01)

NRCMeetingWindsor

June 2008

Perceptions of institutions and society - 1

• Trust in groups and institutions (H01)– Good scaling properties for CIVED items and

interest in single-item reporting for not scaled items

– Addition of “people in general”– “Standardisation” of optional items– PAC suggestion: Consider asking about

media separately (Newspaper, TV, radio, internet) and ask about “people in general” first

• Attitudes toward country (H02)– Generally, good scaling properties except for

three items (two of them should be deleted)

NRCMeetingWindsor

June 2008

Perceptions of institutions and society - 2

• Government responsiveness (external political efficacy) (H03)– Analysis revealed two dimensions

(“responsiveness” and “cynicism”)– Both scales have weak reliabilities

• Satisfaction with country’s achievements (H04)– Generally, good scaling properties– Relatively high correlations with attitudes toward

country– Deletion proposed on content grounds– Mixed PAC advice

NRCMeetingWindsor

June 2008

Perceptions of institutions and society - 3

• Support for political parties (H05)– Interesting results– Sufficient proportions in each category

NRCMeetingWindsor

June 2008

Perceptions of participation in society - 1

• Citizenship self-efficacy (I01)– Good scaling properties for all but one

item (“raising money for people in need”)

• Protest activities (I02)– Two dimensional-structure (legal and

illegal protest)– Shortening of legal protest scale (two

items proposed for deletion)

NRCMeetingWindsor

June 2008

Perceptions of participation in society - 2

• Participation as an adult (I03)– Two dimensions (expected electoral

participation and expected active political participation)

– Good scaling properties for all items

• Participation as a young person (I04)– First two items reflect different dimension

(community-based participation)– Good scaling properties for item

measuring informal participation

NRCMeetingWindsor

June 2008

International option regarding religion

• Religious denomination (J01)

• Participation in religious services (J02)

• Attitudes toward religion (J03)– Good scaling properties for five items– Three items do not really reflect attitudes

and should be deleted

NRCMeetingWindsor

June 2008

Proposed student questionnaire for main survey

• With 5 items per minute the estimated length is 34 minutes

• There are about 170 to 180 items in the instrument

• The CIVED questionnaire had 162 items in the perceptions part (for 30 minutes)

NRCMeetingWindsor

June 2008

Proposed changes

• Reduction in questionnaire length of about 10 minutes

• Generally, only slight modifications or deletions are proposed

• The only more fundamental change for democratic value beliefs a format change has been dropped following PAC advice– Risk of using an un-trialled question format in

main survey!

NRCMeetingWindsor

June 2008

Questions or comments?

NRCMeetingWindsor

June 2008

Group discussions

• No complete re-writing of questions possible• No additional questions

– unless previously used in surveys of the same age group...

• Go through the original field trial material (tables in document 6) and discuss– Agreement with proposed selection– Prioritisation of questions – Feedback on proposed – Suggestions of (minor) changes to wording

NRCMeetingWindsor

June 2008

Group allocations

• From six groups

• Three groups start from the beginning of questionnaire material (those sitting in front)

• Three groups start from the end (those sitting towards the back)

• Try to form mixed groups (from different countries and regions)