how to interpret your school's icas reports
TRANSCRIPT
How to Interpret Your School’s ICAS Reports
Nicholas Marosszeky
Educational AssessmentAustralia
LEARNING OBJECTIVES
• Key Concepts / Making Comparisons
• Information in ICAS Reports - How data is presented to describe results
• Scaling methods to develop a common scale
• Analysis of ICAS questions and student results to produce a formative assessment
2
Key Concepts
• Raw Score / Total Score
• Sub-Scale Score
• Mean
• Distribution / Spread
• Strengths and Weaknesses
• Percent Correct
• Rank Order / Percentile Rank
• Common Scaled Score
3
Making Comparisons
• ICAS Report allows us to make comparisons and judgements– Individual
• Individual with Group• Individual with Locale• Individual over Time
– Group (School Class / School Year Level)• Group with Locale• Group over Time (also Locale over Time)
• Combine this information with what we already know about the student and school class
4
Presenting Results
• Reports should transfer information not test scores !
• Personalisation / Individualisation of Student Assessment
5
Presenting Results
• For group results must report the Mean and Distribution of scores
• You can manipulate the mean and distribution
6
1 EACH Min Max R1
1 0 0 7
2 0 0 9
3 1 0 9
4 1 0 2
5 2 0 3
6 2 0 2
7 3 0 4
8 3 0 1
9 4 0 0
10 4 0 9
11 5 9 5
12 5 9 7
13 6 9 6
14 6 9 0
15 7 9 9
16 7 9 5
17 8 9 4
18 8 9 5
19 9 9 1
20 9 9 5
total 90 90 93
n 20 20 20
mean 4.5 4.5 4.7
sd 2.9 4.6 3.0
Key Concepts – Test Construction
• Evaluating item quality
–Item Difficulty (easiness of item –percent correct)
–Item Discrimination (describes high and low scoring groups – item / total correlation, point bi-serial correlation)
7
Key Concepts – Data Analyst
• What do I expect ?
• What do I observe ?
8
Information in ICAS Reports
• Page 1 – Total Test Score versus Locale Average Total Test Score (NB: the cleaned distribution – middle 80%)
• Section 1 – All school results on Common Scale* with Locale results
• Section 2.1 – Total Test Score versus Locale Average (NB: the cleaned distribution – middle 80%) (plus number of questions, standard deviation and highest score)
• Section 2.2 – Total Subscale Score versus Locale Average (Plus number of questions and highest score)
• Section 2.3 – Question Analysis – Easy versus Hard Questions, Percentage correct versus Locale Percentage correct (Strengths and weaknesses code)
• Locale = Region
• Chart generated if more than 10 students
• * = Based on Item Response Theory 9
Information in ICAS Reports
• Section 2.4 – Student Response Analysis – Looks at percentage of students answering the various response options A, B, C, D (i.e. correct option and the others). Also includes non attempts.
• Section 2.5 – Student Results – Alphabetical Order (Total Score, Award, School Percentile, Locale Percentile)
• Section 2.6 – Student Results in Achievement Order (Easy versus Hard Questions, High versus Low Scorers) Pattern of learning for the class of students
• Locale = Region
• Chart generated if more than 10 students
• * = Based on Item Response Theory
10
Information in ICAS Reports (TAP Schools Only)
• Section 2.7 – Development of Students over Time* – Performance of the current cohort of students over time
• Section 2.8 – Comparison of Different Students over Time* –Performance of the Year 3 school class over time
• Section 2.9 – Individual Student Development* – Performance of individual students over time
• Locale = Region
• Chart generated if more than 10 students
• * = Based on Item Response Theory11
Information in ICAS Student Reports
• Page 1 – Performance – Student Percentile Rank, Student Total Test Score and Sub-Test scores compared with mean and distribution of the Year Level in the Locale (NB: the cleaned distribution – middle 80%)
• Page 1 – Comparative Performance* – Student over time, Cohort in Locale mean and distribution over time, Cohort in Locale in the Year Level Above (mean and distribution)
• Page 2 – Analysis By Question – Correct Answer, Student Answer, % correct in Locale
• Locale = Region
• Chart generated if more than 10 students
• * = Based on Item Response Theory 12
ICAS School Reports
Raw Scores
Mean and Distribution
Placed on a Common
Scale
Strengths +Weaknesses
PercentCorrect
Rank / Percentile
Page 1
Section 1
Section 2.1
Section 2.2
Section 2.3
13
ICAS School Reports
Raw Scores
Mean and Distribution
Placed on a Common
Scale
Strengths +Weaknesses
PercentCorrect
Rank / Percentile
Section 2.4
Section 2.5
Section 2.6
14
ICAS School Reports (TAPS Schools)
Raw Scores
Mean and Distribution
Placed on a Common
Scale
Strengths +Weaknesses
PercentCorrect
Rank / Percentile
Section 2.7
Section 2.8
Section 2.9
15
ICAS Student Reports
Raw Scores
Mean and Distribution
Placed on a Common
Scale
Strengths +Weaknesses
PercentCorrect
Rank / Percentile
Page 1
Table 1
Table 2
Table 3
16
Development of the Common Scale
• Large Scale Assessment is like a juggernaut
17
- How does data get to my desk ? - How does information get to your desk ?
Oregon DATA Project - Triangulation
Development of the Common Scale
• Calculate total raw score for the test
• Apply our measurement model to the data –based on Item Response Theory (Latent Trait / Probabilistic / Modern Test) to produce an equal interval measure of ability for each year level
• This measure of ability is then vertically equated
• And then the test is historically equated
• Finally converted to a common scale for ease of interpretation
19
Progression in Maths before Applying Vertical Scaling
-3.0
-2.5
-2.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Y3 Y4 Y5 Y6 Y7 Y8 Y9 Y10 Y11 Y12
20
Progression in Maths after Applying Vertical Scaling
-3.0
-2.5
-2.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Y3 Y4 Y5 Y6 Y7 Y8 Y9 Y10 Y11 Y12
21
Formative Assessment
• See Sections 2.3, 2.4 and 2.6 (School Report)
• See Page 2 – Analysis by Question (Student Report)
• Look at test questions
• Look at patterns in questions – sub-scales
• Distractor Analysis
• Group and Individual approaches
• Can download data from the web-site
22