secondary assessment and the nceas€¦ · exemplification. supporting documents • assessment...
TRANSCRIPT
![Page 1: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/1.jpg)
Secondary Assessment and theNCEAs
2011
![Page 2: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/2.jpg)
Changes to Setting and Marking –Why?
• Continuous improvement
• Research based
• Monitored by external experts
![Page 3: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/3.jpg)
Basis of the changes
• Test Dimensionality
• Item characteristics
• Candidate characteristics
![Page 4: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/4.jpg)
Item response curves
![Page 5: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/5.jpg)
Grade thresholds
![Page 6: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/6.jpg)
DIF (differential item functioning)
• Provides information about whether candidates
perform differently based on their particular
demographic.
• Example 1. Males/females in particular question
• Example 2. Asian/non-Asian in particular question
![Page 7: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/7.jpg)
How we use the Information?
![Page 8: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/8.jpg)
Assessment Principles
• Discuss and identify the key principles for all
assessment with those around you
• Write down six adjectives that describe good
assessment which would be used as part of the
assessment principles
![Page 9: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/9.jpg)
Assessment Principles
Assessment must be:
• Appropriate
• Fair and inclusive.
Evidence gathered must be:
• Valid
• Sufficient.
Assessor judgements must be:
• Consistent
• Transparent.
![Page 10: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/10.jpg)
We ask examiners to Include(where relevant to the task)…
• Content accessible to all
• Content that is interesting and relevant
• Tasks that encourage response from a diverse
perspective
• Gender neutral titles and names
• Gender balance.
![Page 11: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/11.jpg)
We ask them to Avoid (unless pertinent to the standard) …
• Controversial issues, people or events
• Topics which may upset
• Stereotypical pictures, cartoons and resources
• Tokenism.
![Page 12: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/12.jpg)
Achievement
standard(criteria and
explanatory notes)
Assessment
schedule(evidence/
judgement
statements)
Assessment
task(the means of
collecting valid
evidence)
![Page 13: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/13.jpg)
Analysis of the standard
• Criteria – determine the difference between each level of performance
• Explanatory notes – identify any restrictions, explanations or specifics
• Curriculum – may provide further clarification or exemplification
![Page 14: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/14.jpg)
Supporting Documents
• Assessment Specifications
• Exemplars
• Assessment reports
• Curriculum
![Page 15: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/15.jpg)
Developing questions
• Verb + Subject + Condition
• Verb must signal clearly what is required and be
consistent with criteria and explanatory notes in
the standard
• Subject must be appropriate to the curriculum,
criteria and explanatory notes
• Condition must be appropriate to the curriculum,
criteria and explanatory notes
• Students must have sufficient opportunity to
achieve at every level.
![Page 16: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/16.jpg)
The Assessment Schedule…
• Designed to achieve consistency of assessor
judgements
• Developed at the same time as the assessment
activity
• Provides examples of expected student evidence
• Specifies minimal requirements for achievement
at each level (quality and quantity).
![Page 17: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/17.jpg)
Quality Check
Check for:
• Excess or areas of deficiency
• Validity – against the standard – use standard
itself
• Level of difficulty
• Time for students to complete task
• Clarity of instructions
• Consistency
• Error-free.
![Page 18: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/18.jpg)
Traps and pitfalls - check…
• All resources are appropriate, clear and accessible.
• Labels and tables – instructions match.
• Diagrams – do they work?
• General accuracy – languages: accents and
characters / history: dates and periods.
• Pictures / images – relevant, good quality.
• Genre/context of task and standard match
• Task numbering is consistent.
• Key words bolded.
![Page 19: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/19.jpg)
Research has affected question structure
• All questions now provide evidence for all levels of
achievement
• Most questions now have scaffolding
![Page 20: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/20.jpg)
What is scaffolding?
• Providing the support (footholds) that enable more
students to demonstrate evidence of higher
performance levels.
• Questions that are scaffolded:
– elicit better responses from students
– have more students attempting the more
challenging parts of questions.
• Some form of scaffolding should be provided for all
questions.
![Page 21: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/21.jpg)
Scaffolding in ChemistryConsider the development of the following question:
• Version 1:
Discuss the different states of fluorine and bromine at room temperature.
• Version 2:
Fluorine, F2, is a gas and bromine, Br2, is a liquid at room temperature. Discuss the different states of these elements at room temperature.
• Version 3:
Fluorine, F2, is a gas and bromine, Br2, is a liquid at room temperature. Discuss the different states of these elements at room temperature. You should include in your answer information on particle separation, energy, particle motion and the attractive forces between the particles.
![Page 22: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/22.jpg)
And finally
• Version 4:
Fluorine, F2, is a gas and bromine, Br2, is a liquid at room temperature.
Discuss the reasons for the different states of these elements at room temperature. You should include in your answer:
• information on particle separation
• energy
• particle motion
• the attractive forces between the particles.
![Page 23: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/23.jpg)
How research supports these changes
![Page 24: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/24.jpg)
How research supports these changes
Biology Level 3 2006 Question Three
Many metabolic pathways are controlled by multiple
genes. An example is the metabolic pathway that
produces normal skin pigmentation. Albinism, which is
the total lack of pigment, can be caused by a mutation in
any one of the genes controlling this pathway.
(e)Discuss the fact that it
is possible for two albino
parents to have a child with
normal skin pigmentation.
![Page 25: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/25.jpg)
BIOLOGY LEVEL 3 2007 QUESTION 3 (C)
ON THE SAME CONCEPT
![Page 26: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/26.jpg)
The same concept again in 2009
The 2007 question The 2009 question
![Page 27: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/27.jpg)
Monitoring
marking
![Page 28: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/28.jpg)
Profiles of expected performance
• PEPs are:
• indicators of ranges of expected results for external
standards
• tolerances developed on the basis of professional
judgement of National Assessment Facilitators (NAFs)
in the Secondary Examinations team, NZQA
statistical/research staff, Subject experts
• tools to ensure results that are consistent with the
achievement standards.
![Page 29: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/29.jpg)
Why are they used?
PEPs were introduced as a result of the 2005 State
Services Commission report into NCEA and New
Zealand Scholarship.
The report addressed concerns about the excessive
variability shown in consecutive years of NCEA
results in some standards. Such variability is not
acceptable in national examinations where cohorts
are large.
The NZQA Board adopted the recommendation that
a range of expected results should be established to
give guidance to examiners and markers.
![Page 30: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/30.jpg)
Do they ever change?
PEPs are:
• set afresh each year
• affected by changes in the registered standards
• affected by changes in cohort performance
• guidelines which can and are broken with justification.
![Page 31: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/31.jpg)
Grade Score Marking (GSM)
![Page 32: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/32.jpg)
Purpose
To improve discrimination between grade levels in NCEA
external examinations.
![Page 33: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/33.jpg)
Research
1. Item response theory (IRT) 2006-2009
2. Score based grading 2007-2009
3. Live Pilot 2010
![Page 34: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/34.jpg)
Outcomes: the NZQA perspectives
• National Assessment Facilitators had increased
confidence in signing off marking schedules
• Tracking of results during marking was straightforward
• All data procedures post marking were business-as-
usual.
![Page 35: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/35.jpg)
Outcomes: the marker perspectives
• Cut score setting enhanced Panel Leaders’ (PLs’)
confidence that grade boundaries reflected standards
• Markers liked the refinement of being able to indicate
high/low performance within each grade
• PLs thought that discrimination was better at
boundaries.
![Page 36: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/36.jpg)
Projected benefits
• Greater accuracy in grade determination
• Fairer to students
• Reduction in year-by-year variability leading to
diminished need for PEPs longer term
• More transparency
• Closer alignment of marking to the standard
![Page 37: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/37.jpg)
Grade Score Marking – what is it?
• Grades are assigned according to the criteria in the
standard
• Each grade is divided into 2; there is also a zero Ø
• Each question receives a grade and score
• The scores are totalled
• A sample of papers on each score is judged by the
panel leader and check marker to set the boundaries
for each grade
• The boundaries are called cut scores and become
the sufficiency statements for the grades.
![Page 38: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/38.jpg)
Scoring each question with GSM
N A M E
NØ N1 N 2 A3 A4 M5 M6 E7 E8
![Page 39: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/39.jpg)
GSM Assessment schedules
• Schedules are based on the criteria in the standard for
A, M and E
• N is below A
• There is an NØ for ‘no response; no relevant evidence’
• Each grade, NAME, is divided into two.
![Page 40: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/40.jpg)
Total score
The scores for the questions are aggregated to give a
total score for the paper.
The total score represents the sum of the grades for the
questions.
![Page 41: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/41.jpg)
What is a cut score?
A cut score sets the grade boundaries by establishing the
range for each grade.
Senior markers use the actual student exam papers to
set cut scores to define each NAME grade.
![Page 42: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/42.jpg)
Results
• Markers write only the total score on the front of the
paper
• Markers enter only the total score online
• The cut-score is entered by the NAF after consultation
with the senior markers and other NZQA staff
• The correct result grade will be generated
automatically.
![Page 43: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/43.jpg)
Results to candidates
• The record of achievement will show only the grade N,
A, M, E.
• The judgement statement on the web will show the cut
scores for the various grades.
• When candidates receive their booklets back they will
also receive an information sheet telling them where to
go on the web to check their grades.
![Page 44: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/44.jpg)
Questions
• What do students need to know to sit the exam?
• How are teaching programmes affected?
• Is this a move away from standards based
assessment ?
• What is the difference between this and using marks?
• Can GSM be used for internal marking (either for
internal assessment or mock exams)?
![Page 45: Secondary Assessment and the NCEAs€¦ · exemplification. Supporting Documents • Assessment Specifications • Exemplars • Assessment reports • Curriculum. Developing questions](https://reader033.vdocuments.net/reader033/viewer/2022043005/5f8d962f026d16785b7f9677/html5/thumbnails/45.jpg)
Jennifer Mackrell and Christine Pallin
Team Leaders - National Assessment Facilitators
Secondary Examinations
NZQA
PO Box 160
Wellington 6140.
Contact emails: