select committee on education and recreation report on the quality assurance of the nsc
DESCRIPTION
19 February 2014 Dr Mafu S Rakometsi. Select Committee on Education and Recreation Report on the Quality Assurance of the NSC. WHAT CAN WE LEARN FROM THE NSC RESULTS? Role of Umalusi, principles, approaches and processes of QA Dr Mafu Rakometsi - CEO of Umalusi. Role of Umalusi. - PowerPoint PPT PresentationTRANSCRIPT
Select Committee on Education and Recreation
Report on the Quality Assurance of the NSC
19 February 2014
Dr Mafu S Rakometsi
WHAT CAN WE LEARN FROM THE NSC RESULTS?
Role of Umalusi, principles, approaches and processes of QA
Dr Mafu Rakometsi - CEO of Umalusi
Role of Umalusi Umalusi is the quality assurer in general
and further education and training of the National Qualifications Framework (NQF)
The Council ensures that the qualifications and curricula within GFET are of quality, that providers of education and training have the capacity to deliver and assess qualifications and are doing so to expected standards of quality, and that assessments are of the required standard
Establishment of Umalusi
Established through the promulgation of the General and Further Education and Training Quality Assurance Act, 2001 (GENFETQA Act number 58 of 2001, as amended 2008)
Two predecessors, namely the Joint Matriculation Board - JMB (1918) and the South African Certification Council - SAFCERT (1986)
Umalusi started work in 2002 having taken over from SAFCERT
Establishment of Umalusi Umalusi was established as a band education
and training quality assurance body under the GENFETQA Act in 2001 (NQF levels 1-4):
Quality Assuring exit point assessments for qualifications in schools (National Senior Certificate), FET Colleges (N3, NCV) and for Adult (GETC)
Accrediting Independent schools, private FET Colleges and Adult Learning Centres as well as private assessment bodies
Umalusi’s brief In 2007 the review of the Implementation of the
NQF was completed and in 2008 the GENFETQA Act was amended creating Umalusi as one of three Quality Councils with extended mandates, the other two being Council on Higher Education and the Quality Council for Trades and Occupations
The object of the amended Act is to enhance the quality of general and further education and training through:
Development and management of a sub-framework of qualifications for GFET
Umalusi’s brief Quality assurance of: Qualifications and curricula Provision through the accreditation of
private providers of education and assessment, to
provide and assess these qualifications Exit point assessments of the
qualifications
Umalusi’s brief
Certifying learner attainments for these qualifications
Conducting research on matters pertaining to the GFET sub-framework of qualifications
Advising the Minister on matters related to the GFET sub-framework of qualifications
Quality Assurance of the DBE 2013 National Senior Certificate
ExaminationEmmanuel Sibanda – Acting Sen. Manager :
Quality Assurance of Assessments
PURPOSE: To ensure that the question papers are of the required
standard (standard captured in the NCS and SAG’s) To ensure that the question papers are relatively:
- fair- reliable- representative of an adequate sample of the
curriculum - representative of relevant conceptual domains
- representative of relevant levels of cognitive challenge
Moderation of question papers
Moderation of the question papers
Approach: Question papers set by panel of
examiners – DBE Internally moderated by DBE Externally moderated by Umalusi Subsequent moderations and approval
Moderation of the question papersCriteria : Technical criteria Internal moderation Content coverage Text selection, types
and quality of questions
Predictability Cognitive skills Marking
memorandum or guidelines Language and bias
Moderation of the question papers
Findings: Areas of Good Practice Percentage of question papers and
memoranda approved after first and second moderation ( Nov 2013- 70% ; Mar 14 – 78%) (out of 130 papers)
Simultaneous moderation of final and supplementary question papers.
Moderation of the question papers
Findings: Areas of Concern Adherence to timeframes and impact on
quality of setting and moderation. Question papers requiring more than four
moderations. 2 papers for November 2013 (Isizulu HP P1, Isixhosa
HL P1) and 2 for March 2014 (Business Studies, Isizulu HL P1)
Definition: Internal assessment refers to any
assessment conducted by the provider , the outcome of which count towards the achievement of the qualifications
Umalusi appoints panels of moderators / subject specialist to carry out this mandate
Moderation of internal assessment
Purpose of Umalusi’s verification: To verify the rigour and appropriateness of the DBE
moderation process – linked to DBE plans Ascertain the degree to which assessment bodies/provinces
are attempting to ensure standardisation across Ascertain the standard and quality of the tasks Determine the extent and quality of internal moderation and
feedback. Determine the reliability and validity of the assessment
outcomes
Moderation of internal assessment
Moderation of Internal Assessment
Approach 1 (June/July 2013)- verifying the DBE SBA
moderationPROVINCIAL EDUCATION DEPARTMENT
Subjects
All Mathematics, Life Sciences, Physical SciencesAccounting, English FAL, History, Geography, Life Orientation,
Moderation of Internal Assessment
Approach 2 (June/July 2013)- Umalusi independent
moderation (own sample)PROVINCIAL EDUCATION DEPARTMENT
Subjects
All Economics, Business Studies, Mathematical Literacy, Music, (Practical subjects)Mechanical Technology, Electrical Technology
Moderation of Internal Assessment
Areas of good practice: DBE conducted very rigorous
moderation and provided useful verbal feedback to PDE’s at the end of each moderation session
General adherence to policy in terms of number of tasks done, presentation of learner evidence/portfolios.
Moderation of Internal Assessment
Areas of concern: While internal moderation is being done in
most schools, much of the focus is on compliance (monitoring) and not on qualitative issues (actual moderation)
Persistent problem of lack of constructive feedback given back to learners after moderation
Teachers are still unable to develop tasks pitched at appropriate cognitive levels: focus is more on lower cognitive level.
Monitoring of Examinations• “State of readiness”• Conduct of examinations• Marking
State of Readiness
Monitored the DBE state of readiness visits in 6 provinces: Eastern Cape, Free State, Kwazulu-Natal, Limpopo, Northern Cape, Gauteng
Comprehensive approach that monitors exam systems (including SBA)
Monitoring of Examinations“State of readiness”
Findings: All provinces have working examination
systems in placeOf concern: Many vacant posts and use of contract staff In some PDE’s closer monitoring printing of
question papers needs attention Inefficient or lack of coordination with districts
wrt exam related processes
Monitoring of the writing phase
Findings:Generally examinations conducted in line with policy Isolated instances of non-compliance (suitability of
venue, identification of learners)
No of exam centres No of exam centers monitored by Umalusi
Total number of Umalusi monitors
6699 159 36
Monitoring of the marking process
Number of marking centres
No of marking centers monitored by Umalusi
Total 118 77
Monitoring of marking processFindings:Marking centres were generally well organised and suitable for the task.Inadequate and inexperienced security and in some provinces,Absence of communication facilities reported at one centre in one province
PURPOSE:Moderation of marking determines the standard
and quality of marking and ensures that marking is conducted in accordance with agreed practices
Umalusi engages the following during the moderation of marking
Pre-marking/memorandum discussion: centralised memo discussions recommended - this will ensure consistency across marking centres
Moderation of marking (centralised and on-site)
Verification of marking
Marking verificationMemo discussion meetings:Areas of good practice: The memo discussions for the approval
of final memoranda went relatively well in 2013. Provision of an extra day of training for marking was very welcome.
Memo discussion meetings Areas of concern: The time between the examination dates and
the memo discussions was generally far too short to allow pre-marking to take place. This was reported in several subjects, and seriously compromised the validity of the process, as meaningful discussion depends on the pre-marking of scripts.
Some provinces sent only one representative or none at all to the memo discussions.
Centralised & on-site marking verification
Areas of good practice: Many external moderators expressed the
opinion that the accuracy of marking had improved slightly.
External moderators unanimous in appreciation of the impact that thorough training at the memo discussion meetings had on the quality of marking.
Centralised & on-site marking verification Areas of concern: Markers still experience problems with regard to interpreting answers to open-ended and higher order questionsUse of rubrics continues to be a serious concern e.g. the inappropriateness of the rubrics used for P3 of HL and FAL: the descriptors do not facilitate good marking. There are still markers marking literature questions, who do not have a thorough knowledge of the stories/dramas/novels/poems they are marking.
Monitoring of the writing phase
Areas of good practice: The DBE exam was generally administered in line with
policy. No major concerns were reported. One can see growth in the administration and conduct of the exams by PDEs.
No of exam centres No of exam centers monitored by Umalusi
No of Umalusi monitors per province
6699 159 36
Examination Irregularities The majority of irregularities were of a technical
nature and these were reported to Umalusi according to the established channels.
Some irregularities were as a result of registration-related problems, e.g. candidates nor appearing on mark sheets, some registered for incorrect subjects.
Umalusi represented on NEIC
WHAT CAN WE LEARN FROM THE NSC RESULTS?
STANDARDISATION PROCESS
Why Umalusi standardises results, and how
Provision of GENFETQA – Council may adjust raw marks.
International practice – large scale assessment systems
Standardisation – process used to mitigate the effect of factors other than learners knowledge and aptitude on the learners performance.
Sources of variability – difficulty in question paper, undetected errors, learner interpretation of questions
Objectives for Standardisation To ensure that a cohort of learners is not
advantaged or disadvantaged by extraneous factors other than their knowledge of the subject, abilities and their aptitude.
To achieve comparability and consistency from one year to the next.
Why Umalusi standardises results & how
Assumptions – for large populations the distribution of aptitude and intelligence does not change appreciably
Process of standardisation Moderation of question papers Review of learner performance against historical
performance of candidates in each subject. Historical average (norm) constructed using past 3 to 5
years data. Pairs analyses provides further comparisons of raw
means Statistical moderation of Internal assessment
Why Umalusi standardises results & how Qualitative input meetings Reports (Moderator, Chief Marker and Internal
Moderator) Umalusi research (maintaining standards & post
exam analysis) Responsibility of Assessment Standards
Committee Committee of Council Responsible for setting and maintaining
assessment standards Observers (SAQA, HESA, Teacher Unions)
Process for Standardisation Continuation of JMB and SAFCERT model Assessment Standards Committee Qualitative Reports Pre-standardisation and Standardisation
meetings Standardisation booklets (data) – subject raw
mark distributions (external written component only) of entire cohort.
Subjects are standardised individually, in a linear and non-iterative manner
Principles applied in the standardisation of
examination marks In general no adjustment should exceed 10% or
the historical average In the case of the individual candidate, the
adjustment effected should not exceed 50%of the raw mark obtained by the candidate
If the distribution of the raw marks is below the historical average, the marks may be adjusted upwards subject to the limitations
Statistical moderation
Scope of standardisation 2013 59 standardised Raw marks accepted: 38 subjects Moderated upward : 5 subjects Moderated downward : 16 subjects
WHAT CAN WE LEARN FROM THE NSC RESULTS?
STANDARDISATION DECISIONSDBE NSC 2013
SUBJECT ADJUSTMENT
Life Sciences½ CA = 0 at 0, scaled to -6 at 87, block -6 up to 162, scaled to 0 at 281, scaled to -3 at 300
Physical Sciences Raw
Mathematics 0 at 0, scale to -6 at 96, block -6 from 96 to 246, scale to 0 at 300
Mathematical Literacy0 at 0, raw to 180, scale 0 to +6 from 180 to 234, scale to 0 at 300
Mathematics: Probability; Data Handling
½ CA = 0 at 0 scaled to -15 at 57, block -15 up to 260, scaled to 0 at 300
SUBJECT ADJUSTMENT
Geography 0 at 0, scale to -12 at 102, block -12 from 102 to 268, scale to 0 at 300
History RawReligion Studies RawLife Orientation Raw
Hospitality Studies0 at 0, scale to -6 at 96, block -6 from 96 to 257, scale to 0 at 300
Tourism½ CA = 0 at 0 scaled to -6 at 47 scaled to 0 at 119, scaled to +11 at 241, scaled to 0 at 300
Consumer Studies Raw
SUBJECT ADJUSTMENT
Agricultural Science Raw
Agricultural Management Practices RawAgricultural Technology RawMusic RawDance Studies Raw
DesignRaw
Dramatic Arts Raw
SUBJECT ADJUSTMENTVisual Arts Raw
Information Technology Raw
Computer Applications Technology
½ CA = 0 at 0 scaled to -12 at 50, scaled to 0 at 269, raw up to 300
Civil Technology ½ CA = 0 at 0 scaled to -14 at 129, scaled to -7 at 300
Electrical TechnologyRaw
SUBJECT ADJUSTMENT
Mechanical Technology0 at 0, scale to -6 at 126, block -6 from 126 to 246, scale to 0 at 300
Engineering Graphics and Design
Raw
Accounting Raw
Business Studies0 at 0, scale to -6 at 96, block -6 from 96 to 246, scale to 0 at 300
EconomicsRaw
SUBJECT ADJUSTMENTAfrikaans HL block -7
English HL RawIsiNdebele HL CA = 0 at 0 scaled to -13 at
90, scaled to +11 up to 163, scaled to +16 at 218, scaled to 0 at 300
IsiZulu HL Raw
Setswana HL Block -6
SUBJECT ADJUSTMENT
Siswati HL
½ CA = 0 at 0 scaled to -10 at 93, scaled to 0 at 154,, scaled to +9 at 258 , scaled to 0 at 300
IsiXhosa HL Raw
Xitsonga HL0 at 0, scale to -12 at 73, block of -12 from 73 to 232, scale to 0 at 300.
Tshivenda HL0 at 0, scale to -10 at 30, block of -10 from 30 to 245, scale to 0 at 300.
Sepedi HL½ CA = 0 at 0 scaled to -13 at 98, block -13 up to 159, scaled to +5 at 276, scaled to 0 at 300
SUBJECT ADJUSTMENT
Sesotho HL0 at 0, scale to -6 at 126, block of -6 from 126 to 233, scale to 0 at 300.
Afrikaans FAL RawEnglish FAL RawIsiNdebele FAL RawIsiZulu FAL Raw
Setswana FAL
CA = 0 at 0 scaled to -30 at 60, block -30 up to 222. scaled to -10 at 259, scaled to -30 at 300
Siswati FAL RawIsiXhosa FAL RawXitsonga FAL Raw
SUBJECT ADJUSTMENT
Tshivenda FALRaw
Sepedi FAL½ CA = 0 at 0 scaled to -15 at 99, scaled to -5 at 142, block -5 up to 269, scaled to -15 at 300
Sesotho FALRaw
Afrikaans SAL Raw
English SAL Raw
IsiNdebele SAL Raw
IsiZulu SAL Raw
SUBJECT ADJUSTMENTSiswati SAL RawIsiXhosa SAL Raw
Sepedi SAL Raw
Sesotho SAL Raw
Quality Assurance of the IEB 2013 National Senior Certificate
ExaminationEmmanuel Sibanda – Acting Sen. Manager :
Quality Assurance of Assessments
Moderation of the question papers
Areas of Good Practice :Percentage of question papers and
memoranda approved after first and second moderation ( Nov 2013- 95% Mar 2014 – 100%)
Moderation of Internal Assessment
Term 4 moderation- focus on learner evidence and
teacher files. IEB Subjects Mathematics, Maths Lit, Life Sciences, Geography,Physical Sciences, Accounting, English HL, History, Life Orientation, Business Studies, Economics, Afrikaans FAL,
Moderation of Internal AssessmentFindings: Areas of Good Practice IEB has very good systems and processes in place for SBA
implementation and support of their educators. This is borne out by the generally good quality of tasks
set by the IEB educators.Findings: Areas of Concern The main challenge is how rubrics are developed: there is
some element of vagueness and subjectivity – this leads to differences in interpretation by educators, resulting in differences in allocation of marks
Moderation at school level can also be improved
Monitoring of the writing phase
Scope:No of exam centres No of exam centers
monitored by UmalusiNo of Umalusi deployed for IEB monitoring
187 22 09
Monitoring of the writing phaseFindings: Management of the examinations: Generally examinations conducted in line with policy There is good and ongoing communication between
the IEB and their chief invigilators who control the writing phase competently and professionally
Isolated instances of non-compliance (candidate identification)
Monitoring of centres occurs but it should be done in a more visible onsite and regular basis
Monitoring of the marking phase
Scope:No of marking centres
No of marking centers monitored by Umalusi
No of Umalusi deployed for IEB monitoring
03 03 01
Verification of markingFocus: Memo discussion and on-site marking
verification
Scope: Accounting, Business Studies, Economics,
English HL, English FAL, Afrikaans FAL, History, Geography, Life Sciences, Mathematical Literacy, Mathematics, and Physical Sciences
Verification of marking Findings:The IEB conducts memo discussions in the presence of sub-examiners/Senior markers who are responsible for groups of markers during marking. The sub-examiners/Senior markers then train markers thoroughly on day 2Additions made to the final memo are ratified by the internal moderator and chief examiner. Commendable – IEB is a small system
Verification of marking (cont)Findings:Marking was found to be fair, valid and reliable: attributed to the continuous feedback sessions between the chief examiner, senior sub-examiners, and sub-examinersThe practice of double marking and therefore verification is highly commendableAttention should be paid to instances where it was indicated that moderation doesn’t always involve full moderation of a script but sampled questions
Statistical moderation
Scope of standardisation 2013:
62 subjects standardised
Raw marks accepted: 47 subjects
Moderated upward : 2 subjects
Moderated downward : 13 subjects
SUBJECT ADJUSTMENT
Life Sciences½ CA = 0 at 0, scaled to -6 at 87, block -6 up to 162, scaled to 0 at 281, scaled to -3 at 300
Physical Sciences Raw
Mathematics 0 at 0, scale to -6 at 96, block -6 from 96 to 246, scale to 0 at 300
Mathematical Literacy0 at 0, raw to 180, scale 0 to +6 from 180 to 234, scale to 0 at 300
Mathematics: Probability; Data Handling
½ CA = 0 at 0 scaled to -15 at 57, block -15 up to 260, scaled to 0 at 300
SUBJECT ADJUSTMENT
Geography 0 at 0, scale to -12 at 102, block -12 from 102 to 268, scale to 0 at 300
History RawReligion Studies RawLife Orientation Raw
Hospitality Studies0 at 0, scale to -6 at 96, block -6 from 96 to 257, scale to 0 at 300
Tourism½ CA = 0 at 0 scaled to -6 at 47 scaled to 0 at 119, scaled to +11 at 241, scaled to 0 at 300
Consumer Studies Raw
SUBJECT ADJUSTMENT
Agricultural Science Raw
Agricultural Management Practices RawAgricultural Technology RawMusic RawDance Studies Raw
DesignRaw
Dramatic Arts Raw
SUBJECT ADJUSTMENTVisual Arts Raw
Information Technology Raw
Computer Applications Technology
½ CA = 0 at 0 scaled to -12 at 50, scaled to 0 at 269, raw up to 300
Civil Technology ½ CA = 0 at 0 scaled to -14 at 129, scaled to -7 at 300
Electrical TechnologyRaw
SUBJECT ADJUSTMENT
Mechanical Technology0 at 0, scale to -6 at 126, block -6 from 126 to 246, scale to 0 at 300
Engineering Graphics and Design
Raw
Accounting Raw
Business Studies0 at 0, scale to -6 at 96, block -6 from 96 to 246, scale to 0 at 300
EconomicsRaw
SUBJECT ADJUSTMENTAfrikaans HL block -7
English HL RawIsiNdebele HL CA = 0 at 0 scaled to -13 at
90, scaled to +11 up to 163, scaled to +16 at 218, scaled to 0 at 300
IsiZulu HL Raw
Setswana HL Block -6
SUBJECT ADJUSTMENT
Siswati HL
½ CA = 0 at 0 scaled to -10 at 93, scaled to 0 at 154,, scaled to +9 at 258 , scaled to 0 at 300
IsiXhosa HL Raw
Xitsonga HL0 at 0, scale to -12 at 73, block of -12 from 73 to 232, scale to 0 at 300.
Tshivenda HL0 at 0, scale to -10 at 30, block of -10 from 30 to 245, scale to 0 at 300.
Sepedi HL½ CA = 0 at 0 scaled to -13 at 98, block -13 up to 159, scaled to +5 at 276, scaled to 0 at 300
SUBJECT ADJUSTMENT
Sesotho HL0 at 0, scale to -6 at 126, block of -6 from 126 to 233, scale to 0 at 300.
Afrikaans FAL RawEnglish FAL RawIsiNdebele FAL RawIsiZulu FAL Raw
Setswana FAL
CA = 0 at 0 scaled to -30 at 60, block -30 up to 222. scaled to -10 at 259, scaled to -30 at 300
Siswati FAL RawIsiXhosa FAL RawXitsonga FAL Raw
SUBJECT ADJUSTMENT
Tshivenda FALRaw
Sepedi FAL½ CA = 0 at 0 scaled to -15 at 99, scaled to -5 at 142, block -5 up to 269, scaled to -15 at 300
Sesotho FALRaw
Afrikaans SAL Raw
English SAL Raw
IsiNdebele SAL Raw
IsiZulu SAL Raw
SUBJECT ADJUSTMENTSiswati SAL RawIsiXhosa SAL Raw
Sepedi SAL Raw
Sesotho SAL Raw
Annexure to Approval Letter to DBE
Specific concerns regarding the quality assurance of the NSC examination and assessment:
There has been some improvement with regard to adherence to timeframes. However, there are subjects where improvements must be made. 25 November and 25 March question papers were submitted in May/June 2013 for first moderation. It is important to note that for the credibility of the NSC examination it is vitally important that every effort is made to adhere to agreed deadlines for setting and moderation of question papers. Any delays affect Umalusi in its quality assurance exercise.
Annexure to Approval Letter to DBE
There have also been some improvements made pertaining to administration of SBA and presentation of learner evidence of performance. Having said this, the following issues were found to be problematic:
Internal moderation reports are generally not available. Lack of constructive feedback given back to learners after moderation. Teachers are still challenged regarding the development of tasks pitched
at appropriate cognitive levels: focus is more on lower cognitive level. Assessment of Practical investigations, Research projects,
Assignments and simulations still remains a major problem. The use and development of rubrics is problematic: descriptors are
unrealisable and vague. Assessment of the Physical Education Task (PET) in Life Orientation
continues to be a problem – inflation of marks.
Annexure to Approval Letter to DBE
A number of problems were reported with regard to the standard and quality of marking and these clearly hinge on the calibre of people appointed as markers. DBE is urged to look closely into the appointment of markers.
Annexure to Approval Letter to IEB
Specific concerns that require remediation:
The quality of question papers submitted at first moderation. 58% of the question papers were not compliant with regard to Umalusi criterion that looks into the technical aspects or the face validity of the question paper. It should be borne in mind that question papers submitted to Umalusi must be in their print-ready form.
The quality of the memoranda submitted for moderation should be looked into.
Annexure to Approval Letter to IEB
The issue of rubrics used for assessment of learner work in SBA needs to be looked into. As Umalusi we strongly believe that rubrics are meant to facilitate marking, and should therefore leave no room for ambiguity on the part of educators who are meant to use these. Having part marks indicated in the rubrics will go a long way in addressing this problem.
It was again observed that in Life Orientation there was a kink at the 80% level. The IEB was requested in 2012 to address this matter. Having noted this, the IEB is again urged to ensure the bulging of marks at the 80% level does not recur. If this problem persists in 2014 Umalusi would have no choice but to take appropriate corrective measures.
Conclusion The findings of the quality assurance processes are a clear
indication of a maturing system that has, on the one hand, made positive strides towards improvement in certain areas of assessment and examination, but, on the other hand, still has a few challenges that need to be addressed.
The quality assurance of each of these processes presented above was conducted based on Umalusi criteria. Umalusi uses criteria that are subjected to constant review and refinement, to ensure that they are in line with current trends in assessment and examinations.
Conclusion… In general Umalusi is pleased with the manner in which the
2013 NSC examination was administered.
Umalusi acknowledges that a number of technical irregularities were reported, but these were addressed in a fitting manner.
Umalusi takes this opportunity to express appreciation to the national & provincial departments of education for their concerted effort in ensuring a credible examination.
Umalusi expresses appreciation also to all the relevant stakeholders for the necessary support given in line with Umalusi quality assurance initiatives.
Thank you!