Portfolio Committee on Basic Education

Report on the Quality Assurance of the NSC

18 February 2014

Dr Mafu S Rakometsi

WHAT CAN WE LEARN FROM THE NSC RESULTS?

Introduction to context, principles, approaches and processes

Dr Mafu Rakometsi - CEO of Umalusi

Regulatory Framework

Quality Assurance of AssessmentNQF Act Section 27 (h)

The QC must develop and implement policy and criteria for assessment for the qualifications on its sub-framework.

Section17 of the GENFETQA Act(5) The Council must, with the concurrence of the Director-General and

after consultation with the relevant assessment body or education

institution, approve the publication of the results of learners if the

Council is satisfied that the assessment body or education institution

has—

(i) conducted the assessment free from any irregularity that may

jeopardise the integrity of the assessment or its outcomes;

(ii) complied with the requirements prescribed by the Council for

conducting assessments;

(iii) applied the standards prescribed by the Council which a learner is

required to comply with in order to obtain a certificate; and

(iv) complied with every other condition determined by the Council.

Framework for QA of Learner Achievement Based on established and existing practices in

assessment for certification Prescribed components of external

assessment (examinations) and Site-based/ internal / continuous assessment

Use of systems, processes, and procedures to evaluate, inspect, monitor and report on examination systems, processes and procedures of public and private assessment bodies.

Framework for Quality Assurance of Assessment Evaluation and /or accreditation of assessment bodies Periodic inspection of assessment systems Ongoing monitoring of assessment systems Quality assurance of external examinations through: Moderation of examination question papers Monitoring and moderation of SBA Monitoring the conduct of examinations Moderation of marking Standardization of assessment outcomes

Approval for the release of Results

Approval is based on the following requirements:

The examinations are conducted compliant to the applicable policies regulating the conduct and administration of the examinations

At the time of approval, there is no serious irregularity which could undermine the credibility of the examinations.

Quality Assurance of the DBE 2013 National Senior Certificate Examination

Emmanuel Sibanda: Acting Sen. Manager :

Quality Assurance of Assessments

PURPOSE:

To ensure that the question papers are of the required standard

(standard captured in the NCS and SAG’s)

To ensure that the question papers are relatively:- fair- reliable- representative of an adequate sample of the curriculum - representative of relevant conceptual domains

- representative of relevant levels of cognitive challenge

Moderation of question papers

Moderation of the question papers Approach:

Question papers set by panel of examiners – DBE

Internally moderated by DBE

Externally moderated by Umalusi

Subsequent moderations and approval

Moderation of the question papers

Criteria:

Technical criteria

Internal moderation

Content coverage

Text selection, types and quality of questions

Predictability

Cognitive skills

Marking memorandum/guidelines

Language and bias

Moderation of the question papersFindings: Areas of Good Practice

Percentage of question papers and memoranda approved after first and second moderation ( Nov 2013- 70% ; Mar 14 – 78%)

Simultaneous moderation of final and supplementary question papers

Moderation of the question papersFindings: Areas of Concern

Adherence to timeframes and impact on quality of setting and moderation

Question papers requiring more than four moderations

2 papers for November 2013 (Isizulu HP P1, Isixhosa HL P1) and 2 for March 2014 (Business Studies, Isizulu HL P1)

Definition:

Internal assessment refers to any assessment conducted by the provider , the outcome of which count towards the achievement of the qualifications

Umalusi appoints panels of moderators / subject specialist to carry out this mandate

Moderation of internal assessment

Purpose of Umalusi’s verification:

To verify the rigour and appropriateness of the DBE moderation process – linked to DBE plans

Ascertain the degree to which assessment bodies/provinces are attempting to ensure standardisation across

Ascertain the standard and quality of the tasks

Determine the extent and quality of internal moderation and feedback.

Determine the reliability and validity of the assessment outcomes

Moderation of internal assessment (cont.)

Moderation of Internal AssessmentApproach 1 (June/July 2013)- verifying the DBE SBA

moderation

PROVINCIAL EDUCATION DEPARTMENT

Subjects

All Mathematics, Life Sciences, Physical SciencesAccounting, English FAL, History, Geography, Life Orientation,

Moderation of Internal AssessmentApproach 2 (June/July 2013)- Umalusi independent

moderation (own sample)

PROVINCIAL EDUCATION DEPARTMENT

Subjects

All Economics, Business Studies, Mathematical Literacy, Music, (Practical subjects)Mechanical Technology, Electrical Technology

Moderation of Internal AssessmentAreas of good practice:

DBE conducted a very rigorous moderation and provided useful verbal feedback to PED at the end of each moderation session

Certain pockets of improvement wrt SBA implementation although far and wide

General adherence to policy in terms of number of tasks done, and presentation of learner evidence/portfolios.

Moderation of Internal AssessmentAreas of concern:

While internal moderation is being done in most schools, much of the focus is on compliance (monitoring) and not on the quality of assessment (actual moderation)

Teachers are still unable to develop tasks pitched at appropriate cognitive levels: focus is more on lower cognitive level

Assessment of practical investigations, research projects, assignments and simulations still a major problem

The use and development of rubrics is problematic: descriptors are unrealisable and vague

Monitoring of Examinations

“State of readiness”

Conduct of examinations

Marking

State of Readiness

Monitored the DBE state of readiness visits in 6 provinces: Eastern Cape, Free State, Kwazulu-Natal, Limpopo, Northern Cape, Gauteng

Comprehensive approach that monitors exam systems (including SBA)

Monitoring of Examinations“State of readiness”Findings:

All provinces have working examination systems in place

Of concern:

In certain PDE’s closer monitoring of printing of question papers needs attention

Inefficient or lack of proper coordination with districts wrt exam related processes

Monitoring of the writing phase.

Findings:

Generally examinations conducted in line with policy

Isolated instances of non-compliance (suitability of venue, identification of learners)

No of exam centres No of exam centers monitored by Umalusi

No of Umalusi monitors per province

6699 159 36

Monitoring of the marking phase

Number of marking centres

No of marking centers monitored by Umalusi

Total 118 77

Monitoring of marking phase:

Findings:Marking centres were generally well organised and suitable for the taskInadequate and inexperienced security in some provincesAbsence of communication facilities reported at one centre in one province

PURPOSE:Moderation of marking determines the standard and

quality of marking and ensures that marking is conducted in accordance with agreed practices

Umalusi engages with the following during the moderation of marking

Pre-marking/memorandum discussion: centralised memo discussions recommended - this will ensure consistency across marking centres

Moderation of marking (centralised and on-site)

Verification of marking

Marking verificationMemo discussion meetings:

Areas of good practice: The memo discussions for the approval of

final memoranda went relatively well in 2013. Provision of an extra day of training in marking was very welcome

Memo discussion meetings (cont)Areas of concern: Not all representatives did pre-marking. Provincial

representatives often experienced problems accessing a sample of scripts to pre-mark before coming for memo discussion meetings

The time between the examination dates and the memo discussions was generally far too short to allow pre-marking to take place. This was reported in several subjects, and seriously compromised the validity of the process, as meaningful discussion and consistency depends on the pre-marking of scripts

Memo discussion meetings (cont)

Areas of concern: Some provinces sent only one representative or

none at all to the memo discussions. The fact that the memo discussions for African

languages, HL, SAL and FAL took place in the same time slot caused problems because there was only one internal and external moderator for these levels in some languages.

Centralised & on-site marking verification

Areas of good practice:

Many external moderators expressed the opinion that the accuracy of marking had improved slightly.

External moderators were unanimous in their appreciation of the positive impact that the thorough training at the memo discussion meetings had had on the quality of marking.

Centralised & on-site marking verification (cont)

Areas of concern: Markers still experience problems with regard to interpreting answers to open-ended and higher order questionsUse of rubrics continues to be a serious concern in particular the inappropriateness of the rubrics used for P3 of HL and FAL: the descriptors did facilitate good marking. This needs revisiting by the DBE as it has a bearing on the marks allocated There are still markers marking literature questions, who do not have a thorough knowledge of the stories/dramas/novels/poems they are marking

Monitoring of the writing phase.

Areas of good practice:

The DBE exam was generally administered in line with policy. No major concerns were reported. One can see growth in the administration and conduct of the exams in PDEs.

No of exam centres No of exam centers monitored by Umalusi

Total number of Umalusi monitors

6699 159 36

Standardisation and verification of resulting

Provision of GENFETQA – Council may adjust raw

marks

International practice – large scale assessment systems

Standardisation – process used to mitigate the effect of factors other than learners knowledge and aptitude on the learners performance

Sources of variability – difficulty in question paper, undetected errors, learner interpretation of questions

Statistical moderation

Scope of standardisation 2013:

59 standardised

Raw marks accepted: 38 subjects

Moderated Upward : 5 subjects

Moderated Downward : 16 subjects

Verification of the Resulting Process

Planned Status

Subject structures Subject structures verified.

Candidate registration System verified during state of readiness visits.

Generation of mark Sheets Monitored

Capturing of Marks Monitored

Standardisation data & Booklets

Data sets received and verified

Capturing of adjustments Verified

Statistical moderation & resulting

Verified

Examination Irregularities The majority of irregularities were of a technical nature

and these were reported to Umalusi according to the established channels.

Some irregularities were as a result of registration-related problems, e.g. candidates nor appearing on mark sheets, some registered for incorrect subjects.

Umalusi represented on NEIC

WHAT CAN WE LEARN FROM THE NSC RESULTS?

STANDARDISATION DECISIONS

DBE NSC 2013

SUBJECT ADJUSTMENT

Life Sciences

½ CA = 0 at 0, scaled to -6 at 87, block -6 up to 162, scaled to 0 at 281, scaled to -3 at 300

Physical Sciences Raw

Mathematics

0 at 0, scale to -6 at 96, block -6 from 96 to 246, scale to 0 at 300

Mathematical Literacy

0 at 0, raw to 180, scale 0 to +6 from 180 to 234, scale to 0 at 300

Mathematics: Probability; Data Handling

½ CA = 0 at 0 scaled to -15 at 57, block -15 up to 260, scaled to 0 at 300

SUBJECT ADJUSTMENT

Geography 0 at 0, scale to -12 at 102, block -12 from 102 to 268, scale to 0 at 300

History RawReligion Studies Raw

Life Orientation Raw

Hospitality Studies0 at 0, scale to -6 at 96, block -6 from 96 to 257, scale to 0 at 300

Tourism

½ CA = 0 at 0 scaled to -6 at 47 scaled to 0 at 119, scaled to +11 at 241, scaled to 0 at 300

Consumer Studies Raw

SUBJECT ADJUSTMENT

Agricultural ScienceRaw

Agricultural Management Practices Raw

Agricultural Technology Raw

Music Raw

Dance Studies Raw

DesignRaw

Dramatic ArtsRaw

SUBJECT ADJUSTMENTVisual Arts Raw

Information Technology Raw

Computer Applications Technology

½ CA = 0 at 0 scaled to -12 at 50, scaled to 0 at 269, raw up to 300

Civil Technology ½ CA = 0 at 0 scaled to -14 at 129, scaled to -7 at 300

Electrical TechnologyRaw

SUBJECT ADJUSTMENT

Mechanical Technology0 at 0, scale to -6 at 126, block -6 from 126 to 246, scale to 0 at 300

Engineering Graphics and Design

Raw

AccountingRaw

Business Studies0 at 0, scale to -6 at 96, block -6 from 96 to 246, scale to 0 at 300

EconomicsRaw

SUBJECT ADJUSTMENTAfrikaans HL block -7

English HL RawIsiNdebele HL CA = 0 at 0 scaled to -13 at

90, scaled to +11 up to 163, scaled to +16 at 218, scaled to 0 at 300

IsiZulu HL Raw

Setswana HL Block -6

SUBJECT ADJUSTMENT

Siswati HL

½ CA = 0 at 0 scaled to -10 at 93, scaled to 0 at 154,, scaled to +9 at 258 , scaled to 0 at 300

IsiXhosa HL Raw

Xitsonga HL

0 at 0, scale to -12 at 73, block of -12 from 73 to 232, scale to 0 at 300.

Tshivenda HL

0 at 0, scale to -10 at 30, block of -10 from 30 to 245, scale to 0 at 300.

Sepedi HL½ CA = 0 at 0 scaled to -13 at 98, block -13 up to 159, scaled to +5 at 276, scaled to 0 at 300

SUBJECT ADJUSTMENT

Sesotho HL0 at 0, scale to -6 at 126, block of -6 from 126 to 233, scale to 0 at 300.

Afrikaans FAL RawEnglish FAL RawIsiNdebele FAL RawIsiZulu FAL Raw

Setswana FAL

CA = 0 at 0 scaled to -30 at 60, block -30 up to 222. scaled to -10 at 259, scaled to -30 at 300

Siswati FAL RawIsiXhosa FAL RawXitsonga FAL Raw

SUBJECT ADJUSTMENT

Tshivenda FALRaw

Sepedi FAL

½ CA = 0 at 0 scaled to -15 at 99, scaled to -5 at 142, block -5 up to 269, scaled to -15 at 300

Sesotho FALRaw

Afrikaans SAL Raw

English SALRaw

IsiNdebele SALRaw

IsiZulu SALRaw

SUBJECT ADJUSTMENT

Siswati SAL RawIsiXhosa SAL Raw

Sepedi SAL Raw

Sesotho SAL Raw

Quality Assurance of the IEB 2013 National Senior Certificate ExaminationEmmanuel Sibanda – Sen. Manager : Quality Assurance of Assessments

Moderation of the question papersAreas of Good Practice:

Percentage of question papers and memoranda approved after first and second moderation ( Nov 2013- 95% Mar 2014 – 100%)

Examining panels must be acknowledged for their favourable response to the External Moderator’s requests for changes and recommendations made

Moderation of Internal AssessmentTerm 4 moderation- focus on learner evidence and

teacher files

IEB Subjects

Mathematics, Maths Lit, Life Sciences, Geography,Physical Sciences, Accounting, English HL, History, Life Orientation, Business Studies, Economics, Afrikaans FAL,

Moderation of Internal AssessmentFindings: Areas of Good Practice IEB has very good systems and processes in place for SBA

implementation and support of their educators.

This is borne out by the generally good quality of tasks set by the IEB educators.

Findings: Areas of Concern The main challenge is how rubrics are developed: there is

some element of vagueness and subjectivity of criteria – this leads to differences in interpretation by educators, resulting in differences in allocation of marks.

Moderation at school level can also be improved.

Monitoring of the writing phase.

Scope:No of exam centres No of exam centers

monitored by UmalusiNo of Umalusi deployed for IEB monitoring

187 22 09

Monitoring of the writing phase.Findings:

Management of the examinations:

Generally examinations conducted in line with policy

There is good and ongoing communication between the IEB and their chief invigilator controlled the writing phase competently and professionally.

Isolated instances of non-compliance (candidate identification)

Monitoring of centres occurs but it should be done in a more visible onsite and regular basis

Monitoring of the markingphase.

Scope:No of marking centres No of marking centers

monitored by UmalusiNo of Umalusi deployed for IEB monitoring

03 03 01

Verification of markingFocus:

Memo discussion and on-site marking verification

Scope:

Accounting, Business Studies, Economics, English HL, English FAL, Afrikaans FAL, History, Geography, Life Sciences, Mathematical Literacy, Mathematics, and Physical Sciences

Verification of marking (cont)Findings:

The IEB conducts memo discussions in the presence of sub-examiners/Senior markers who are responsible for groups of markers during marking. The sub-examiners/Senior markers then train markers thoroughly on day 2.

Additions made to the final memo are ratified by the internal moderator and chief examiner. Commendable – IEB is a small system.

Marking was found to be fair, valid and reliable: attributed to the continuous feedback sessions between the chief examiner, senior sub-examiners, and sub-examiners.

The practice of double marking and therefore verification is highly commendable.

Attention should be paid to instances where moderation doesn’t always involve full moderation of a script but sampled questions.

Statistical moderation

Scope of standardisation 2013:

62 subjects standardised

Raw marks accepted: 47 subjects

Moderated Upward : 2 subjects

Moderated Downward : 13 subjects

SUBJECT ADJUSTMENT

Life Sciences

½ CA = 0 at 0, scaled to -6 at 87, block -6 up to 162, scaled to 0 at 281, scaled to -3 at 300

Physical Sciences Raw

Mathematics

0 at 0, scale to -6 at 96, block -6 from 96 to 246, scale to 0 at 300

Mathematical Literacy

0 at 0, raw to 180, scale 0 to +6 from 180 to 234, scale to 0 at 300

Mathematics: Probability; Data Handling

½ CA = 0 at 0 scaled to -15 at 57, block -15 up to 260, scaled to 0 at 300

SUBJECT ADJUSTMENT

Geography 0 at 0, scale to -12 at 102, block -12 from 102 to 268, scale to 0 at 300

History RawReligion Studies Raw

Life Orientation Raw

Hospitality Studies0 at 0, scale to -6 at 96, block -6 from 96 to 257, scale to 0 at 300

Tourism

½ CA = 0 at 0 scaled to -6 at 47 scaled to 0 at 119, scaled to +11 at 241, scaled to 0 at 300

Consumer Studies Raw

SUBJECT ADJUSTMENT

Agricultural ScienceRaw

Agricultural Management Practices Raw

Agricultural Technology Raw

Music Raw

Dance Studies Raw

DesignRaw

Dramatic ArtsRaw

SUBJECT ADJUSTMENTVisual Arts Raw

Information Technology Raw

Computer Applications Technology

½ CA = 0 at 0 scaled to -12 at 50, scaled to 0 at 269, raw up to 300

Civil Technology ½ CA = 0 at 0 scaled to -14 at 129, scaled to -7 at 300

Electrical TechnologyRaw

SUBJECT ADJUSTMENT

Mechanical Technology0 at 0, scale to -6 at 126, block -6 from 126 to 246, scale to 0 at 300

Engineering Graphics and Design

Raw

AccountingRaw

Business Studies0 at 0, scale to -6 at 96, block -6 from 96 to 246, scale to 0 at 300

EconomicsRaw

SUBJECT ADJUSTMENTAfrikaans HL block -7

English HL RawIsiNdebele HL CA = 0 at 0 scaled to -13 at

90, scaled to +11 up to 163, scaled to +16 at 218, scaled to 0 at 300

IsiZulu HL Raw

Setswana HL Block -6

SUBJECT ADJUSTMENT

Siswati HL

½ CA = 0 at 0 scaled to -10 at 93, scaled to 0 at 154,, scaled to +9 at 258 , scaled to 0 at 300

IsiXhosa HL Raw

Xitsonga HL

0 at 0, scale to -12 at 73, block of -12 from 73 to 232, scale to 0 at 300.

Tshivenda HL

0 at 0, scale to -10 at 30, block of -10 from 30 to 245, scale to 0 at 300.

Sepedi HL½ CA = 0 at 0 scaled to -13 at 98, block -13 up to 159, scaled to +5 at 276, scaled to 0 at 300

SUBJECT ADJUSTMENT

Sesotho HL0 at 0, scale to -6 at 126, block of -6 from 126 to 233, scale to 0 at 300.

Afrikaans FAL RawEnglish FAL RawIsiNdebele FAL RawIsiZulu FAL Raw

Setswana FAL

CA = 0 at 0 scaled to -30 at 60, block -30 up to 222. scaled to -10 at 259, scaled to -30 at 300

Siswati FAL RawIsiXhosa FAL RawXitsonga FAL Raw

SUBJECT ADJUSTMENT

Tshivenda FALRaw

Sepedi FAL

½ CA = 0 at 0 scaled to -15 at 99, scaled to -5 at 142, block -5 up to 269, scaled to -15 at 300

Sesotho FALRaw

Afrikaans SAL Raw

English SALRaw

IsiNdebele SALRaw

IsiZulu SALRaw

SUBJECT ADJUSTMENT

Siswati SAL RawIsiXhosa SAL Raw

Sepedi SAL Raw

Sesotho SAL Raw

Annexure to Approval Letter to DBE

Specific concerns regarding the quality assurance of the NSC examination and assessment:

There has been some improvement with regard to adherence to timeframes. However, there are subjects where improvements must be made. 25 November and 25 March question papers were submitted in May/June 2013 for first moderation. It is important to note that for the credibility of the NSC examination it is vitally important that every effort is made to adhere to agreed deadlines for setting and moderation of question papers. Any delays affect Umalusi in its quality assurance exercise.

Annexure to Approval Letter to DBE

There have also been some improvements made pertaining to administration of SBA and presentation of learner evidence of performance. Having said this, the following issues were found to be problematic:

Internal moderation reports are generally not available.

Lack of constructive feedback given back to learners after moderation.

Teachers are still challenged regarding the development of tasks pitched at appropriate cognitive levels: focus is more on lower cognitive level.

Assessment of Practical investigations, Research projects, Assignments and simulations still remains a major problem.

Annexure to Approval Letter to DBE

The use and development of rubrics is problematic: descriptors are unrealisable and vague.

Assessment of the Physical Education Task (PET) in Life Orientation continues to be a problem – inflation of marks.

A number of problems were reported with regard to the

standard and quality of marking and these clearly hinge on the

calibre of people appointed as markers. DBE is urged to look

closely into the appointment of markers.

Annexure to Approval Letter to IEB

Specific concerns that require remediation:

The quality of question papers submitted at first moderation. 58% of the question papers were not compliant with regard to Umalusi criterion that looks into the technical aspects or the face validity of the question paper. It should be borne in mind that question papers submitted to Umalusi must be in their print-ready form.

The quality of the memoranda submitted for moderation should be looked into.

Annexure to Approval Letter to IEB

The issue of rubrics used for assessment of learner work in SBA

needs to be looked into. As Umalusi we strongly believe that

rubrics are meant to facilitate marking, and should therefore leave

no room for ambiguity on the part of educators who are meant to

use these.

It was again observed that in Life Orientation there was a kink at

the 80% level. The IEB was requested in 2012 to address this

matter. Having noted this, the IEB is again urged to ensure the

bulging of marks at the 80% level does not recur. If this problem

persists in 2014 Umalusi would have no choice but to take

appropriate corrective measures.

Conclusion The findings of the quality assurance processes are a

clear indication of a maturing system that has, on the one hand, made positive strides towards improvement in certain areas of assessment and examination, but, on the other hand, still has a few challenges that need to be addressed.

The quality assurance of each of these processes presented above was conducted based on Umalusi criteria. Umalusi uses criteria that are subjected to constant review and refinement, to ensure that they are in line with current trends in assessment and examinations.

Conclusion… In general Umalusi is pleased with the manner in which the

2013 NSC examination was administered.

Umalusi acknowledges that a number of technical irregularities were reported, but these were addressed in a fitting manner.

Umalusi takes this opportunity to express appreciation to the national & provincial departments of education for their concerted effort in ensuring a credible examination.

Umalusi expresses appreciation also to all the relevant stakeholders for the necessary support given in line with Umalusi quality assurance initiatives.

Thank you!