linda meyer - moderation of fsas - best practice (1)

Moderation of the FSA –

Best Practice

Linda Meyer

INTEGRATED FSA

The FSA for the qualification will focus on the extent to which a learner can demonstrate applied competence.

Applied competence, in terms of the NQF is evidenced through learner’s ability to integrate concepts, ideas and actions in authentic, real-life contexts and is expressed as practical, foundational and reflexive competence

PURPOSE

The final integrated summative assessment refers to the process of making judgments about achievement. This is carried out when a learner is ready to be assessed at the end of a learnership or full qualification. The FSA will be set by a registered assessor appointed by the Certification/Outsourced Partner after notification from the provider/assessor that the candidates are ready.

FSA

Practical competence - demonstrated ability to perform a set of tasks and actions in authentic contexts

Foundational competence - demonstrated understanding of what we are doing and why we are doing it

Reflexive competence - demonstrated ability to integrate our performances with our understanding so that we are able to adapt to changed circumstances and explain the reason behind these adaptations.

ASSESSMENT METHODS

In setting the FSA, SETQAA would like

Certification/ Outsourced Partners

and/or assessors to take note of the

different assessment methods and

encourage the use of a wide variety of

methods.

SETTING OF FINAL INTEGRATED

SUMMATIVE ASSESSMENT (FSA)

• Function and role of the

Certification Partner/Outsourced Partners

The Certification/Outsourced

Partner is responsible for setting of

the FSA at NQF Level 4, with a

memorandum of model answers

and a grid showing its linkages

with the assessment criteria of the

qualification.

FSA

All FSA tools and memoranda, even

in draft form, are the property of the

Services SETA ETQA and may not be

revealed to, or discussed with any

unauthorized person. The highly

confidential nature of these

assessment instruments must be

safeguarded at all times.

FSA

If the assessment instrument and

memoranda are stored on computer,

thorough security measures must be

taken to prevent any illegal access to

these files.

The contents of the assessment

instrument must reflect the relevant

unit standards and qualification and

must not be biased towards any

particular textbook or learning

material.

A second assessment instrument

should be set as standby assessment

instrument. SETQAA have the right to

propose the use of either assessment

instruments for the FSA, if necessary.

FSA

The Certification/Outsourced Partner must ensure that the assessment instruments are not identical.

The Certification/Outsourced Partner must also ensure that neither of these assessment instruments are the same as any previous assessment instruments.

FSA

MODERATION CRITERIA

The moderator must ensure that the

assessment instrument is aligned with

the purpose and rationale of the

qualification and that it complies with

the applicable SAQA level descriptor.

Draft assessment instruments must fall

within the ambit and in accordance

with the guideline document.

Functions and role of the

moderator

After the Certification/Outsourced Partner has set the FSA instrument, with a memorandum of model answers, an internal moderator, appointed by the Certification/Outsourced Partner, must moderate all draft assessment instruments and memoranda for the assessment instruments under his/her jurisdiction.


moderator

During the moderation, the

assessment instrument and

memoranda are to be objectively

assessed by the moderator with

regard to the following:


moderator cont.

The moderator must verify that papers set for the FSA`s are not identical.

The moderator must verify that the Unit Standards are appropriate to the registered qualification.


moderator

The moderator must ensure that the final summative integrated assessment instrument is aligned with the purpose and rationale of the qualification and that it complies with the applicable SAQA level descriptor.


moderator

The moderator must ensure that the final

integrated assessment instruments must

fall within the ambit and in accordance

with this guideline document.

The extent to which the prescribed Unit

Standards are covered as well as the

assurance that nothing is asked that is

outside the scope of the Unit

Standard/Qualification must be verified.


moderator

The spread and number of marks allocated according to the weighting

and credits as per the registered learnership/qualification must be

assessed.


moderator

The effectiveness of the type of

assessment methods used, written

questions, essays, paragraphs, case

studies, must be assessed.


moderator

The questions, terminology, numbering, spelling, punctuation, sentence construction, units, and symbols must be clear, unambiguous and grammatically correct so that candidates will know exactly what is expected of them.


moderator

The length of the assessment instrument must take into account the time allocated as well as reading time.

The draft assessment instrument must be of such a length that a well-prepared learner will be able to answer it comfortably within the time allocated, with reasonable time remaining for revision.


moderator

It should be borne in mind that it

takes considerable time to read

through certain assessment

instruments.

The memorandum must ensure that

model answers give enough detail

for the assessor of the FSA to conduct

an accurate and fair assessment.

Criteria for moderation of the FSA

instrument

Standard of the paper (validity, reliability and fairness)

Questions of various types e.g. Multiple

Choice Questions, Case studies,

paragraphs, data response, essay, etc

Questions from which candidates are to

choose – are they of equal difficulty

level

Correct distribution in terms of cognitive

levels (Bloom’s Taxonomy)

Bloom’s Taxonomy In 1956, Benjamin Bloom, a professor at the

University of Chicago, shared his famous "Taxonomy of Educational Objectives". Bloom had identified six levels of cognitive complexity that have been used over the past 40 years to make sure that instruction stimulates and develops learners' higher-order thinking skills. The Levels are: Knowledge : Rote memory skills (facts, terms, procedures, classification systems) Comprehension : The ability to translate, paraphrase, interpret or extrapolate material. Application : The capacity to transfer knowledge from one setting to another. Analysis : The ability to discover and differentiate the component parts of a larger whole. Synthesis : The ability to weave component parts into a coherent whole. Evaluation : The ability to judge the value or use of information using a set of standards.

Bloom’s Taxonomy 1. Knowledge: remembering of previously learned material; recall (facts

or whole theories); bringing to mind. Terms: defines, describes, identifies, lists, matches, names. 2. Comprehension: grasping the meaning of material; interpreting

(explaining or summarizing); predicting outcome and effects (estimating future trends).

Terms: convert, defend, distinguish, estimate, explain, generalize, rewrite.

3. Application: ability to use learned material in a new situation; apply rules, laws, methods, theories.

Terms: changes, computes, demonstrates, operates, shows, uses, solves. 4. Analysis: breaking down into parts; understanding organization,

clarifying, concluding. Identify parts: See Related Order; Relationships; Clarify. Terms: distinguish, diagrams, outlines, relates, breaks down,

discriminates, subdivides. 5. Synthesis: ability to put parts together to form a new whole; unique

communication; set of abstract relations. Terms: combines, complies, composes, creates, designs, rearranges. 6. Evaluation: ability to judge value for purpose; base on criteria;

support judgment with reason. (No guessing). Terms: appraises, criticizes, compares, supports, concludes,

discriminates, contrasts, summarizes, explains.

Bloom’s Taxonomy .Knowledge (finding out)

a. Use - records, films, videos, models, events, media, diagrams, books... b. observed behavior - ask match, discover, locate, observe, listen.

2. Comprehension (understanding) a. Use - trends, consequences, tables, cartoons.... b. observed behavior - chart, associate, contrast, interpret, compare.

3. Application (making use of the knowledge) a. use - collection, diary, photographs, sculpture, illustration. b. observed behavior - list, construct, teach, paint, manipulate, report.

4. Analysis questions (taking apart the known) a. use - graph, survey, diagram, chart, questionnaire, report.... b. observed behavior - classify, categorize, dissect, advertise, survey.

5. Synthesis (putting things together in another way) a. use - article, radio show, video, puppet show, inventions, poetry, short story... b. observed behavior - combine, invent, compose, hypothesis, create, produce, write.

6. Evaluation (judging outcomes) a. use - letters, group with discussion panel, court trial, survey, self-evaluation, value, allusions... b. observed behavior - judge, debate, evaluating, editorialize, recommend.


instrument

Overall: how does the standard of the assessment instrument compare in relation to other qualifications/s assessment instruments and previous assessment instruments.


instrument

• Intellectually challenging and

allowing for creative responses from

candidates

•Suitability of examples and

illustrations

Relationship between mark

allocation, level of difficulty and

time allocation


instrument

FSA do not count more than

continuous assessment of providers –

the objective for the FSA is to check

the standard across providers and

across qualifications -

STANDARDISATION

Technical Criteria

•Cover page with all relevant details

such as time, unit standards and

instructions to candidates

•Clarity of instructions to candidates

•Lay out: learner friendly

•Correct numbering

Technical Criteria

•Mark/weighting allocation clearly

indicated

•Quality of illustrations, graphs, tables

etc must be print ready

•Complete memorandum with model

answers with mark/weight allocation

and provision for alternatives

Registered Unit Standards and

Qualifications

•Relevance to registered

Qualification, i.e. unit standards,

specific outcomes, assessment

criteria

•Levels of questions

•Coverage of unit standards

•Weighting and spread of contents

History of tool

Full history of FSA (when it will be conducted, who will be assessed, when and where) with all drafts, internal moderators’ comments etc. must accompany the assessment instrument each time it is submitted to the external moderator (SETQAA)

Cognitive skills

What conceptual constructs of the unit standards does the assessment instrument deal with?

e.g. – reasoning ability

• ability to communicate

• ability to translate from verbal to symbolic

• ability to compare and contrast

• ability to see causal relationship

• ability to express an argument clearly

Cognitive skills

Are these constructs representative of the

best and latest developments in the training of this knowledge field?

Are the questions challenging and allowing for creative responses from candidates?

Suggested application of cognitive levels for an NQF Level 4 qualification is: • 10% knowledge • 20% comprehension • 40% application • 30% analysis, synthesis, evaluation

Language and bias

• Correct terminology

• Appropriate language register – for the level of the learner

• Avoidance of gender, race, cultural, provincial bias

• Clear and unambiguous specification of instructions within questions e.g. list, describe

Time

• Relationship between difficulty level of questions and time

• Relationship between time and mark allocation and answer requirements

Internal Moderation

Is there evidence that the paper has been internally moderated?

Quality, standard and relevance of input from internal moderator.

Competency of Assessor

• Knowledge of assessor setting the FSA with regard to qualification.

• Knowledge, experience, expertise in assessment

Marking Memorandum

Correctness of memorandum

Correspondence with questions.

Alternative answers provided

Does it facilitate assessment?

Alignment Grid

Did they use the grid showing the alignment of the instrument with the purpose, rational, assessment criteria and NQF level descriptor of the qualification as issued by the SSETA used?

Overall impression of the paper

• Fairness of the assessment instrument as whole

• Will the assessment instrument as a whole assess the achievement of the purpose of the qualification and the exit level outcomes

• Recommendations for improvement or maintenance of qualification/unit standards.

• final acceptance/rejection of individual assessment questions and whole assessment instrument needs to be substantiated

REFERENCES OR RECORDS

ETQA-F 005 Final Integrated Summative Assessment Instrument

Only after written approval from the

external moderator, SETQAA will

approve and sign off the final

assessment instrument and notify the

Certification/Outsourced Partner to

proceed with the distribution of the

assessment instrument and conduct

the assessment.

STAKEHOLDERS INVOLVED IN

QA OF ASSESSMENT

PROCESSES Provider internal assessors

Provider internal moderators

Certification/Outsourced Partner

assessors

Certification/Outsourced Partner

moderators

SETQAA external moderators

SETQAA external verifiers

TRAINING PROVIDER ASSESSMENT Assessment should happen on a CONTINUOUS

basis during learnership/training – formative as

well as summative

Evidence gathered in PoE

Internally moderated and reports submitted to

SETQAA at regular specified intervals (See

moderation policy)

Externally moderated/verified by SETQAA

PoE submitted for external moderation by CP at

end of learning

FSA (PRACTICAL

OBSERVATION) Should be set and conducted

CONTINUOUSLY and/or AT END of

learnership/ qualification

Set and assessed by registered

assessors, moderated by registered

moderators appointed by CP

Externally verified by SETQAA

FSA (WRITTEN COMPONENT)

Conducted AT END of

learnership/Qualification

Set and moderated by registered

assessors and moderators appointed

by CP

Assessed and moderated by registered

assessors and moderators appointed

by CP

Externally moderated/verified by

SETQAA appointed moderators

PRACTICAL COMPONENT THEORETICAL

COMPONENT

CONTINUOUS ASSESSMENT VS FINAL SUMMATIVE ASSESSMENT

SETQAA DOC

FSA Moderation process

APPLICATION FOR MODERATION OF FINAL INTEGRATED SUMMATIVE ASSESSMENT INSTRUMENT RECEIVED BY SSETQA

EXTERNAL MODERATOR APPOINTED BY SSETQA

NO COMMUNICATION PERMITTED BEWEEN DEVELOPER AND MODERATOR

MODERATION OF FINAL INTEGRATED SUMMATIVE ASSESSMENT INSTRUMENT / RPL TOOL

FSA Moderation process PRELIMINARY REPORT PREPARED BY

MODERATOR AFTER THE FSA/RPL TOOL HAS BEEN MODERATED - REMEDIATION IDENTIFIED

TOOL SENT BACK TO DEVELOPER FOR REMEDIATION TO BE ADDRESSED

THE DEVELOPER MAY REMEDIATE / REQUEST ADDITIONAL INFORMATION OR APPEAL THE MODERATION DECISION THROUGH THE SSETA APPEALS PROCEDURE

FSA Moderation process

REMEDIATION COMPLETED

SENT TO EXTERAL MODERATOR TO

CONFIRM REMEDIATION & FINAL

REPORT SUBMITTED

ELECTRONIC VERSIONS SENT

FSA/RPL TOOL REGISTERED ON

DATABASE

The Certification/Outsourced Partner assures the rectification is done and sends the final draft together with the Moderators report to SETQAA for external moderation.

Only after written approval from the external moderator, SETQAA will approve and sign off the final assessment instrument and notify the Certification Partner to proceed with the distribution of the assessment instrument and conduct the assessment.

Thank you