Download - Consistency of Assessment (Validation) Webinar – Part 2 Renae Guthridge WA Training Institute (WATI)
Consistency of Assessment(Validation)
Webinar – Part 2
Renae GuthridgeWA Training Institute (WATI)
RQI - 321
What three things do you recall about yesterday’s webinar?
What two questions do you have about yesterday’s?
What was one interesting point about yesterday’s webinar?
Foxes and a Vixen
Principles of Assessment
Rules of Evidence
FourVeryRampantFoxes
ASatisfiedCunningVixen
Ensuring Validity - Unpacking the Unit of Competence
Mandatory unitrequirements include:• Pre-requisites and/or co-
requisites• Elements • Required Skills• Required Knowledge• Range Statement (‘must’)• Critical Aspects of
Evidence• Other “must”
requirements
Mandatory unit requirements include:
• Pre-requisite(s)• Elements • Foundation Skills• Performance Evidence • Knowledge Evidence
These can be listed in a Unit Requirements List…
Unit Requirements List
Unit of competency: SHOP123A Review stock
Mandatory unit requirementsElements of competency1. Identify stock requirements2. List stock in store3. List stock shortages4. Prepare stock order formsFoundation skills1. Literacy skills (reading and writing)2. Numeracy skillsPerformance evidence1. Accurate list of stock on display2. Accurate list of stock in store-room3. Accurate list of stock to order4. Two categories of stockKnowledge evidence1. Organisation procedures2. Range of stock management processesPre-requisites:SHOP101A
Unit of competency: SHOP123A Review stock
Mandatory unit requirementsElements of competency1. Identify stock requirements2. List stock in store3. List stock shortages4. Prepare stock order formsRange Statement (musts)1. Stock on display2. Stock in storeroomCritical Aspects of EvidenceCAE 1 Accuracy and timelinessCAE 2 Full review of stockRequired SkillsRS 1 Communicates with staffRS 2 Correct forms selectedRS 3 Correct calculationsRequired Knowledge (See list)Pre-requisites: SHOP101AOther (specify): Two contexts
Old Unit Structure New Unit Structure
Ensuring Validity -Identifying Mandatory Unit Requirements
• Highlight all mandatory requirements in a unit of competency.
• Prepare a list of all the mandatory requirements.
• Validate your list with a colleague by comparing lists.
• Prepare a flexible and fair evidence-gathering plan.
• Validate your plan with a colleague.
Planning to Gather Evidence
The Unit Requirements List can now be extended to show the relationship between the unit requirements and evidence sources.
Unit of competency: SHOP123A Review stock
Mandatory unit requirementsSources of evidence
WP1 WP2 SIM1 SIM2 Q/AElements of competency 1. Identify stock requirements 2. List stock in store 3. List stock shortages 4. Prepare stock order forms Range Statement 1. Stock on display 2. Stock in storeroom Critical Aspects of Evidence CAE 1 Accuracy and timeliness CAE 2 Full review of stock Required Skills RS 1 Communicates with staff RS 2 Correct forms selected RS 3 Correct calculations Required Knowledge (see list) Pre-requisites: SHOP101A Other (specify): Two contexts
WP1: Context of workplace assessment
WP2: Context of workplace assessment
SIM1: Context of simulation
SIM2: Context of simulation
Assessment Planning Matrix
Shade the cells that don’t apply.
The blank cells in a column provide a space to record evidence later on when it has been gathered.
Developing Evidence-gathering Tools - Assessing Required Knowledge / Knowledge Evidence
Assessing ‘know-how’…
…tell me…
…written or oral short answer tests.
Ensuring Validity, Reliability and Sufficiency -Assessing Required Knowledge / Knowledge Evidence
Is the AQF level of the questions appropriate?
Is the context and content of the questions relevant to the unit?
Do the questions extend beyond the elements and demonstrate transfer?
Is ALL required knowledge covered?
If other knowledge is covered, has it been distinguished from the required knowledge?
Is ALL required knowledge demonstrated before signing off on the unit record?
Developing Evidence-Gathering Tools -Assessing Elements and Required Skills / Performance Evidence
Assessing can do…
…show me…
…observation in the workplace or a simulation of a workplace.
Unit of competency: SHOP123A Review stock Name of student: Joe Bloggs
Mandatory unit requirements Performance Criteria Comments Result
E1. Identify stock requirements 1.1 Contextualised for third party observer.
The observer would write a narrative of what was observed for the assessor to review.
1.21.31.4
E2. List stock in store 2.1
2.22.3
RK1 Stock on display Display stock not reviewed N/a
RK2 Stock in storeroom CAE1 Accuracy and timeliness Review completed to company standards
CAE2 Full review of stock All storeroom stock reviewed
RS1 Communicates with staff RS2 Correct forms selected Assessment context: Two different contexts
requiredFrozen foods
Observer’s comments:
Observer’s name: M. Ployer Observer’s position: SupervisorSignature and date: M. Ployer 5/8/10 Student signature and date: Joe Bloggs 5/8/10
Workplace Observation Tool 1 (WP1) (3rd Party Observer)
Unit of competency: SHOP123A Review stock Name of student: Joe Bloggs
Mandatory unit requirements Performance Criteria Comments Result
E3. List stock shortages 3.1
3.2E4. Prepare stock order forms 4.1
4.24.3
RK1 Stock on display RK2 Stock in storeroom Storeroom stock not reviewed N/aCAE1 Accuracy and timeliness Met our company standards
CAE2 Full review of stock Full range of display stock reviewed
RS1 Communicates with staff Did not ask for advice or report problems to supervisor
RS2 Correct forms selected RS3 Correct calculations Assessment context:
Two different contexts required
Dry goods
Observer’s comments:
Observer’s name: Sam Smith Observer’s position: RTO AssessorSignature and date: Sam Smith 7/9/10 Student signature and date: Joe Bloggs 7/9/10
Workplace Observation Tool 2 (WP2) (RTO Observer)
Unit of competency: SHOP123A Review stock Name of student: Joe Bloggs
Mandatory unit requirements Performance Criteria Comments Result
E1. Identify stock requirements 1.1
1.21.3 1.4
E2. List stock in store 2.1
2.22.3
E3. List stock shortages 3.1 Incorrect stock list used.
3.2E4. Prepare stock order forms 4.1
4.24.3
RK1 Stock on display
RK2 Stock in storeroom N/aCAE1 Accuracy and timeliness CAE2 Full review of stock RS2 Correct forms selected RS3 Correct calculations Assessment context: Two different contexts
requiredDry goods
Observer’s comments:
Observer’s name: Sam Smith Observer’s position: RTO AssessorSignature and date: Sam Smith 9/10/10 Student signature and date: Joe Bloggs 9/10/10
Simulation Observation Tool (SIM)
Unit of competency: SHOP123A Review stock Name of student: Joe Bloggs
Required knowledge
itemsQuestions Verbatim answers or comments Correct Score Criterion Result
RK1. 1 2 2
3 RK2. 2
2 3 6
Hot water is mixed with soap powder to make a warm nourishing drink
7 8
RK3. 4 1 1 RK4. 5
4 3
9 10 11
13 Record the incorrect answer here ….
RK5. 12 1 1 RK6. 14
2 2 15
RK7. 16 2 2 17
18 Overall unit result Extra knowledge Questions EK1. 19
EK2.20
Sandals can be worn in the cool room
EK3. 21
EK4. 22
Observer’s name: Sam Smith Observer’s position: RTO AssessorSignature and date: Sam Smith 9/8/15 Student signature and date: Joe Bloggs 9/8/15
Question & Answer Tool (Q/A)
Ensuring Validity, Reliability and Sufficiency - Assessing Elements and Required Skills / Performance Evidence
Are clear instructions provided to the observer?
Are clear instructions provided to the candidate?
Are all four dimensions of competency (task skills, task management skills, contingency management skills, and job/role environment skills) met?
If third-party evidence-gatherers are used, have the performance criteria been contextualised by writing them in familiar language (plain English) and with examples, and has space been provided for narrative to explain the or assigned to each criterion?
Recording evidence Use the Assessment Planning Matrix as a
record of individual evidence by adding the name of the student and a ‘Result’ column and recording evidence from the tests and observation checklists into the blank cells.
Unit of competency: SHOP123A Review stock Name of student: Joe Bloggs
Mandatory unit requirementsSources of evidence
WP1 WP2 SIM1 SIM2 Q/AElements of competency 1. Identify stock requirements 2. List stock in store 3. List stock shortages 4. Prepare stock order forms Range Statement 1. Stock on display N/a 2. Stock in storeroom N/a N/a N/a Critical Aspects of Evidence CAE 1 Accuracy and timeliness CAE 2 Full review of stock Required Skills RS 1 Communicates with staff RS 2 Correct forms selected RS 3 Correct calculations Required Knowledge (See list) Pre-requisites: SHOP101A Pre-requisite unit achieved in 2007 at XYZ CollegeOther (specify): Two contexts Frozen
foodsDry goods Dry goods Dry goods
WP1: Frozen foods in storeroom
WP2: Dry goods on display
SIM1: Dry goods on display
SIM2: Supplementary assessmentName of assessor: Sam Smith Student signature and date:Signature and date: Result reported date:
Assessment Record
Meeting the Rules of Evidence -Recording evidence of competency
Are checklists used to record evidence as it happens?
Is poor quality evidence discarded?
Is evidence gathered and collated from checklists?
Is evidence from different units in a cluster distributed to appropriate unit records?
Making the Assessment Judgement
The Assessment Record can now become a Judgement Tool showing which requirements had been met in the ‘Results’ column.
Unit of competency: SHOP123A Review stock Name of student: Joe Bloggs
Mandatory unit requirementsSources of evidence
ResultWP1 WP2 SIM1 SIM2 Q/A
Elements of competency
1. Identify stock requirements 2. List stock in store 3. List stock shortages 4. Prepare stock order forms Range Statement
1. Stock on display N/a 2. Stock in storeroom N/a N/a N/a Critical Aspects of Evidence
CAE 1 Accuracy and timeliness CAE 2 Full review of stock Required Skills
RS 1 Communicates with staff RS 2 Correct forms selected RS 3 Correct calculations Required Knowledge (See list) Pre-requisites: SHOP101A Pre-requisite unit achieved in 2007 at XYZ College Other (specify): Two contexts Frozen
foodsDry goods Dry
goodsDry
goods
Result WP1: Frozen foods in storeroom
WP2: Dry goods on display
SIM1: Dry goods on display
SIM2: Supplementary assessment
Name of assessor: Sam Smith Student signature and date: Joe Bloggs 6/11/15Signature and date: Sam Smith 6/11/15 Result reported date: 14/11/15
Judgement Tool
Meeting the Foxes and Vixen -Making the assessment judgement
Has sufficient evidence been gathered for each mandatory requirement?
Is the evidence authentic and current?
Have alternative arrangements been offered for candidates with specific needs?
Has each mandatory requirement been demonstrated?
Has the record been signed off by person(s) with assessor qualifications and vocational expertise?
Have other uses of the record been quarantined from the competency judgement?
Training and Assessment Plan
The Assessment Planning Matrix, Assessment Record and Judgement Tool can now morph into a Training Delivery and Assessment Plan by listing the learning resources for each unit (and any extra) requirements.
Unit of competency: SHOP123A Review stock Name of student: Joe Bloggs
Mandatory unit requirementsLearning resources Sources of evidence
ResultSession Reference WP1 WP2 SIM1 SIM2 Q/A
Elements of competency 1. Identify stock requirements 1 - 3 pp 15 - 23 2. List stock in store 4 pp 37 - 39 3. List stock shortages 5 - 7 Handout
1
4. Prepare stock order forms 9 Handout 3
Range Statement 1. Stock on display 8 Handout
2N/a
2. Stock in storeroom 8 Handout 2
N/a N/a N/a
Critical Aspects of Evidence CAE 1 Accuracy and timeliness 4 p. 38 CAE 2 Full review of stock 4 p. 39 Required Skills RS 1 Communicates with staff 7 Handout
1
RS 2 Correct forms selected 8 Handout 2
RS 3 Correct calculations 7 Handout 1
Required Knowledge (See list) 1 - 9 All Pre-requisites: SHOP101A Student records Pre-requisite unit achieved in 2007 at XYZ College Other (specify): Two contexts 1 - 9 All Frozen foods Dry
goodsDry goods Dry
goods
Result WP1: Frozen foods in storeroom
WP2: Dry goods on display
SIM1: Dry goods on display
SIM2: Supplementary assessment
Workplace or regulatory requirements: OH&S: Use of steps & cool room access
Name of assessor (Assessment): Sam Smith Name of Assessor (Vocational): Sam SmithStudent signature & date: Joe Bloggs
6/11/15Signature and date: Sam Smith 6/11/15 Signature and date: Sam Smith 6/11/15 Result reported date: 14/11/15
Training and Assessment Record
Thoughts…What do you think?Would you use this approach?What do you like / dislike about this approach?
Strategies for Systematically Validating Assessment Procedures
Review assessment strategies
Compare assessment evidence
Evaluate assessment outcomes
How Can We Review Assessment Strategies?
Reviewing involves the inspection of assessment processes and products to determine the ‘face validity’ of the assessment strategy…‘does it look right’?
What criteria could be used to guide the review of your assessment processes?
Review Criteria…
How Can We Compare Assessment Evidence?
Comparing assessment processes involves comparing assessment documentation and comparing assessment outcomes to determine content validity, reliability, consistency, and reproducibility of the assessment strategy.
What criteria could be used to guide the comparison of your assessment processes with those of another assessor?
Comparison Criteria…
How Can We Evaluate Assessment Outcomes?
Evaluating assessment processes involves gathering stakeholder feedback to determine the predictive validity, impact, effectiveness and credibility of the assessment strategy.
What criteria could be used to guide the evaluation of your assessment processes.
Evaluation Criteria…
How Can We Improve Assessment Processes?
The quality cycle
How can you follow this cycle to improve your assessment processes?
What Records of Validation Should We Keep and Why?
Records of Validation…Create a separate file for each unit of competency. Each file to contain:
• UoC;• Assessment plan;• Current assessment instruments and instructions;• Validation form;• A schedule for the administration of validation forms
showing past and future activities;• A summary of evidence gathered to date from each
validation source;• A summary of problems identified; and• A summary of actions taken to remedy them.
Before You Go…
Answer the following with a yes, no or maybe…1. I have learned something about validation
during these two sessions. 2. I will change some of the assessment
processes I currently use. 3. I am more confident about validating with
others and sharing my assessment processes.
Is there anything else you need to know about validation?
Thank you and watch out forFoxes and Vixens!
All pictures in these slides were taken from http://search.creativecommons.org/