sterling practices in designing and scoring performance-based exams ronald c. rodgers, ph.d....
TRANSCRIPT
Sterling Practices in Designing and Scoring Performance-Based Exams
Ronald C. Rodgers, Ph.D.Continental Testing Services, Inc./
Employment Research Institute
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Design Principles
• Select work products that can be tested uniformly for all candidates (materials, etc.)
• Blind Scoring: If possible, have examiners score work products anonymously
• Observed Scoring: Checklists of observable steps may limit bias if blind scoring not feasible
• Apply industry standards in scoring criteria
• Special security steps may be required
Sample 1: Plumbing Practical Exam
Residential Drawing Three-story apartment building with fixturesFinish plumbing specifications for all vents and
drains in compliance with Plumbing CodeMay use isometric or diagrammatic response
Copper Pipe AssemblyBring specified 5′ & 2′ copper tubing & fittingsMeasure, cut, assemble and solder pipe left- or
right-hand rolling offset around obstruction
Sample Copper Pipe Specifications
Blind Scoring: Plumbing
Residential Drawing (Did candidate follow Code?) Bathroom soil stacks, drains, vents, sizing (50%) Kitchen waste stack, drains, vents, sizing (24%) Basement ejector, laundry, sewer connection (26%)
Copper Pipe Assembly (Quality of workmanship) Measurement, Assembly & Alignment (53%)
(deduct for each ¼″ difference over ±¼″) Pressure Test at 100 psi under water (30%) Quality of soldering & reaming (17%)
Plumbing Practical Exam Security
Two building drawings with same fixtures; candidates seated with alternate drawings
Two copper projects; no two candidates with same project placed together in shop
Pipe dimensions change for every testMust show calculations for 4 pipes not givenRepeat candidates receive different copper
pipe project and drawing for any retesting
Sample 2: Court Reporters
• Professionally recorded 5-minute opening or closing argument at 200 wpm
• Professionally recorded 5-minute testimony in two contrasting voices at 225 wpm
• Two-minute practice test in same voice(s)
• Word & syllabic counts controlled for each 15-second interval throughout dictations
• CD controls playback speed & intervals
Blind Scoring: Court Reporters
Define scoring criteria in candidate study guide Wrong, omitted, added or misspelled words Wrong or missing dates or numbers Omitted capital letters for specified words Also specify what is NOT scored as an error
(misplaced commas & periods, spacing, etc.) Passing score = 95% accuracy in transcript* 50 errors or less in 200 wpm transcription 56 errors or less in 225 wpm transcription* National Court Reporters Association standard
Court Reporters Exam Security
Exam CDs repeat in two-year rotationProctors have directions to erase digital
memory from transcription machinesAll cell phones and recording devices must
remain outside room during testingPaper-less transcription machines prohibited
(proctors would need to erase exam from three digital memory sites in machine)
Sample 3: Carpentry/Rafters Test
Quality and Safety Requirements• Layout and cutting ± 1/16" of specifications• Layout must be performed in required
measurement and marking sequence• Layout/cutting must be done within time limits• The test must be stopped and candidate scored as
failed if a candidate removes safety glasses while working on project and/or fails to follow any other OSHA safety procedures
Observed Scoring: Carpentry
Cutting
All common rafter cuts ± 1/16" of plans or specifications
Tracing Second Rafter
Traced common rafter marks ± 1/16" of plans or specifications
Followed Measurement and Marking Sequence Finished Cutting & Tracing within Time Limit
Contact Information
Ronald C. Rodgers, Ph.D.Director of Measurement Services,
Continental Testing Services& President, Employment Research Institute809 Ridge Road, Suite 201Wilmette, IL 60091847-256-5240 Fax 847-256-5241