7 ways that online assessments help ensure compliance

Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

7 Ways That Online Assessments Help Ensure Compliance

John Kleeman, Chairman, Questionmark

Brian McNamara, Marketing Director, Questionmark


What do we mean by “assessment”


Good practice in using assessments

Q&A

Agenda

Slide 2


Brian McNamara

John Kleeman Chairman and Founder of Questionmark Active in the development of industry and

technical standards relating to online assessment Frequent contributor of blog articles focusing on

assessment good practice

Marketing Director, Questionmark 15+ years working with organizations in the

training/ learning technology space

About the Presenters

Slide 3


Slide 4

About Questionmark

Founded in 1988 Assessment Management Systems

Create, deliver and report on quizzes, tests, surveys and exams Measure knowledge, skills and attitudes Commonly used for corporate learning, certification and

compliance related assessment

More than 1,100 current customers worldwide Corporate: Many customers in highly regulated industries such

as Energy & Utilities; Financial Services; Manufacturing; Healthcare; Medical (Pharmaceuticals, Biotech, Medical devices); Transportation; Retail.

Government and Military Higher Education


What we mean by the term “assessment”

Slide 5


Examples of Suppliers: Hogan, SHL, Insights, Myers Briggs, etc.

Examples of Suppliers: Questionmark

Survey Monkey and numerous others

Examples of Suppliers to author, deliver and analyze:

Questionmark

Examples of Suppliers of pre-built tests:

Total Testing, eSkill, etc

Not normally used in companies or education.

Assessments

Personality assessment

Cultural Fit

Job Fit

Survey

Customer Satisfaction

Employee Satisfaction

Course Evaluation

JTA

180/360

On The Job Performance

Formative

Diagnostic

Summative

Observational

Medical and Psychological

Psychiatrists. Courts, etc.

Types of Assessment

All assessments provide a stimulus and collect data to

evaluate

Type of Suppliers: 1. Prebuilt Instruments

/Assessments 2. Tools to create assessments (i.e.

Authoring, Delivery and Analytics)

Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 7

Quizzes, Tests, Exams

•Questions and scoring algorithms

•Scores and feedback at question, topic and assessment level

Self-Assessments

•Questions related to knowledge, skills, abilities and attitudes

•Retains responses for future review and change

Surveys

•Questions relate to a person’s opinions on a topic

•Course evaluations, employee attitude surveys

Observational Assessments

•Someone assessment a person’s performance, knowledge, skills, and abilities

180 and 360 Assessments

•Groups of related individuals are asked questions to assess a person

•Leverages on the wisdom of a crowd to help individual development

Assess Content and Contributors

•People assess value of learning content

•Expert locator and acknowledgement

Types of Assessments


Learning

Assessments through the Learning Process

Evidence Accumulated

Windows CDPrepare For Learning

Windows CDIn/As Learning Assessments

Windows CDOf-Learning Assessments

Mentoring Instructor Collaborative & E-learning Work Place


Learning

Assessments through the Learning Process




Windows CDOf Learning Assessments

Windows CDJob Task Analysis Windows CDCourse Evaluations

Windows CDNeeds Analysis Mentoring Instructor Collaborative & E-learning Work Place

Windows CDSlow Forgetting Curve


Learning

Pre-Learning Assessments



Purpose:

• Create intrigue • Route to course • Inform instructors • Establish benchmarks





Learning

In-Learning Assessments



Purpose:

• Encourages focus on learning • Provide instant feedback to instructor • Stimulate and strengthen memory • Correct misconceptions





Learning

Post-Learning Assessments



Purpose:

• Was learning environment okay? • Knowledge transfer occurred? • Reduce Forgetting Curve • Certifications





Criterion referenced vs. Norm referenced assessment

Norm Referenced What is the participant’s knowledge …compared to others in a group?

•Typically used for standardized college placement tests like the SAT, ‘competition’ tests (e.g. advancement test for limited number of openings)

•Pass/Fail can only be determined after the assessment

Criterion Referenced What is the participant’s knowledge …compared to a benchmark?

•Typically used for certification tests, licensing exams, end of course exams, etc.

•Typically Pass/Fail score is determined before the assessment

Compliance oriented assessments are generally

criterion referenced


Reliable: dependable, repeatable, consistent

Valid: measures the specific knowledge and skills that were intended to be measured

Reliability and validity

Slide 14

Figure 1: Reliable but not Valid

Figure 2: Not reliable, not Valid

Figure 3: Reliable and Valid



Slide 15


Regulators encourage them

Demonstrate commitment

Find problems early Document

understanding after training

Increase knowledge / reduce forgetting

Reduce cost & time for compliance training

Reduce human error

7 reasons why assessments mitigate compliance risk

Slide 16


Reason #1 : Regulators encourage them

Slide 17


US FDIC: “Once personnel have been trained on a particular

subject, a compliance officer should periodically assess employees on their knowledge and comprehension of the subject matter”

WHO good manufacturing principles

“Continuing training should also be given, and its practical effectiveness periodically assessed.”

US Dept of Health & Human Services

“Contractors should consider using tests or other mechanisms to determine the trainees’ comprehension of the training concepts presented.”

Three of many, many, many examples

Slide 18


Reason #2 : Demonstrate commitment

Slide 19


Because employees follow their leaders Charles Jennings (August 2012)

“… common thread that runs through almost all high-profile compliance catastrophes. It is that the top-tier executives and middle managers in the organisations simply don’t model the behaviours that will lead to a culture of compliance” http://charles-jennings.blogspot.co.uk/2012/08/compliance-training-does-it-really-work.html

Why is it important to demonstrate top level commitment?

http://charles-jennings.blogspot.co.uk/2012/08/compliance-training-does-it-really-work.html
















If you require all employees and managers to Take assessments every 6 or 12 months

Remedial training and retake assessment if fail

Assessments seen as fair

Major stride in demonstrating commitment

How do assessments demonstrate commitment?

Slide 21


“Prosecutors also should determine whether the corporation's employees are adequately informed about the compliance program and are convinced of the corporation's commitment to it. This will enable the prosecutor to make an informed decision as to whether the corporation has adopted and implemented a truly effective compliance program”

US Department of Justice

Slide 22


Reason #3 : Find problems early

Slide 23


Look at topic and assessment scores By department

By geography

Over time

Look at wrong answers

Look at test failures

Assessments touch every employee: a unique tool

How tests/quizzes/exams can find problems early


Surveys identify issues in culture and ethics

Most employees want to help. From employee of bank involved in 2012 LIBOR scandal: “Always happy to help, leave it with me, Sir”.

UK Ministry of Justice Bribery Act Guidelines “Staff surveys, questionnaires and feedback from training can also provide an important source of information on effectiveness and a means by which employees and other associated persons can inform continuing improvement of anti-bribery policies”.

How surveys can find problems early

Slide 25

http://www.cftc.gov/ucm/groups/public/@lrenforcementactions/documents/legalpleading/enfbarclaysorder062712.pdf

http://www.justice.gov.uk/downloads/legislation/bribery-act-2010-guidance.pdf


Reason #4 : Document understanding after training

Slide 26


Because regulators require it common regulatory requirement, e.g. OECD recommends on

ethics/compliance programme:

measures designed to ensure periodic communication, and documented training for all levels of the company

Because makes business sense Trained employees more effective

“If you can not measure it, you can not improve it.“

Why document training for compliance?


Take attendance

Get employee signatures

Test to check understanding

Common Methods of Documenting Training

Slide 28


How to do Trainer records attendance or takes photo of attendees

Record online attendees

Pros Easy to do

Measures who turns up

Cons Did they stay the whole time?

Did they pay attention?

Did they get it?

Take attendance

Slide 29


How to do Get employees to sign a form confirming

that they attended training AND that they understand the rules

Pros Gets employee confirmation

Employee committed to follow rules

Cons People may sign but not take seriously

People may sign without attending or understanding

Get employee signatures

Slide 30


Not where safety is involved, according to 2005 United States Appeal Court: “merely having an individual sign a form acknowledging his responsibility to read the safety manual is insufficient to insure that the detailed instructions contained therein have actually been communicated.”

Complete General Const. Co. v. OSHRC, 2005 U.S. App. LEXIS 5197 (6th

Cir.), March 29, 2005

Does getting an employee to sign give legal protection?

Slide 31


How to do: After training, test employees People who fail test undergo

remedial training

Pros Genuinely check understanding Allows improvement of weaknesses Feedback to improve training

Cons More effort You have to follow through & take

action on failures

Test to check understanding

Slide 32


Reason #5 : Increase knowledge / reduce forgetting

Slide 33


Compliance Training

Slide 34

Receive information

Working memory

Long term memory


Cue

• Question

• Other stimulus

Search • Retrieve from long

term memory

Act • Act or respond

Research shows retrieval practice helps future retrieval

Successful retrieval modifies memory making it more retrievable in future

Training useful if can retrieve

Slide 35


120 students divided into groups randomly

Given reading comprehension passages used in TOEFL

Some students Study for 14 mins

Some students Study for 7 mins

Recall material for 7 mins

Compared results after 5 mins and 1 week

Roediger & Karpicke experiment 2006

Slide 36


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

After 5 mins After 1 week

Restudy

Quiz

How much learners retained

Slide 37


Learning in the Actual World They don’t get it All!

Slide 38


Learning Curve

Slide 39

Ouch! The Forgetting Curve

Time

Kn

ow

led

ge/S

kills

Lear

nin

g

Exp

erie

nce

Lea

rnin

g

Exp

erie

nce

Lea

rnin

g

Exp

erie

nce

Qu

esti

on

s Q

ues

tio

ns

Purpose: • Content repetition • Memory retrieval practice • Strengthens memory recall

Purpose: • Content repetition • Memory retrieval practice • Strengthens memory recall


In the Actual World… we forget some of it!

Slide 40


Slow the Forgetting Curve

Slide 41

Time

Kn

ow

led

ge/S

kills

Lear

nin

g

Exp

erie

nce

Lea

rnin

g

Exp

erie

nce

Lea

rnin

g

Exp

erie

nce

Qu

esti

on

s Q

ues

tio

ns

Qu

esti

on

s Q

ues

tio

ns

Purpose: • Memory retrieval practice • Strengthens memory recall • Reduce Forgetting Curve

Purpose: • Memory retrieval practice • Strengthens memory recall • Reduce Forgetting Curve


Reason #6 : Reduce cost & time for compliance training

Slide 42


Take test If pass, skip

training Otherwise do training

Testing out of compliance training

Slide 43


Management Report Example

A) Number of people 1,000

B) Time for training if no testing (2 hours each) 2,000 hours

C) Time to take test (20 mins / test) 333 hours

D) 25% pass test and skip training

E) 75% do compliance training 1,500 hours

F) Hours of training avoided (B – C - E) 177 hours

G) Average cost / hour $100

H) Money saved (F * G) $17,700

Saving money by “testing out”

Slide 44


Many regulators permit “testing out” to reduce compliance training ABA Bank Compliance magazine article 2009 interviewed

some US regulators

OTS: Testing out acceptable

OCC: Agree, providing testing program well structured. May be some exceptions

FDIC : Agree

Check with your regulator

Will the regulator allow?

http://www.aba.com/Products/bankcompliance/Documents/8bdcd6492072495e84d8322ce10cd0e9BCJF2009Training.pdf


Reason #7 : Reduce human error

Slide 46


22%

23%

29%

15%

6% 5%

Root causes of human error (data from UK MHRA 2011, graph by Questionmark)

Process/procedure incorrect Procedural steps omitted Concentration error

Training misunderstood Training missing Poor communication/rushing


Assessments can fix Training misunderstood (15%) Training missing or out of date

(6%) Procedural steps omitted

(23%) Process or procedure incorrect

(22%)

Other causes of error Poor communication / rushing

(5%) Concentration error (29%)

Assessments

Slide 48

22%

23%

29%

15%

6% 5%

Process/procedure incorrect

Procedural steps omitted

Concentration error

Training misunderstood

Training missing

Poor communication/rushing


Regulators encourage them

Demonstrate commitment

Find problems early Document

understanding after training

Increase knowledge / reduce forgetting

Reduce cost & time for compliance training

Reduce human error

Summary on why to use assessments

Slide 49


Good practice in using assessments

Slide 50


Create scenario questions Assess application not just knowledge


Evaluation

Synthesis

Analysis

Application

Comprehension

Knowledge

Questions better if use application / comprehension

NOT just knowledge

Blooms Taxonomy


What does a yellow traffic light mean? Stop

Go

Caution

If you are driving toward an intersection and the light turns from yellow to red, what should you do? Speed up and cross the

intersection

Stop suddenly

Stop gradually

Example

Slide 53


Use feedback at the topic level It is actionable

Slide 54


What to do Divide assessment by topic areas

Give individual feedback

Topic scores

Resources or advice to improve weak topics

Analyze at department or organization level

Topic feedback is actionable Direct participant to specific resources to improve learning

Direct compliance to areas knowledge is weak

How topic feedback can help


Feedback based topics of questions

Slide 56

Corrective feedback points participant to appropriate learning

resources

“Topics” of questions

answered in assessment

Score on topic


“participant can immediately see where they need further facilitation”

“We have done extremely well in terms of increasing our overall pass mark … by using topic feedback”

Real life financial services compliance

Slide 57


Consider observational assessments Measure skills and behavior

Slide 58


Traditional assessment

• Participant is presented with questions for direct response

Observational assessment

• Observer is presented with questions to ‘rate’ participant to measure knowledge, skills and abilities

Observational assessment vs. ‘traditional’ quiz, test or exam

Slide 59


Preparation Performing the

task Wrap Up

Observer typically rates participant in 3 areas

Smartphones/tablets make more practical


Real life example of observational assessments for learning & compliance

$10 billion medical equipment manufacturer Company certifies employees by administering

observational assessments Assessments are key to both learning and compliance initiatives

Observational assessment process: Sales rep is trained on a particular product Field trainer

Observes rep demonstrate the product’s use

Using an iPad, completes an observational assessment

Rates performance on a 3-point scale: “Performed without Prompting”

“Performed with Prompting”

“Unable to Perform.”


Use item analysis Pick out the bad questions

Slide 62


Difficulty index: P value = % answered correctly Number from 0.0 to 1.0

Too high means too easy

Too low means too hard or perhaps confusing

Correlation/discrimination : correlation between item score and test score < 0.2 low discrimination, review item

> 0.3 good

Item analysis key numbers


Slide 64


Set a pass score

Slide 65


A test with a pass score of 70%



It depends on how hard the questions are

Which is the best test for compliance?


Think about pass or “cut” scores

Good compliance practice Not so good practice

Set pass or cut score to be minimal acceptable competence

Passing test demonstrates competence

Someone who fails does not

Pick a number out of a hat to use that as the pass score

Slide 67


Angoff method: What is the % chance that a borderline candidate will get question right?

How it works Poll SMEs

Consider marginal candidates and the probability of getting specific questions right (0-100%)

Make a mathematical calculation to determine cut score

Angoff Method

Slide 68


Angoff Method Example What is the % chance that a borderline candidate will get question right?

SME A SME B SME C Total

Q1 75% 75% 75% 75.00%

Q2 70% 80% 80% 76.67%

Q3 65% 75% 70% 70.00%

Q4 60% 85% 90% 78.33%

Q5 80% 80% 85% 81.67%

Q6 80% 80% 80% 80.00%

Q7 75% 80% 75% 76.67%

Q8 65% 90% 65% 73.33%

Q….. 75% 80% 75% 76.67%

Q50 65% 85% 65% 71.67%

Totals 71% 81% 76% 76%

Slide 69


One of our customers in the US military summed it up this way… this method is:

Defensible

Easy to use

Requires only minimal training to implement

Widely accepted

Why use the Angoff Method?

Slide 70


Have participants agree a code of conduct Help them be honest

Slide 71


Fraud needs: Motivation

Opportunity

Rationalization

Fraud triangle

Slide 72


Motivation depends on stakes

Slide 73

Higher Stakes

High

Medium

Low

Medium Stakes

Low Stakes

What’s at Stake?

Life and Limb

Promotion & Jobs & Legal Concern

Educational Exams

Tests

Elearning & Surveys


Impersonation Content Theft Cheating

Opportunity

Slide 74

Monitoring/proctoring helpful, e.g. having manager supervise employees


Reduce rationalization to cheat Make the program fair

Communicate widely that it is fair

A code of conduct (honor code) can help

Idea borrowed from universities

Employees commit at start of assessment to be honest

Rationalization

Slide 75


1. Use scenario questions – test above knowledge

2. Use topic feedback

3. Consider observational assessments

4. Use item analysis

5. Set a pass score

6. Use a code of conduct

Summary on good practice

Slide 76


Excellent book on assessment Criterion-Referenced Test

Development

by Shrock and Coscarelli

More good practice

Slide 77


Questionmark White Papers

www.questionmark.com/go/whitepapers

More good practice

Slide 78

http://www.questionmark.com/go/whitepapers


What questions do you have?

Slide 79


Thank you!

John Kleeman, Chairman, Questionmark

Brian McNamara, Marketing Director, Questionmark

Slide 80

7 ways that online assessments help ensure compliance

Education

questionmark brian mcnamara

questionmark survey

founder of questionmark

term assessment slide

compliance related assessment

online assessments help

assessment management

assessments qa agenda