cets 2011, ken phillips, handout for construct the ultimate multiple-choice test! stop measuring...

The “How-To” Measurement & Evaluation Series

Increasedproductivity

through skill building

Construct theUltimate MultipleChoice Test!Stop MeasuringRecall and StartMeasuringApplication

Chicago e-Learning & Technology ShowcaseAugust 16, 2011

Today’s Presentation

About Your Presenter — KEN PHILLIPS

Ken Phillips is founder and CEO of Phillips Associates, aconsulting company with expertise in performancemanagement, measurement and evaluation of learning andsales performance. Ken has more than 25 years experiencedesigning learning instruments and assessments and hasauthored more than a dozen published instruments. Healso regularly speaks to American Society for Training andDevelopment (ASTD) groups and university classes. Forthe past four years, he has spoken at the ASTDInternational Conference on topics related to measurement

and evaluation of learning.

Prior to pursuing a Ph.D. in the combined fields of organizational behavior andeducational administration at Northwestern University, Ken held managementpositions with two colleges and two national corporations. In addition, he haswritten articles that have appeared in T+D magazine, Training Today andHR.com.

Currently Ken is serving his seventh year on the board of directors of theChicagoland Chapter of the American Society for Training and Development(CCASTD), and was chapter president in 2008. He earned the CertifiedProfessional in Learning and Performance (CPLP) credential from nationalASTD in 2006 as a pilot pioneer and was recertified in 2009. Ken is alsocertified in Measuring and Evaluating Learning and Test Design and Deliveryby ASTD and in the Kirkpartick Four LevelsTM Evaluation. In January 2009,he also was appointed to serve a three-year term (2009-2011) as a member ofthe ASTD National Advisors for Chapters (NAC) committee.

Ken can be reached at (847) 231-6068 or [email protected].

Ken is a frequent presenter atASTD chapter meetings and otherindustry events. Here’s a preview ofwhere he’ll be speaking next.

16 Chicago e-Learning &Technology Showcase

Topic: Construct theUltimate Multiple-ChoiceTest! Stop Measuring Recalland Start MeasuringApplication

14 Training & PerformanceManagement PDN(Chicago)Meeting

Topic: Learn 12 SurprisingTechniques for ObtainingPowerful Data from Level 1Evaluations

Copyright ©2011 by Phillips Associates All rights reserved. Printed in the United States of America. Except as permitted under the United States Copyright Act of 1976, no partof the publication may be reproduced or distributed in an form or by any means, or stored in a database or retrieval system, without theprior written permission from Phillips Associates

Do you feel like you know how

much participants truly learn from

your elearning programs? If not,

you’re not alone. According to a

2009 ASTD research study, 81% of

organizations measure participants’

learning (Level 2 evaluation), yet

only 55% of these organizations

view the data collected to be of real

value. The problem is that the

validity of our results is only as

good as the quality of our test

questions. Often, we develop

questions that are too simple, that

are overly difficult, or that test

knowledge and not application. In

this session, you will learn the art

and science of Level 2 multiple

choice test creation. Together, we

will analyze a sample test and learn

how to identify and correct common

mistakes. Whether you are new to

elearning or an advanced

professional, you will walk away

with skills to help you improve your

online Level 2 multiple-choice tests.

“Great speaker and subject matter.

Ken’s expertise really made the event

worthwhile.” –South Central Wisconsin ASTD

Chapter Member

“This was very helpful. Obviously

many people write tests for adults that

can be flawed. This session provided

good guidelines for writing valid tests.” –South Central Wisconsin ASTD

Chapter Member

1 www.phillipsassociates.com

Sample Level 2 Multiple Choice Knowledge Test

1.

Interpersonal communication is often rendered ineffective due to the presence of communication barriers. What are the three categories of communication barriers?

A. Environmental, Interpersonal and Intrapersonal

B. Environmental, Interpersonal and Cultural

C. Structural, Interpersonal and Cultural

D. Organizational, Intrapersonal and Structural

Correct Answer: A. Environmental, Interpersonal and Intrapersonal

2.

_________________________ are used to report specific, observable actions or details in an objective, non-evaluative manner.

A. Behavior descriptions

B. Open probes

C. Evaluative statements

D. Direct statements

Correct Answer: A. Behavior descriptions

3.

What are questions that begin with words such as is, are, could, would, have and shall known as?

A. Clarifying questions

B. Open-ended questions

C. Close-ended questions

D. Bridging questions

Correct Answer: C. Close-ended questions

4.

A Collaborative Communication Style is also known as a?

A. Win/Win communication style

B. Low concern for self and high concern for other communication style

C. Engaging and outgoing communication style

D. Intermediate concern for self and high concern for other communication style

Correct Answer: A. Win/Win communication style


5.

What are “You” messages?

A. Non-ownership statements that convey feelings, but do not actually describe the feelings.

B. Statements that contain the pronoun “you” two or more times.

C. Statements that point out what another person has done or was supposed to do.

D. Statements that describe behavior and begin with the word “You.”

Correct Answer: A. Non-ownership statements that convey feelings, but do not actually describe the feelings.

6.

A benefit of being an effective listener is?

A. Facilitates getting to know another person better

B. Increases the likelihood of having a mutual exchange

C. Helps another person to get to know himself or herself better

D. All the above

Correct Answer: D. All the above

7.

What are Hidden Agenda Questions?

A. An indirect or “pseudo” form of communication.

B. A question used to clarify what someone has said.

C. A question used to direct a conversation in a different direction.

D. A technique for getting another person to talk.

Correct Answer: A. An indirect or “pseudo” form of communication.

8.

What are questions that begin with words such as what, where, when, how and why known as?

A. Bridging questions

B. Clarifying questions

C. Leading questions

D. Open-ended questions

Correct Answer: D. Open-ended questions


9.

A perception check is a?

A. Technique for moving a conversation from the objective, where the focus is on facts, to the subjective, where the focus is on feelings.

B. Technique for pulling together the important parts of one long conversation or a series of conversations.

C. Technique for helping someone to express his or her real self.

D. Technique for conveying interest in what another person has said.

Correct Answer: A. Technique for moving a conversation from the objective, where the focus is on facts, to the subjective, where the focus is on feelings.

10.

Effective listening is defined as?

A. Attempting to understand the person speaking from his or her perspective and not your own.

B. Encouraging another person to talk.

C. Consolidating the important parts of a discussion.

D. Creating a win-win communication situation.

Correct Answer: A. Attempting to understand the person speaking from his or her perspective and not your own.

11.

A proven technique for improving interpersonal communication effectiveness is?

A. Replace generalizations and evaluative statements with behavior descriptions.

B. Seek first to understand before attempting to be understood.

C. Replace hidden agenda or pseudo questions with direct statements.

D. All the above.

Correct Answer: D. All the above.

12.

Which is not one of the four communication styles?

A. Intuitive

B. Avoiding

C. Aggressive

D. Collaborative

Correct Answer: A. Intuitive


13.

Which is not a use for paraphrasing?

A. Pulling together the important parts of one long conversation or a series of conversations.

B. Conveying interest in what another person has to say.

C. Testing for understanding when accuracy and clarity of communication is important.

D. Helping to achieve mutual understanding when strong feelings are present.

Correct Answer: A. Pulling together the important parts of one long conversation or a series of conversations.

14.

What communication technique is used to consolidate what has been previously discussed and to move a conversation from one topic to another?

A. Arguing

B. Interrupting

C. Summarizing

D. Initiating

Correct Answer: C. Summarizing

15.

____________________ are non-verbal cues such as head nods and brief verbal comments such as “right”, “okay” and “umhum”.

A. Minimal encouragements

B. Communication fillers

C. Confirmations

D. Acknowledgements

Correct Answer: A. Minimal encouragements

16.

According to Robert Merabian, what percentage of interpersonal communication is transmitted non-verbally through tone of voice and body language?

A. 90 percent.

B. 60 percent

C. 75 percent

D. 50 percent

Correct Answer: A. 90 percent


Level 2 Multiple Choice Test Question Job Aid

% of live classroom vs. technology based programs evaluated at Levels 1-5 *

Live Classroom Technology Based

Level 1 81% ______%

Level 2 50% ______%

Level 3 25% ______%

Level 4 14% ______%

Level 5 6% ______%

Level 2 evaluation facts *

• 81% of organizations evaluate some learning programs at Level 2

• 55% of organizations view data collected as having high or very high value

Multiple Choice Test Item Pros **

• Versatility in measuring all levels of cognitive ability

• Highly reliable

• Easy to score

• Minimizes guessing factor

Multiple Choice Test Item Cons **

• Difficult and time consuming to create

• Lead to creation of test items that measure recall of simple facts


Multiple choice test creation tips **

• Avoid use of “All the above” “None of the above”

• If administering electronically, use diminishing response technique

• Write stem as question – response can finish question but don’t use fill-in-the-blank

• Test for understanding not just knowledge

• Keep response options equivalent in length and writing style

• Include central idea and most of phrasing in stem

• Question should not reveal correct answer

• Be sure there is only one correct response

• Spread correct answers across A – D

• All response choices must be plausible

• Eliminate excessive verbage or irrelevant information from stem

• Write questions same way material taught – avoid “null” questions

* ASTD research study, “The Value of Evaluation: Making Training Evaluations More Effective,” 2009. ** Adapted from ASTD Test Design and Delivery Certificate program participant workbook, 2006.

Training Resources Available from Phillips Associates

No Pain—All Gain! PerformanceAppraisal Meeting Inventory Help managers conduct performanceappraisal meetings that improveemployee performance and areviewed as valuable and worthwhileusing the Performance AppraisalMeeting Inventory. The instrumentpinpoints a manager’s strengths and development needsin six skill areas and outlines a simple six-step processfor conducting effective performance appraisal meetings.

Getting On the Same Page: Objective SettingHandbooks Easy to read booklets (one for managers and one fordirect reports) filled with proven tips and techniques ontransforming the objective setting process from littlemore than an annual ritual into a valuable, worthwhileexperience for everyone involved. Includes a set ofworksheets the reader can use to prepare a set ofobjectives.

Acting the Part: Performance Management Role Plays Ready-to-use role play scenarios designed to helpmanagers, supervisors and team leaders apply andpractice the skills needed for coaching effectively andconducting productive performance appraisals.Includes half-day workshop design and PowerPoint slides.

Coaching to Win: Coaching Meeting Inventory The Coaching Meeting Inventory assesses a manager’sstrengths and development needs in eight skill areas andoutlines a logical step-by-step process for conductingeffective coaching discussions.

Superior Service Starts Here: Customer Service SkillsInventory (CSSI)CSSI consists of 32 statements divided into four keys ofproviding superior customer service: Developing arelationship of Interpersonal Trust, Listening, ProblemSolving and Professionalism. Based on how they answerthe statements, customer service reps gain valuableinsight into their greatest strengths and most pressingdevelopment needs in delivering superior customerservice.

Taking Stock for Better Communication: Communication Style Inventory (CSI)CSI consists of 24 sets of paired statements describingthe four most commonly used communication styles:Aggressive, Avoiding, Accommodating and Collaborative.By choosing the statement that most nearly describeshow they would communicate in a particular situation,respondents generate a profile of their use of the fourcommunication styles.

Insightful Learning Products Turnkey training resources for organizations and individuals Phillips Associates offers an array of learning instruments, multi-rater assessment systems, handbooks (job aids),manuals and games designed to develop specific skills in the areas of performance management and salesperformance. Some of our best-sellers include:

For descriptions of all Phillips Associates Learning Products, visit www.phillipsassociates.com

PERFORMANCE MANAGEMENTPRODUCTS• Performance Appraisal Meeting

Inventory

• Managing Your Own PerformanceAppraisal Inventory

• Coaching Meeting Inventory

• Keys to Effective Coaching

• Objective Setting Inventory

• Interpersonal Feedback Inventory

• Objective Setting Games

• Performance Management RolePlays

• Objective Setting Handbooks

• Performance Appraisal Handbooks

SALES PERFORMANCE PRODUCTS• Face-to-Face Sales Skills Inventory

• Cross-Selling Skills Inventory

• Territory Management StyleInventory

• Customer Service Skills Inventory

MULTI-RATER ASSESSMENT PRODUCTS• Performance Management Profile

System

• Selling Skills Profile System

FOUNDATION SKILLS PRODUCTS• Communication Style Inventory

Browse and buy online from the PHILLIPS ASSOCIATES

Training Resources Online Catalogwww.phillipsassociates.com

Save 10% on your entire order

when you mention this code: CELTS811(offer expires 9/16/11)

Performance Management and Sales Performance Programs

In addition to our full offering of learning products,Phillips Associates delivers interactive programs to helpemployees achieve and maintain better on-the-jobperformance. Our highly participative approach teachesmore than just concepts and practices—it builds lastingskills. Some of our most popular programs include:

• Exceeding Expectations: Managing the PerformanceAppraisal Process

• From Basics to Best in Class: Customer Service Skills

• Coaching to Win: Coaching Employee Performance

For a complete list of programs offered, please visit us atwww.phillipsassociates.com/seminars.

CANADA Training & Development Materials ofCanada372 Moonstone road east, PO Box 38Moonstone, ontario L0K 1N0800.283.4964

UNITED KINGDOMManagement Learning Resources, Ltd.PO Box 28, CarmarthenDyfed SA31 1DT Wales, United Kingdom01267-281661

34137 N. Wooded Glen Drive | Grayslake, IL 60030 847.231.6068 | [email protected]

INTERNAT IONAL D ISTR IBUTORS

The “How-To” Measurement & Evaluation SeriesOnly Phillips Associates shows you how!

Presentations (75-90 minute learning programs with audience participation)

• Learn 12 Surprising Techniques for Obtaining Powerful Data from Level1 Evaluations

• Crack the Code of Test Question Design and Take Your Level 2Evaluations to 2.5!

• How to Capture Elusive Level 3 Data: The Secrets of Survey Design

Workshop (1 or 2 day presentation with participant interaction)

• Demonstrate Value and Get Your “Seat at the Table” 51 Tips and BestKept Secrets for Designing Level 1 – 4 Evaluations

For more information contact Ken Phillips at 847.231.6068 or [email protected]

cets 2011, ken phillips, handout for construct the ultimate multiple-choice test! stop measuring...

Business