ardms mentor to item writers trainingwriters help us reach our goals. new item writers receive...

Mentor to Item Writers Training Part One – The Vision

AgendaOverview of the Mentoring Program

Mentoring in EPIC

Mentoring Tips

Mentoring toward Higher-Level Items

Evaluating Items and Mentees

Overview

We need many high-quality items.We need approximately 4,000 new items written each year across all our exams to keep them fresh and current. The value of our credentials depends on it.

The better the original quality of an item, the faster it passes through the review process. This saves time, effort, and money in getting the items onto exams.

High-quality items more accurately assess the potential for high-quality patient care.

Mentors to Item Writers help us reach our goals.New Item Writers receive training on writing items that follow our guidelines.

Their skills are honed and developed by the one-on-one support they receive from a Mentor.

As a Mentor, you have the opportunity to share your knowledge and experience by guiding others to a higher level, leading to assessments that are relevant and valid and predict quality patient care.

Vision for the Mentoring Program

New Item writers

Mentors

ExperiencedWriters

Advanced Item Types (including hot spots, Semi-Interactive

Consoles, PACSims), higher-level items, and hard to write areas

Steps for New Item Writers

Complete training.

Submit items after receiving Mentor’s approval.

Evaluate your mentor.

Write 10 items per 3-month writing project.

Incorporate Mentor’s comments and send item back for additional review.

Become paired with a Mentor.

Send each item to Mentor for feedback.

Steps for Mentors to Item Writers

Complete training.

Become paired with an item writer and contact her/him within a week.

Review items within a week of receiving them.

Encourage your Mentee throughout the writing project.

Evaluate each of your Mentees.

New Item Writer…

…paired with Mentor…

…leads to high-quality patient care.

Summary of Vision

Mentor to Item Writers Training Part Two – Mentoring in EPIC

Mentoring in EPIC

Step 1: You will receive an e-mail.

From: donotreply@strasz.com [mailto:donotreply@strasz.com] Sent: Thursday, October 23, 2014 8:03 PMTo: Your NameSubject: Notification - An item has been moved to your queue

Dear Your Name,

You are receiving this email because Mentee’s Name has sent item 233012 to you for review under project Q2 PE Project. Click on the link below or copy it into your browser to login to the Item Authoring and Review System.

https://ARDMS-PROD-RIW.EPIC-EXAM.COM

The item is available in the Mentor Review queue under the project.

You do not need to reply to this message. The status of the item, along with necessary comments and information, is available at the web address listed above.

Thank you.

From: donotreply@strasz.com [mailto:donotreply@strasz.com] Sent: Thursday, October 23, 2014 8:03 PMTo: Your NameSubject: Notification - An item has been moved to your queue

Dear Your Name,

You are receiving this email because Mentee’s Name has sent item 233012 to you for review under project Q2 PE Project. Click on the link below or copy it into your browser to login to the Item Authoring and Review System.

https://ARDMS-PROD-RIW.EPIC-EXAM.COM

The item is available in the Mentor Review queue under the project.

You do not need to reply to this message. The status of the item, along with necessary comments and information, is available at the web address listed above.

Thank you.

Step 2: Keep track of information.

This is an example of how you might want to keep track.

Keep a log of your Mentees’ names, item numbers, and your comments so that evaluation of your Mentees at the end of the writing project will be easier to do.

You may use whatever system works for you.

Step 3: Log into EPIC.

Click on the writing project that was noted in the e-mail you received.

Check to see if you have more than one screen of projects.

Step 4: Click on the item to review.

When you click the Training Project, it will show the items ready to be reviewed.

This screen shows you have two items to review.

Click on the item number to review the item.

Step 5: Review the item.

Review Stem.

Review Content (classification).

Review Answer Choices.

Scroll all the way down the page to review references and rationales.

Step 6: Add your comments.

Insert your comments here.

Click Add Comment.

You might find it easier to write in Word and cut-and-paste your comments into EPIC.

Save comments in Word for reference when evaluating your Mentee.

Step 7: Click Review Complete.

Click “Remove” if you want to edit your comment. Rewrite it and re-enter it.

When you are finished, click Review Complete.

After clicking Review Complete, you will not be able to retrieve the item until your Mentee sends it back to you.

When mentoring in EPIC…TRACK COMMENTS TO MENTEES YOU MAY ONLY COMMENT ON THE

ITEM, NOT REVISE IT.

Mentor to Item Writers Training Part Three – Mentoring Tips

Mentoring Tips

1. Communicate regularly.

E-mail your Mentees within a week of being paired.

Let your Mentees know in advance if you will not be available for a while.

If you haven’t heard from your Mentee in two weeks, contact her/him.

Review an item within five days of receiving it.

Remind Mentees about a week prior to the end of the writing-project.

2. Encourage Mentees to begin writing.

“Start writing no matter what. The water does not

flow until the faucet is turned on.”

-Louis L’Amour

3. Suggest how to “turn on the faucet.”

1. Select a task on the content outline.

2. Choose an image.

3. Then just start.

Many people find editing is easier than creating.

4. Provide constructive feedback.

Specific

Clear

Objective

Productive

Encouraging

This is not constructive feedback.

Your stem is great, but one of

your distractors is weak.

Be Specific: Define abstract terms.

WHAT DOES “GREAT” MEAN?

• “Stem is focused.”

• “Stem is concise and grammatical.”

• “Stem contains only necessary information.”

• “Stem follows ARDMS guidelines.”

WHAT DOES “WEAK” MEAN?

• “A is not a plausible answer.”

• “A and B are too similar.”

• “The key is obvious because it is the longest answer.”

• “Distractors vary too much in length.”

Specific Your stem is great focused and

follows all ARDMS guidelines,

but distractor A is weak not

plausible.

Be Clear: Avoid ambiguity.

Avoid although, but, however, etc. These words tend to negate the

positive comments that proceeded them.

Put each comment into a separate sentence.

SpecificClear

• Your stem is great focused and

follows all ARDMS guidelines, but .

• Your distractor A is weak not

plausible.

Be Objective: Critique the writing, not the writer.

AVOID THESE WORDS

• “You”

• “You need to…”

• “You should…”

REPLACE WITH THESE WORDS

• “The stem or key or distractor,…”

• “Perhaps try…”

• “A suggestion for…”

SpecificClearObjective

• Your The stem is great focused

and follows all ARDMS guidelines,

but .

• Your Distractor A is weak not

plausible.

Be Productive: Guide toward improvement.

Propose suggestions of resources to reference.

Give direction for improving the stem or options.

Critique only those things that can be changed.

Coach toward higher-level questions.

SpecificClearObjectiveProductive

• The stem is focused and follows all

ARDMS guidelines.

• Distractor A is not plausible.

• Try looking in [Book Title] for another

pathology that might be more plausible.

Be Encouraging: Build a feedback sandwich.

Start with something good.

Finish on a positive note.

Add the “meat” of the message.

Examples of Encouraging Statements

“This item really improved since I last saw it. Good job!”

“You have some great images of [xxx]. We can really use those in the [xxx]

domain.”

“Seems like you have a good grasp on what it takes to write a good item.”

“The quality of your items keep getting better with each one.”

SpecificClearObjectiveProductiveEncouraging

The stem is focused and follows all

ARDMS guidelines. Distractor A is not

plausible. Try looking in [Book Title] for

another pathology that might be more

plausible. You are very close to having an

excellent item!

Mentoring toward Higher-Level Items

Levels of QuestionsQuestions can be written to assess different cognitive levels of learning.

ARDMS and APCA exams focus on Remembering/Understanding and Applying/Analyzing.

Bloom’s Taxonomy

Action Verbs Associated with Each Level

Coach toward application and analysis.

ITEM SENT BY MENTEE MENTOR’S COMMENTS

“The image is clear and the structure of the item is good. It assesses remembering, which is the lowest level on Bloom’s Taxonomy.

Try to write an item that assesses application, a higher cognitive level, by asking for an interpretation of the image or what the driver should do with the information.”

What is this image of a car dashboard showing?

A. The amount of gas in the gas tankB. The amount of air in the tiresC. The amount of oil in the oil tankD. The amount of coolant in the engine

Lower-Level versus Higher-Level Item

REMEMBERING AND UNDERSTANDING APPLICATION AND ANALYSIS

What is this image of a car dashboard showing?

A. The amount of gas in the gas tankB. The amount of air in the tiresC. The amount of oil in the oil tankD. The amount of coolant in the engine

If a driver saw this, what should the next step be?

A. Drive to a gas station and fill upB. Take it to a service station to find out what’s wrongC. No problem, keep drivingD. Buy some antifreeze

Mentor to Item Writers Training Part Four– Evaluation

Evaluating the Items and the Mentees

Evaluate the overall quality of the items.Language and terminology are clear and concrete.

Each item has one stem and only one key and three distractors.

Items contain proper grammar, punctuation, and spelling.

Items use proper nouns (e.g., the patient) rather than names (e.g., Paul) or pronouns (e.g., he, she).

Each item has a reference.

The majority of items require application rather than just recall.

The key is not given away: There are no similar words in the stem and in the key.The key is neither the longest nor the shortest response option.All distractors are plausible response options.

Common wording is placed in the stem.

Each item is independent.

Items are written in the positive voice, avoiding words such as no and not.

Evaluate the content of the items.Items are relevant to the content area for which they are written.

Items focus on critical knowledge required in daily clinical practice.

Evaluate the stem.The question is focused.

The question is succinct.

It is not a fill-in-the-blank question.

It is not a true/false question.

If there is an image attached to the item, the image is referenced in the question.

Evaluate the response options.Distractors are plausible.

Distractors are indisputably incorrect.

Each response option contains only one answer, not a combination such as “A and B.”

Response options do not include excessive use of measurements.

Common (repetitive) wording is in the stem, not in the response options.

There is only one correct response (the key).

The key is not given away because of how it is written.

“None of the above” nor “All of the above” are response options.

Is your Mentee ready to move on without a Mentor?

At the end of the three-month writing project, you will be asked whether your Mentee is ready to write items without mentoring support.

This is when it will be useful to refer to your tracking sheet of names, items, and comments.

How well did the Item Writer follow item-writing guidelines?

Did the items written require the test-taker to demonstrate an appropriate level of knowledge and application?

Should the Item Writer be invited to continue writing?

Is the Item Writer ready to write without a Mentor?

About each of your Mentees, you will be asked…

…and each of your Mentees will be asked about you. Did your Mentor provide feedback within a week of receiving an item from you?

Were your Mentor’s comments specific, clear, objective, productive and encouraging? SCOPE!

Did your Mentor motivate you to continue writing and improving?

Did your items improve after working with your Mentor?

Thank You!The Exam-Development Team