item writing for cmas item bank

Item Writing for CMAS Item Bank

April 23, 2004American Association of Medical Audit Specialists

10th National ConferenceNashville, TN

Marilyn Balcita RN CMASCMAS Council

2Marilyn Balcita CMAS Council

Do you remember your CMAS Exams?

• Four Parts– Professional Standards– Audit Process and Methodology– Audit Skill– Medical Audit Environment

• 200 mostly multiple choice questions• Few matching type questions• Collection of questions from members

?


Before and After• Not “officially” linked to Job Practice

Analysis Studies (Role delineation study)

• Item Writing principles not formally followed

• Need more contributions from Subject Matter Experts (SME) representing the broad spectrum of medical audit

• For future CMAS Item Banking • To fulfill NOCA requirements


How are questions created ????

Benjamin Bloom's Taxonomy of the Cognitive Domain (1956)

Classification of Educational Goals and Objectives

SynthesisEvaluation

AnalysisApplication

ComprehensionKnowledge

*Adapted from: Bloom, B.S. (Ed.) (1956) Taxonomy of educational objectives: The classification of educational goals: Handbook I, cognitive domain. New York ; Toronto: Longmans, Green.


Taxonomy for Learning, Teaching, and Assessing*

A Revision of Bloom's Taxonomy of the Cognitive Domain

RememberUnderstand

ApplyAnalyzeEvaluateCreate

*Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. New York: Addison Wesley Longman.


Bloom's Taxonomy of the Cognitive Domain

RememberKeywords: list,

define, tell, describe,

identify, show, label, collect, examine, tabulate, quote, name, who, when, where

UnderstandKeywords:

summarize, describe, interpret, contrast, predict, associate, distinguish, estimate, differentiate, discuss, extend



ApplyKeywords: apply,

demonstrate, calculate,

complete, illustrate, show, solve, examine, modify, relate, change, classify, experiment,

discover

AnalyzeKeywords:

analyze, separate, order, explain, connect, classify, arrange, divide, compare, select, explain, infer



EvaluateKeywords: categorize, combine, compile, compose, create, devise, design, explain, generate, modify, organize, plan, rearrange, reconstruct, relate, reorganize, revise, rewrite, summarize, tell, write

CreateKeywords: appraise, compare, conclude, contrast, criticize, critique, defend, describe, discriminate, evaluate, explain, interpret, justify, relate, summarize, support


What do we need for the CMAS Exams???

RememberUnderstand

ApplyAnalyzeEvaluateCreate

RecallHave

Enough..

MoreMore

More …


CMAS Item Development Process

Define Medical Audit Functions

(JTA)

Develop Med. Audit Test Blueprint

Develop, collect and

validateitems

Item Bank storing

Randomly select items for testing

Refine content

ItemReady

for Testing


The Basis of the CMAS Exams

• Job Practice Analysis (Role Delineation Study) Constructed from ….. Job descriptions AAMAS membership information – bulletins, informal

communication, networking Onsite and telephonic interviews SMEs – Medical Auditors, Employers, Consultants,

Academicians Healthcare delivery reports Official books and publications, journals, and other

references


Job Task Analysis Survey

• Initial JTA instrument design developed and tested

• Beta testing by AAMAS and non AAMAS volunteers

• Focused on CMAS respondents• One state chapter volunteered• Tabulated and analyzed• Analysis used for the first time in the 2004

CMAS Examination• Demographic information included to gauge

AAMAS general membership make-up


• Measures“Criticality: how important or critical

it is" to the overall performance of your job

Frequency: "its frequency" to the overall performance of your job

• Ideally, should be measured every five years

Job Task Analysis Survey(Role Delineation Study)


Job Testing Analysis SurveyInitial Result

10%

40%

30%

20%

Professional Standards And Audit Behavior

Medical Audit Process and Methodology

Audit Skill

Medical Audit Environment


Medical Audit Process and

Methodology (40%)• Investigate and verify

charges against medical record documentation

• Audit Process, Work Flow, and Audit Findings

• Other Relevant Medical Audit Responsibilities

• Quality Improvement Activities, Education and Training

• Compliance and Special Investigations

• Contracts and Negotiations

• Denial and Appeals Management

• Health Information Management (Medical Records)

• Health Informatics and TechnologySee

Detailed List


Medical Audit Environment (30%)

• HIPAA• Medicare/Medicaid

Policies• National Audit

Guidelines• OIG Compliance

Guidance/Work Plan • JCAHO

• Medicare Integrity Program

• GAAP/GAAS Principles • CIA-AICPA Audit Rules• NCQA/Hedis• DHHS Audit Process• US Sentencing Rules• Health Insurance

reimbursement methodology

Application of laws, guidance, standards, guidelines and other accrediting bodies


Audit Skill (20%)• Interaction and Communication• Leadership and Managerial • Medical Audit Specific Knowledge and Skill

Set RequiredClinical Judgment and Nursing

ProcessAccounting/FinanceProblem Solving StatisticsNegotiationQuantitative AnalysisResearchProject Management


Professional Standards And Audit Behavior

(10%)• Principles of objectivity • Principles of independence • Internal control • Goal setting, strategic planning, and

mission/vision development • Code or standards of conduct • Conflict of Interest• Patient access and confidentiality • Enforcement, expectations and systems of

accountability


Why Multiple Choice??PROS

• Quick and easy to score

• Measures all levels of candidates’ ability

• Allows objective scoring

• Can be analyzed for effectiveness

• Covers a broad range of knowledge in limited time

CONS• Challenge in measuring

individual differences in skill

• Hard to develop good items

• Tendency to measure simple recall


National Hospital Billing Audit Guidelines require that:

A. Auditors must be certifiedB. Hospitals must charge audit fees in

excess of $275C. Auditors must conduct themselves in a

professional manner, adhere to ethical standards, confidentiality requirements, and objectivity

D. All of the above

STEM

OPTIONS

ITEM

What is the correct answer?What are the distractors? Is this a good item ?

What’s wrong with it?

Reference: National Health Care Billing Audit Guidelines. AAMAS 2003 Directory


CMAS Item Writing Guiding Principles

Ask yourself: Can a medical auditor with at least 2 years of basic medical audit experience answer this question correctly?

Measure knowledge, skill, abilities (KSA) that is generally known in the medical audit field = JTA

Present only ONE central idea, and ONE correct answer

Provide 4 options to support the stemAvoid constructing items for the “street-smart” test

takersCheck for grammars all the timeNo questions with connotations of race, creed,

religion, or ethnic/national background


• Present stem in a complete sentence, even if it is in a question. Bad Example: A medical audit specialist is …

• Ideally – no more than 50 words• Try to present stems positively. Avoid using: All (or which)

of the following except, which of the following is NOT or Never

• Avoid trick questions, or measure trivial things• Good item writers begin with: Who, What, Where, How, and

When “Which” should be followed by a noun “What” should be followed by a verb

• Use “should” instead of “would/could”• Avoid using always, all, and never• No double negatives – negative stem, and negative options

CMAS Item Writing Guiding PrinciplesThe Stem


• No! No! --- All of the above, None of the above, No slang, No silly options

• Correct answer must be 100% correct• Distracters MUST be the wrong answer• If numeric, arrange in logical sequence• Equally balance the options – long/short, or

logical choice of options. Ex: 3 ICD codes, one CPT ---- don’t make it obvious …

• Do not overuse Partial K type items – answers with (A) 1 and 2; (B) 3 and 4 etc.

CMAS Item Writing Guiding Principles

The Options


• Include author, title/edition, publisher, publication, date, (month for journals), and page numbers

• Official, widely known and readily available references only• References must clearly support the correct answer• For practical type questions i.e., mathematical calculation -

include the steps in calculation• DO NOT copy the exact wording from the reference source.

Exception: Legal quotes, federal regs, contracts• Use official ONLINE reference sources ONLY – CMS, DHHS,

OIG, other government sites. Cite the website in html if possible

Examples: http://www.gao.govaud/d02870g.pdf/ http://www.cms.hhs.gov/manuals/10_hospital/ho200.asp#_1_16

Baker, Judith, (2000) Health Care Finance: Basic Tools for Nonfinancial Manager. Aspen Publishers Inc. pg 165-168

Abbey, Duane. “Designing and Maintaining an Effective Chargemaster." Healthcare Financial Management, (2001 March): 50-55.

CMAS Item Writing Guiding PrinciplesReferences

http://www.gao.govaud/d02870g.pdf/

http://www.cms.hhs.gov/manuals/10_hospital/ho200.asp#_1_16


CMAS Subject Matter Expert Technical Advisory Group Team

SME-TAG• Like that name????• Who are we …• What is it for me ??

– Ability to share your knowledge– Prestige as an item developer– Earn points applicable for

recertification and membership fees, discounts to AAMAS marketplace, etc.

– Great learning experience• What do I need to do??• ……………………. Care to

join??????The TAG Team


CMAS SME TAG Team• CMAS certified fellows only• Well rounded experience in medical audit• Experience in test item development

processes and review• Talent in writing and editing• Sound knowledge of psychometric analysis,

validation, and statistics • Agree to sign CMAS certification security

agreement• Committed and willing to learn• Have the time …..• Have the “tools” to work from – PC at a secured place, appropriate software, compatible system

C’mon down!!


Now let’s take a look at this item one more

timeNational Hospital Billing Audit Guidelines require that:

A. Auditors must be certifiedB. Hospitals must charge audit fees in excess of $275C. Auditors must conduct themselves in a

professional manner, adhere to ethical standards, confidentiality requirements, and objectivity

D. All of the above


Ready to have fun???Let’s play ….

• Find your group --- introduce yourself• Choose a team lead, recorder, reporter, and

timekeeper• Choose one item to develop (15 minutes)

– Identify what is being measured: Application vs. Analysis

– Match with Test Blueprint (measure med. audit function)

– Follow checklist, “guiding principles”– Validate with official resource

• Each group: Be ready to present (5 min. ) and critique (5 min.)

• Good Luck!!!


Thanks for coming !!!Hope to hear from you soon !!!

[email protected] – 686 -0180 (East Coast time)

mailto:[email protected]

item writing for cmas item bank

Documents