Reverse Multiple-Choice Method (RMCM)

Indu M. AnandMathematics DepartmentUniversity of Massachusetts – Lowell

AMCOA Conference, February 9, 2012

RMCM -- The MethodReverse Multiple-Choice Method:

Presents content for Teaching or Testing

Introduced in Patent No. US 7,033,182

Uses multiple-choice question format

Formalizes reinforcement &

explanationHarnesses power of “wrong”

answers

RMCM -- The Method (contd)Requires examinee to:

(i) examine each answer choice in the context of the query;

(ii) identify key information that makes the answer correct or incorrect; then,

(iii) change the given facts to make an incorrect answer choice the "correct" answer to changed query.

RMCM Method (contd)…Correct answer, no modification

For a “wrong” answer choice Examinee:--

Must examine answer choiceMust understand why answer is

incorrect Must communicate why it is incorrect

More efficient than Open format / essay May be automated for gradingMay be used for most subject areas

How it works – Examinee’s view…Examinee, typically, is asked: To select correct answerThen --select one (or more) wrong answerchange the query so that

selected wrong answer is the correct answer for changed query

Modification criteria may be provided

How it works – RMCM QuestionQuestion is key data structure of

system. Components of the Question object are:

Query – “Narrative” + Call of the question

Answer choices – Multiple , One correct others incorrect

RMCM Questions:May be written from the scratch, orMay be developed from other multiple-

choice questionsStored in persistent databanks of

questions

RMCM Question (contd)

Special features of RMCM question:

Fact Objects (FO’s) = Facts / segments of the query on which the answer turns;

FO = True or False for an answer choice, depending on its relevance to answer choice

Fact Object Values (FV’s) – Values assigned to relevant FO’s for original or modified query.

Structure…

Test Question

Query

AnswerChoice

Body

Call

FactObject FactValue

Value Score

1

1..*

1..*

RMCM Question - StructureAnswer Choices should be closely related

1 Answer choice

FO/FV

2 5

3 Query 4

How it works – Examiner’s view…Select incorrect answer choices

strategically in order to --Emphasize the facts provided in

the questionIdentify “Fact Objects” for each

answer choice; Provide “Fact Values” of the

“True” Fact Objects ( for each answer choice)

Provide score / partial score for each FO for each answer choice

RMCM – Examiner’s view (contd)Examiner must:Define the Objectives for the question

Figure out what concept to assessKnow where and how students miss the

conceptForm answer choices around common errors

Work BackwardsCorrelate facts / segments in query with errors

Determine importance of each FOApportion relative credit for the facts / segments

How it works – System viewThe RMCM system:Assists Examiner with

Editorial support to write or import questions;

Selecting “Fact Objects” for an answer choice with (proprietary) segmentation software;

Maintaining FO/FV databases, and credit generation databases

Generating computer-assisted queries

Assists Examinee with navigation

How it works System view (contd)Compare Examinee’s answers with

Examiner’s data Identifying FO’s for selected wrong answer

choice + Fact Values for (True) FO’s for selected

wrong answer

Calculate score for question based on comparison Identification, or not, of correct answer;

and Identification of FO’s for selected wrong

answer Fact Values for FO’s for the selected wrong

answer

Calculate score for test

How the system works – Logic I

How the system works – Logic II

Example 1What is the area of a rectangle with sides 3 and 4?

A. 5B. 12C. 14D. 16E. 9

Screen mock up for Example 1

Screen mock up continued

Screen mock up continued

Screen mock up continued

Example 2Divide and simplify the expression: (10) 2 =

(A) 8(B) 22(C) 5(D) 25(E) 23 This question is taken from REA’s SAT preparation guidebook (1990)

Example 2 (contd)Correct answer is C.

(A) would be the correct answer for the expression: (10-2);

(B) would be correct for the expression: 2 * ( 5 - 1 );

(D) would be correct for the expression: (10 * 2);

(E) would be correct for the expression:

(10 + 2).

Example 3

Example 3 (contd)What is the current in the 4 Ω resistor while switch S is open?

(A) Is correct if 12 V is changed to 0 V;

(B) Is the correct answer;(C)Is correct if 12 V is changed to

24 V;(D)Is correct if 12 V is changed to

40 V;(E) I correct if 12 V is changed to 60

V.

Example 3 & DiscussionOther variants possible with different

sets of Fact Objects; e.g., Resistors with other resistance values;

or Changes of both resistance and

voltage.One interesting case is where a Fact

Object is “Switch S is open or closed.”In general, FO holds information –

may be numeric or non-numericExamples: Mathematical operation;

Word order in a sentence; Sequence of lab activity

Example 4 – Math word problem*The sophomore class plans to sell T-

shirts with the school’s name on them. The cost of each T-shirt alone is $3.50, and the printing cost of each shirt is $0.75. If the class plans on selling each printed T-shirt for $11, what expression can you use to calculate the class profit for selling n printed T-shirts?

A. 11.00 – (3.50 + 0.75)nB. 11.00n – (3.50 + 0.75)C. 11.00 – 3.50 – 0.75nD. (11.00 – 3.50 – 0.75)n* From 2001 MCAS test for 10th grade

Example 4; non-numeric FOThe sophomore class plans to sell T-

shirts with the school’s name on them. The cost of each T-shirt alone is $3.50, and the printing cost of each shirt is $0.75. If the class plans on selling each printed T-shirt for $11, what expression can you use to calculate the class profit for selling n printed T-shirts?

A. 11.00 – (3.50 + 0.75)nB. 11.00n – (3.50 + 0.75)C. 11.00 – 3.50 – 0.75nD. (11.00 – 3.50 – 0.75)n

Non-numeric FO highlightedThe sophomore class plans to sell T-

shirts with the school’s name on them. The cost of each T-shirt alone is $3.50, and the printing cost of each shirt is $0.75. If the class plans on selling each printed T-shirt for $11, what expression can you use to calculate the class profit for selling n printed T-shirts?

A. 11.00 – (3.50 + 0.75)nB. 11.00n – (3.50 + 0.75)C. 11.00 – 3.50 – 0.75nD. (11.00 – 3.50 – 0.75)n

Other question FO’s highlighted

The sophomore class plans to sell T-shirts with the school’s name on them. The cost of each T-shirt alone is $3.50, and the printing cost of each shirt is $0.75. If the class plans on selling each printed T-shirt for $11, what expression can you use to calculate the class profit for selling n printed T-shirts?

A. 11.00 – (3.50 + 0.75)nB. 11.00n – (3.50 + 0.75)C. 11.00 – 3.50 – 0.75nD. (11.00 – 3.50 – 0.75)n

Examiner’s selection & save of FOThe sophomore class plans to sell T-

shirts with the school’s name on them. The cost of each T-shirt alone is $3.50, and the printing cost of each shirt is $0.75. If the class plans on selling each printed T-shirt for $11, what expression can you use to calculate the class profit for selling n printed T-shirts?

A. 11.00 – (3.50 + 0.75)nB. 11.00n – (3.50 + 0.75)C. 11.00 – 3.50 – 0.75nD. (11.00 – 3.50 – 0.75)n

selling each printed T-shirt

RMCM query modifications…A: Change ‘selling each printed shirt’

to ‘selling all printed shirts.’B: Change ‘cost of each T-shirt’ to ‘cost

of all T-shirts,’ and change ‘printing cost of each shirt’ to ‘printing cost of all shirts.’

C: Change ‘cost of each T-shirt’ to ‘cost of all T-shirts,’ and change ‘selling each printed shirt’ to ‘selling all printed shirts.’

D: (may indicate that student lacks understanding of the material tested and may lead to complete loss of credit for the question).

Another RMCM embodiment of Example 4…Examinee Task:

choose an answer that you believe is incorrect.

find facts /segments from LIST, so that chosen incorrect answer becomes correct answer to changed question

LIST: --‘cost of each T-shirt;’ ‘alone;’ ‘$3.50;’ ‘printing cost of each shirt;’ ‘$0.75;’ ‘selling each printed shirt;’ ‘$11;’ ‘profit;’ ‘selling n printed shirts’

LIST: ‘cost of each T-shirt;’ ‘alone;’ ‘$3.50;’ ‘printing cost of each shirt;’ ‘$0.75;’ ‘selling each printed shirt;’ ‘$11;’ ‘’profit;’ ‘selling n printed shirts.’

Expected answers:A: ‘selling each printed shirt.’B: ‘cost of each T-shirt’ and ‘printing cost of

each shirt.’C: ‘cost of each T-shirt’ and ‘selling each

printed shirt.’D: (test-taker lacks understanding of

subject matter)

Wrap up - Advantages of RMCM

Multiple-choice format used for RMCM:--

Allows context definition economically

Questions may be reusedScoring may be automatedOther advantages of automation

e.g. computer-aided generation of queries

Forces analysis/ compares / contrasts of similar and related concepts

Wrap up - Disadvantages mitigated by RMCM

Unlike traditional Multiple-choice RMCM:

Discourages answering without careful reading

Encourages analysis of questions

Finds depth of student’s learning

Databank of questions has lasting value

Disadvantages of open format/ essay mitigated by RMCM

Encourages identification of context

Discourages padding of answers with irrelevant writing

More efficient

Computer-aided assessment possible

InvitationIf you would like to get involved

--

Effort is underway to:Assemble groups of educators /

others Assemble RMCM Questions

databanks Develop self-study coursesDevelop instructor-led courses

You are also invited to contribute to further design and development of RMCM software

Thank You!Contact

Indu M. Anand

indu_m_anand@yahoo.comindu_anand@uml.edu