blueprint of exam questions

Prof. Dr. Shahid Hassan MHPE

H/O Unit of Medical Education

Faculty of Medicine and Health Sciences Universiti Putr Malaysia

Blueprint of Examination Questions

1st Professional Examination of MD Program in FMHS at UPM

Definition: A blueprint of examination also known as the test specification provides examination strategy of an institution at a glance.

•  Blueprint is the matrix or chart reporting the number and type of test questions represented across the topics in content area, consistent with learning objective and relative weight on test given to each topic.

•  A blueprint also identifies the % weighting of cognitive dimensions as the level of competence tested in each konowledge domain.

Blueprint of Examination Questions 1st Professional Examination of MD Program in FMHS at UPM

Purpose of Blueprints: The purpose of constructing examination using a blueprint is to provide a conceptual map of examination format and the content area represented in assessment. Blueprint of examination is the widely used assessment tool that provides list of information about: 1. Type of measurement tools and proportion of each question format in assessment shown in respective weighting colunm.

2. Topics and the level of training for each topic and the relevant learning objectives.


Purpose of Blueprints: 3. Topics and the relative weighting given to each topic in content area. 4. Competencies as unidimesional cognitive process tested in knowledge and skills domains of Bloom’s taxonomy of educatioal objective and Miller’s pyramid of competence learning respectively. 5. Knowledge as multidimensional process of learning proposed as taxonomy table by Anderson including factual knowledge, conceptual knowledge and procedural knowledge.



Learning Objective/Outcome

Cognitive dimension

1 2 3 4 5 6

Knowledge

Comprehension

Analysis

Application

Synthesis

Evaluation

Bloom’s Unidimensional Taxonomy of Educational Objective


Learning Domains è

The Cognitive Process Dimension

Knowledge Dimension ê

Recall

Understand

Apply

Analyze Evaluate Create

Factual Knowledge Conceptual Knowledge Procedural Knowledge Metacognitive Knowledge

Anderson’s Multidimensional Taxonomy Table

Who will Use the Blueprint: The blueprint is a globally accepted tool used to construct examination questions for ready reference of stakeholders engaged in assessment process. Following stakeholders can use the blueprints readily available. 1. Candidates to use examination blueprints to ensure prpeparation of assessment for its time management and strategy to achieve the desired outcom of passing the examination.


Who will Use the Blueprint: Eevery student however, must be informed that: •  All topics in blueprint may not be listed and a curriculum

detail on content should always be sought by students. •  Topics not listed in blueprints are still part of content,

which can be tested through assignments, student centered learning (SCL) and continuous (formative) assessment.

•  Students must take the responsibility of acquiring a broad-

based knowledge of all those topics not listed in blueprint.


Who will Use the Blueprints: 2. Teaching faculty inclusive of package and module coordinators to ensure the validity of assessment in term of assessment questions being representative of content area documented in curriculum. 3. External examiners to know the assessment specification with optimal information in given time to help facilitate the assessment and review the learning process.


Who will Use the Blueprint: 4. Members of vetting committee to ensure specification of questions being consistent with konowledge and skills domain versus cognitive dimension as per the learning objectives of each module. •  Blueprint also provides a readily available opportunity to

vetting members to ensure that the essential topics are not overlooked.


Essentials of Blueprints: The objective of examination at different level of training will vary from topic to topic. A blueprint of examination should not only be consistent with those objectives listed for each topic but should also reflect to achieving them for reasons: 1.  Candidats’ are required to attain the competencies listed in

assessment in order to be be promoted to next level in training.

•  A blueprint therefore must reflect that a candiadte has

acquired the competencies to be able to advance to next level.


Essentials of Blueprints: 2. Blueprint is seen as readily available document, which contains the list of topics covered under each module with its identified learning objectives as blueprint. •  However, to develop the blueprint of examination

questions in MD program, we take overall objectives of individual modules in each package as the prelude for listing the outcome compentency of each objective.

.


Essentials of Blueprints: 3. Learning outcome is viewed as module competencies essential for each module and a blueprint provide the eveidence of as many modules as possible reperesented in percentage weighting.

•  This will reflect on content validity of assessment in blueprint.

•  A blueprint representating the percentage weighting of content across assessment is one of the essentials of assessment validity.


Guideline to Prepare Blueprints: Blueprint of examination is produced as three separate tables:

Table 1:

A table reflecting the type of questions and their weighting respectively.

•  Ideally each measurement tool and items in a mesure should also have percentage weighting depending on cognitive demand of question.

•  In fact time allocation should also vary between questions of same construct such as MTF vs. SBAQ.


Guideline to Prepare Blueprints:

•  Currently all question regardless of cognitive demand carried the same marks weighting. We however, follow the practice of marks weighting between the tools but not between the items (all MTF/SBAQ questions carry equal marks).

•  We have allocated equal marks to each item in MCQ

irrespective of complexity of cognitive demand it is intended to test.

•  The current practice somehow sends a wrong message to students that questions with factual recall is as mportant as applications and analysis of knowledge.


Guideline to Prepare Blueprints: Table 2: A series of tables showing the contents with its objectives and percentage weighting and preferably the level of training for each topic/module and this may need several tables for each topic or module in a package


Guideline to Prepare Blueprints: Table 3: A table potraying the ultimate blueprint developed as a chart representing each module. Total number of questions with its percentage weighting and number of questions weighted against their ability to test the levels of cognitive dimension described by Bloom’s taxonomy of knowledge is the characterstics of blueprint


Structural Components of Blueprint: Table 1. Type of Measurement Tools in Assessment: •  This table comprises of examination question guided by

type of assessment tools and their approximate weighting. This table is the first step towards developing the actual blueprint

(See Table 1)


Structural Components of Blueprint: Table 2: Topics with objectives and learning outcome: •  The next table comprised of a series of matrix containing

the code and the title of course content (topics/module), learning objectives, learning outcome (module competencies), weighting and preferably the level of training

(See Table 2)

•  This may help student and faculty to know about competencies to be learned in the course, highlighted in blueprint.


Structural Components of Blueprint: Table 2: Topics with objectives and learning outcome: •  A summary of detail weighting of content area from each

module in a package currently practiced in 1st professional eaxamination is developed with % weighting is calculated.

(See Table 3)


Table 3: Summary of weighting of each package and their respective modules calculated for number of weeks of study (content area) per module versus weighting currently practiced in 1st Professional Exmination.

Structural Components of Blueprint: Table 2: Topics with objectives and learning outcome: •  It can be ensured that the weighting of assessment in term

of content area calculated from number of weeks allocated to deliver that module is accordingly represented in assessment.

•  An obvious impression from eyeball rolling over this

blueprint clearly shows that the content is not evenly distributed. Some of the modules are not even given a place in weighting whilw other are overrepresented in assessment of professional examoination.


Structural Components of Blueprint: Table 3: Number and Weighting of Questions in Assessment: •  After knowing the learning objectives of each contentent

areas as per their modules in preclinical phase of training (see table 2) and the type of measurement tools in 1st professional examination (see table 1) it is easy to finally create the examination blueprint.

(See Table 4-6)



Table 4: Template of examination blueprint showing number of questions (%) determined by content area (weeks of study) as defined for each packages/modules and their learning domain (cognitive dimension) determined by their respective lerning objectives in 60 questions MCQ/SBAQ (30/30) of 5 marks each from Paper I/II in 1st professional examination of MD course.

* Indicates recommended percent wise representation of questions with respect to their level of competence (cognitive dimension) in different modules however, exact number may not follow this strickly and can be determined in vetting


Table 5: Template of examination blueprint showing number of questions (%) determined by content area (weeks of study) as defined for each packages/modules and their learning domain (cognitive dimension) determined by their lerning objectives in 12 questions SEQ/MEQ (8/4) of 15 marks each from Paper I/II in 1st professional examination. Of MD course



Table 6: Template of examination blueprint representing the number of station demarcated into packages and modules with skills domains (level of competence) determined by their lerning objectives in 15 stations OSPE and 5 stations OSCE of 5 marks each in 1st professional examination of MD course.


Structural Components of Blueprint: Table 3: Number and Weighting of Questions in Assessment: •  Examination blueprint is tabulated as chart representing the

total number of questions within each module and their distribution as per level of competence (cognitive dimension) in term of knowledge and skills domains respectively.

•  This will ensure the desired coverage of topics in term of modules and their relative importance per their lerning objectives and content area in the curriculum.


Rationale of Blueprints: 1. Assessment is a complex and multifaceted aspect of learning and teaching, which may be viewed differently by different users. However, a valid assessment determins the acquired competencies identified as learning outcome in curriculum. 2. The document that provides an overview to assessment and can be readily used on stakeholders (particularly the external examiners, MQA, MMC and Senate) demand of quality assessment evident from its blueprint.


Rationale of Blueprints: 3. Another important aspect of assessment, the feedback on students’ progress and teachers’ delivering of curriculum is less often practiced simply for not well understood assessment process by faculty.

•  A well design assessment documented as blueprint provides an opportunity and confidence to faculty to give feedback with realtive importance to measures and their weighting.


Rationale of Blueprints: 4. From student’s point of view assessment always define curriculum.

•  What is to be learnt (content area in assessment) and how to be learnt (superficial to deep learning approaches) will depend how assessment is designed and how it is being made available (blueprint) to students and assessors alike.


Rationale of Blueprints: 5. We all agree that most students demonstrate a preference for learning approach. •  In this perspective teaching and learning context

determined as a blueprint of examination question in assessment can influence the students’ actual approach in practice.


Rationale of Blueprints: 6. We in FMHS would like to develop a blueprint of examination question not because it is demanded by the external examiners but to guide our students to adopt a learning approach, moving from superficial to deeper level of learning domain.

•  This will enable students’ in handling the learning task that our assessment approach not only tends to present but now it is also provded as a blueprint of examination questions.


Rationale of Blueprints: 7. Besides the external pressure to develop blueprint of exam questions, assessment imperatives such as validity, reliability and managebility takes the cetral position in idea behind bluepriting.

•  Bloom’s taxonomy of knowledge as test of cognitive dimension is primarily utilized to develop the entire written and some aspect of practical questions.

•  Miller’s pyramid is used for judgment of practical and clinical skills of competency to test students’ affective and psychomotor abilities ensured in blueprint.


Systemic Reporting of Questions for Blueprint: •  Further to blueprint a guideline for faculty member to categorize each question into relevant columns of cognitive dimension that the question intend to test will be requested.

•  It will be mandatory for each question setter to fill up the proposed table while writing and submitting the examination questions.

(See Table 7)


Table 7: Guidelines for questions writer to categorize each question to a relevant column og cognitive dimension that the question is set to test.

Systemic Reporting of Questions for Blueprint: •  All the questions categorized to one or more level of competence refering to cognitive dimension will also be vetted by the vetting committee in their routine meetings. •  This practice will ensure that the examination questions are consistent with target set for weghting of question in blueprint.

•  This will also help the academic office to prepare the blueprint of examination questions well on time for ready referal of faculty and external examiners.



THANK YOU