cas examination process julie stenberg, fcas cane meeting march 20, 2007

CAS EXAMINATION PROCESS

Julie Stenberg, FCAS

CANE MeetingMarch 20, 2007

2

CAS Admissions Process Audit

• Fall 2000 – CAS Issues RFP for External Review of Admissions Processes

• The Chauncey Group (Subsidiary of ETS) Selected

• Spring 2001 – Chauncey Group Conducts Audit of CAS Admissions Processes

3

Audit Findings

The CAS Does Many Things Well:

• Good Communication with Candidates

• Sound Procedures for Maintaining Confidential Information

• Exams are Administered with Appropriate Controls and Standardized Procedures

4

Audit Findings

Several Areas for Improvement:• Need Better Link Between Learning

Objectives and Exams/Readings• Learning Objectives and Exam

Blueprints Should be Published• Need Better Training of Item Writers• Need to Consider Alternative Processes

for Selecting Pass Marks

5

Major Objectives

The Chauncey Group Engaged to Help CAS with Three Issues:

• Write Better Learning Objectives and Establish Links to Readings/Exams

• Develop a Process for Training Item Writers

• Pilot an Alternative Process for Selecting Pass Marks

6

Major Milestones• August 2001 – Chauncey Began Facilitating

Meetings to Write Learning Objectives• October 2001 – Piloted Pass Mark Panel

Process for Exams 6 & 9• March 2002 – Piloted Item Writer Training

Classes for Exams 6 & 9• April 2002 – Pass Mark Panels Meet for Exams

5, 7 & 8• June 2002 – Item Writer Training for Exams 5,

7 & 8

7

Major Milestones

• October 2002 – Pass Mark Panels Meet for Exams 6 & 9

• February 2003 – Executive Council Agrees to Fund Item Writer Training and Pass Mark Panels as Ongoing Processes

• April 2003 – Executive Council Approves New Learning Objectives for Exams 3, 5-9

8

Recent Improvements

• Evolution of CBT

• Improvement to Pass Mark Panel Process

• Expanded Sample Answer Sets

• Increased Communication

• CAS Board White Paper

9

Learning Objectives

The way things were

– What topics should successful candidates understand

– What readings should they know?

The way things are now

– What should successful candidates be able to DO?

10

Learning Objectives

The way things were

– Individual topics and readings were the basis for assigning the writing of exam questions


– Learning Objectives are the basis for assigning the writing of exam questions

11

Learning Objectives

The way things were

– Syllabus “blueprints” were the documents governing the review of the Syllabus and the construction of Exams


– Learning Objective Documents are the basis for the review of the Syllabus and the construction of Exams

12

Learning Objectives

The Syllabus Committee has developed Learning Objective Documents for CAS Exams 3, 5, 6, 7-US, 7-Canada, 8 and 9

13

Learning Objective Documents

Five Elements

• Overview Statement for a Group of Learning Objectives

• Learning Objectives

• Knowledge Statements

• Syllabus Readings

• Weights

14


Overview Statements

• Certain Syllabus Sections Can Have Multiple Learning Objectives (e.g., Ratemaking)

15


Learning Objectives• What successful candidates should be able to

do• Learning Objectives Should:

Clearly state a main intentReflect a measurable outcomeSupport an attainable behaviorRelate to the learner’s needs or job

functionHave a definitive time frame

16


Knowledge Statements

• Support Learning Objectives

• In order to accomplish the objective, what does the candidate need to know?

17


Readings

• An individual reading may be listed under more than one learning objective

• Readings listed under multiple objectives may facilitate more synthesis/reasoning/cross-topic Exam questions

18


Weights (by Learning Objective)

• Shown as ranges

• The ranges are guidelines and are not intended to be absolute

• Ended practice of candidates calculating de facto weights by reading or topic from past Exams

19

IV. Ratemaking This section contains objectives covering ratemaking in broad, general principles, as well as specific detail. Candidates should have a thorough understanding of the basic principles of ratemaking, so that they can analyze data, select an appropriate technique, and develop a solution to a numerical problem. In addition, the candidate should be able to compare specific ratemaking techniques in terms of advantages and disadvantages as they are applied to specific situations and for different lines of business. Weight: xx-xx% 1. Explain the role of exposure bases in the

ratemaking process. Weight: xx-xx%

Definition of exposure base Characteristics of exposure base Impact of exposure change Coverage provisions

Reading(s): McClenahan, Finger, Principles, Bouska, Feldblum (WC), Graves & Castillo, Jones 2. Use appropriate premium data to

estimate premium input into the overall rate level indication, adjusting for the following:

Coverage and benefit level changes

Rate level changes Premium trend

Weight: xx-xx%

Compilations of experience (CY/PY/AY) Written versus earned premium Rate changes Policy terms Distributional shifts/changes in volumes (tend over time) Parallelogram method Extension of exposures Definition of exposures Impact of law changes

Reading(s): McClenahan, Finger, Principles, Feldblum (Asset Share), Feldblum (WC), Jones, ASOP #13

20

Learning Objectives and the Syllabus

• Learning Objective Documents Provide High Level Guidance

– Review of Current Syllabus Material

– Identification of Topics Requiring New Syllabus Material

• Weights help Syllabus Committee Target Specific Objectives

21

Future Changes to Learning Objective Documents

• These are Living Documents– Never Perfect– Subject to Change

• Updates – When and How Often?– Once a Year Per Exam Seems

Reasonable– At Least Disruptive Time for Candidates

22

Future Changes to Learning Objective Documents

• CAS Executive Council (VP-Admissions) Performs Oversight and Final Approval of Any Changes

– Just as it does with changes to the Syllabus

– Just as it has with the current Learning Objective Documents

23

Learning Objective Summary• Transition to Published Learning

Objectives Should Help the CAS Achieve:– Better Syllabus Content and Exam

Questions– More Transparent Basic Education

Process– Better Model for Evaluating Future

Changes to the Syllabus– Better Model for Evaluating Future

Changes to the Desired Education of Casualty Actuaries

24

Writing Exam QuestionsQuestion 1 – According to Miller, “Writing Exam Questions”, which of the following is true?

I. Writing exam questions is the same now as it was 6 years ago

II. Writing exam questions is easier now than it was 6 years ago

III. Writing exam questions is harder now than it was 6 years ago

A. II onlyB. I and II onlyC. I and III onlyD. II and III onlyE. I, II and III

25

Writing Exam QuestionsI. Writing exam questions is the same now as

it was 6 years ago

True – Question writers have always wanted to write good, fair, high quality questions.

1. Still takes the same time commitment

2. Still requires studying assigned readings

3. Still involves choosing the areas you want to test

4. Still involves peer review by others

26

Writing Exam QuestionsII. Writing exam questions is easier now than

it was 6 years ago

True – We have more tools to work with

1. We have identified objectives

2. They identify the readings tied to those objectives

3. Question writing skills are taught – what to do and what to avoid

4. We have a common language with which to make constructive criticisms

27

Writing Exam Questions III. Writing exam questions is harder now

than it was 6 years ago

True – We have been conditioned by years of studying old questions

1. Triple True-False are often easier to write than short answers for Multiple Choice

2. The easiest questions to write may not always fit the objectives

3. The bar is higher and we don’t like to fail

28

Writing Exam Questions

Question 1 – Solution: E

A. Some candidates will think nothing has changed

B. Some candidates will think the new process could not possibly make it harder to write questions

C. Some candidates always guess C when they don’t know the answer

D. I used to guess D

E. Correct answer

29

What Have We Learned From The Chauncey Initiatives?

• Questions should be focused on learning objectives, rather than individual papers

• Triple True/False is not the only kind of multiple choice question

• Art of selecting good “wrong” multiple choice answers

30

What Changes Should The Candidates See On The Exams?

• Better questions

• Questions with many possible full-credit answers

• Less “according to” and “based on” questions

• Heavy “list” papers have become open-book

31

Setting the Pass Mark

• Identify Purpose of the Pass Mark

• Convene Pass Mark Panel

• Analyze Exam Statistics

• Prepare Recommendation

• Proceed through Approval Process

32

Purpose of the Pass Mark

• Pass Minimally Qualified (or better) Candidates– Those who have demonstrated a

sufficient grasp of the syllabus material

• Fail Others

• There is no predetermined pass ratio

33

Purpose of the Pass Mark

Failers Passers

Minimally Qualified Candidate

34

Pass Mark Panel

• Panel includes:New Fellows (1-3 years)Fellows experienced in practice areaOfficers of exam committee

• Recommends a pass mark independent of the normal exam committee procedures

35

Pass Mark Panel

• Defines Minimally Qualified CandidateWhat he or she should will knowWhat he or she will not knowWhat he or she will be able to

demonstrate on the exam

• Relates Criteria to Learning Objectives for defining the minimally qualified candidate.

36

Pass Mark Panel

• Each panelist independently estimates how 100 minimally qualified candidates will score on each question (and sub-part of each question).

• Scores are assembled and shared in a group format.

• Group discusses ratings and may change estimates

• Facilitator compiles ratings and shares results with exam committee officers

37

Analyze Exam Statistics (back at the Grading Session)

• Collect Initial Scores for All Candidates• Review/Discuss Key Measures

High, Low, MeanPercentiles, Percentile RelationshipsPass Mark Panel RecommendationPrior statistics from previous examsCAS Board goal, “…that 40% or more of the candidates

should get a score of 70% or more on any given exam; and all candidates that get such a score should pass.”

• Pick an initial pass mark and re-grade candidates within certain range of pass mark (+/- 3 points, for example)

38

Prepare Recommendation

• Recollect scores if any have changed and review all relevant statistics again.

• Repeat process until only looking at the 5 exams above and the 5 exams below the recommended pass mark.

• Justify Recommended Pass Score

39

Approval Process

• Part Chair

• General Officer (Spring / Fall)

• Exam Committee Chair (Arlie Proctor)

• VP-Admissions (Jim Christie) – The final decision on the pass mark is the responsibility of this position.

40

Appeal Process

• In the event of a candidate appeal, a grader may

be called upon to review the appeal and

reconcile the score with the grading key.

41

Join the Exam Committee

Fill out the annual CAS participation survey

or

Contact the exam committee recruiter directly

Rhonda Walker

[email protected]

42

Questions and Comments

Contact

Julie Stenberg at

[email protected]

Or

Arlie Proctor at

[email protected]