Faculty of Medicine Institute of Medical Education Assessment and Evaluation Unit

Report on the ESMO Examination 2016 in Medical Oncology

Rabea Krings, Dr. phil.

Rabea Krings, Dr. phil. Scientific co-worker Konsumstrasse 13 3010 Bern rabea.krings@iml.unibe.ch www.iml.unibe.ch

Content

1. Introduction ............................................................................................................................... 3  

2. Method ....................................................................................................................................... 3  2.1   Candidates ........................................................................................................................ 3  2.2   Format ............................................................................................................................... 3  

3. Performance of measurement .................................................................................................. 3  

4. Results ....................................................................................................................................... 4  4.1   Score distribution ............................................................................................................... 4  4.2   Passing score .................................................................................................................... 4  4.3   Detailed analyses .............................................................................................................. 4  4.4   Differentiated feedback to the individual candidate ............................................................ 5  

5. Summary .................................................................................................................................... 6  

Difficulty distribution, Difficulty/discrimination index diagram ................................................. 7  

Score distribution total ................................................................................................................. 8  

Score distribution (location) ........................................................................................................ 9  

Score distribution (region) ......................................................................................................... 10  

Composition of the exam (Blueprint 1) ...................................................................................... 11  

Composition of the exam (Blueprint 2) ...................................................................................... 12  

Composition of the exam & Subscores ..................................................................................... 13  

Sample feedback letter ............................................................................................................... 14  

 

3

1. Introduction This report was prepared by the Institute of Medical Education, University of Bern, Switzerland. The 2016 examination in Medical Oncology conducted by the European Society for Medical Oncology (ESMO) took place on October 8th, 2016, in the locations Berlin, Cairo, Chennai, Copenhagen, and Olten. 351 candidates participated in this examination, which consisted of 100 multiple-choice questions written in English.

2. Method 2.1 Candidates

351 candidates participated in the examination: 30 in Berlin, 29 in Cairo, 73 in Chennai, 187 in Copenhagen, and 32 in Olten.

2.2 Format

The examination 2016 was composed of 100 multiple choice questions: 80 type A questions (single choice), and 20 type K’ questions (quadruple correct/incorrect decision).

3. Performance of measurement The diagrams on page 7 give an overview of the psychometric properties of the examination. The upper diagram shows the distribution of the items in terms of difficulty, the diagram underneath in terms of the discrimination index. At the bottom of this page the reliability (Cronbach’s Alpha = 0.88), and the standard error of measurement (4.1) are given.

4

4. Results

4.1 Score distribution

The score distribution is shown on page 8. On average, candidates answered correctly to 59.8% (2015: 67.2%) of the questions, with a standard deviation of 12.1% (2015: 12.8%). 4.2 Passing score

On November 16, 2016, the ESMO Examination Working Group discussed the questions which displayed statistical deviations. For content reasons, two A questions and one K’ question were eliminated from the evaluation, so that 97 questions remained in the analysis and final evaluation of each candidate. After psychometric analysis of the difficulty of the examination, in particular with the Rasch model, the Examination Working Group decided to set the pass/fail limit at 54.6% correct answers (2015: 60.8%), in order to keep the passing requirement fair and comparable to previous years. 245 attendees (69.8%) passed the examination, and 106 attendees (30.2%) failed the examination (2015: 72.8% passed and 27.2% failed).

4.3 Detailed analyses

On page 9, the score distributions for the different locations of the examination are given. These results are shown in the following table in %. Location n mean % SD % success %

Berlin 30 60.4 11.5 83.3 Cairo 29 48.5 10.4 27.6 Chennai 73 59.6 11.2 64.4 Copenhagen 187 60.6 11.4 75.4 Olten 32 65.2 14.6 75.0

Total 351 59.8 12.1 69.8 The candidates were then grouped into 5 regions according to the address given upon inscription. The countries with the largest numbers of attendees were rated separately. In comparison to last year, the number of candidates from each region were almost the same. Because only five groups are possible, candidates from Switzerland and Germany

5

were rated together, as well as candidates from Spain and Portugal. These score distributions are shown on page 10, and in the following table in %. Region n mean % SD % success % Egypt 26 48.7 10.0 26.9 Germany and Switzerland 72 63.3 13.5 83.3 India 71 59.8 11.4 64.8 Spain and Portugal 42 60.3 10.2 73.8 Other countries 140 59.9 11.6 72.1

Total 351 59.8 12.1 69.8 The blueprint analyses are shown on pages 11 and 12. On the graph concerning

blueprint 1, it can be seen that candidates had more difficulties answering questions on “Gy-

necologic malignancies” than questions on “Carcinoma of unknown primary site”.

For the graph concerning blueprint 2, a further example shows that questions on “Complica-

tions of treatment, Supportive and Palliative Care. Communication and Ps” were answered

correctly more often than those about “Screening and prevention”. However, this does not

allow conclusion on the origin of these differences: The attendees could have performed bet-

ter in some fields, or the questions were more difficult. Comparisons are also difficult be-

cause some fields are represented by few questions (e.g. “Geriatric oncology, Cancer in ado-

lescents and young adults, Cancer and pregnancy” and “Screening and prevention” are rep-

resented by 2-5 items).

The graph on page 13 shows the expected as well as the true values for each field in blue-

print 1, separated for each location. It can be seen that candidates in Berlin were better or

equal to the expected level in most blueprint fields (10 out of 13), whereas candidates in all

other locations could not exceed the expected values in several blueprint fields (8 out of 13).

4.4 Differentiated feedback to the individual candidate

Feedback letters with their individual results detailed according to blueprint 1 were given

to all candidates. An example of such a feedback letter is given on page 14.

6

5. Summary 351 candidates participated in the 2016 written examination in Medical Oncology by the European Society for Medical Oncology, in Berlin, Cairo, Chennai, Copenhagen, and Olten. The exam consisted of 100 multiple choice questions, three items had to be eliminated from the evaluation. The exam shows a reliability of 0.88 (Cronbach’s Alpha). On average, candidates answered correctly to 59.8% of the questions, with a standard deviation of 12.1%. The pass/fail limit was set at 54.6%. 245 (69.8%) candidates passed the examination. The next examination will be held on Saturday, 9th September 2017.

7

Difficulty distribution, Difficulty/discrimination index diagram

©AA

E/IM

L, 1

2.Ja

nuar

201

7, 1

0:35

K’ évaluation à demi-pointsK’ Halbpunktbewertung

Examination in Medical Oncology08.10.2016ESMOTotal

0 - 10 11 - 20 21 - 30 31 - 40 41 - 50 51 - 60 61 - 70 71 - 80 81 - 90 91 - 1000

10

20

Schwierigkeitsverteilung Distribution de difficulté

Anza

hl /

nom

bre

% richtige Antworten / % de réponses correctes

Items mit einfacher Wahlitems à choix unique

Items mit vier richtig/falsch-Beurteilungenitems à quatre décisions juste/faux (type K’)

Totaltotal

eliminierte Itemsitems éliminés

0 10 20 30 40 50 60 70 80 90 100

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Schwierigkeits-/Trennschärfediagramm Diagramme difficulté/sélectivité

Tren

nsch

ärfe

/ sé

lect

ivité

(r ) it

% richtige Antworten / % de réponses correctes

Items mit einfacher Wahlitems à choix unique

Items mit vier richtig/falsch-Beurteilungenitems à quatre décisions juste/faux (type K’)

eliminiertéliminé

Reliabilität (CRONBACH-alpha) 0.880 Fiabilité (r )tt

auf 100 Items gerechnet 0.883 calculée pour 100 itemsErreur type (s = s 1-r )e x ttStandardmessfehler (s = s 1-r )e x tt 4.1+_

auf 100 Items gerechnet calculée pour 100 items4.1+_

0 Ni 0 2 2 5 0 12 0 11 1 19 0 19 0 15 0 9 0 5 0elim

45

47

K18

8

Score distribution total

©A

AE/

IML,

12.

Janu

ar 2

017,

10:

41

Examination in Medical Oncology08.10.2016ESMO

Punkte- und NotenverteilungK’ Halbpunktbewertung

Répartition des points et des notesK’ évaluation à demi-points

Total

Note X X% f cumf% Histogramm

Np 351Anzahl KandidatInnen nombre de candidatsNi 97mögliches Punktemaximum score maximum possible

x 57.99 x% 59.78Mittelwert moyennes 11.79 s% 12.15Standardabweichung écart type

eliminierte Fragen (Nummern) questions éliminées (numéros)45, 47, K18

22 22.7 1 0.3

27 27.8 1 0.6

29 29.9 1 0.9

32 33.0 3 1.733 34.0 3 2.634 35.1 1 2.835 36.1 1 3.136 37.1 3 4.037 38.1 1 4.338 39.2 4 5.439 40.2 5 6.840 41.2 5 8.341 42.3 2 8.842 43.3 6 10.543 44.3 5 12.044 45.4 8 14.245 46.4 9 16.8

47 48.5 7 18.848 49.5 8 21.149 50.5 7 23.150 51.5 5 24.551 52.6 9 27.152 53.6 11 30.253 54.6 10 33.054 55.7 13 36.855 56.7 15 41.056 57.7 12 44.457 58.8 12 47.958 59.8 10 50.759 60.8 13 54.460 61.9 9 57.061 62.9 12 60.462 63.9 15 64.763 64.9 12 68.164 66.0 10 70.965 67.0 6 72.666 68.0 9 75.267 69.1 13 78.968 70.1 5 80.369 71.1 7 82.370 72.2 10 85.271 73.2 11 88.372 74.2 3 89.273 75.3 9 91.774 76.3 3 92.675 77.3 4 93.776 78.4 2 94.377 79.4 3 95.278 80.4 4 96.379 81.4 1 96.680 82.5 1 96.981 83.5 3 97.782 84.5 4 98.983 85.6 1 99.184 86.6 1 99.4

86 88.7 1 99.787 89.7 1 100.0

x

-s

+s

Note =>X f f%

n i c

h t

b e

s t

a n

d e

nb

e s

t a n

d e

n

nicht bestanden 0 106 30.2

bestanden 53 245 69.8

9

Score distribution (location)

Examination in Medical Oncology08.10.2016ESMO

Anzahl KandidatInnen1 Skalenstrich = 1 Kand.

Nombre de candidats1 cand. par division

Punkteverteilungen Répartitions des pointsK’ Halbpunktbewertung K’ évaluation à demi-points

©A

AE/

IML,

12.

Janu

ar 2

017,

10:

49 Tota

lN

=35

1x

=57

.99

s=

11.7

8

010

2030

4050

6070

8090

Olte

nN

=32

x=

63.2

5s

=14

.31

Cop

enha

gen

N=

187

x=

58.7

7s

=11

.08

Che

nnai

N=

73x

=57

.77

s=

10.8

6

Cai

roN

=29

x=

47.0

7s

=10

.09

Berli

nN

=30

x=

58.6

3s

=11

.13

n i c

h t

b

e s

t a

n d

e n

b e

s t

a n

d e

n

10

Score distribution (region)

Examination in Medical Oncology08.10.2016ESMO

Anzahl KandidatInnen1 Skalenstrich = 1 Kand.

Nombre de candidats1 cand. par division

Punkteverteilungen Répartitions des pointsK’ Halbpunktbewertung K’ évaluation à demi-points

©A

AE/

IML,

12.

Janu

ar 2

017,

10:

35 Tota

lN

=35

1x

=57

.99

s=

11.7

8

010

2030

4050

6070

8090

Oth

ers

N=

140

x=

58.0

7s

=11

.25

Port

ugal

_Spa

inN

=42

x=

58.5

0s

=9.

87

Indi

aN

=71

x=

57.9

9s

=11

.05

Ger

man

y_Sw

itzer

land

N=

72x

=61

.44

s=

13.1

2

Egyp

tN

=26

x=

47.1

9s

=9.

65

n i c

h t

b

e s

t a

n d

e n

b e

s t

a n

d e

n

11

Composition of the exam (Blueprint 1)

©A

AE/

IML,

30.

Nov

embe

r 201

6, 0

6:56

ESMO Examination in Medical Oncology08.10.2016

Prüfungszusammensetzung Composition de l’examennach Inhalt selon le contenuK’ Halbpunktbewertung K’ évaluation à demi-points

Code Subscore Items K’[%]

1 Hematologic malignancies 9 22.2

2 Chest malignancies 11 18.2

3 Breast cancer 9 22.2

4 Gynecologic malignancies 4* 25.0

5 Head and neck cancers 4* 25.0

6 Central nervous system malignancies 5* 20.0

7 Genitourinary cancers 9 33.3

8 Gastrointestinal cancers 7 28.6

9 Skin cancers 4* 0.0

10 Sarcomas 4* 0.0

11 Carcinoma of unknown primary site 2* 0.0

12 Endocrine cancers 4* 25.0

13 Others/Clinic 25 16.0

Total 97 19.6

* Bei kleinen Subscores ist die Aussagekraft eingeschränkt (Einfluss des Zufallsfehlers)L’interprétation des petites chapitres devient incertaine (influence de l’erreur aléatoire)

Subscoreanalyse Analyse par subscores

30

40

50

60

70

80

nach ansteigender Leichtigkeit angeordnet arrangé selon la facilité des chapitres

Proz

ents

atz

richt

ige

Ant

wor

ten

Pour

cent

age

de ré

pons

es c

orre

ctes

BerlinCairoChennaiCopenhagenOltenmean

Total 4 10 7 9 8 2 6 1 13 3 12 5 11

12

Composition of the exam (Blueprint 2)

©A

AE/

IML,

30.

Nov

embe

r 201

6, 0

7:05

ESMO Examination in Medical Oncology08.10.2016

Prüfungszusammensetzung Composition de l’examennach Inhalt2 selon le contenu2K’ Halbpunktbewertung K’ évaluation à demi-points

Code Subscore Items K’[%]

1 Biology, Pathology incl. tumour markers, Laboratory medicine 17 17.6

2 Epidemiology, Etiology, Clinical research 5* 20.0

3 Screening and prevention 5* 40.0

4 Clinic, Differential diagnosis, Staging, Imaging 15 26.7

5 Therapy, Surgery, Radiation oncology, Anticancer agents, Biologic therapy, Respo 46 17.4

6 Complications of treatment, Supportive and Palliative Care. Communication and Ps 7 0.0

7 Geriatric oncology, Cancer in adolescents and young adults, Cancer and pregnancy 2* 50.0

Total 97 19.6

* Bei kleinen Subscores ist die Aussagekraft eingeschränkt (Einfluss des Zufallsfehlers)L’interprétation des petites chapitres devient incertaine (influence de l’erreur aléatoire)

Subscoreanalyse Analyse par subscores

30

40

50

60

70

80

nach ansteigender Leichtigkeit angeordnet arrangé selon la facilité des chapitres

Proz

ents

atz

richt

ige

Ant

wor

ten

Pour

cent

age

de ré

pons

es c

orre

ctes

BerlinCairoChennaiCopenhagenOltenmean

Total 3 7 5 4 2 1 6

13

Composition of the exam & Subscores ©

AA

E/IM

L, 3

0.N

ovem

ber 2

016,

07:

05

ESMO Examination in Medical Oncology08.10.2016Subscoreanalyse Analyse par subscoresnach Inhalt selon le contenuHalbpunktbewertung Evaluation avec des demi-pointsItems K’[%]

erzielte Leistungerwartete Leistung

performance réaliséeperformance attendue

1 Hematologic malignancies 9 22.2

2 Chest malignancies 11 18.2

3 Breast cancer 9 22.2

4 Gynecologic malignancies 4 25.0

5 Head and neck cancers 4 25.0

6 Central nervous system malignancies 5 20.0

7 Genitourinary cancers 9 33.3

8 Gastrointestinal cancers 7 28.6

9 Skin cancers 4 0.0

10 Sarcomas 4 0.0

11 Carcinoma of unknown primary site 2 0.0

12 Endocrine cancers 4 25.0

13 Others/Clinic 25 16.0

Total 97 19.6

Berlin

30

40

50

60

70

80%

Subsc.

Cairo

30

40

50

60

70

80%

Subsc.

Chennai

30

40

50

60

70

80%

Subsc.

Copenhagen

30

40

50

60

70

80%

Subsc.

Olten

30

40

50

60

70

80%

Subsc.

1 2 3 4 5 6 7 8 9 10 11 12 13

1 2 3 4 5 6 7 8 9 10 11 12 13 1 2 3 4 5 6 7 8 9 10 11 12 13

1 2 3 4 5 6 7 8 9 10 11 12 13 1 2 3 4 5 6 7 8 9 10 11 12 13

14

Sample feedback letter

ESMO Head OfficeEducation Department

Via Luigi Taddei 4CH 6962 Viganello-Lugano / Switzerland

Dr

Informal feedback on your exam results

On October 8, 2016 you participated in the ESMO examination in Berlin. In order to pass, 53 out of 97 questions had to be answered correctly (54.6%). Your results are shown in detail in the table below.

Questions have been grouped in 'subscores' according to their content. For each subscore you find how many questions you answered correctly ('points'), the number of questions per subscore ('of') and how you compare with the other candidates ('sucess'). Subscores with less than 6 questions (*) are not evaluated due to low measuring reliability.

/

/

30

Miklos Pless, Chair, ESMO Examination Working Group

The absence of an arrow indicates that your results were between the 26th and the 75th percentile.

You scored within the top 25 %.

ESMO Examination in Medical Oncology, October 2016

points / of success subscore total score: 71 result: passed

5 9 Hematologic malignancies

8.5 11 Chest malignancies

7 9 Breast cancer

✳2 4 Gynecologic malignancies

✳4 4 Head and neck cancers

✳3.5 5 Central nervous system malignancies

7 9 Genitourinary cancers

6 7 Gastrointestinal cancers

✳4 4 Skin cancers

✳2 4 Sarcomas

✳2 2 Carcinoma of unknown primary site

✳3 4 Endocrine cancers

16.5 25 Others/Clinic

/

//////////

You scored within the bottom 25 %, but not in the weakest 5 %.

You scored within the weakest 5 %.