who wants to be an ich euro billionaire?

PHARMACEUTICAL STATISTICS

Pharmaceut. Statist. 2008; 7: 4–8

Published online 16 March 2007 in Wiley InterScience

(www.interscience.wiley.com) DOI: 10.1002/pst.270

Who wants to be an ICH Euro

billionaire?

Kerry Gordon*,y

Quintiles Ltd, Station House, Market Street, Bracknell RG12 1HX, UK

The ICH E9 guideline on Statistical Principles for Clinical Trials is a pivotal document for

statisticians in clinical research in the pharmaceutical industry guiding, as it does, statistical aspects

of the planning, conduct and analysis of regulatory clinical trials. New statisticians joining the

industry require a thorough and lasting understanding of the 39-page guideline. Given the amount of

detail to be covered, traditional (lecture-style) training methods are largely ineffective. Directed

reading, perhaps in groups, may be a helpful approach, especially if experienced staff are involved in

the discussions. However, as in many training scenarios, exercise-based training is often the most

effective approach to learning. In this paper, we describe several variants of a training module in ICH

E9 for new statisticians, combining directed reading with a game-based exercise, which have proved to

be highly effective and enjoyable for course participants. Copyright # 2007 John Wiley & Sons, Ltd.

Keywords: ICH E9; induction training; TV quiz format

1. INTRODUCTION

New statisticians and statistical programmers tothe pharmaceutical industry (for instance, newMSc graduates) usually undergo a period ofinduction training, including on-the-job trainingunder supervision, prior to their introduction toimportant project statistical roles. Depending onthe specific staff member’s background qualifica-tions and experience, such training may includecertain technical statistical topics (such as survival

analysis), aspects of SAS1, aspects of clinical trials(e.g. randomization, study design), companyprocedures, standards and policies as well astraining in relevant statistical regulatory guide-lines. In the latter category, the InternationalConference on Harmonization (ICH) E9 guidelineon Statistical Principles for Clinical Trials [1] isgenerally considered to be the primary statisticalsource of guidance for regulatory clinical trials,although of course there are many other highlyrelevant statistical and therapeutic regulatorydocuments (see the Regulatory area of the PSIwebsite [2] for a complete listing). Thus, training inICH E9 is inevitably a central part of anyinduction course and, given the size of theyE-mail: [email protected]

*Correspondence to: K. Gordon, Biostatistics, Quintiles Ltd,Station House, Market Street, Bracknell RG12 1HX, UK.

Copyright # 2007 John Wiley & Sons, Ltd.

guideline (39 pages) and the broad range of topicsit covers, consideration must be given to the mostsuitable format for such training.

2. CHOICE OF TRAINING FORMAT

As the level of understanding of ICH E9 needs tobe very high, including an awareness of very manyspecific detailed items, traditional training meth-ods may not be the most effective. In particular, alecture-style presentation would need to be quitelong, almost certainly split over several sessions,and so detailed that many important aspects maybe forgotten by course participants. Studies ofandragogy (e.g. Knowles [3]) indicate that, sinceadult learners have a tremendous amount of lifeexperience, the role of the instructor is to guide thelearner to their own knowledge, rather thansupplying them with facts. In this regard, directedreading of ICH E9, combined with group discus-sions, can be very useful, especially when thegroup includes one or more experienced membersof staff, who can shed light on some of the lessroutine (or more controversial) issues.

In addition, it is well known that exercise-basedtraining, where learners are offered the opportu-nity to test their understanding, is often moreeffective than lecture-style learning (see, forexample, Kolb [4]). In my own experience, more-over, people enjoy playing games even when theyare no longer children, and exercises in a gameformat can provide stimulating and effectivelearning. Indeed, if the game is of a kind wheremore than one team can be involved, a prize forthe winning team can provide an extra incentive tolearning. In the section below, we outline a fewvariations of such games which, in combinationwith directed reading and group discussion, havebeen used to great effect on several occasions.

3. SOME ICH E9 GAMES

Three ICH E9 games are described below. In eachcase, the game is preceded by a period of directed

reading within small groups, facilitated by thecourse leader. Individuals (or pairs) are invited toconcentrate on certain sections of the guideline, tobecome especially familiar with them, and after asuitable time period (at least an hour is recom-mended), each group member in turn describeswhat they believe to be the key issues in theirsections, and the group then discuss any experi-ences or uncertainties about these issues. Thechoice of game format to follow then largelydepends on class size; however, other aspects (suchas expected familiarity of the course participantswith the chosen format) may also be important.

3.1. Who wants to be a millionaire?

This version of the game has proved to be mosteffective with class sizes of up to 12, though can beused with larger groups too. It is also suitable fornon-native English speakers, as the questions (andanswers) are written rather than spoken. Assumingwe do indeed have 12 participants, these aredivided into two teams of six and an announce-ment is made that there will be a prize for thewinning team. This is done in advance of theperiod of directed reading, so that the teams aremotivated to maximize their understanding ofICH E9 in this time. At the end of the readingperiod, players are asked to form pairs (so thateach team now consists of three pairs of players).

Firstly, the rules of the game are explained alongthe following lines (based on the format of the TVshow). Players (in pairs) are presented with writtenquestions, each with a choice of four possibleanswers (see Figure 1, for example; using MS

Question 3: 12 points

Which parts of a protocol should bereviewed by a Biostatistician?

A: No Biostatistical review required

D: Entire protocolC: Stats Methods and

Study Design

B: Stats Methods Only

Figure 1. A mid-complexity question using the ‘Who

wants to be a millionaire?’ format.

Copyright # 2007 John Wiley & Sons, Ltd. Pharmaceut. Statist. 2008; 7: 4–8DOI: 10.1002/pst

Who wants to be an ICH Euro billionaire? 5

PowerPoint works well for this). Typical pointsawarded for correct answers might be 3, 6, 12, 25,50 and 100 for questions of increasing difficulty,with the 6-point and 25-point questions as mile-stones. If players choose the correct answer, theystay in the game and move onto the next question.Otherwise they lose the points gained since theirlast milestone and they are ‘out’ (the turn is passedonto the other team). At each question, playerscan decide either to select their preferred answer or‘bank’ the points accumulated so far (in which caseplay passes to the next team). Note that playerscan make this decision after seeing the question sothat, for example, players faced with a questionworth 100 points (in which case they will havealready answered all previous questions correctly)might decide to bank the 50 points gained by theprevious correct answer rather than risk goingback to 25 points, should they get the 100-pointquestion wrong. As per the TV show, lifelines mayalso be permitted, such as ‘50/50’ (where two of theincorrect options are removed, leaving the correctanswer and one incorrect answer, Figure 2) or ‘askthe team’ (the playing pair can seek advice fromothers in their group of six). Each lifeline may onlybe used once. The team of six with the mostbanked points at the end is declared the winner.

3.2. University challenge

In this game, a quizmaster (i.e. the course leader)reads out a starter question (worth 10 points for acorrect answer). Players must not confer and onlythe first player to ‘buzz in’ is permitted to offer an

answer; if incorrect, the question is passed to otherteam (again, no conferring). In order to increasethe chance of buzzing in first, players are permittedto interrupt the quizmaster before a starterquestion has been read completely, however, theylose five points for an incorrect interruption (nopoints are lost if an incorrect answer is given at theend of the question). Once a player has correctlyanswered a starter question, that player’s team getsthe corresponding bonus questions, generally alinked group of three questions worth five pointseach. The full team can confer for bonus questionsand there is no penalty for incorrect answers.Further starter and bonus questions then follow,until a pre-specified time has been reached (up toone and a half hours has been manageable).Examples of starter and bonus questions (withanswers for the benefit of the reader) are givenbelow:

Starter Question: Apart from trials to showsuperiority, name one othertype of comparison de-scribed in ICH E9?

Answer: Equivalence, non-inferiorityor dose–response relation-ships.

Bonus questions on equivalence:Bonus Question 1: For an equivalence trial,

where should the margin ofequivalence be described?

Answer: This will need to be in theprotocol (and justified clini-cally), since the primarystudy objective (hence sam-ple size) will depend on this.The same is true for a non-inferiority trial.

Bonus Question 2: For an equivalence trial,one must show that the truetreatment difference is likelyto lie between lower andupper limits of the clinicallyacceptable equivalence mar-gin. What must be shownfor a non-inferiority trial?

Question 3: 12 points

Which parts of a protocol should be reviewed by a Biostatistician?

A: No Biostatistical review required

D: Entire protocol

Figure 2. As Figure 1, after use of the 50/50 lifeline.


6 K. Gordon

Answer: That the true treatmentdifference is likely to be atleast above the lower limitof such a margin.

Bonus Question 3: What is the main difficultyassociated with the use ofan active control group inan equivalence trial?

Answer: There is a lack of internaltrial validity, in the sensethat you may show thetreatments to be equivalent,but it may not be possibleto say whether they areequally effective or equallyineffective (compared withplacebo). External valida-tion might be needed to dothis. The same considera-tions apply to a non-infer-iority trial.

Although the official (TV) version of this game isplayed with two teams of four, I have been able toplay the ICH E9 version with up to three teams ofeight (although, in this format, use of an indepen-dent referee/score-keeper has been quite helpful).Four of the eight players per team sit in the frontrow, and only they can answer starter questions;however, all eight team members can consult forbonus questions. In this case, it is good practice toallow players to swap around at the half-way pointof the game.

There is, of course, the challenge of the buzzers,for which a little imagination is required (forinstance, the use of children’s toys has beenriotous, but effective!).

3.3. Pub quiz

After the usual directed reading period, the pubquiz version of the game can cater for the largestclass sizes. This version of the game consists of fiverounds of 10 questions (2 points per question). Atthe outset, the quizmaster announces the titles ofthe five rounds, and teams must decide on whichround they wish to play their ‘joker’ (in advance ofhearing the questions); they then get double points

for correct answers in the chosen round. Duringeach round, the quizmaster reads the questionsaloud and teams complete an answer sheet whichthey hand in at the end of the round. A short breakafter round 3 can be helpful to participants, atwhich time interim scores may be determined.Final scores are calculated at the end of the quizand the quizmaster should then go through thecorrect answers to each question, discussing asappropriate. Typical questions are given below(these might all appear in a ‘Study Design’ round):

1. What kind of trial has hypotheses that aredirectly related to the trial’s primary objective,stated in advance, and evaluated as pre-definedwhen the trial is complete? [Answer: A con-firmatory trial]

2. Give one reason why an individual clinical trialmight not be totally representative of all futureusers of the drug? [Answer: Some future usersmight not have satisfied the trial inclusion/exclusion criteria]

3. What is usually considered the biggest disad-vantage of a crossover trial? [Answer: Thepossibility of a carryover effect]

4. DISCUSSION

All three variations of the ICH E9 game describedabove have received consistent, positive feedbackfrom course participants. For instance, to thequestion ‘Would you recommend this course toothers?’, this training has always achieved 100%‘Yes’ responses. Unprompted compliments, suchas ‘a great help for a new career’ and ‘madeeverything fun and interesting while still drivinghome the important points’, have been common-place. As mentioned in Section 3.2, the ‘Whowants to be a millionaire?’ format works particu-larly well for non-native English speakers, how-ever, all versions have proved to be effective forUK and non-UK staff with even a moderateunderstanding of spoken English. One beneficialaspect of this approach to training in ICH E9 (inall versions) is that all course participants get tohear all of the questions and all of the answers,

Who wants to be an ICH Euro billionaire? 7


even when it is not their team’s ‘turn’. Indeed, thecompetitive aspect often helps to keep partici-pants’ concentration level high throughout.

For ICH E9, we have found that this training ismost effective when participants have alreadyspent 6–12 months in their new role, and so canput many of the principles described by theguideline into context. Of course, the trainingformat described is also applicable to many othersituations. Indeed, on occasions, all three gamesdescribed above have been used to cover differenttraining topics in the same course. For instance, aswell as for other regulatory guidelines (such as thevarious European Medicines Agency (EMEA)‘Points to Consider’ documents, see 2), the ‘Whowants to be a millionaire?’ format has also beenused successfully for SOP training, a topic manytrainers often struggle to bring to life.

ACKNOWLEDGEMENTS

I would like to acknowledge my Quintiles colleagues,Kevin England, Deborah McPhail and Maria Efstathioufor their suggestions and input.

REFERENCES

1. ICH E9: Statistical Principles for Clinical Trials.Available at http://www.ich.org/cache/compo/276-254-1.html (date of access: October 2006).

2. http://www.psiweb.org/resources/resources.asp?par-entfolderid=286&subgroup id= (date of access:October 2006).

3. Knowles M. Andragogy in action: applying modernprinciples of adult education. Jossey-Bass: San Fran-cisco, 1984.

4. Kolb DA. Experiential learning: experience as thesource of learning and development. Prentice-Hall: NJ,1984.


8 K. Gordon

who wants to be an ich euro billionaire?

Documents