identifying predictors of performance on usmle step 1 …

IDENTIFYINGPREDICTORSOFPERFORMANCEONUSMLE®STEP1

by

SACHINSHAH

DISSERTATION

PresentedtotheFacultyoftheMedicalSchoolTheUniversityofTexasSouthwesternMedicalCenter

InPartialFulfillmentoftheRequirementsFortheDegreeof

DOCTOROFMEDICINEWITHDISTINCTIONINMEDICALEDUCATION

TheUniversityofTexasSouthwesternMedicalCenterDallas,TX

©CopyrightbySachinShah2017AllRightsReserved

ACKNOWLEDGMENTS

ThisresearchwassupportedbytheUTSouthwesternOfficeofMedicalEducation,OfficeofStudentAffairs,andStudentAcademicSupportServices.Ithankmymentors,Dr.DorothySendelbach,Dr.AngelaMihalic,andDr.ArleneSachs,fortheirunwaveringsupportandguidancethroughoutthisresearchproject.IthankSholaRogersforhereffortscompilingthedatabase,andIthankMs.CarolWorthamandMs.AnneMcLanefortheirpartsin

compilingdata.IalsothankDr.JohnSadlerforhisguidanceinobtainingIRBexemption,andDr.JoanReischforherearlyinputonproposedstatisticalmethods.

1

TABLEOFCONTENTSINTRODUCTION....................................................................................................................................2BACKGROUND........................................................................................................................................3METHODS................................................................................................................................................7RESULTS................................................................................................................................................10DISCUSSION..........................................................................................................................................19CONCLUSION........................................................................................................................................25LISTOFTABLES...................................................................................................................................28LISTOFFIGURES.................................................................................................................................33REFERENCES........................................................................................................................................46VITAE......................................................................................................................................................48APPENDIX.............................................................................................................................................49

2

INTRODUCTION

USMLE®Step1isconsideredbyresidencyprogramdirectorstobeoneofthemost

importantfactorsinselectingmedicalstudentsforinterviews,soitisimportantforboth

studentsandmedicalschoolstomaximizescores1.However,despiteitsimportance,verylittle

hasbeenstudiedtodetermineindicatorsofperformanceontheexam.Itwasthegoalofthis

projecttolookatundergraduateGPAandMCAT®scores,demographicinformation,medical

schoolexamperformance,andself-reportedsurveydatafromtheUTSouthwesternMedical

SchoolClassof2018todiscovercorrelationsthatmayallowmedicalschoolstobetterpredict

studentperformanceontheexamandguidestudentpreparationfortheexam.Importantly,

thisinformationwouldallowadministratorstoidentifyfuturestudentswhoareatriskoffailing

ordoingpoorlyontheexamandinterveneearlyinordertopreventpoorperformanceonthe

exam.Furthermore,ifstudentsareabletobetterunderstandhowtobestpreparefortheexam

andtobeabletoidentifypersonalredflags,theymaybemorelikelytoindependentlyadjust

studyinghabitsearlyoninordertooptimizetheirperformanceontheexamandultimatelybe

morecompetitiveforresidency.Itisforthesereasonsthatwebelieveitisimportanttoreview

andidentifypredictorsofUSMLEStep1performance.Thisprojecthasthreemaingoals:1)

determiningifdataavailablepriortoadmissioncanpredictStep1performance,2)analyzing

schoolperformanceandsurveydatatosuggeststudyingtips,and3)developingamodeltoflag

studentsatriskofperformingpoorlyontheexamforearlyintervention.

3

BACKGROUND

USMLEStep1isthefirstofthreephysicianlicensureexamstakenbymedicalstudents

andcoversthebasicsciencematerialtaughtduringthepre-clinicalyearsofmedicaleducation.

Itwasdesignedinitiallytoseparatethosewithadequateversusinadequateknowledgeasan

indicatorforpreparednessforclinicaltraining,butithasbecomeatoolusedheavilytostratify

applicants2.Astudent’sscoreonthisexamisconsideredbyresidencyprogramdirectorstobe

thetopfactorinselectingstudentsforinterviews3.OntheUSMLE®Step1exam,itisgenerally

agreedthatascorebelow220willsignificantlydecreaseastudent’schanceofgettingintoatop

residencyprogram,asthisisoftenaminimumscorerequiredtoevenapplytomanyprograms,

eveninlesscompetitivespecialties4.Assuch,itisimportantforbothstudentsandformedical

schoolstodowhattheycantomaximizetheirscorestooptimizetheirchancesofbeing

consideredfortopresidencyprograms.

Startingwiththeadmissionsprocess,medicalschoolsuseMCAT®scoresand

undergraduateGPAsastheirtopfactorsinselectingstudentstointerview,withtheassumption

thathighMCAT®scoresandGPAswillpredictsuccessinmedicalschool,andspecificallysuccess

ontheUSMLE®Step1exam.Anumberofstudiesindicatethatthesepre-admissionsdatamay

bemildpredictorsofStep1scoresbutarewaryoftheweightthatisappliedtothesefactorsin

theadmissionsprocessbecauseofhighvariabilityinpredictivevalue5.Sincethesestudiesall

includeonlymatriculatingmedicalstudents,itispossiblethatthisfactorexhibitsathreshold

4

effect,whereafteracertainpointthepredictivevalueofthesedatapointsbecomesless

significant.Furtherlarge-scalestudiesareneededtobeabletomakeamoreaccurate

assessment.

Thesuccessofapre-clinicalmedicalschoolcurriculumisalsojudgedlargelybythe

performanceofstudentsonStep1,andmedicalschoolprogramsassumethathigh

performanceoninternalexamsindicatethatstudentsarepreparedforStep1.Studentsat

manycampuses,includingUTSouthwestern,areoffereddedicatedstudyingtimeforStep1,

andtherearedozensofresourcesavailabletopreparefortheexam,mostlychosenbyword-of-

mouthrecommendations.Allofthesefactorsplayintoone’sperformanceonStep1,andyet

verylittlehasthusfarbeenstudied.OnlyonestudyatAlbertEinsteinlookedatspecific

resourcesandfoundthatquestionbanksaremorepredictivethanotherresources,andthat

theamountoftimespentstudyinghasnocorrelationwithscores6.

BecauseofthesignificantconsequencesofperformanceonStep1,studiesshowthat

thisexamishighlyrelatedtomedicalstudentburnout.Muchofthisstressarisesfromprograms

teachinginformationthatdoesn’toverlapdirectlywithStep1,with35%ofstudentsreporting

thattheyshifttheirstudyfocusawayfromtheirschools’curricula7.Itwouldthereforebenefit

programstobetterunderstandhowtoguidestudentsinpreparingforthisexamtonotonly

improvescoresonStep1,butalsotoimprovefocusontheschool’suniquecurriculum.Itwould

alsobeusefulforprogramstobeabletoaccuratelyidentifystudentswhoareatriskofpoor

performancesothattheymightbeabletoprovideearlyintervention.Fewstudieshavelooked

5

intodevelopingamathematicalmodel,andevenfewerhavehadsufficientpredictivesuccess,

withonenotableexceptionatMarshallUniversityinDecember20168.Byexpandingonthose

studiesandlookingatawidervarietyofdata,itmaybepossibletoguideotherschoolson

developingindividualizedmodelsthateffectivelyflagat-riskstudents.

DuetotheimportanceoftheUSMLEStep1exam,theprojectaimsinclude:

1. ToidentifyearlypredictorsofperformanceonUSMLEStep1.First,analyzingpre-

matriculationdataofundergraduateGPAandMCAT®scoresanddemographic

factorsmayindicategoodpredictorsofStep1scores.Thisinformationwould

potentiallyallowadmissionsofficestobetterdeterminewhatweightshouldbe

giventothesefactorsindeterminingwhichstudentsareofferedadmission.

2. Toevaluateifsuccessinpre-clinicalcoursesatUTSouthwesterncorrelatetosuccess

onStep1.EvaluatingtherelationofoverallGPAtoexamperformanceand

determiningifspecificcourseorup-trendingordowntrendingscoresthroughout

theyearmayserveaspredictorsforexamscores.

3. Todeterminecorrelationsbetweenspecificresourcesorstudyingstrategiesand

Step1scorestobetterinformstudentsandthemedicalschoolonoptimal

preparationfortheexam.StudentsatUTSouthwesternhavecompletedtwosetsof

surveys–oneduringthededicatedstudyperiodtodetermineintervalscoreson

practicetestsaswellasprogressthroughstudyingmaterials,andthesecondafter

theexamtoretrospectivelydeterminehowmuchtimestudentsspentstudyingand

whichresourcestheyused..Analysisofthesesurveyswillbeperformedtoidentify

6

factorsthatareassociatedwithhigherscoresandmayprovideguidancetofutureat

riskstudents;preventingfailuresorpoorscoresonUSMLEStep1.

4. TodevelopamathematicalmodelthatpredictsperformanceonUSMLEStep1in

orderforadministratorstobeabletoidentifyat-riskstudentsandprovideearly

intervention.

Usingpriorstudiesasaguide,wefirsthypothesizethatpre-admissionsdatawillprovide

onlymildpredictivevalueofStep1scoresandshouldbeusedcautiouslyasadmissionscriteria

andmayexhibitathresholdeffect.Second,performanceinmedicalschoolisdirectlycorrelated

withStep1scores,butdecreasinggradesasstudentsgetclosertotheexamdatewillnot

necessarilypredictperformanceonStep1.Third,studentsusingmoreactiveresourcesand

startstudyingearlywillperformbetterontheexam,butwillseediminishingreturnsaftereach

weekofstudywithscoresmaximizingbyweeks5and6.Finally,thatapredictivemodelcanbe

developedtocategorizestudentsaslow-riskorhigh-risk.

7

METHODS

DataCollectionandSubjectProtection

DatawascollectedfortheUTSouthwesternMedicalSchoolClassof2018(n=238).An

independentdatabrokerwasusedtocompileadatabaseofstudentundergraduateGPA,

MCAT®scores,medicalschooltestscores,ComprehensiveBasicScienceSelf-Assessment

(CBSSA)testscores,anddemographicinformation.Demographicinformationincludedage,sex,

race,yearofundergraduategraduation,priordegrees,concurrentdegrees,leavesofabsence,

andsocioeconomiccategoryonadmission.

Thedatabrokeralsoobtaineddatafrom2surveysthatwerealreadybeingadministered

bytheUTSouthwesternStudentAcademicSupportServicesandtheOfficeofStudentAffairs

viaSurveyMonkey(APPENDIXAandB).Thefirstwasaweeklysurveyduringthededicated6-

weekStep1preparationperiodtodeterminestudent’sscoresontheirpracticetestsand

students’progressintheirstudymaterials.Thesecondsurveywasapost-examsurveywhich

askedstudentswhentheystartedstudying,howmanyweekstheystudied,howmanydaysper

weekandhoursperdaytheystudied,whichresourcestheyused,howmanypracticeteststhey

took,andhowinvolvedtheywerewithextracurricularactivities.Thesesurveyshadstudent

identifiers,whichthedatabrokerwasabletode-identifyusingthesamecodesfromthe

databasedescribedinthepriorparagraphinordertolinkthetwodatabases.Nostudentswere

excludedfromthestudy,butonlystudentswhotookStep1inSummer2016weresurveyed.

Studentswereprovidednoincentivesforsurveycompletion.

8

ThecollectivedatawasassembledbytheresearchteaminanMSExcelspreadsheet

alongwiththestudent’sUSMLEStep1scores.Allstudentswereassignedaparticipantnumber

andtheiridentitieswereprotectedbythe‘databroker’personnelwhomaintainedthe

confidentialrecordoflinksbetweentheuniqueparticipantdataandprojectIDnumber.The

investigatorsandotherlistedteammembersdidnothaveaccesstostudentidentityor

demographicinformation.ThisprojectwasdeemedexemptbytheIRBbecausestudent

identitieswereprotected,anduseofthedatadidnotrequireconsentperFERPA34CFR§

99.319.

Pre-AdmissionsDataAnalysis

ThefirststepofanalysisentailedidentifyingcorrelationsbetweenStep1scoresand

MCAT®,undergraduateGPA,anddemographicdata.Duringthispartoftheanalysis,MSExcel

wasusedtoperformsimplelinearregressiontoidentifyifcorrelationsexisted,withr>0.5

suggestingstrongcorrelation,0.3<r<0.5suggestingmoderatecorrelations,and0.1<r<0.3

suggestingweakcorrelation,perhumanbehaviorstudystandards10.Thiscorrespondedtoour

useofR2>0.25indicatingstrongcorrelationand0.10<R2<0.25indicatingmoderatecorrelation.

P-valuesof<0.05wereconsideredsignificant.

MedicalSchoolGradesandCBSSAScoresDataAnalysis

Asabove,MSExcelwasagainusedtoperformsimplelinearregressiontoidentifyif

correlationsexistedbetweenmedicalschoolgradesorpracticeCBSSAscoresandStep1scores.

Independentgroupst-testswereperformedwithStatPlusinordertobetterunderstand

differencesbetweenthosescoringaboveandbelow220.

9

SurveyDataAnalysis

Fortheweeklyprogresssurveys,simplelinearregressionwasusedinMSExcelto

determineifthereweresignificantcorrelationsbetweenprogressmadeinFirstAid,progress

madeintheQ-bank,andcumulativepercentcorrectintheQ-bankandpracticeexamscoresfor

thatweek.Averagepracticescoresandtheaveragedeltaofpracticescoreswerealsoanalyzed

viasimpleplottingperweekinordertodeterminehowscoreschangedoverthecourseofthe

dedicatedstudyperiod.

Similaranalysiswasperformedonthepost-examsurvey.Toanalyzemedicalschool

gradesandsurveyinformation,linearregressioninMSExcelwasusedtoidentifyifcorrelations

existbetweenStep1scoresandtotalhoursstudied,numberofweeksstudied,hoursof

extracurricularinvolvement,Q-bank%completion,andnumberofpracticeteststaken.

Independentgroupst-testswerealsoperformedtocompareusersofaresourcetonon-users

ofaresourceforeachofthecommonlyusedtest-prepresourcesinordertoidentifyresources

thatresultinhigherStep1scores.

DevelopingaModel

Theprimarygoalofthemodelistopredictifastudentisatriskofscoringunder220on

Step1.Therefore,independentgroupst-testswereperformedinStatPlusinordertoidentify

factorsthatweredifferentbetweenthosewhoscoredbelow220andthosewhoscored220or

higher,withap-valueof<0.05consideredsignificant.Acombinationofthefindingsfromthe

linearregressionmodelsinprioranalysisandtheset-testswereusedtotestthemorehighly

correlatedfactorsviastepwisemultipleregressionanalysisinStatPlusinordertocreatea

predictivemodeltoidentifystudentswhoarelikelytoscorebelowa220.

10

RESULTS

Pre-AdmissionsData

Thedistributionofpre-admissionsscoresforall228studentsintheUT

SouthwesternClassof2018canbereviewedinTable1.EachsectionoftheMCAT®,as

wellasthecombinedbiologicalandphysicalsciencesscoreandthecumulativescore,were

correlatedwithStep1scoresandmedicalschoolgrades(Figures1and2).TheR2values

canbeseeninTable2.Oftheindividualsections,thebiologicalsciencesandphysical

scienceshadR2valuesof0.067and0.095,respectively,indicatingonlyaweakcorrelation

withStep1scores;theverbalreasoningsectionhadanR2valueof0.004,indicatingno

correlation.Thesummationofthebiologicalandphysicalsciencessectionsresultedinan

R2of0.127,indicatingamoderatecorrelationbetweenthesciencesectionsoftheMCAT®

andStep1scores.

Whencorrelatedwithmedicalschoolperformance(cumulativesecondyeargrade

percentage),thecorrelationswithMCAT®scoreswereevenweaker(Figures3and4).

Eventhestrongestpredictor,thesummationofthebiologicalandphysicalsciences

sections,hadanR2valueofonly0.072,indicatingonlyveryweakcorrelationbetween

medicalschoolperformanceandMCAT®scores.

UndergraduateGPAexhibitedsimilartrends,butwereevenlesspredictivethan

MCAT®scores(Figures5and6).AnR2valueof0.104indicatesmoderatecorrelationwith

Step1scores,andanR2of0.058showsonlyminimalcorrelationwithmedicalschool

grades.

11

Ananalysisofdemographicfactorsindicatesthatrace,undergraduatemajor

(sciencevsnon-science),andgenderarenotsignificantlydifferent,asthestep1score

averagesofeachgrouparewithinthemarginoferror(p>0.05foreachrace,maleversus

female,andsciencevsnon-science)(Figure7).Althoughatfirstglanceitseemstherewas

nosignificantdifferenceamongagegroups,afterisolatingonlythoseunderandabove30,

therewasasignificantdifferenceinmeansofalmost8points(237.7vs229.7,p<0.03)

betweenthoseunder30andthose30andolder.Stepscoredifferencesamongthe

socioeconomicgroupswasalsonotsignificant(seeAPPENDIXCforfulldetailsonhow

studentsweresortedintoeachcategory).

MedicalSchoolGradesandCBSSAScores

TheCBSSAexamwasadministeredtoeachstudenttowardstheendofthepre-

clinicalcurriculum,andpriortothededicatedtimeforStep1study.Twohundredelevenof

thestudentsintheclasstooktheexamduringthistimeperiod.Ofthose,theaveragescaled

scoreontheexamwas193(SD±28).TheaveragechangebetweentheCBSSAscaledscore

andtheactualStepScorewas44points(SD±18),witharangefrom-6to+85.Agraphof

Step1scoresversusCBSSAscoresshowsanR2of0.599,indicatingaveryhighcorrelation

(Figure8).Ofallthevariablesstudiedoverthecourseofthisstudy,CBSSAscaledscore

hadthehighestpredictivepoweroffinalStep1scores.

Thesecondmostsignificantpredictivefactorwasmedicalschoolperformance,as

indicatedbythesecond-yearmedicalschoolcumulativegradepercentage(Figure9).217

studentshadcompletedallcoursesandtakentheStep1exam,withanaveragegradeof

84.5%,SD±5.0,rangingfrom69.3%to98.0%.Whensecondyeargradesarebrokendown

byquarter,R2valuesindicatethateachquarterismoreandmorepredictiveofthefinal

12

Step1scores,withtheexceptionofthefinalquarter,whichisfarlesspredictivethanthe

otherthree,butstillshowingastrongcorrelationwithStep1scores(Table3).

Therewasverylittlechangeinscoresbetweeneachquartergradingterm,bothfor

individualsandonaverage(Table4).However,whenplottingjusttheindividualchange

betweenthethirdandfourthquartergradescomparedtoStep1scores,individualswhose

scoresdecreasethemostbetweenthethirdandfourthquartersendupwithhigherStep1

scores(Figure10).AnR2of0.137onthisplotindicatesthatthereisamoderatenegative

correlationbetweenthosewhosegradesdropbetweenQ3andQ4andtheirStep1scores.

Thissamerelationshipisnotseeninthechangebetweenthesecondandthirdquarters.

SurveyData

SASSWeeklySurvey

WeeklysurveysweredistributedduringthededicatedStep1preparationperiod

andadministeredbytheStudentAcademicSupportServices.Thesesurveyshad198

uniquerespondersovera6weekperiod,with174respondersinweek1,taperingto43

respondersinweek7.Becauseidentifierswereused,wehadtheabilitytotrackthesame

individualstudentresponsesoverthe7-weekperiodthatthesurveyswereadministered.

Themostpredictivedatapointonthesesurveyswasthecumulative%correctin

theUWorld™and/orUSMLERx™questionbank,thetwoquestionbankresourcesusedby

everystudentsurveyed.Aplotofthe%correctversusthepracticeexamscoreforthat

weekshowsanR2of0.507,indicatingastrongcorrelation(Figure11).Lesspredictiveand

moderatepredictorsweretheprogressmadethroughtheQ-bank(s)andFirstAid™,

respectively,withR2valuesof0.241and0.200(Figures12and13).Thepercentages

13

includemultiplepassesbutdon’tdistinguishbetweendifferentresources,sosomeonewho

washalfwaythroughasecondpassoftheU-World™Q-bankwouldhave150%,aswould

someonewhocompleted100%oftheU-World™Q-bankand150%ofUSMLERx.

EachconsecutiveweekofstudyproducedsmallergainsinpointsontheUSMLE1.

Figure14showsthatingeneral,scorescontinuetoincreasethroughoutthefirst6weeks

ofstudy,butlessandlesseachweek,withasignificantdropbyWeek7.Itisimportantto

notethatstudentswereonlygiven6weeksofdedicatedpreparationtime,sothosewho

spentlongerthan6weekshadtorequestextratime.Forthefirst6weeks,aquadraticline

ofbestfitwasdeterminedtobe:

StepScore=-0.4365x(#weeks)2+8.5953(#weeks)+201.16

withanR2of0.972,indicatingaverystrongfittothedata.Thisequation’svertexis243.56

at9.87weeks,indicatingthatbetweenweeks6and10,astudentmaypotentiallygain6

points,orlessthan2pointsperweekofstudy.

Figure15includesonlydatapointsforstudentswithtestscoresintwoconsecutive

weeks,andshowsthatwhilestudentscouldgainanaverageof13.9pointsinthefirstweek

(SD15.1),thegainsdropprogressivelyweek-to-week,withanaverageincreaseofjust4.1

pointsbetweenweeks5and6(SD12.0),andmanystudentsnotincreasingoreven

droppingscoresatthatpoint.Thedifferencebetweenweeks6and7wasnotplotted

becausetherewereonly5studentswhosubmittedbothweek6andweek7practiceexam

scores.Alinearbest-fitlineonthisgraphindicatesthatafterweek6,onaveragenofurther

scoreincreasesshouldbeexpected,withanR2valueof0.938indicatingthestrong

predictivestrengthofthismodel.

14

Post-ExamSurvey

Thepost-examsurveyadministeredwithinafewweeksoftheStepexamhad140

responsesoutof228totalstudentsintheclass.Thedistributionofscoresofthosewho

completedthesurveycomparedtotheentireclassisshowninTable5.Thosewho

completedthesurveyscored,onaverage,5pointshigherthantheclassasawhole,witha

lowerstandarddeviationamongsurveyrespondentsindicatingthatasignificant

proportionofthosewhodidn’tcompletethesurveyperformedworseontheexam.

Acomparisonofusersversusnon-usersofeachresourceindicatesthatthosewho

useKaplan™,NBME®practiceexams,UWorld™SelfAssessments,BRS™Physiology,and

USMLERx™scoredhigher(Figure16).ThosewhousedDoctorsinTraining™andSketchy

Pharm™scoredlower.Inthisstudy,onlySketchyPharm™hadp<0.05,sothedifferences

seeninthegrapharenotstatisticallysignificant.

ThenumberofquestionsansweredineithertheUWorld™,USMLERx™,orKaplan™

questionbankhadamildtomoderatecorrelationwithanR2of0.094.However,

consideringthatUWorld™is~2350questions,groupingthedataintothoseanswering

fewerthan50%oftheQ-bankquestions,thoseanswering50-150%,andthoseanswering

greaterthan150%,thesignificanceofdifferencebecomesmoreapparent,especiallyatthe

higherend(Figure17).Studentswhoansweredfewerthan1700questionsscoredan

averageof230±10.6(95%CI),thosewhoanswered1701-4000questionsscoredan

averageof241±2.4(95%CI),andthosewhoansweredgreaterthan4000questions

scoredanaverageof254±5.3(95%CI).

StudentswhostartedstudyingforStep1earlierperformedbetter,althoughthedata

doesn’tsuggestaclearidealstarttime,sincethereisnosignificantdifferencebetweenany

15

oftheadjacentcategoriesbeginning1year,6months,3months,or6weekspriortothe

exam(Figure18).Thosewhostartedtostudy1yearpriortotheexamscoredonaverage

254±8.5(95%CI),thosewhostarted6monthspriorscored252±6.9(95%CI),those

whostarted3monthspriorscored242±3.2(95%CI),andthosewhowaiteduntilthe

dedicated6-weekpreparationtimescored237±3.6(95%CI).

Whenaskedhowmanyhoursstudentsstudiedwithinthe6-weekpreparationtime,

therewerenosignificantdifferencesfoundintermsofnumberofweeksspentstudyingor

totalhoursstudyingoverthecourseofthe6weekperiod(Figures19and20).Statistical

erroraccountedfordifferencesineverycategory.

Somewhatsurprisingly,extracurricularinvolvementappearstohavenoimpacton

Step1scores(Figure21).Infact,thosewhostartedtostudy1yearand6monthspriorto

theexamhadhigherlevelsofextracurricularinvolvementat7.7hoursand8.8hours,

respectively,comparedtothosewhostartedstudyinglaterat6.0and6.3hours.Therewas

alsonosignificantdifferenceintheextracurricularinvolvementandtotalhoursstudied

duringthededicatedstudyperiod.

Studentswereaskedtonamethelastpracticeexamtheytookandthescorethey

receivedpriortotakingtheStep1exam.Figure22showsthatNBME®18isanaccurate

andprecisepredictorofone’sfinalscoreonStep1,withanaveragedifferenceof0.06

points±2.0.UWorld™SelfAssessment2(+2points±5.0,95%CI)andNBME®15(-2.5

points,±7.2)areaccuratebutlessprecise.NBME®13(+5.5points,±4.8,95%CI)and

NBME®17(-4.5points,±3.4,95%CI)arelessaccuratepredictors,andalsonotasprecise.

NBME®14(+1point,±13,95%CI)isimprecise.NBME®16(+3points,±2.9,95%CI)isless

preciseandaccuratethanNBME®18,butisotherwiseagoodpredictor.

16

DevelopingaModeltoPredictStep1Performance

Todevelopamodeltoidentifythoseatriskofscoringbelow220onStep1,t-tests

wereperformedforanumberofcategoriesthatsuggestedcorrelationinthepre-

admissionsdatathesurveys,andthemedicalschoolgradesandCBSSAscoresforthose

studentsscoringbelow220comparedtothosescoring220orhigher.Table6showsthe

averages,standarddeviation,andp-valuesobtainedfromthet-testsforeachtested

variable.

Thet-testsindicatedthat6categorieshadsignificantdifferencesbetweenthe

groups:undergraduateGPA,MCAT®®®score,#ofquestionsansweredintheQ-Bank,total

hoursstudying,CBSSAscore,andcumulativeMS2grades.

Basedontheaveragesandstandarddeviationsshown,aroundedthresholdroughly

1-2standarddeviationsbelowtheaverageforthe<220groupwaschosenandapositive

predictivevalueforselectingastudentwhowouldscore<220foreachofthetestswas

calculated(Table7).Mostnotably,astudentwithaCBSSAscoreoflessthan160hasa64%

chanceofscoringbelow220,andastudentinthebottom15%oftheclasshasa60%

chanceofscoringbelow220.ThenextmostpredictivefactorsweretheundergraduateGPA

andMCAT®scores,withanundergraduateGPAlessthan3.5predictinga45%chanceof

scoringbelow220,andanMCAT®lessthan30witha48%chance.Theleastpredictive

wereansweringfewerthan2000Q-bankquestionsorspendinglessthan250totalhours

studying,withapositivepredictivevalueof27%and30%,respectively.

Usingthesefindings,aseriesofmultiplelinearregressionsweretested,usinga

mixtureofcontinuousvariablesandcategoricalvariablesasseenonTable7.Sincethe

purposeofthemodelwastoflagstudentsatriskofperformingpoorlyearly,thenumberof

17

Q-bankquestionsansweredandtotalhoursstudyingwereleftoutofthemodel,asthose

arefactorsthatwouldn’tbeknownuntiltheendofthestudyperiod.Sincetherewasa

significantdifferenceinMCAT®scoresandundergraduateGPAsusingathreshold,those

factorswereleftasconstantseitheraddedorleftoutofthemodeltoincreasethe

predictivevalue.Finally,theinitialCBSSAandcumulativesecondyeargradepercentage

hadsuchstrongcorrelationsthatthosewereincludedwithlinearcoefficients.Theresultof

themultiplelinearregressionwasthefollowingmodel:

USMLE1Score=100.84962+4.85204(ifUndergraduateGPA<3.5)+3.66261(ifMCAT®<31)+0.31215*[CBSSAScore]+0.8223*[CumulativeMS2Grade%]

TheregressionstatisticsareshowninTable8.UndergraduateGPAandMCAT®

scoresdidnothavesignificantp-valueswhenrunthroughthemodel,buttheyimproved

theoverallpredictivevalueofthemodel,sotheywerehelpfultoinclude.AnR2valueof

0.64indicatesthatthismodelaccountsfor64%ofthevariabilityinthepopulation,anda

standarderrorof8.87indicatesthatthismodelwillpredict67%ofscoreswithin8.87

points,and95%ofscoreswithin17.74points.

Whentestingthismodeltoidentifythoseatriskofscoringbelow220,thestandard

errorwasusedastheupperendofthemodel.Thatis,anystudentpredictedbythemodel

toscore228.87orlowerwasconsideredapositivetest.Whenappliedtothegroupasa

whole(186studentshadalldatapointsnecessarytorunthismodel),thisresultedina

sensitivityof81%(25/31),specificityof86%(134/155),positivepredictivevalueof54%

(25/46),andnegativepredictivevalueof96%(134/140).Thatis,eventhoughthemodel

onlypredicts67%ofscoreswithinthestandarderror,usingthistoflagstudentscould

18

catch81%ofthosewhoultimatelyscoredbelow220.Ifastudentwasfoundtoscore

higherthana228.87bythemodel,thenthereisa96%chancethatthestudentwillnot

scorebelow220.Ofnote,38%ofthefalsepositives(8/21)scored225orbelow.

19

DISCUSSION

Pre-AdmissionsData

AlthoughsignificantweightistraditionallyplacedonMCAT®andundergraduate

GPAs,clearlyonlyamild-to-moderatecorrelationexistsbetweenStep1scoresandMCAT®

scores,onlyweakcorrelationsbetweenStepscoresandundergraduateGPAsandeven

weakercorrelationsbetweenMCAT®orGPAstomedicalschoolperformance.Theverbal

sectionoftheMCAT®isparticularlypooratpredictinganylevelofperformanceinmedical

school.However,thesetwofactorsarestillthebestmeasuresavailabletoeasilysort

throughapplicants.Infact,afterputtingtogetheramodeltopredictperformance,itisclear

thatthosewithMCAT®scores30orbelow,orGPAslowerthan3.5areatasignificantrisk

ofperformingpoorlyonStep1.ThisdatamaythereforebeskewedbythefactthatUT

SouthwesternstudentshavegenerallyveryhighMCAT®scoresandundergraduateGPAs.

Thisindicatesathresholdeffect–ifastudentscoresabovea30ontheMCAT®andhas

greaterthana3.5GPA,perhapsnoadditionalconsiderationshouldbemadewithregards

tothosescores.However,thisdatamaysuggestthatadmissionspersonnelbewaryof

studentswhodonotmeetthosethresholds.

Race,undergraduatemajor,socioeconomicclass,age,andgenderhavenosignificant

effectonStep1scores,withanydifferencesofaverageswellwithinthemarginoferror.

Studentscertainlyreportbenefitingfromadiverseclassintheannualgraduation

questionnaire,andampleresearchsuggeststhemanybenefitsofdiverseexposuresand

experiences11.Admissionscommitteesmayconsiderusingthesefactorstohelpcreatea

20

diverseclasstoimprovethelearningatmosphereasawhole,butdemographicfactorshave

nobearingonhowastudentislikelytoperformonStep1.

MedicalSchoolGradesandCBSSAScores

ItisclearthatmedicalschoolgradesandCBSSAscoresarethestrongestpredictors

ofStep1performance,bothshowingveryhighcorrelations.Althoughmanystudentshave

atendencytoshifttheirfocusawayfrommedicalschoolexamsastheybegintopreparefor

Step1,itshouldbemadecleartostudentsthat,atleastatUTSouthwestern,doingwellin

courseswilltranslatedirectlytoperformingwellonStep1.

CBSSAscoresarethesinglehighestpredictorofStep1scores.Thisiseasyto

understand,asthisexamisthesameformatandstyleoftheactualexam,andisafair

assessmentofthebaselineknowledgeofamedicalstudentbeforegoingintoadedicated

preparationperiod.Becausestudentsareallgivenessentiallythesameamountoftime,it

makessensethatthosewhostartaheadwilltendtostayahead.Thatsaid,thereis

significantvariationinhowmuchstudentscanlearnoverthat4-6weekperiod,withsome

studentsincreasingtheirCBSSAscoresbyasmuchas85points.Thosescoringbelow160

(scaledStepscore)haveasignificantriskofdoingpoorlyontheexamandmaywarrant

moreinterventionearlyon.

Itwassurprisingtoseethatstudents’gradesdidnotvarysignificantlybetweenQ2

(Quarter2)andQ3,andevenstudentswhosegradesdecreased,presumablybecausethey

startedtoshiftfocustowardsstep,didnotperformbetteronStep1.Thatwasnottrue

betweenQ3andQ4.OutofallofthesecondyeargradesQ4gradeswerebyfartheleast

predictiveofStepscores,andstudentswhodroppedbetweenQ3andQ4tendedtoscore

higheronStep1.Thismaybeexplainedbyselectionbiasandconfoundingfactors.Students

21

whocouldaffordto“slackoff”anddroptheirQ4gradesweremostlystudentswhowere

notalreadystruggling.Furthermore,bytheendofQ3,UTSouthwestern’scurriculumhad

coveredabulkofthecoreStep1material,withonlydermatology,psychiatry,and

integratedsystemslefttocoverinQ4.Becausethosetopicsareonlyasmallpercentageof

whatiscoveredonStep1,itispossiblethatstudentsbenefitedbystudyingmorehigh-yield

Step1topicsduringthatquarter,whichcouldresultinhigherscores.Thesearethesame

studentswhowouldbeexpectedtohavestartedstudyingbetween3monthsand6months

priortotheexam,whichwasalsocorrelatedwithhigherStep1scores.

Overall,theconclusionremainsthatdoingwellonmedicalschoolexamswill

translatetodoingwellonStep1,somethingadministratorsandeducatorswillappreciate.

However,itispossiblethatasonegetsclosertotheexam,theremaybesomebenefitto

shiftingfocusawayfromschoolandtowardstheexam,particularlyiftheschoolcurriculum

leaveslower-yieldtopicsfortheendoftheyear.

SurveyData

Althoughthesurveyparticipantsscoredhigherthantheclassasawhole,the

surveysprovideusefulinformationtoguidepreparationstrategies.

Inthecomparisonofusersofresourcesversusnon-users,noneofthedatapoints

werestatisticallysignificantexceptthatusersofSketchyPharm®performedsignificantly

worsethannon-users.However,thisislikelyaresultofthepowerofthisportionofthe

study.Ingeneral,itcanstillbeconcludedthatusersofquestion-basedresources–Kaplan™

QBank,NBME®assessments,UWorld™SelfAssessments,andUSMLERx™–performed

betteronexamsthannon-users.Thismayindicatethatthebestpreparationstrategiesare

toanswerasmanyquestionsaspossible.Thisisfurthersupportedbythefactthatthose

22

whoansweredmorethan4000Q-Bankquestionsoutperformedthosewhoanswered

fewer.UsersofthetwovideoresourcesDoctorsinTraining™andSketchyPharm™were

theworstperformers,whichmaybeduetothefactthatvideoresourcestakealongtimeto

gothroughandimpedetheabilitytodomorepracticequestions.

Lookingattheweeklysurveys,thefactorsmostcorrelatedwithhighpracticeexam

scoresarethecumulative%correctintheQ-bankandprogressthroughtheQ-bank,with

progressthroughFirstAid™alsocorrelatedwithhigherscoresbutlessso.Thissupports

theassertionthatquestionbankresourcesarethehighest-yieldresourcesinpreparation

forthisexam.

Thisdataalsoprovidesveryusefulinformationinguidingstudentsonwhentostart

preparingandforhowlong.StudentsatUTSouthwesternwereallgiven6weeksofffor

dedicatedstudy,andmoststudentsdidn’tstartintensestudyinguntilthistime.However,it

isclearthatthosewhostartedstudying1yearor6monthspriortotheexamperformed

betterthanthosewhostartedstudyingeither3monthsprioror6weeksprior.Thatsaid,

oncethededicatedstudyperiodstarted,theresultswerelessclear,astherewasno

significantdifferenceinscoresamongdifferentweeksstudyingortotalhoursstudying

duringthededicatedpreparationperiod.Theaveragescoresforthosewhostudiedfewer

than250totalhoursandthosewhostudiedover600hourswerewithinthemarginof

error,alongwitheachintervalinbetween.Moreimportant,itseems,wasthelevelatwhich

studentscameintothepreparationperiod.Thatsaid,therewasasignificantdifferencein

hoursstudiedbetweenthosewhoscoredbelow220andthosewhoscored220orhigher,

soitwouldnotbeunreasonabletosuggestthatstudentsshouldstillaimtostudy300

hours,whichis1standarddeviationbelowtheaverageforthosewhoscored220orhigher.

23

Thiscorrelatestoroughly50-60hoursperweekfor5-6weeks.Ofnote,thiswas

retrospectivedataandmaynotaccuratelyrepresenttheexacttimestudying,asitmay

includetimespentonbreaks,stretching,eating,etc.

Thisnumberisfurthersupportedbytheweeklysurveys.Itisclearthatstudents

havediminishingreturnsaftereachweekofstudy.Fittingaquadraticfunctiontothefirst

sixweeks(week7wasanoutlierbecausethesewerestudentswhorequestedextratime

andwerethelowestperformers)suggeststhatafterWeek6,studentsshouldexpect,on

average,nomorethan1-2pointsperweek,andeveniftheystudy10weeks,theyare

unlikelytogetmorethan6points.Bylookingatjustthedelta,thelineofbestfitiseven

moreconservative,statingthatstudentsareunlikelytomakeanyincreaseinscoresafter

week6.Ofcourse,thereareexceptions,butformoststudents,itappearsthatscoresmay

starttoeitherstabilizeorevendecreaseafterweek6.Giventhisdata,itappearsthat5-6

weeksisoptimalfordedicatedpreparationtimeamongstudentsatUTSouthwestern.

DevelopingaModeltoPredictStep1Scores

Therewereanumberoffactorsthatdidnotshowstrongcorrelationswhenlooking

attheentiregroupofstudents,butwhenonlycomparingthosewhoscored<220tothose

whoscored220orhigher,afewmoresignificantrelationshipsemerge.Undergraduate

GPAsandMCAT®scores,asexpected,weresignificantlydifferentbetweenthetwogroups.

Alsosignificantwerethenumberofquestionsansweredandthetotalhoursstudying,both

withpvalues<0.02.Thissuggeststhatalthoughonthewholethecorrelationmaybeweak,

thosewhoperformbettermeetatleastacertainthresholdofstudying.Theothernotable

significantfactorsaretheCBSSAscoresandcumulativeMS2grades,alsoasexpectedbased

onthepriordata.

24

Themodeldevelopedcanserveasaveryusefulresourceforeducatorstoflag

studentsatriskofpoorperformanceonStep1,definedasscoringbelowa220.Itusesa

thresholdforundergraduateGPAandMCAT®scores,withpointsdeductedifastudenthas

lowerthana3.5orscores30orbelowontheMCAT®,anditappliesacoefficienttothe

students’CBSSAscoreandmedicalschoolgrades.TheundergraduateGPAandMCAT®

don’tindividuallyhavelowp-valuesinthemodelandcontributeminimallycomparedto

theCBSSAandmedicalschoolgrades,buttheydostillimprovethepredictivevalueofthe

model.

Themodelisabletopredict67%ofscoreswithin8.87points,and95%ofscores

within17.39points.Moreimportantly,though,itcanserveasagoodwaytoflagstudents

atriskofscoringpoorlyontheexam.Ifweusethe67%confidenceintervalandmarkthe

upperlimitofthemodelat228.87,itwillcatch81%ofthosewhowouldscorebelow220

with86%specificity.Italsohasanegativepredictivevalueof96%,soitcanalsoalleviate

concernsaboutamajorityoftheremainderofthestudentbody.Ofnote,althoughthe

modelcanserveasausefulresourceforeducatorsandadministratorstoflagat-risk

students,individualstudentsshoulduseitwithcaution.Someofthosewhoarepredicted

toscorelowmayverywelldoverywellontheexam,justasthosepredictedtoperform

wellmaydopoorlyontheexam.Themodelisausefulguide,butitdoesnotpredictdestiny.

25

CONCLUSION

Thisprojectdemonstratesthatobjectivedataanalysiscanbeusedtoexplain

studyingstrategiesandtoguidestudentstowardsoptimalpreparationforStep1.Thesame

datacanbeusedbyeducatorstoflagstudentsatriskofpoorperformancetointervene

early.

Thefirstaimoftheprojectwastoanalyzepre-matriculationdata.AtUT

Southwestern,itwasshownthatmostpre-matriculationdataserveasonlyweak

predictorsofmedicalschoolorUSMLE®examperformance,andthatdemographic

informationhasnoinfluenceonperformanceinmedicalschool.However,thereisacertain

thresholdthatindicatesbasiccompetence,underwhichstudentswillbeatriskofpoor

medicalschoolperformance.Thissuggeststhatmedicalschooladmissionsofficersshould

useGPAandMCAT®scorestomakesurestudentsreachacertainthreshold,butbe

cautiousaboutplacingtoomuchweightonthesecharacteristicsbeyondthatpointandlook

insteadtootheraspectsoftheapplications.Futurestudiesinthisareacouldfurther

explorethepredictivepowerofMCAT®andGPAsforamorepowerfulstudy,andalso

exploreotheraspectsthatmaybeusefulinsortingapplications.

Thesecondaimoftheprojectwastodetermineifperformanceinmedicalschool

correlatedtoperformanceontheStep1exam.Itisabundantlyclearthatmedicalschool

gradesaresomeofthestrongestpredictorsofStep1performance,sostudentswho

performwellinmedicalschoolshouldbeexpectedtoperformwellonboardexams.

However,itisimportanttonotethesmallcaveatshowninstudentswhoshiftedtheirfocus

inthelastfewmonthsoftheschoolyearduringlesshigh-yieldtopics–thosewhosescores

26

wentdowninthelastquarterperformedbetteronStep1.Althoughtheimplicationsofthis

dataislimitedatthispoint,asthisclasswasthefirsttoswitchtopass/failatUT

Southwesternandthecurriculumhasbeensincereformedforfutureclasses,this

highlightstheimportanceofensuringthatthemedicalschoolcurriculumremainshighly

relevantthroughoutthepreclinicalyears,andperhapspointstowardsaneedformore

Step-focusedcourseworktowardstheendofthepre-clinicalyears.

Thethirdgoalofthisprojectwastoidentifyifthesurveysmightprovideuseful

studyingtips.Thedataindicatesthatthebestwaytopreparefortheseboardexamsistogo

throughasmanypracticequestionsaspossible.Thedataalsoindicatesthatstudying

earlierwillleadtohigherscores,butthatoncethededicatedpreparationperiodhasbegun,

5-6weeksseemsoptimal.Twodistinctmodelsbothshowthatscoreswillincreaseonly

minimally,ifatall,afterthe5-6weekmark,sostudyingfurtherthanthatamountwouldbe

apooruseoftime.Furtherstudiesshouldlookintonewresources,aswellasaimformore

powerforeachresourceinordertobetterdeterminewhatthebestresourcestrulyare.

Finally,thisprojectaimedtodevelopamodeltocatchthoseatriskofscoringbelow

220onStep1.UtilizingathresholdofundergraduateGPAandMCAT®scores,anda

coefficientmultiplierformedicalschoolgradesandinitialCBSSAscores,amodelwas

developedwithsignificantpredictivepotential.Thisprojectshowsthatamathematical

screeningtoolmightbeabletoresultintangiblesuccessbyvirtueofpreventingfailurefor

asignificantnumberofatriskstudents.Nextstepsincludevalidatingthismodelacross

differentclassesatUTSouthwesternandacrossmultiplecampusesbeforeitcanbe

confidentlyusedtostratifystudents,butitatleastprovidesguidanceonwhatmedical

schooleducatorsandadministratorsshouldlookforastheypreparestudentsforStep1

27

exams.Futurestudiesshouldalsodetermineifearlyinterventionwillimprovetheoutcome

forat-riskstudents,becauseitispossiblethatthesestudentsmaynotperformsignificantly

higherevenwithappropriateinterventions.

28

LISTOFTABLES

Table1.DistributionofPre-AdmissionsScoresforUTSouthwesternClassof2018.

Variable Average StandardDeviation

MCAT®Score 33.79 2.94

Undergraduate

GPA

3.83 0.18

Table2.R2valuesforMCAT®sectionscomparedtoStep1scoresandMedicalSchoolGrades

MCAT®Section R2forStep1Scores R2forMedicalSchoolGrades

BiologicalScience(BS) 0.06717 0.06577

PhysicalScience(PS) 0.09543 0.02945

VerbalReasoning(VR) 0.00041 0.00081

BS+PS 0.12715 0.07188

TotalScore 0.10168 0.05981

Table3.CorrelationstatisticsforsecondyearquartertermgradescomparedtoStep1scores.

GradingTerm R2forStep1Scores

Quarter1 0.4425

Quarter2 0.45742

Quarter3 0.4886

Quarter4 0.29051

Cumulative 0.52221

29

Table4.DistributionandcorrelationstatisticsforgradechangebetweensecondyearquartertermgradescomparedtoStep1scores.Terms Average

Grade

Change(%)

Standard

Deviation

R2forStep1Scores

Q1-Q2 -2.36 3.32 0.13717

Q2-Q3 1.48 3.26 0.0088

Q3-Q4 2.81 3.52 0.0184

Table5.DistributionofscoresofsurveypopulationversusactualpopulationofUTSouthwesternClassof2018 Survey Actual

NumberofStudents 140 228

Average 242 237

StandardDeviation 14 17.5

Min 202 169

Max 272 272

30

Table6.Distributionandt-testsstatisticscomparingstudentswhoscored<220versusthosewhoscored220orhigheronStep1 <220 220+

Category AVG SD AVG SD p-value

UgrdGPA 3.71 0.29 3.83 0.184 0.01741MCAT® 31.77 2.94 34.15 2.55 0.00003#Questions 2173.33 568.48 2527.49 686.76 0.01838#WeeksStudying 5.73 0.729 5.73 0.928 0.4937#DaysperWeek 6.47 0.48 6.54 0.6 0.28827TotalHoursStudying 350 109 420 120 0.01804PassesThroughFirst

Aid

2.09 0.99 2.46 1.02 0.097

NumberofPractice

ExamsTaken

4.2 1.47 4.75 1.47 0.09245

CBSSAScore 159 14 199 25 0Changebetween

NBMEandStep

scores

46.4 15.8 43.2 18.5 0.15894

CumulativeMS2

Grades%

78.98 3.84 85.51 4.52 1.42E-11

31

Table7.Positivepredictivevalueforscoring<220onStep1forvariousriskfactors

RiskFactorsofScoring<220

Postitive

Predictive

Value

UndergraduateGPA<3.5 45%

MCAT®<31 48%

<2000Q-bankquestionsanswered 27%

<250totalhoursstudied 30%

InitialCBSSAscaledscore<160 64%

CumulativeGrades<80%(inbottom15%of

class) 60%

32

Table8.RegressionstatisticsformultiplelinearregressionmodelforpredictingUSMLEStep1scoresRegressionStatistics

R 0.80236

R-square 0.64379

StandardError 8.86639

N 142

USMLE®Step1Score=100.84962+4.85204(ifUndergraduateGPA<3.5)

+3.66261(ifMCAT®<31)+0.31215*[CBSSAScore]+0.8223*

[CumulativeMS2Grade%]

Coefficient StandardError p-level H0(5%)

Intercept 100.84962 14.81136 2.83E-

10

rejected

UndergraduateGPA 4.85204 4.70029 0.30376 accepted

MCAT®Score 3.66261 2.98453 0.22185 accepted

CBSSAScaledStep1Score 0.31215 0.04042 2.15E-

12

rejected

CumulativeMS2Grade% 0.8223 0.22227 0.00031 rejected

33

LISTOFFIGURESNote:Errorbarsonallgraphsindicatea95%confidenceinterval.

Figure1.CorrelatingUSMLEStep1scoreswithMCAT®Scores.R2indicatesmoderatecorrelation.

Figure2.CorrelatingUSMLEStep1scoreswithsumofBiologicalSciencesandPhysicalSciencesMCAT®Scores.R2indicatesmoderatecorrelation.

R²=0.10168

160

180

200

220

240

260

280

25 27 29 31 33 35 37 39 41 43

Step1Score

MCATScore

Step1ScorevsMCAT

R²=0.12715

160

180

200

220

240

260

280

15 17 19 21 23 25 27 29 31

Step1Score

MCATBS+PSScore

Step1ScorevsMCATBS+PS

34

Figure3.CorrelatingsecondyearmedicalschoolgradeswithMCAT®Scores.R2indicatesweakcorrelation.

Figure4.CorrelatingsecondyearmedicalschoolgradeswiththesumofBiologicalSciencesandPhysicalSciencesMCAT®Score.R2indicatesweakcorrelation.

R²=0.05981

6065707580859095100

25 27 29 31 33 35 37 39 41 43

MS2Cum

ulativeAverage(%

)

MCATScore

MS2GradesvsMCAT

R²=0.07188

6065707580859095100

15 17 19 21 23 25 27 29 31

CumulativeGrade%

MCATBS+PSScore

MS2GradesvsMCATBS+PS

35

Figure5.CorrelatingStep1scoreswithundergraduateGPA.R2indicatesmoderatecorrelation.

Figure6.CorrelatingsecondyearmedicalschoolgradeswithundergraduateGPA.R2indicatesweakcorrelation.

R²=0.10355

160

180

200

220

240

260

280

2.60 2.80 3.00 3.20 3.40 3.60 3.80 4.00

Step1Score

UndergraduateGPA

Step1ScorevsUndergraduateGPA

R²=0.05751

6065707580859095100

2.60 2.80 3.00 3.20 3.40 3.60 3.80 4.00

MS2Cum

ulativeAverage(%

)

UndergraduateGPA

MedicalSchoolGradesvsUndergraduateGPA

36

Figure7.AverageStep1scoressplitbydemographicfactors.(A)showsaveragesamongdifferentraces,(B)showsscienceversusnon-sciencemajors,(C)showsaveragesamongdifferentagegroups,(D)showsaveragessplitbygender,and(E)showssplitbasedonsocioeconomiccategories,withAbeinglowsocioeconomicstatusandDindicatingaffluentfamilybackground.

200

220

240

260

White Asian Hispanic BlackAverageStepScore

Race

RaceandStepScores

225230235240245

Science NotScience

AverageStep1Score

UndergraduateMajor

UndergraduateMajorvsStepScore

210220230240250

<26 26-29 30+

StepScore

Age

EffectsofAgeonStepScore

225230235240245

M F

AverageStepScore

Gender

EffectsofGenderonStepScores

180200220240260

A B C D

AverageStep1Score

SocioeconomicClass

EffectsofSocioeconomicClassonAverageStepScore

A B

C D

E

37

Figure8.CorrelatingStep1scorewithinitialCBSSAscaledscore.R2indicatesverystrongcorrelation.

Figure9.CorrelatingStep1scorewithsecondyearmedicalschoolcumulativegradeaverage.R2indicatesverystrongcorrelation.

R²=0.59936

160180200220240260280300

120 140 160 180 200 220 240 260 280

Step1Score

InitialCBSSAScore(PredictedStepScore)

Step1ScorevsCBSSAScore

R²=0.52221

160180200220240260280

65 70 75 80 85 90 95 100

Step1Score

MS2CumulativeGradeAverage(%)

Step1ScorevsMS2CumulativeGradeAverage

38

Figure10.Correlationbetweenthechangeingradesbetweenthirdquarterandfourthquartergradingtermsandastudent’sStep1score.ThissuggeststhatstudentswhosegradesdropbetweenQ3andQ4performbetterontheexam,withR2indicatingmoderatecorrelation.

Figure11.Correlatingsurveydataofcumulative%correctintheUWorld,Kaplan,orUSMLERxQ-bankswithpracticeexamscoresinthesameweek.R2indicatesstrongcorrelation.

R²=0.13717

160

180

200

220

240

260

280

-10 -5 0 5 10 15

Step1Score

DeltaQ3toQ4(%points)

Step1ScorevsChangeinQ3toQ4Grades

R²=0.50715

140160180200220240260280

30 40 50 60 70 80 90 100

PracticeExamScore

Cumulative%CorrectinQ-Bank

PracticeExamScoresvsCumulative%CorrectinQ-Bank

39

Figure12.CorrelatingsurveydataofprogressthroughUWorld,Kaplan,and/orUSMLERxQ-bankwithpracticeexamscoresinthesameweek.150%canindicateeitherfinishingtheQ-bankonceandrepeating50%,orfinishing1fullQ-bankand50%ofanother.R2indicatesmoderate/strongcorrelation.

Figure13.CorrelatingsurveydataofprogressthroughFirstAidwithpracticeexamscoresinthesameweek.150%canindicateeitherfinishingtheQ-bankonceandrepeating50%,orfinishing1fullQ-bankand50%ofanother.R2indicatesmoderatecorrelation.

R²=0.24149

130150170190210230250270290

0 50 100 150 200 250

PracticeExamScore

ProgressThroughQ-Bank(%)

PracticeExamScoresvsProgressThroughQ-bank

R²=0.19987

130150170190210230250270290

0 50 100 150 200 250 300 350 400 450 500

PracticeExamScore

ProgressThroughFirstAid(%)

PracticeExamScoresvsProgressThroughFirstAid

40

Figure14.Surveydatashowingaveragepracticeexamscoresforeachweekofthesurvey.Aquadraticbest-fitfunctionforjustweeks1through6issuperimposed,withtheR2indicatingveryhighcorrelationbetweenthemodelequationandthedata.Week7wasanoutlierbecausethereweresofewstudents,allofwhichspecificallyrequestedextratimetostudy.

y=-0.4365x2+8.5953x+201.16R²=0.97238

180

190

200

210

220

230

240

250

260

1 2 3 4 5 6 7

PracticeExam

Scores

WeeksofStudy

AveragePracticeExamScoresover7weeksofStudy

41

Figure15.Surveydatashowingthechangeinscoresfromoneweektothenext.Onlyconsecutivedatapointswereincluded.Alinearbest-fitfunctionforweeks1through6issuperimposed,withtheR2indicatingveryhighcorrelationbetweenthemodelequationandthedata.Week7wasanoutlierbecausetherewereonly5studentswithconsecutivedatapoints,allofwhichspecificallyrequestedextratimetostudy.

13.88

11.79

8.36

3.86 4.08

y=-2.7525x+16.652R²=0.93793

0

5

10

15

20

25

Weeks1-2 Weeks2-3 Weeks3-4 Weeks4-5 Weeks5-6

ChangeinPracticeExam

Scores

WeeksofStudy

AverageΔinPracticeScoresBetweenWeeks

42

Figure16.Averagescoresofusersversusnon-usersofeachresource.OnlySketchyPharmhadp<0.05.Asterisksindicatefewerthan10usersofaresource.KaplanQ-bank,Firecracker,andUSMLERxarequestionbanks.NBMEandUWorldSelfAssessmentsarepracticeexams.DIT,SketchyMicro,andSketchyPharmarevideo-basedresources.RapidReviewandBRSaretexts.

Figure17.AverageStep1scorebrokendownbynumberofQ-bankquestionsanswered.ThefullUWorldQbank,themostcommonlyusedresource,isroughly2350questions.1700thenindicateslessthan50%oftheQ-bankand4000indicates150%.

-8

-6

-4

-2

0

2

4

6

8

KaplanQBank*

NBM

EComprehensive

BasicScienceSelf

UWorldSelf

Assessments

Doctorsin

Training*

Flashcard

programs/Anki

Firecracker

RapidReview

Pathologyby

Goljan

SketchyM

icro

SketchyPharm

BRSPhysiology

USMLERx

AverageScoreDifference

ScoresofUsersvsNon-UsersofResources

190200210220230240250260270

<1700 1701-4000 4001+

AverageStep1Score

#ofQuestions

StepScorevs#ofQ-BankQuestionsAnswered

43

Figure18.AverageStep1scoresbasedonhowmuchpriortotakingtheexamthestudentbegan“intense”studyforStep1.6monthscorrespondedtowinterbreak,and3monthscorrespondedtospringbreak.6weekspriorwasthedefault,asthatwashowmuchtimeeachstudentwasguaranteedfordedicatedstudytime.

Figure19.Averagestepscoresbrokendownbynumberofweeksofdedicatedstudy.

215220225230235240245250255260265270

1yearprior 6monthsprior 3monthsprior 6weeksprior

AverageStep1Score

WhenIntenseStudyBegan

WhentoStartStudyingforStep1

200

210

220

230

240

250

260

4 4.5 5 5.5 6 6.5 7 >7

AverageStep1Score

#WeeksStudying

#WeeksStudyingvsStepScore

44

Figure20.Averagestepscoresbrokendownbytotalhoursstudyingduringthededicatedstudyperiod.

Figure21.AverageStep1scoresbrokendownbylevelofextracurricularinvolvement.

205210215220225230235240245250255

<250 251-300 301-350 351-400 401-450 451-500 501-550 551-600 601+

AverageStep1Score

TotalHoursStudying

TotalHoursStudyingvsStepScore

200210220230240250260270

Signi]icantlylessinvolved(0-1hr)

Somewhatlessinvolved(2-4hr)

Average(4-6hr) Somewhatmoreinvolved(7-9hr)

Signi]icantlymoreinvolved(10+hr)

AverageStep1Score

ExtracurricularInvolvementvsScores

45

Figure22.Averagepredictivevalueofpracticetestscomparedtotheactualexamscore.Theseexamsweremostlytakenwithinaweekoftheactualexam.Errorbarsindicate95%confidenceinterval.

-15

-10

-5

0

5

10

15

Uworld2 NBME13 NBME14 NBME15 NBME16 NBME17 NBME18

AverageScorePrediction

PredictiveValueofPracticeTests(ComparedtoActualUSMLEScore)

46

REFERENCES

1. NationalResidentMatchingProgram.(2016,June).Resultsofthe2016NRMP

ProgramDirectorSurvey.http://www.nrmp.org/.RetrievedFebruary1,2016.

2. London,D.A.,Kwon,R.,Atluru,A.,Maurer,K.,Ben-Ari,R.,&Schaff,P.B.(2016).More

onHowUSMLEStep1ScoresAreChallengingAcademicMedicine.AcademicMedicine,

91(5),609-610.doi:10.1097/acm.0000000000001159

3. NationalResidentMatchingProgram.(2016,June).Resultsofthe2016NRMP

ProgramDirectorSurvey.http://www.nrmp.org/.RetrievedFebruary1,2016.

4. SearchtheAMAResidency&FellowshipDatabase™.(n.d.).RetrievedMarch25,

2017,fromhttps://www.ama-assn.org/life-career/search-ama-residency-

fellowship-database

5. Kleshinski,J.,Khuder,S.A.,Shapiro,J.I.etal.AdvinHealthSciEduc(2009)14:69.

doi:10.1007/s10459-007-9087-x

6. Johnson,J.,Jordan,E.,Burton,W.,&Silhiger,S.(n.d.).AreQuestionstheAnswer?The

EffectofPopularStudyResourcesonUSMLEStep1Performance.Lecture.Retrieved

March10,2017,fromhttps://members.aamc.org/eweb/

7. Prober,CharlesG.MD;Kolars,JosephC.MD;First,LewisR.MD;Melnick,DonaldE.

MD.(2016).APleatoReassesstheRoleofUnitedStatesMedicalLicensing

47

ExaminationStep1ScoresinResidencySelection.AcademicMedicine,91(1),12-15.

doi:10.1097/ACM.0000000000000855

8. Gullo,C.A.,Mccarthy,M.J.,Shapiro,J.I.,&Miller,B.L.(2015).PredictingMedical

StudentSuccessonLicensureExams.MedicalScienceEducator,25(4),447-453.

doi:10.1007/s40670-015-0179-6

9. 34CFR99.31-Underwhatconditionsispriorconsentnotrequiredtodisclose

information?CornellLII.(n.d.).RetrievedMarch23,2017,from

https://www.law.cornell.edu/cfr/text/34/99.31

10. PearsonProduct-MomentCorrelation.(n.d.).RetrievedMarch23,2017,from

https://statistics.laerd.com/statistical-guides/pearson-correlation-coefficient-

statistical-guide.php

11. MedicalSchoolGraduationQuestionnaire2016AllSchoolsSummaryReport.(Rep.).

(2016,June).RetrievedMarch23,2017,fromAAMCwebsite:www.aamc.org/

48

VITAE

SachinShah(July301991-present)isafourthyearmedicalstudentatUTSouthwestern

MedicalSchoolandwillbestartinghispediatricresidencyatUTSouthwesterninJune.He

grewupinPlano,Texas,andwasaEugeneMcDermottScholarasanundergraduate

studentattheUniversityofTexasatDallas.HegraduatedwithaB.S.inBiologyin2013and

immediatelybeganmedicalschool.Inmedicalschool,SachinservedasCo-Presidentofhis

medicalschoolclassforallfouryearsandwasdrawntomedicaleducation.Sachinwill

graduatewithhisM.D.andadistinctioninMedicalEducationinJune2017,andhehopesto

pursueacareerasanattendingphysicianatanacademicteachinghospital.

PermanentAddress:

8024GrandCanyonDr.

Plano,TX75025

49

APPENDIX

APPENDIXA–WEEKLYSTUDENTACADEMICSUPPORTSERVICESSURVEY

APPENDIXB–POST-EXAM1STUDENTAFFAIRSSURVEY

5/2/16, 9:54 PM[SURVEY PREVIEW MODE] Step 1 Preparation 2016 Survey

Page 1 of 12https://www.surveymonkey.com/r/Preview/?sm=WX8nu_2FGtw2v2ITfa…ngPSRUCRFUZYyi1O4jb4DqlthKzf_2B5ndVsIVd3jve_2FT21aCo0sNUYHJ0C

Thank you so much for participating in this survey. Your participation will really help futurestudents. We are asking for your name in order to link these responses to other data includingthe weekly surveys you filled out during the dedicated study period. Per IRB protocol, onlyCarol Wortham will be able to link a person to a survey response and will remove all identifiersbefore providing the research team with a completely de-identified data set. If you prefer not tohave your survey responses linked to other de-identified information, you can skip questionone. You also don’t have to answer any question with which you feel uncomfortable.

Thanks again for your time!

1. Step 1 Preparation

Step 1 Preparation 2016

Exit this survey

1. Please select your name from the drop-down list.

2. Which USMLE Step 1 materials/courses did you use?

First Aid for Step 1

USMLE World

Kaplan Q Bank

NBME Comprehensive Basic Science Self Assessments (CBSSA)

UWorld Self Assessments

Doctors in Training Course

Pathoma

Picmonic

Flashcard programs/ AnkiPREVIEW & TEST


Page 2 of 12https://www.surveymonkey.com/r/Preview/?sm=WX8nu_2FGtw2v2ITfa…gPSRUCRFUZYyi1O4jb4DqlthKzf_2B5ndVsIVd3jve_2FT21aCo0sNUYHJ0C

Firecracker

Rapid Review Pathology by Goljan

SketchyMicro

SketchyPharm

Med Bullets

Online MedEd

BRS Physiology

Osmosis

Other (please specify)

3. How did you decide what resources to use?

4. How many Kaplan Qbank or USMLE World questions did you complete? (UWorld has2350 questions. You can estimate if unsure.)

5. When in the MS2 year did you start serious (weekly) review/study for Step 1?

1st Semester

January

Spring Break

After Second Year



6. How many weeks did you study full-time for this exam, after MS2 classes ended?

7. When studying full-time, on average how many days per week did you study?

8. When studying full-time, on average how many hours per day did you study?

9. How many passes through First Aid did you make? (Enter 0 if you did not use.)

Why, or why not?

10. Did you find the NBME Comprehensive Basic Science Self Assessment (CBSSA)given after spring break helpful?

Yes

No

11. How many practice exams did you take in addition to the one the school provided?

12. What was the FINAL practice exam that you took? (NBME #, UWorld SelfAssessment #)

13. Did your practice test over- or underestimate your score?



By how many points?

Underestimated

Accurate estimate

Overestimated

14. How many days before your actual test date did you take your last practice exam?

If yes, do you regret this decision?

15. Did you end up moving your USMLE Step 1 exam date?

Yes, I moved it earlier.

Yes, I moved it later by a few days.

Yes, I moved it later by a week or so.

Yes, I moved it later by a month.

No

16. How much money did you spend on Step 1 preparation materials/courses? (notincluding exam administration cost)

17. If finances influenced your choice of materials, please comment.



Please add comments if your agreement or disagreement has a caveat.

18. Please indicate your level of agreement with the following:

I felt very prepared for the USMLE Step 1 based on the curriculum at UT Southwestern.

Strongly Agree

Agree

Undecided

Disagree

Strongly Disagree

Comments

19. Please indicate your level of agreement with the following:

UT Southwestern provided me with enough dedicated study time to prepare for USMLEStep 1.

Strongly Agree

Agree

Undecided

Disagree

Strongly Disagree

20. Compared to your peers, how involved are you in extracurriculars, volunteering,and/or research?



Significantly less involved

Somewhat less involved

Average

Somewhat more involved

Significantly more involved

21. On an average school week, how many hours did you spend on extracurriculars,volunteering, and/or research?

22. Performance on USMLE Step 1 would be improved by placing a larger emphasis onthe following topics in the MS1 MS2 curriculum. (Please list specific things)

23. My best piece of GENERAL advice for Step 1 preparation is:

StronglyAgree Agree Undecided Disagree

StronglyDisagree Not Utilized

Results of thisSurvey from thePrior Class

24. Please rate your level of agreement with the following statement as it applies to eachservice you used.

I found this service beneficial to my preparation for Step 1. (SASS= Student Academic Support Services)


Page 7 of 12https://www.surveymonkey.com/r/Preview/?sm=WX8nu_2FGtw2v2ITfa…ngPSRUCRFUZYyi1O4jb4DqlthKzf_2B5ndVsIVd3jve_2FT21aCo0sNUYHJ0C

AOA Step-up toStep 1 reviewsessions

AOA/SASS PanelDiscussion on Step1 Preparation

Library SimulatedPractice Exams

SASS Step 1Resource Fair andPacket

SASS CalendarClinic

Personal Consultsin SASS

SASS Step 1Preparation Course

SASS reaching outto me during thepreparation period

Comments

StronglyAgree Agree Undecided Disagree

StronglyDisagree

NotApplicable

First Aid for Step 1

USMLE World

25. Please rate your level of agreement with the following statement as it applies to eachresource you used.

I found this resource beneficial to my preparation for Step 1.



Kaplan Qbank

NBME CBSSA

U World SelfAssessments

Doctors in Trainingcourse

Pathoma

Picmonic

Flashcardprograms/ Anki

Firecracker

Rapid ReviewPathology byGoljan

Sketchy Micro

Sketchy Pharm

Med Bullets

Online MedEd

BRS Physiology

Osmosis

26. If applicable, please provide your feedback/comments on First Aid for Step 1.

27. If applicable, please provide your feedback/comments on USMLE World.



28. If applicable, please provide your feedback/comments on Kaplan Qbank.

29. If applicable, please provide your feedback/comments on NBME ComprehensiveBasic Science Self Assessments.

30. If applicable, please provide your feedback/comments on U World SelfAssessments.

31. If applicable, please provide your feedback/comments on the Doctors in Trainingcourse.

32. If applicable, please provide your feedback/comments on Pathoma.

33. If applicable, please provide your feedback/comments on Picmonic.



34. If applicable, please provide your feedback/comments on Flashcard programs/ Anki

35. If applicable, please provide your feedback/comments on Firecracker.

36. If applicable, please provide your feedback/comments on Rapid Review Pathologyby Goljan.

37. If applicable, please provide your feedback/comments on SketchyMicro.

38. If applicable, please provide your feedback/comments on SketchyPharm.

39. If applicable, please provide your feedback/comments on Med Bullets.



40. If applicable, please provide your feedback/comments on Online MedEd.

41. If applicable, please provide your feedback/comments on Osmosis.

42. If applicable, please provide your comments on other resources you used to preparefor Step 1.

43. Before starting the dedicated study period, what was your initial goal for Step 1.

44. In order to evaluate effectiveness of preparation strategies to benefit futurestudents, please indicate your actual score on Step 1.

Thank you very much for your time. The MS2s will be quite grateful as are we.

APPENDIXC–RULESFORCALCULATINGSOCIOECONOMICCATEGORY

SOCIOECONOMICSTATUSQUESTIONS&POINTVALUES GROUPDESIGNATIONSWITHRANGEOFSCORES:Maximumpointsavailabletobeassigned=85

GroupA=45-85totalpointsassignedGroupB=30-40totalpointsassigned GroupC=20-25totalpointsassignedGroupD=0-15totalpointsassigned PARENT’SEDUCATIONALLEVEL: Onlyonesetofpoints,15maximum,willbeincludedintheSEScalculationforparent’seducationallevel.Thepointswillbeassignedtotheparentorsignificantparentalrolepersonwiththelowestlevelofeducationalattainment.Thesignificantparentalrolepersonisconsideredonlyiftheapplicantlivedwiththispersonatleast3yearswhileattendinghighschool.Ifapplicantlivedwiththesignificantparentalpersonforatleast3yearswhileinhighschool,thatpersoniseligibletobeconsideredinassigningtheParents’EducationalLevelpointsintheSEScalculation.Applicantmustanswerinthefollowingway:SignificantMale:

• Didsomeoneotherthanyourbiologicalfatherplayasignificantmaleparentalroleinyourlife?YES

• DidyoulivewiththispersonforatleastthreeyearswhileattendingHighSchool?YES• Ifyes,pleaseprovideeducationlevel.

SignificantFemale:

• Didsomeoneotherthanyourbiologicalmotherplayasignificantfemaleparentalroleinyourlife?YES

• DidyoulivewiththispersonforatleastthreeyearswhileattendingHighSchool?YES• Ifyes,pleaseprovideeducationlevel.

Theprogramwillcheckeducationalleveloffather,motherandeacheligiblesignificantparentalpersonandassignonlyonesetofSESpointsbasedonthelowestlevelofeducationalattainment.

ParentsEducationLevel:

PointsAssigned Choices15 Non-H.S.Graduate10 HighSchoolGraduate/GED5 SomeCollege5 Associate’sDegree0 BachelorDegree0 CollegeBeyondBachelorDegree0 Master’sDegree0 DoctoralDegree0 PostDoctoralStudies

APPLICANT’SHOMETOWNAREA:Howwouldyoudescribetheareawhereyougrewup?

PointsAssigned Choices10 Rural0 Urban10 InnerCity0 Suburban0 MilitaryorGovernmentInstallation0 Other

HOUSEHOLDINWHICHAPPLICANTWASRAISEDORSPENTMAJORITYOFLIFEFROMBIRTHTOAGE18:SizeofHousehold:

PointsAssigned Choices

0 4orLess10 515 620 7orMore

EstimatedValueofResidentialProperty(OwnedorRented):

PointsAssigned Choices15 0-$50,00010 $50,001–$75,0005 $75,001-$100,0000 $100,001-$150,0000 $150,001-$200,0000 Over$200,000

ADDITIONALSESQUESTIONS:

1. Areyouamemberofthefirstgenerationinyourfamilytoapplyto,attendorgraduatefromanundergraduateprogram?a. Yes=10pointsb. No=0points

2. Wereyourequiredtocontributetotheoverallfamilyincome(asopposedtoworkingprimarilyforyourowndiscretionaryspendingmoney)whileattendingelementaryand/orhighschool?a. Yes=15pointsb. No=0points

identifying predictors of performance on usmle step 1 …

Documents