How to Select Students Likely to Pursue PhD Programs

Rick McGee, PhDAssociate Dean

Faculty Recruitment and Professional DevelopmentNorthwestern University, Feinberg School of Medicine

Director, IMSD/CLIMB Program

From an art to a science…

How DO we predict the future – i.e. select students who will start and complete a PhD?

Evidence of appropriate skills and abilities Evidence of appropriate motivation

Nature of the motivational goals? Internal vs. external motivations?

Do we reward past experiences and accomplishment or really attempt to predict the future? Are they the same? Differences between medical and graduate school…

How do we assess the past and predict the future differently for students from atypical life experiences? Can we? Are criteria similar or different?

To what extent do we validate our selections – remember our logic…treat each selection as an experiment…collect data to see if the selection criteria worked?

Gathering data to predict and select

Objective data – GPA, test scores, count publications, etc. Even though objective, still boils down to subjective

interpretation – not definitive or necessarily predictive Subjective/qualitative data

Answers to questions on forms Interviews Is the goal to just see what someone will say or look for

specific answers? BIG DIFFERENCES! Again, to what degree do we treat our selections as

experiments and collect data to validate them? Emphasis of Northwestern IMSD and my previous jobs –

educate faculty on validity(?) of typical selection data especially for students from atypical backgrounds –

unlike many faculty (underrepresented, disadvantaged, others)

For research programs, how do you identify and choose the right students?

Grades, especially in science courses What other information do you ask for and use? What are your favorite questions to ask on paper/web

applications to your programs, or when interviewing? Why do you want answers to these questions? What data do you expect the questions to give you? What will you do with the data? How will it contribute to

your selection? What evidence do you have that the data the questions

provide predicts persistence to the PhD and/or a career as a scientist, or anything?

Think of it all as oral data – even if written

Asking questions, interpreting answers

Applies to both written (paper or web) and oral Most commonly we ask questions to just hear what

the student says – often not consciously listening for a particular response How we interpret the data is all over the map Our ‘evidence’ for the importance or predictive validity

of the answers is largely visceral, almost never grounded in objective evidence or outcomes

Even if listening for something, often unwilling to make definitive decisions from what we hear because we have not validated the data

Question design REALLY MATTERS!!!! Especially on a form or when you can’t ask for

elaboration/clarification

How did we get started on this topic?

One of the first IMSD programs awarded in 1997 – Mayo Clinic New requirements for robust evaluation plan/process Renewed emphasis on PhD and MD/PhD as outcomes

for undergraduate programs – e.g. summer research Core question – Does high yield reflect quality of

program or quality of selection, or both? Experience with summer undergraduate researchers

fed hypothesis that selection critical, but poor program or research experience might deflect

Also, we had HUGE problem with MD/PhD students dropping the PhD in first two years!

Random life event – met Dr. Jill Keller, education qualitative researcher from U of AZ, at reception in Washington, DC

Basic question…

Is it possible to predict with high reliability those college students likely to go into research, often vs. clinical medicine? If so, how, when, and using what criteria and methods?

Delicate issue…

NO desire to come up with formulaic approach that could limit opportunities and student development

Many URM students enter college with no clue about research as a career and what a PhD is all about – programs essential for raising awareness and interest – can’t constrict too much

But resources are limited and NIGMS had to show it was accomplishing program goals

MOST importantly, our job is to help students find their niches….not good to push square pegs into round holes or push them into unsatisfying lives

More delicacies…

Basic scientists often want to avoid taking ‘pre-meds’ into their labs – Why? Mentor-based research training is predicated on

interactions with those with like interests Pre-med students seldom get ‘turned on’ to the lab and

research – frustrating when they can’t be converted Feels like taking opportunity away from someone who

really needs experience for PhD or MD/PhD But learning about research is valuable to future MDs Many medical schools put high premium on UG

research – some REQUIRE it – so need is high Many pre-meds have no idea about research

How do we do this kind of research?

We ask students questions – called interviews – but interested in the answers with no intent to judge them or use the data for selection

Basic questions start the conversations but then we follow it wherever it goes

Conversations are recorded and transcribed

We look for patterns in what people tell us

Sounds pretty simple….

Qualitative Research - A long history in social sciences

NOT an oxymoron

Also includes surveys, focus groups, observation

Used by anthropologists, sociologist, some psychologists, education researchers, etc.

Powerful way to study behavior, beliefs, experiences

Can find common traits that ‘define’ a group – basis for Cultural Anthropology, Sociology

Can test hypotheses or generate them

Data Analysis Using Grounded Theory

Grounded in Reality – something that is concrete and an important part of the life of almost everyone in the group

If it is an important trait people will express it naturally, without thinking, and in high frequency

Should be discoverable with as few as 8-10 individuals

Our study subjects

Undergraduates doing summer research at the Mayo Clinic – from all over the US, including Puerto Rico – rising JR and SR students

Semi-structured interviews at the start, the end and 9-12 months after summer research

Vast majority go on to PhD, MD or MD/PhD but outcome unknown at the start

No trouble getting them to talk about themselves and their experiences!

Interviews done by trained interviewer Transcripts analyzed by R. McGee and Jill Keller

Example entrance interview questions…

How did you “get here” - i.e. past experiences?How did you “get here” - i.e. past experiences? What roles have family, friends, teachers and What roles have family, friends, teachers and

mentors played in your experiences?mentors played in your experiences? What are your career plans and goals at this point?What are your career plans and goals at this point? What do you think you will get form this career or the What do you think you will get form this career or the

careers you are choosing between?careers you are choosing between? Etc.Etc.

The interviewer listens for what is important to the The interviewer listens for what is important to the student and asks questions to extend and clarifystudent and asks questions to extend and clarify

Example exit interview questions…

Describe the research you did this summer.Describe the research you did this summer. What do you think will stay with you from the What do you think will stay with you from the

summer?summer? Can you identify people who have acted as mentors Can you identify people who have acted as mentors

to you this summer? What did they provide?to you this summer? What did they provide? What is your current view of and interest in research? What is your current view of and interest in research?

Do you see research as part or all of your career? Do you see research as part or all of your career? Why or why not?Why or why not?

Answers have no impact on students – very different Answers have no impact on students – very different from application/interview questionsfrom application/interview questions

Etc.Etc.

Results – Two themes consistent across almost all students…

High self-confidenceHigh self-confidence Action-orientationAction-orientation Equally present in those who went on to Equally present in those who went on to

research and clinical trainingresearch and clinical training Qualities of successful applicants who rose to Qualities of successful applicants who rose to

the top of the SURF applicant poolthe top of the SURF applicant pool

5 Themes predicted the 18 (of 28) who went toward PhD, MD/PhD, or MD to do research…

1. Curiosity to discover the unknown…. 1. Curiosity to discover the unknown…. especially something that nobody had ever especially something that nobody had ever seen beforeseen before expressed by 15 of the 18 students who started expressed by 15 of the 18 students who started

toward researchtoward research 1 of the 3 who did not show curiosity changed to 1 of the 3 who did not show curiosity changed to

DVM after 1 year – we predicted it was not for herDVM after 1 year – we predicted it was not for her only seen in 1 of 8 other students who went on to only seen in 1 of 8 other students who went on to

clinical MD or other directionclinical MD or other direction

Curiosity to Discover the Unknown

...it’s like what do you call it? Like frontier land, ...it’s like what do you call it? Like frontier land, sort of. You know, it’s like people have sort of. You know, it’s like people have general ideas about things, I mean it’s based general ideas about things, I mean it’s based on a lot of factors, some of it’s theory, but it on a lot of factors, some of it’s theory, but it seems like it’s always changing and you can seems like it’s always changing and you can always make new discoveries. I guess I like always make new discoveries. I guess I like the fact that it seems like you can always find the fact that it seems like you can always find something or be on the brink of discovery.something or be on the brink of discovery.

2. Enjoyment of Problem Solving

Similar but slightly different from discovery of Similar but slightly different from discovery of the unknownthe unknown

10 of 12 students who went on to PhD or 10 of 12 students who went on to PhD or MD/PhD MD/PhD

Did not come out as strongly or emotionally Did not come out as strongly or emotionally as curiosityas curiosity

Enjoyment of Problem Solving

… ….when I first started working in the lab, it just .when I first started working in the lab, it just killed me to see nothing for three months. But I killed me to see nothing for three months. But I still learned from it and I learned what I was still learned from it and I learned what I was doing wrong and different ways to fix it and you doing wrong and different ways to fix it and you get... real creative ... you might spend a whole get... real creative ... you might spend a whole day thinking what can I do to prevent this gel day thinking what can I do to prevent this gel from leaking out the bottom…. So finally you from leaking out the bottom…. So finally you look at everything at the lab and you find look at everything at the lab and you find something sticky and you say, "I'll try this". So something sticky and you say, "I'll try this". So you try Vaseline or something so it makes you you try Vaseline or something so it makes you very creative. So when you're trying to rig very creative. So when you're trying to rig something up - it just makes you creative when something up - it just makes you creative when things aren't going right.things aren't going right.

3. Independence

AllAll of the students who expressed the intent of the students who expressed the intent to do research in the future talked of wanting to do research in the future talked of wanting to being able to think or act on their own with to being able to think or act on their own with limited consultation or guidance from others.limited consultation or guidance from others.

None of the students who went on to other advanced training expressed this as a strong trait or something they sought

Independence

I: So what makes you feel that you want a career as a scientist? S: I’ve always loved science. I haven’t been that good in math,

and I wanted to be a scientist…I went to a medical magnet to experience different things and through there I got involved in different researches. It was like, “That’s what I want to do. I want to work in a lab.” I like finding something using my analytical skills. It’s like you’re trying to search something and I really enjoyed that. It’s very independent, like you go in there and they teach, and you’re on your own and no one’s looking over your shoulder, “you need to do this-- you need to do that.” I like that independence -- learning new things by yourself that they just give me a protocol and I can do it. Maybe not perfect the first time. But I am my own woman and you learn little tricks so it’s very independent and I like that. It’s an individual job.

4. Helping others indirectly through research

24 of 26 expressed a desire to help others Not unexpectedly, those who went on to

medicine expressed this strongly and directly – seeing the direct impact of their caregiving

Virtually all of those who headed toward research did too expressed as a desire to work on research

that might help others, realistically knowing the chances were not high they would see it

Helping others indirectly through research – went on to PhD

I: But when you think of ‘making a difference’, how do you define making a difference?

S: Many ways. One of the ways that comes more clear to me would be changing someone’s life from worse to better… -- let’s say I’m working with asthma or something and I develop something that will prevent them from suffocating or needing a ventilator every time they get wheezy. I have made a difference in their life making them more comfortable.

5. Approaches to the Future….

Very surprising theme that took a long time to emerge

Clearly differentiated two patterns among students

Approaches to the Future Pattern #1

Clear and often well thought out goals for the future and plans of how to attain them Often included both professional and personal

goals and strong sense they were attainable Planning often included strategic decision-making

to achieve sequential steps along the way to goals

Approaches to the Future Pattern #2

Minimally structured view of the future Long-term goals more vaguely defined, often

expressed as several possible outcomes Minimal concern about which outcome achieved Sometimes exasperated that everyone kept

asking them about what they planned to be/do Enjoyed seeking out or creating options rather

than focus on moving directly toward a single goal

Often part of both professional and personal lives

Minimally Structured Approach to the Future

I: So you're definitely planning to have research as some part of your career.

S: Right now. Who knows....? I just like to leave the door open because just talking to people - the professors or something - most people don't end up in what they think they're going to end up in college so I'd like to say for sure but right now that seems like where I'd like to be.

Additional Tidbits

Able to replicate results with online survey at the end of the summer – wording VERY critical…

“If you are currently planning to pursue a research career as a PhD or MD/PhD, what do you find attractive about doing research? Why have you chosen it as a career path?”

High fraction came up with curiosity/discovery as primary reason

Similar pattern with MDs who expressed strong intent to continue research after completing Masters in Clinical Research

Words say it all….

“I feel it is rewarding to discover new things, especially those that can help people. It is also just really interesting.”

“I like the idea of discovering things that no one has known before and that the research I do will contribute to the greater scientific knowledge about a topic. I also hope that my work will someday be used help to treat human disease.”

Some other themes emerging in current research… Decisions commonly driven by interests

For some these interests can flip quickly Others hold on to them tenaciously – immunologists,

neuroscientists Don’t expect research-focused students to be

interested in everything…. we aren’t Mismatch between labs and interests can be

misinterpreted – interest alignment important – don’t take it lightly

Motivation seems to be more internally driven, but it is difficult to separate internal and external motivation

Acquiring an identity as a scientist emerging in work of others and our current research as pivotal step in commitment to PhD

Take-away messages…

It may be easier to predict future scientists than we thought, but what you look for is different

At least by late college, tendency toward research is largely set – explains great difficulty of converting pre-medical students to research

We do see it in some students consciously choosing to do MD as a route to get there w/o MD/PhD, and later in other MDs, but quite limited

Can use to cautiously help students figure out if research is a good fit for them

Whole different approach to gathering and using selection data required

Can you apply this?

Mayo MD/PhD program story Mayo SURF story Rescue of a failing MARC program story

Caveats on the characteristics…

Only beginning to study how soon these tendencies are set, how to find them prior to doing research

Have not tried to systematically detect their expression outside of the research context but do have ideas

May take months to years for student to know what to do when they discover a ‘fit’ with research if it conflicts with earlier goals/plans

Only beginning to study these characteristics at college application

Also only beginning to study the relationships between these characteristics and progression through the PhD and beyond…

So how do you detect/find these qualities? Stage of development and life experiences matter a

lot In high school Entry into/early college – RISE, IMSD, Bridges,

REU Big developmental step ~ junior year – RISE,

MARC, IMSD, Bridges, REU Late bloomers/slow committers – PREP, Bridges PhD – Bridges, IMSD Postdoc

So how do you detect/find these qualities? Start with the easy ones… Easiest in verbal exchanges but can be done with

carefully crafted written questions Best to use open ended, tangential questions that

reveal what comes out naturally Easiest case – student has just returned from well-

designed summer research experience in good lab “Tell me about your time at U of X this summer.”

Reveals what was most important/meaningful Hopefully research will come out very early Asking follow-up questions to pull out reasons for

what they talk about – reveals what is important

Questions continued…

“What did you like best about your time in the lab?” Listening for curiosity/discovery/problem solving If not, tease out if the absence bothered the student

“Sounds like you had a pretty established project to work on, not a lot new to discover or problems to solve. Was that ok with you?”

If it didn’t bother them, or other most important things come out, probe to see what they were looking for in the experience to reveal insights into them

Questions continued…

“Why DO you like doing research?” If there is a key question, this is it, but becomes

revealing only AFTER student has experienced REAL research

Research experience need not be highly successful, get good data or publication – absence of success actually probably more revealing

But should be positive experience from social or developmental context

Listening for curiosity, discovery, solving problems or puzzles – ideally the first thing that comes to mind and pops out

Often hear elements of making a difference, other things, but listening for these being secondary

Questions continued…

If someone has been in more than one lab…… “What has been your most rewarding experience in any of your research to date? When was your biggest ‘high’?”

Again looking for discovery and problem solving – best when content with small pieces they see add up

Sometimes will hear the thrill of seeing something for the first time nobody has ever seen before

Getting a paper published, seeing your name in ‘print’ also comes up – likely this should not be the primary motivation given it is so infrequent – also suggests external motivation more than internal

Questions continued…

“Have you had periods of time when nothing was working in the lab? How do you deal with it? How does it affect you?”

Listening for high tolerance of failure More mature students will talk about how you learn

new things even from failure – small problems solved If answer is no, “How do you think you will handle it?”

Harder situation – no prior research

“What got you interested in science?” Listening for curiosity-driven experiences and pure interest

“What have been your favorite sciences classes? What made them your favorites?”

Listening for classes that encouraged development of student’s interests, problem solving, experiments with unknown outcomes – topic aligned with or triggered student interests

Allowed independence Be caution with things like ‘teacher made it really easy,

teacher was very clear on what we needed to know’, etc. “Since you have never done research before, what makes you

think you will like it?” Ideally listening for accurate idea of what research is

about, solving problems, finding new information Tricky handling this kind of situation

Contrast with the classic question

“Why did you apply for this program?” Student very likely trying to figure out what you want

to hear Much less likely to get true, unfiltered answer Still can be listening for what seems like honesty… Friend who is doing research and it sounds like “fun” I need to make money and it will be a lot more

valuable than working at McDonalds – this is ok, don’t loose sight of students’ realities – would treat this as neutral information but honest

Pre-med is unlikely to tell you to improve their resume Fine to ask, harder to know what to do with the

information – like a poorly designed experiment

View of the future

“Where do you see yourself in 15-20 years?” Purposely pushing out to long vision Listening for open frame or branching options Concern when there is a very clear and narrow outcome Fine line if totally clueless or too uninformed

“What will you feel like if that is NOT where you end up?” Listening for flexibility, comfort with alternative outcomes

“How will you feel if near the end of your career you have been very successful, gotten grants, published papers, but nothing you have discovered has made any measurable impact on health or well being of others?”

Listening for no problem with that – often will easily respond ‘that will be ok’…things like others may have used their work, or trained others have had an impact – should be clearly no problem with it

Be cautious with the long pause or negative reaction

Independence

Harder to explicitly draw out – we struggle in our own research

Comes out spontaneously and in multiple contexts “When you are working in the lab or on school projects,

do you prefer to have your own project, knowing it might go slower or problems may be harder to solve, or work on a group project?”

Have not used much but might reveal independence Despite rise in research collaborations, independence

likely a preference driver at early stages Often comes out as wanting to have own project

“When you run into a tough problem in the lab or school, how do you approach solving it?”

Listening for drive to solve it themselves but also able to get guidance from others if need be

What questions would you like to try?

Throw out a question have used or have considered using or might use now after what you have heard…

Are there specific issues you would like help figuring out how to get specific data?

Acknowledgements - Our Team…

“Scientific Careers Research and Development Group” “Career Decision-Making of Future Minority Biomedical

Faculty” – NIGMS-funded Jill Keller, PhD (Arizona) – Co-Investigator Patricia Campbell, PhD, Campbell-Kibler Assoc. – Co-

Invest. Robin Remich, MAT, MEd – Research Associate Bryan Breau – Research Study Coordinator

“Pivotal Career Decisions Guiding Potential Women Science Faculty” – NINR-funded Patricia Campbell, PhD – Co-Investigator Jill Keller, PhD – Co-Investigator Sandy LaBlance, PhD – Research Associate Bryan Breau – Research Study Coordinator

Our Team…

Integrating multiple social science theories to interpret career decisions and guide intervention design – NIGMS Lynn Gazley, PhD, MPH – Research Associate

“Mentoring for Success: Developing Fundamental Skills for Biomedical Research” – NIGMS Steve Lee, PhD – Assistant Director Karl Keller, MA – Communications Professional

“Translating Theory to Practice to Diversify the Biomedical Research Community” – NIH Pathfinder – “The Academy for Future Science Faculty”

the whole team plus… Michelle Naffziger, PhD – Research Associate Jennifer Richardson, MA, PhD – Research Associate Simon Williams, PhD – Research Associate Beth Morrissey, MA – Research Project Coordinator

My contact information

Rick McGee, PhDAssociate Dean for Faculty Recruitment and Professional

Development

Northwestern University, Feinberg School of Medicine

Chicago, IL

312-503-1737

r-mcgee@northwestern.edu

References

McGee, R. and Keller, J.L.: (2007) Identifying Future Scientists: Predicting persistence into research training. CBE-Life Sciences Education 6(4), 316-331. http://www.lifescied.org/cgi/content/full/6/4/316

McGee, R and DeLong, M.J. (2007) Collaborative co-mentored dissertations spanning institutions: Influence on student Development. CBE Life Science Education 6(2), 119-131. http://www.lifescied.org/cgi/content/full/6/2/119

McGee, R., Almquist, J, Keller, J.L. and Jacobsen, S. J.: Teaching and Learning Responsible Research Conduct: Influences of Prior Experiences and Conflicting Messages. (2008) Accountability in Research 15, 30-62.