Finding the ‘Right’ Students:

The Search for Predictive Validity in Applicant Screening

for the Health Sciences

Rosemary Lysaght, Ph.D.Catherine Donnelly, M.Sc.Michelle Villeneuve, M.Sc.

School of Rehabilitation Therapy

Background• Impetus for examination of our admissions

process was the creation of the new masters entry-level program in 2004

Goals for admission included: • An evidence-based admissions process• Increase the applicant pool• Efficiency in admission process

Overall Aim of SelectionTo select students who will:

• Succeed in the academic program• Perform credibly in professional practice • Possess the traits of character and ethical

values desired of a professional person

(Nayer, 1992)

Commonly Used Selection Criteria in Health Sciences• Pre-admission academic grades• Discipline-specific aptitude tests (MCAT, HOAE)• Interviews• Written submissions• Letters of reference• Prerequisites

(Auriemma, 2002; Salvatori, 2001)

Predictors of academic success

• Evidence across disciplines supports the predictive validity of – Pre-admission academic grades

• No clear support found for – Discipline-specific aptitude tests – Interviews– Written submissions– Letters of reference– Prerequisites

(Bridle, 1987; Caplan et al, 1996; Howard & Jerosh-Herold, 2000; Lewis & Smith, 2002; Kirchner et al., 2000; Kirchner & Holm, 1997; Salvatori, 2001)

Predictors of clinical performance

• Unclear relationship between pre-admission performance and clinical performance

• Valid outcome measures difficult to identify• Lack of consistent raters• Variability across settings

(Howard & Jerosch-Herold, 2000; Kirchner & Holm, 1997; Katz & Mosey, 1980; Tan et al., 2004)

Research Questions

1. What admissions screening tools best predict academic performance in a masters level OT program?

Selected academic courses and Clinical performance

2. Does undergraduate coursework predict success in topically-related coursework?

Method

• Analysis of existing data for 128 students admitted to the OT masters program– Multiple regression – Models created for each research question

• Project received approval by the Queen’s University’s Research Ethics Board

• Sample included 3 cohorts of students (1st three years of the new professional masters program)

Factors Considered in OT Admissions

– GPA– Academic transcript– Letter of intent– 2 referee rating forms/letters– Supporting data (foreign/non-traditional

applicants)

Predictor Variables

• Undergraduate Grade Point Average (GPA) – from ORPAS

• Letter of Intent (LOI) • Average rating, 2 raters

• Referee Rating• 5 point scale, 12 items• Average total score of 2 external referees

Predictor Variables (con’t)• Additional ratings were created for each student

based on file review (1 = weak to 5 = exceptional) Physical Sciences Experience/Preparation

• e.g. Anatomy, physiology, kinesiology (r for 2 raters = .98)

Social Sciences Experience/Preparation• e.g. Psychology, Sociology, Family Studies (r=.91)

Experience with People with Disabilities/Vulnerable Populations

• Rated for number, duration, and relevance (r=.79)• Information drawn from Letter of Intent, referee letters and

experience questionnaire

Criterion Variables

• Program GPA (1st year)• Communication Skills Practicum• Targeted course grades in:

Physical Determinants of Occupation Cognitive-Neurological Determinants Psycho-Emotional Determinants

Descriptive Statistics - Predictor Variables

Variable N Mean SDUndergraduate GPA 129 3.24 .278

Letter of Intent Rating 129 71.70 12.86

Referee Rating 129 4.72 .25

Bkg. Physical Sciences 127 3.92 1.35

Bkg. Social Sciences 127 2.90 1.66

Bkg. Experience with PWD 127 3.70 .88

Descriptive Statistics - Criterion Variables

Variable N Mean SDFirst Year GPA 128 80.29 3.0

0

Communication Skills Rating 112 8.44 1.19

Physical Determinants Grade 129 77.72 4.05

Cogntive-Neuro Det. Grade 126 78.97 5.13

Psycho-Emotional Det. Grade 126 78.89 5.03

Results – First Year GPA

• Predictor Variables: GPA, Letter of Intent, Referee Rating• Model is significant (R2 = .101; F= 4.65, p <

.001)• GPA is only variable with significant beta

score (p = .005)• Significant correlations between all three

variables and program GPA

Results – Communication Skills • Predictor Variables: GPA, Letter of

Intent, Referee Rating, all experience ratings,

• Model is not significant (R2 = .021; F= .62)• Referee rating is significantly correlated

with performance rating• No background experience ratings were

correlated with performance rating

Results – Physical Determinants

• Predictor Variables: GPA, all experience ratings

Result:• Model not significant (R2 = .009; F=.275)• No significant correlations between any

background experience rating and outcome

Results – Cognitive-Neuro

• Predictor Variables: GPA, all experience ratings

Result:• Model is significant (R2 = .14; F=5.0, p = .004)• GPA only variable with significant beta score (p

= .00)• Background in physical sciences was

significantly negatively correlated with outcome

Results – PsychoEmotional Det.

• Predictor Variables: GPA, all experience ratings

Result:• Model is significant (R2 = .082; F = 2.6, p< .05)• Social science bkgd only variable with

significant beta score (-.2 )• GPA is significantly positively correlated with

outcome, while SS bkgd is significantly negatively correlated.

Additional Observations• The contribution of the undergraduate GPA to all

models was reduced by the addition of the 3rd cohort, which had significantly higher GPA ratings than the first 2 cohorts and less spread in scores

• Referee ratings also had small but significant correlations with – physical determinants grade (r = .18, p = .02)– communication skills grade (r = .25, p = .003)

Conclusions

• Findings relative to GPA as a significant predictor of academic performance in a health science program supports previous research

• Other positive correlates with 1st year GPA:– Referee ratings– LOI ratings

suggest that these measures have some value in the admissions process.

• Referee ratings have even broader potential value, given positive correlations with performance in Physical Determinants & Communications Skills courses.

Conclusions (con’t)

• No support for the requirement of specific academic pre-requisites

• Previous experience with PWD/vulnerable populations did not affect academic grades or communication skills performance

• No support for the requirement of specific academic pre-requisites

• Previous experience with PWD/vulnerable populations did not affect academic grades or communication skills performance

Discussion

• Fairness of admissions process– Pre-requisite requirements not justified if predictive validity of courses

not substantiated– No control over authorship of LOI

• Inherent biases in process– LOI requirement may bias selection towards persons from the same

culture who highlight issues salient to that culture, females, and strong writers

– Applicants with strong social science background may be over-represented if GPA is primary selection factor

• Impact on program– Elimination of pre-requisites broadens applicant pool, may result in

higher calibre class– More diversity of students in program/field

Discussion

• Role of Pre-Requisites and other Screening Tools– LOI, Referee ratings useful as screen out, rather

than selection criteria?– Certain pre-requisites may make academic

progress easier for student– Admissions requirements may have value beyond

identification of best applicants• Credibility of process and applicant• Interviews may help form relationships, promote program

Limitations

• Findings limited to one health science program format within a Canadian context, and may not generalize

• Other potential screening tools not available for consideration in this study

• Course grades are subject to unsystematic scoring errors

• Ratings of background experience and LOI subject to rater error

References• Auriemma, D. (2002). Admission methods of professional occupational therapy

programs in the united states. Education Special Interest Section Quarterly, 12(3), 1-4.

• Bridle, M. J. (1987). Student selection: A comparison of three methods... queen's university occupational therapy program. Canadian Journal of Occupational Therapy, 54(3), 113-117.

• Caplan, R.M, Kreiter, C., & Albanese, M. (1996). Preclinical science course ‘preludes’ taken by premedical students: do they provide a competitive advantage? AMJ, 71 920-922.

• Howard, L., & JeroschHerold, C. (2000). Can entry qualifications be used to predict fieldwork and academic outcomes in occupational therapy and physiotherapy students? British Journal of Occupational Therapy, 63(7), 329-334.

• Katz, G.M., & Mosey, A.C. (1980). Fieldwork performance, academic grades, and pre-selection criteria of occupational therapy students. American Journal of Occupational Therapy, 34(12), 794 – 800.

• Kirchner, G. L., & Holm, M. B. (1997). Prediction of academic and clinical performance of occupational therapy students in an entry-level master's program. American Journal of Occupational Therapy, 51(9), 775-779.

• Kirchner, G. L., Stone, R. G., & Holm, M. B. (2000). Use of admission criteria to predict performance of students in an entry-level master's program on fieldwork placements and in academic courses. Occupational Therapy in Health Care, 13(1), 1-10.

• Lewis, M., Smith, S. (2002). Selection of pre-registration physiotherapy students: Changing to a more objective process. Physiotherapy, 88(11), 688 – 698.

• Nayer, M. (1992). Admission criteria for entrance to physiotherapy schools: How to choose among many applicants. Physiotherapy Canada, 44, 41 – 46.

• Salvatori, P. (2001). Reliability and validity of admissions tools used to select students for the health professions.see comment. Advances in Health Sciences Education, 6(2), 159-175.

• Tan, K. P., Meredith, P., & McKenna, K. (2004). Predictors of occupational therapy students clinical performance: An exploratory study. Australian Occupational Therapy Journal, 51(1), 25-33.