Assessing verbal communication skills of medical students

J VogesE Jordaan *

L KoenDJH Niehaus

Department of Psychiatry, University of Stellenbosch and Stikland Hospital

* Biostatistics Unit: Medical Research Council, Bellville

Positioning of the study

Large project:Correlation of communication skills with

academic performance of medical students Sub-studies:

Facial affect recognitionOral examination marks in psychiatryNon-verbal communication skillsVerbal communication skills

Introduction

Communication is one of six required competencies identified by the ACGME

Effective communication associated with: Improved patient and doctor satisfactionTreatment complianceStrong predictor of medical school success

Assessment of communication skills is complex and difficult to implement

AimFirst phase: To evaluate the usability of the Liverpool

Communication Skills Assessment Scale for assessing the communication skills of medical students of the University of Stellenbosch

Second phase: To determine effectiveness of undergraduate

medical students’ communication skills using the Liverpool Communication Skills Assessment Scale

To determine if there is a correlation between communication skills and overall academic performance

Measurement equivalence

Central issue in determining the applicability of instrument cross-nationally and cross-culturally

Factors to consider:Content equivalenceSemantic equivalenceTechnical equivalenceCriterion equivalenceConceptual equivalence (Flaherty et al, 1988)

Methods

Subjects:Medical students completing late rotation5 min. semi-structured interview with patient

that was videotapedPermission granted by Faculty of Health

Sciences and Ethics committee of SU Venue:

5-week Psychiatry rotation at Stikland hospital

Methods Assessment tool:

Liverpool Communication Skills Assessment Scale (LCSAS) Consists of 12-items measuring several aspects of

communication Mixed method of using both a checklist and a rating approach 4-point ordinal rating scale ranging from Unacceptable to Good Ease of use, acceptable reliability

Raters: 2 independent raters, additional training Third rater included, instruction given Help sheet with additional descriptors to guide scoring

Primary statistical evaluation: Inter-rater reliability

Marginal homogeneity (Chi-square statistic, p-value<0.01 as significant)

Agreement (Cohen’s weighted Kappa index for ordinal data)

Results: Distribution of score by rater

Intra-class correlation coefficient = 0.8 (0.71-0.87)

Results: Distribution of score by item

Results: Distribution of score by item

Results: Distribution of score by item

Agreement between raters Agreement for items that had marginal homogeneteity

(Cohen’s weighted Kappa index for ordinal data, 95% confidence interval)

Items Rater 1 vs. Rater 2 Rater 1 vs. Rater 3 Rater 2 vs. Rater3

Item 1: Greeting 0.71 (0.52-0.91) 0.54 (0.33-0.75) 0.57 (0.38-0.77)

Item 2: Introduction

0.76 (0.63-0.70) 0.79 (0.66-0.92) 0.92 (0.83-1.00)

Item 4: Eye-contact 0.66 (0.41-0.91) 0.38 (0.10-0.65) 0.69 (0.47-0.91)

Item 8: Questions 0.62 (0.43-0.81) 0.37 (0.14-0.60) 0.55 (0.37-0.74)

Discussion LCSAS evaluated for usability to measure communication skills

in medical students Additional training Inclusion of additional rater and help sheet Total score: high level of correlation Inter-rater reliability

Marginal homogeneity 4 of 12 items

Agreement Additional training – greater agreement

Reliability of measure: Continue with development and standardisation of assessment scale for

use in South Africa Training

Limitations and future directions Limitations:

Small sample Inter-rater reliability

Marginal homogeneity and agreement Training

Future directions:Re-evaluation of the scoring categories to

promote understandingLanguageGenderCulture

Conclusion

Use of LCSAS in South African setting Correlation for total score Training necessary to improve agreement

for each item Further development necessary Use in education context of South Africa

by various health professionals

Epstein, R.M. Campbell, T.L., Cohen-Cole, S.A., McWhinney, I.R. & Smilkstein, G. (1993). Perspectives on patient-doctor communication. Journal of Family Practice 37(4): 377–388.

Flaherty, J.A., Gaviria, F.M., Pathak, D., et al. (1988). Developing instruments for cross-cultural psychiatry needs. Journal of Nervous and Mental Disorders 176(5): 257-263.

Humphris, G.M. & Kaney, S. (2001). The Liverpool Brief Assessment System for Communication Skills in the Making of Doctors. Advances in Health Sciences Education 6: 69–80.

Parker, G. (1993). On our selection: predictors of medical school success. Medical Journal of Australia 158(11): 747–751.

Project supported by funding from FINLOFaculty of Health Sciences

Selected references