REVIEWING CONFERENCE ABSTRACTSPat O’Sullivan

Research and Development in Medical Education

Acknowledgments

Paul Hemmer, MD, MPHUniformed Services University of the Health Sciences

G. Dodd Denton, MD, MPHUniformed Services University of the Health Sciences

Karen Szauter, MDUniversity of Texas Medical Branch School of Medicine

Sebastian Uijtdehaage, PhDUniversity of California Los Angeles

Brian Mavis, PhDMichigan State University

Creative Commons LicenseAttribution-NonCommercial-Share Alike 3.0 Unported

You are free:• to Share — to copy, distribute and transmit the work • to Remix — to adapt the work Under the following conditions:• Attribution. You must give the original authors credit (but not in

any way that suggests that they endorse you or your use of the work).

• Noncommercial. You may not use this work for commercial purposes.

• Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one.

See http://creativecommons.org/licenses/by-nc-sa/3.0/ for full license.

Workshop Objectives

1. Define and describe major structural components of a conference abstract

2. Recognize what makes each component effective

3. Describe the elements required in a “Methods” and “Results”/ “Evaluation Plan” section

4. Avoid common pitfalls in a “Discussion” section

5. Describe the peer-review process for a conference abstract

6. Write reviews for abstracts being considered for a conference

Abstracts are different than manuscripts

• Often describe preliminary or pilot studies• Word restrictions (400 max) limit the scope • Exclude supporting documentation such as references,

tables, etc.

Anatomy of abstract

Title

1. IntroductionPurpose/Background

2. Methods

3. Results/Evaluation Plan

4. Discussion/Dissemination

5. Reflective Critique

THE INTRODUCTION

The Introduction provides background or rationale• Reference to what is already known• Reference to general knowledge or accepted

standards of practice (“conceptual framework”)• Addresses the “So what?” question

A clear Problem Statement/Purpose must be included

• Reviewer should clearly understand:• What question was being investigated, and• Why this question is important

Abstract is unacceptable without a clear purpose/research question

Problem Statement, Purpose, or Hypothesis

• Problem statement/Research Question • “We investigate the relationship between USMLE Step 1

scores and self-assessed study skills” • Statement of Purpose

• “Our purpose is to develop a curriculum for students to improve skills in communicating bad news”

• Hypothesis• “We hypothesized that our students would demonstrate

improvement in exam scores after participating in on-line remediation”

References to the literature are not required

• Word limits preclude full reference citations

• Authors might include abbreviated citation, e.g.: Smith, 2008.

• Allows author and reviewer to differentiate between “opinion” and building on another’s work

Example 1

INTRODUCTION 

Increasing the enrollment of students from underrepresented groups in medical schools continues to be a challenge. Recent data (Smith, Jackson & Jones, 2011 ) showed that only 7% percnt  of physicians are minority whereas this proportion is 25% in the US population overall. Pipeline programs are designed to improve diversity in medical schools. This study focused on the academic performance of underrepresented (UR) students on courses that are required for applying to medical school (“gateway courses”).

Example 2

INTRODUCTION

Increasing the enrollment of students from underrepresented population in medical schools continues to be a challenge. Minority students’ achievement in college courses that our required for application in medical school (“gateway courses”) is key to successful admission in medical schools. We sought to determine (a) whether minority students receive lower grades than non-minority students in pre-health gateway courses; (b) the extent to which lower grade performance might be explained by the differences in precollege academic achievement; and (c) whether minority students are less likely than non-minority students to persist in completing at least four gateway courses.

METHODS

General Considerations for Methods

• Operationalizes the research question or the innovation• Describes a quantitative, qualitative design, or mixed

designOR• Describes the framework for development of innovation• Explicit statement about IRB approval or review is optimal,

but not required

Methods describe:“Who, where, what, when, how?”

• Who is sampled?

• Patients, students, residents, fellows, faculty?

• What is the sample size?

• What is the setting?

• What is the source of data?

• Survey, papers, examinations, etc.

• What is the study design?

• When is this study done?

• How is the data analyzed?

What reviewers look for in Qualitative Studies• Was sampling appropriate to the question?• Was data collection method appropriate?

• Interviews, focus groups, survey responses, written comments

• Was data analysis done with rigor?• Who analyzed the data?• How were coding difference resolved? • Triangulation?

Innovation or Curriculum Development

• Guiding framework for development• Methods used for steps taken

• Ensure the quality of the innovation or curriculum developed• Data planned or collected

Example 1

METHOD

Using regression analysis, we analyzed academic performance of 5,000 college students in three California colleges who matriculated in the 2009–2010 or 2010–2011 academic year. We compared grade performance of students across ethnic groups in gateway courses and the degree in which they completed at least four gateway courses, using regression methods to control for students’ college admission test scores. Statistical significance was set at 0.05 .  

Example 2

METHOD

We obtained retrospective, administrative data from 3 California colleges on 5,000 college students who matriculated in the 2009–2010 or 2010–2011 academic years and enrolled in at least one college gateway course. We compared students across ethnic groups in gateway course grade performance and persistence in completing at least four gateway courses, using regression methods to control for students’ college admission test scores.

Curricular Example 1• METHOD

We held weekly seminar taught by faculty from medicine and law schools for ten weeks. The curriculum focuses on areas where medicine and law intersect, including foster care, prisoner health, caring for refugees, and homeless health policy. Participants are exposed to academic and community leaders who express interest in developing mentoring opportunities with students. Eleven law and thirteen medical students enrolled in the seminar during the first year. Seminar sessions are conducted in lecture-based format for one hour followed by half hour small-group discussions. Students simultaneously interact with each other in the clinical setting when providing direct services with faculty supervision, and are encouraged to discuss these experiences.

Curricular Example 2• METHOD

We developed the joint medical and law seminar following a curriculum development framework where we analyzed needs, set objectives, chose teaching strategies, implemented and evaluated the seminar. Needs assessment indicated selecting topics where medicine and law intersected such as foster care, prisoner health, caring for refugees, and homeless health policy. Since we were emphasizing the intersection, we chose a format that allowed for sharing as well as didactics. Didactics were reviewed by the development team prior to sessions. Success was assessed by numbers enrolling from both schools, attendance, and learner and faculty satisfaction.

RESULTS

Results must present key findings• Actual data (numbers) must be presented• Congruent with Methods section• Descriptive data usually present• Inferential analyses, with p values, should be present

as appropriate.

Results: Essential components

• Response rate to a survey

• Participation rate in cohort studies

• Numbers in control and intervention groups

Preferably:• Numbers and/or reasons for exclusion

• Demographic information

Data and p values must be reported

RIGHT:

“Group A scored higher than group B (means: 10 ± 2 versus 4 ± 1, p<.05)”

WRONG:

“Group A scored significantly higher than group B (p < .05)”

What influences statistical significance?

• Sample size • As sample size increases, smaller differences needed for

significance

• Multiple comparisons• As number of statistical tests increases, likelihood of at least one

statistically significant finding by chance alone increases• the p value can and should be adjusted for multiple comparisons

Statistical vs Functional Significance

• Statistical analyses with significant p values ideally should be accompanied by a measure of functional significance

• Examples: Effect size, confidence intervals, proportion of variance explained (R2)

• Such measures help to show the importance of the statistical difference

“Negative” studies

• Negative findings can be important for learning what works (or doesn’t work) in medical education

• However, need to consider if negative or non-significant findings due to lack of statistical power

• Most common reason is insufficient sample size

• With negative findings, author should include a comment about power in the abstract• If absent, this is a feedback point in the written comments

Results: Qualitative Studies

Description of themes and sub-themes that emerged• Word limit may preclude example quotes

Differences among participants, groups may be discussed

Quantification of the results often not included or relevant

Results: Common Pitfalls

• No data (actual numbers) are reported• Results “will be presented at the meeting” is

unacceptable [*may depend on conference but helps if have preliminary or provide plan]

• No or low response rate (with surveys)• Not enough detail about measures• Statistical analyses inappropriate for type of data• Authors interpret data instead of just listing results:

“Analyses support our hypothesis that our intervention was effective and should be broadly implemented (60 vs. 35, p<.05)”

Example 1

RESULTS

Gateway courses included Biology, Organic Chemistry, Calculus and Physics . Calculating the GPA for these courses, we found that on average , the GPA was lower for UR students compared to non-UR students (p<.05). This gap could not be explained by differences in prior academic performance. However, no statistically significant differences were found in the rate in which UR students completed at least four gateway courses compared to white  students. However , after accounting for the lower grades of UR students in their initial classes, UR students were more likely than non-URM students to complete four or more gateway courses (p<.01).

Example 2

RESULTS

On average, the GPA on gateway courses was 0.63 lower for students who are UR (N=1,800) compared to students who are not UR (N=3,200, R2=0.3, p<.05). This gap persisted after adjusting for measures of prior academic performance. However, logistic regression showed that students who are UR were nearly as likely students who are not UR to persist in completing at least four gateway courses. After accounting for the lower grades of students who are UR in their initial classes, UR students were more likely than non-UR students to complete four or more gateway courses (adjusted odds ratio = 1.25, p<.01).

EVALUATION PLAN OPTION

When evaluation is planned but not completed

• Most appropriate for curricular innovations where work has been done to develop the curriculum but it has not yet been implemented

• Indicate type of data to be collected• From whom• How analyzed• Timeline

• In a way, an extended methods section

DISCUSSION

Discussion describes what the results mean

• Interpretation of results

• General reaction to findings

• Statements of limitations

• Thoughts on study’s impact

• Proposals for next steps

Discussion pulls all information together

• Discussion must relate to problem statement/ hypothesis

• Interpretation of study’s findings• What do the results mean?• Do they answer the original hypotheses or problem statement?

Discussion: Common pitfalls• Assertions are made that are not supported by the data• Rehash of the results without interpretation:

“In this study we found that the intervention group performed significantly better than controls”

• Obvious limitations are ignored

Example 1

DISCUSSION

Scores in Gateway courses were statistically lower for UR students compared to white students. However, they completed four or more of these courses at the same rate.  Our study clearly shows that academic support for UR students in college targeting gateway courses is crucial for increasing the diversity in medical schools.

Example 2

DISCUSSION

Despite academic challenges the minority students experience in college, many persist in their pre-health course requirements. Interventions at the college level to improve minority student performance in gateway courses are particularly important for increasing the diversity of medical schools

REVIEW CRITERIA

UCSF Education ShowcasePurpose: learly describe the project's goals and objectives. If you have a hypothesis, please state it here.Background: Describe how project was informed by knowledge of previous scholarly work or knowledge of educational principles.Methods: Describe the approaches used for this project. Who was studied or who will evaluate this project? What data were collected and how?Results/Evaluation Plan: Describe the findings of the study/evaluation including response rate, results, and statistical testing where appropriate.ORDescribe the outcomes that will tell you whether your curriculum is effective.

Discussion/Dissemination: State concisely the data-driven conclusion of your study/evaluation and how the results of the project are being (or will be) made available to colleagues.ORDescribe your plans for making the results of your project available to colleagues (talks, posters, publications, websites, etc.).Reflective Critique: Describe the steps taken to solicit feedback regarding your project and how it has been (or will be) modified based on the feedback.

WGEA: Oral Abstract Presentations & Posters

• :

Clarity of Purpose

Strength of Design

Reliability and Validity of Methods

Significance of Results/Outcomes

Soundness of Conclusions/Reflective Critique

Reviewers make a final recommendation

Definitely Include Highly rated in most/all categories; highly relevant

Probably Include Few concerning areas; can be improved before presenting. Relevant

Probably Exclude Several areas of concern, will require significant revision to remedy

Definitely Exclude Many areas of concern; fatal flaws present; needs substantial revision

REVIEWERS ETIQUETTE

Conforming to Submission Requirements

• As a reviewer, it is important to be familiar with the submission requirements

• Some submission systems reject an abstract that exceeds the word count• If the abstract seems long, check the word count and comment on

it in the review if excessive

General Rules• Always write comments

• Engage authors in feedback

• Your chance to convey:• Concerns/suggestions for improvement• Compliments• Reasons for ratings of abstract elements

• Consider writing comments FIRST, then use the rating form

Written Comments • Be respectful

• Be specific

• Be constructive

• Be focused

Example of Comments

• “The worst abstract I’ve ever read!”

• “Data presentation was pitiful.”

• “Wonderful study!”

• “The authors need to attend to several areas before resubmitting; these include:”

• “The authors need to present the actual data and statistical analyses.”

• “This was an excellent study for the following reasons:”

Poor Improved

Are you Available to Review? • Most reviews must happen within a discrete (and often

short) time period

• Look at your calendar before you agree to review

• If the timing is bad, politely DECLINE!

Comfortable with the Content?• Are you comfortable addressing the specific topics of

each submission?

• If you are not an appropriate reviewer, return it immediately so other reviewers can be found

REVIEWING EXERCISE