assessment of process skills in ecology and conservation biology Cultivating skills for 21st century professionals: Development &

Ana Luz Porzecanski1, Eleanor Sterling1, Nora Byrum2, Adriana Bravo1, Romi L. Burks3, Andrés Gòmez1, Kimberley Landrigan1, Barbara J. Abraham4, Gillian Bowser5, Michelle Cawthorn6, John A. Cigliano7, Liliana M. Dàvalos8, Denny S. Fernandez9, Laurie Freeman10, Stuart R. Ketcham11, Tom A. Langen12, Joshua Linder13, John F. Mull14, Matthew I. Palmer15, David Roon16, Douglas Ruby17, Juerel Singleton17, Terry Theodose18 and Donna W. Vogler19

(1) Center for Biodiversity and Conservation, American Museum of Natural History, New York, NY, (2) Nicholas School for the Environment, Duke University, (3) Biology, Southwestern University, Georgetown, TX, (4) Biological Sciences, Hampton University, Hampton, VA, (5) Warner College of Natural Resources, Colorado State University, Fort Collins, CO, (6) Biology, Georgia Southern University, (7) Biological Sciences, Cedar Crest College, Allentown, PA, (8) State University of New York at Stony Brook, (9) Biology, University of Puerto Rico at Humacao, Humacao, PR, (10) Fulton Montgomery Community College, (11) Division of Science and Mathematics, University of the Virgin Islands, Kingshill, US Virgin Islands, (12) Biology, Clarkson University, Potsdam, NY, (13) Anthropology, James Madison University, (14) Weber State University, (15) Department of Ecology, Evolution and Environmental Biology, Columbia University, New York, NY, (16) Center for Natural Resources, University of Idaho, (17) Biology, University of Maryland Eastern Shore, (18) University of Southern Maine, (19) Biology, SUNY College at Oneota

GoalsA concern exists that US undergraduate science students do not currently develop important process skills needed as professionals. Our project aims to:

• Develop instructional materials that promote key student skills important for conservation biology and other integrative fields.

• Pilot these materials in the classroom to collect data on the development of those skills and determine what works best in terms of instruction.

Study Questions• Do students improve their abilities in relation to the targeted process skill? • Do students’ confidence in their abilities change in relation to the targeted

skill? • Do students accurately diagnose their own level of development in relation

to the targeted skill?• Are changes in process skill performance correlated with content-related

performance?• Is the intensity of the teaching intervention correlated with the overall gains

in process skills?

Exercise 2

Semesters 1 and 2

JUSTIFICATION

EXPERIMENTAL DESIGN

33 33 Skills Targeted• Critical thinking: the habit of mind characterized by the comprehensive

exploration of issues, ideas, artifacts, and events before accepting or formulating an opinion or conclusion.

• Data analysis: the systematic process of exploring issues, questions or problems through the collection and analysis of evidence that results in informed conclusions or judgments.

• Oral communication: a prepared, purposeful presentation designed to increase knowledge, to foster understanding or to promote change in the listeners’ attitudes, values, beliefs, or behaviors.

Student GainsIn Spring 2011, we (Sterling) piloted preliminary instructional materials developed for oral communication skills in a Conservation Biology class at Princeton University (N = 21).

Content: The class showed medium gains in content understanding between pre and post content assessments (Normalized gains pre-post: g = 0.51 ± 0.28; Figure 1A).

Confidence: Student confidence on their oral communication skills also increased between exercises (Wilcoxon signed-rank test: V = 16.5, P < 0.01; Figure 1B).

Skill: Oral communication skills increased significantly from the first exercise to the second (Normalized gains pre-post: g = 0.31 ± 0.83; Figure 1C). Overall, there were significant gains in delivery, content and supporting material, visual aids and text, and timing (Wilcoxon signed-rank test: P ≤ 0.05).

Although significant gains occured in content understanding, skill, and confidence in oral communication, there was no significant correlation among them at this stage (Spearman’s rank-correlation test: skill vs. content: P = 0.13; skill vs. confidence: P = 0.43).

Figure 1A. Frequency distribution of normalized content gains among students (N = 19); B. Pre and post student self-assessed confidence of their skill level. Box plots show the median, upper and lower quartiles, highest and lowest data values (N = 21); C. Frequency distribution of normalized skill gains among students in oral communication (N = 21).

PRELIMINARY RESULTS

Project participants at Black Rock Forest, NY, April 2011

Next StepsRevised, final materials for the three target skills will be piloted by at least 17 professors from a diversity of institutions during the next four semesters. For more information about this project: ncep@amnh.org.

This project is supported by the National Science Foundation CCLI/TUES Program (DUE-0942789). Any opinions, findings, and con-clusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Exercise 1

A Closer Look at the Assessment ToolsAssessment tools are used in conjunction with the exercises:

Professors use an Assessment

Rubric to evaluate and grade the

exercise and also the students’ level

of skill. Students complete a Post-Content Assessment about their knowledge of the subject.

Exercise 1

Students complete a Self-

Assessment about their level of confidence in

the skill.

Before EX1, students complete a short Pre-Content Assessment of their knowledge of the subject.

Two levels of teaching intervention, an intensive class study and a light, in-dividual student study, will take place between EX1 and EX2 in the first and second semester, respectively.

Teaching Intervention

Pre-Content Assessment Self-Assessment

Assessment RubricPost-Content Assessment

Pre-Content Assessment Self-Assessment

Assessment RubricPost-Content Assessment

A professor applies two exercises during the semester.

Both exercises target the same skill, such as “oral communication”.

frequ

ency

(stu

dent

s)

pre-assessment post-assessment

stud

ent s

elf-a

sses

smen

t sco

re

normalized skill gain (g)

frequ

ency

(stu

dent

s)

Key Questions Within SemestersGains in content?Gains in skill?Changes in confidence?Do they corrrelate with each other?

Key Question Across SemestersIs the intensity of the teaching intervention correlated with the overall gains in process skills?

B Cg > 0 gaing ≤ 0 no gain

normalized content gain (g)

g > 0 gaing ≤ 0 no gain

A