+On assessing student understanding of the nature of scientific knowledge

Nancy RuggeriDepartment of Curriculum & InstructionUniversity of Wisconsin-Madison

+Ways of Knowing in the Sciences Integrated Liberal Studies 153

General science course for non-majors

Taught in the ILS program on campus

~40 students per semester

Activity and discussion-oriented classroom

Four modules: Planetary motion, Plate tectonics, Evolutionary theory, Global climate change

+Course goals

To understand how scientific knowledge develops (the how instead of the what)

To develop critical thinking skills via scientific reasoning

To increase student interest in science

To increase student awareness of science and its integral relationship with society

+Course objectivesAfter taking ILS 153, students will gain a better understanding of:

• How scientists use different ways of knowing via empirical (inductive) and theoretical (deductive) approaches to answer questions about the natural world

• The methods scientists use to answer questions (observation, experimentation and modeling)

• The notion that scientific knowledge is subject to change given new data, interpretations and models

• The types of uncertainties that are inherent in scientific data and interpretations

• The ways in which scientists address uncertainty in scientific data and interpretations

• The role of the scientific community in placing checks and balances on the acceptance of scientific findings

• The ways in which politics, economics and religion can influence scientific practice

+Selected Readings Bryson, B. A Short History of Nearly Everything. . Broadway Books. New York,

NY.

Shermer, M. 2002. Why People Believe Weird Things: Pseudoscience, superstition, and other confusions of our time. Henry Holt and Company. New York, NY.

Huff, D. 1954. How to Lie with Statistics. W.W. Norton & Company. New York, NY.

Feynman, R.P. 1998. Uncertainty of Science. In: The Meaning of it All: Thoughts of a Citizen-scientist. Perseus, New York, NY , USA.

Oreskes, N. 1999. The Rejection of Continental Drift: Theory and Method in American Earth Science. Oxford University Press.

Weiner, J. 1994. The Beak of the Finch: A Story of Evolution in Our Time. Vintage Books. New York, NY.

Pollack, H. 2003. Uncertain Science… Uncertain World. Cambridge University Press.

Weart, Spencer R. 2003. The Discovery of Global Warming. Harvard University Press. Cambridge, MA.

NYT Science Times

Excerpts from Darwin, Galileo (Great Books series)

+Self-reported majors from students in ILS 153 (n= 33 with 37 reported majors)

+How would you describe the instructional approach taken in this class?

• “Open-minded and creative”

• “Lots of interactive lectures and class discussions”

• “Very much participation based. We used each other’s knowledge to build our understanding of concepts”

• “Professors tried to shift the paradigm of science by developing critical thinking skills, not just deliver knowledge of science which already prove by others”

+At issue: How to best elicit student thinking about the nature

and process of science?

How are the complex elements of the process of science identified in the course objectives integrated into students’ understanding?

+Assessment in ILS 153

Formative assessment Administered pre-instruction or during instruction to

inform both teaching and learning processes

Summative assessment Measure extent to which students have achieved a

particular learning goal

+Assessments Implemented in ILS 153Assessment Formative Summative Graded

Pretest/Posttest

√

Reflective Journals

√ √ √

Writing Assignments

√ √ √

In-class Group Activities

√ √

Large Group Discussions

√ (√)

Small Group Discussions

√

Minute Papers √

Quizzes & Exams

(√) √ √

Student Assessment of Learning Gains (SALG)

(√)

+PRETEST responses: How would you describe how scientists (biologists, chemists, earth scientists) do their work? For example, what types of activities do they do to learn about the natural world?

Jake: Scientists do their work by looking for problems. Through experiments, scientists come to conclusions by the results and evidence ~ support. They essentially solve problems and find answers

Susan: They must observe and do experiments. They must take detailed recordings down and apply them to their question at hand.

Elina: They read scientific journals, observe the world around them, and theorize based on these two things.

+POSTTEST responses: How would you describe how scientists (biologists, chemists, earth scientists) do their work? For example, what types of activities do they do to learn about the natural world?

Jake: There is no one way of doing science. Generally, scientists work inductively or deductively. They collect data from observation or come to conclusions about principles. Then they test these findings to see if they hold up, and publish all aspects for peer review.

Susan: They take information and produce models of how certain natural processes work. They must be highly objective and review others’ work as a community.

Elina: Scientists are in the business of observation and explanation. Essentially they find something interesting, observe a data field regarding it (be it natural, experimental, or quantitative) and attempt to explain the data. They can also reverse the order and come up with a theory, finding data to support it.

+Concerns How do we avoid having students just telling us what

we want to hear?

Are they simply memorizing aspects of the process of science that we highlight, and regurgitating them?

Will this “understanding” be forgotten two weeks after the semester is over, much like other content knowledge?

+Possible solutions

Create assessments that require students to integrate the information in context

Students make personal connections to their understanding

Reflections and essays responses can provide a broader demonstration of student understanding

+Weekly Journal Reflections

Reiterate themes from reading and lecture

Opportunity for students to synthesize complex notions

Ideally provides an opportunity to see one’s own changes over the course of the semester

+Example Journal Reflection Questions

This week’s reflection question is associated with this blog on the NY Times:http://judson.blogs.nytimes.com/2009/02/10/guest-column-letting-scientists-off-the-leash/

We would like you to write a 1-2 page response to the article, as if you were going to post online.  We would like you to write it from your perspective as an undergraduate at a research university, and explicitly include at least one piece of information that you have learned in this class.  

Course objective: The ways in which politics, economics and religion can influence scientific practice

+Example Journal Reflection Questions

“Scientists are not free agents, historians and sociologists have argued, and the social context of their work not only delimits their options but may even determine the content of their knowledge. And if all knowledge is socially constructed, then objectivity is a chimera. This radical claim strikes at the heart of scientists’ beliefs about their enterprise.”

After reading the text surrounding this quote from Naomi Oreskes (1999), discuss the valid points she makes as a historian, and compare them to the valid points Pollack (2003) makes (in the reading for today) as a scientist. Be sure to support your argument with evidence.

+Final Journal Reflection

For Monday's reflection, please choose two of the overarching themes from the course overview and discuss how the themes interrelate with one another, drawing on specific examples from the whole semester.  

The more examples you use, the better. If you can think of other themes from the course, you are welcome to write about these as well. We would be very interested to hear them. This assignment will help you to prepare for the final.

+Some excerpts from students’ final reflections:

“The nature of science is very much a mess of interrelating ideas. Scientists have to deal with society, politics, objectivity, honesty, uncertainty, and many other factors when they create and present their work.”

“I would argue that… the idea that science is influenced by cultural norms, is essentially the same thing as saying that science is done by humans that are inherently biased. Scientists do, indeed, strive for objectivity, but at the end of the day, they are still (as shown by the op/ed article we read) very dependent on governmental and private donations to finance their research.”

+

+Final Exam

Had there been a final paper for this course instead of an exam, write a short summary or outline of the 6-8-page paper you would have written. Provide the main thesis and some of the examples from the class you would have included in your paper. (Obviously, you cannot include everything we have talked about this semester, so you must choose a thesis and be selective about your examples!)

Do you think your notions of how science is done and its implications for society have changed as a result of taking this course? If so, how do you think they have changed and why? If not, explain why not.

+FINAL EXAM: Do you think your notions of how science is done and its implications for society have changed as a result of taking this course? Is so, how do you think they have changed and why? If not, explain.

John: “Yes. I never put together the pieces of how science in society are and so intertwined. The article about the funding of science made it really clear to me how science can’t function without funding with requires the scientists to cater to these funders and often practice fast science rather than good science…”

Elina: “My assumptions about science have changed because I actually view it as more certain now. I was a major skeptic when I came into the class. I did not know the power behind a scientific theory. Things that people take as fact like evolution and climate change…I understand, now, the difference between what is scientific and what is societal and how much society can dilute the certainties of science. I really appreciate how much more certain scientists actually are when they call something a theory.”

+SALG Data: How much did the Journal Reflections help your learning? (n=30)

+

What do students think they learned as a result of taking ILS 153?

+SALG: Please comment on how your understanding of the subject has changed CHANGED as a result of this class “I learned a ton from this class, not only about subject

material but about ways of knowing and scientific process”

“I am much more knowledgeable on all of these topics and feel more confident in talking about them. I am also more interested in each of them than I previously was.”

“I came in knowing a lot about science, but I definitely think more critically about things like science in the news”

“I am more skeptical of what I hear from people but I am also more confident in true scientific findings, because I know how much work it really takes.”

“greater appreciation for what scientist do”

“i am more interested in science now”

+SALG: What will you carry with you into other classes of other aspects of your life? “Using evidence to support arguments was a huge gain

from this class. It is easier to get my points across because I know more effective ways of going about it now.”

“I feel like analyzing data and being critical of information and ideas presented to me will really help in reading research in my area of interest/career, but I don't see using the big concepts at all in my daily life.”

“I will be more skeptical of data and the arguments of scientists that are not backed by the scientific community as a whole.”

“Definitely being aware of uncertainty, and those that make outrageous claims with shady evidence.”

“I'm surprised how much I've talked about this class and used the information in conversations and to my benefit.”

Assessment Strengths Weaknesses

Pretest/Posttest Has potential to measure targeted learning gains and

identify prior conceptions

Difficult to design adequate instrument; student conceptions

weak; tendency to tell us what we want to

hear

Reflective Journals Excellent opportunity for students to “play

with ideas”; instructors get sense of how well students

are integrating understanding

Very labor and time intensive to grade;

clumsy to collect and hand back; difficult to

grade

Student Assessment of Learning Gains

(SALG)

Very informative; preliminary data analysis; online;

anonymous

Lengthy; requires students to complete online; response rate

can be low

Assessments: Strengths and Weaknesses

+Course goals

To understand how scientific knowledge develops (the how instead of the what)

To develop critical thinking skills via scientific reasoning

To increase student interest in science

To increase student awareness of science and its integral relationship with society

+Challenges

Complexity in describing the nature and process of science

The moment we isolate individual elements, they lose the importance of context

Contextual nature of this makes it challenging

+Future changes for ILS 153 assessment

Consider assigning a final paper to synthesize complexity of ideas

Build stronger connections between assessments that target one course theme over the semester