researching student learning in the geosciences steve semken this session is intended for those who...

Researching Student Learningin the Geosciences

Steve Semken

This session is intended for those who have a working understanding of how learning research is applied to teaching.

It will include a practical survey of quantitative and qualitative methods that can be used to collect and analyze data on student learning in different settings.

July 2010

22

Why conduct research on your students’ learning?

It makes you a better teacher.

Feedback on how well your students learn is an indispensable part of reflective, professional teaching.

Sharing your findings with colleagues could also benefit them.

It could benefit you professionally.

Your findings may be publishable in any of a number of refereed journals: Journal of Geoscience Education, Journal of College Science Teaching, Science Education, Journal of Research in Science Teaching, International Journal of Science Education…even Science!

Your work could seed or enhance a grant proposal: NSF, NASA, NOAA, DOE science proposals typically require a Broader Impacts or Education & Outreach component. These agencies also offer many funding programs that specifically target science education (both research and practice).

It could even further the contributions of geoscience to the scholarship of teaching and learning….

33

Before we begin, consider these points….

Human beings are what we study in teaching and learning research.

The theories that inform this research and the tools we use to carry it out come from the social and behavioral sciences.

Many of these ideas and methods may seem foreign to researchers who have been trained solely in the natural sciences.

Research involving students is thus human subjects research.

The well-being and protection of students should always be our foremost concerns (and are legal and ethical obligations…more on this later).

Shared interest in improving the quality and effectiveness of science education has led to unprecedented collegiality and collaboration among scholars of science and scholars of education!

Bush, S. D., Pelaez, N. J., Rudd, J. A., Stevens, M. T., Tanner, K. D., & Williams, K. S. (2008).Science faculty with education specialties. Science, 322, 1795-1796.

44

How do I conduct research on student learning?

1. How do I devise my research project?

2. What kinds of data should I collect?

3. What methods and tools will I use to collect the data?

4. How will I analyze the data and interpret the results?

5. What permission will I need to do this?

6. How do I disseminate my findings?

7. What else should I consider in doing learning research?

These discussions will be followed by a fewillustrative examples of recent studies of geoscience learning.

55

1. How do I devise my research project?

Learning research should be guided by a question to answer, an observation to explain, a hypothesis to test. …How do I come up with one?

What problems have you or your colleagues encountered in teaching?

Have you found relevant prior work in the literature that you could refine or extend?

Has a funding agency issued a Request for Proposals with a problem they’d like to see addressed?

What interests you about student learning?

…Is my proposed project feasible?

What is the setting (a course, a section, an exercise, a field trip, etc.)?

How many students will be involved? What level of cooperation will be needed?

How much time will it require? (Not just for data collection, but analysis, writing, etc.)

Are there any potential ethical conflicts?

…What similar work has been done?

Review the geoscience-ed literature to avoid reinventing any wheels, to identify potential pitfalls, and to help you select the right methods and tools for your project.

Trochim, W., & Donnelly, J. P. (2007). The research methods knowledge base (3rd ed.).Mason, OH: Atomic Dog Publishing. Online version at http://socialresearchmethods.net

66

How do we define and characterize our students’ learning?

The three domains of learning

2. What kinds of data should I collect?

Can a student recall the definition of an igneous rock?Can a student summarize the theory of plate tectonics?How does a student account for differences in water quality?Can a student critically review a published research article?How well can a student map the geology of an unfamiliar area?

Psychomotor“manipulating”Honing of motor

skills, manipulative abilities, athleticism

Affective“feeling”

Changes in interest, attitudes, values, and motivation

Cognitive“thinking”

Recall or recognition of knowledge;

develop-ment of intellectual abilities

and skills

Does a student really care about learning geology?Is a student comfortable working in the field?Does a student value knowledge relevant to environmental sustainability?What motivates a student to study hard and do well on an exam?

Terra celebra Terra nova Terra incognita

Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: Cognitive domain. New York: David McKay.Krathwohl, D. R., Bloom, B. S., & Masia, B. B. (1964). Taxonomy of educational objectives: Affective domain. New York: Longman.

How do physical ability (or disability) impact field learning?Do reflexive actions play a role in field geoscience?

77


Qualitative methodsDescriptive data captured verbally, textually, or bymeans of observations; interpreted and coded

Quantitative methods

Data captured numerically and analyzed by

mathematical and statistical methods

“Mixed-methods”research combinesboth quantitativeand qualitative

methodsQuizzes, examsConcepTestsContent assessmentsConcept inventoriesAttitude surveysCourse evaluations(quantitative)

InterviewsFocus groupsObservationQuestionnairesStudent feedback(e.g., minute papers)Artifacts (e.g., assignments)Course evaluations(descriptive)Strongly

Agree1

Agree 2

Neutral 3

StronglyDisagree

5

Disagree 4

The Likert scale

To be most useful and effective, instruments must be bothvalid: a test tests what you expect it to testand reliable: a test works the same way every time it is usedValidity and reliability are established by expert review and statistical tests.

88

When do I collect the data? That depends on what you are looking for!

For example:


Formativeassessment

Front-end assessment

Pre-test

In-course

Post-test

Summative assessment

Student trackingTime

Course or activitystarts

Course or activityends

Ability to performa task

Du

rati

on

Comparison of different courses or methods

Retention ofcontent knowledge

Effects of course onsubsequent courseworkor career path

Changes in attitudesover course

Changes in contentknowledge over course

Preconceptions ofcontent knowledge

Pre-courseattitudes aboutsubject

Feedback on learning during a lesson

Test peer learning

Behaviors in class or field

99

If you are just starting out, don’t start from scratch!First try working with existing, well-used, valid and reliable methods and tools, which you can find in sources and references such as these:

Assessment Primer at SERC Starting Point: serc.carleton.edu/introgeo/assessment

Student Learning at SERC Cutting Edge: serc.carleton.edu/NAGTWorkshops/assess

Online Evaluation Resource Library: oerl.sri.com

Research Methods Knowledge Base: socialresearchmethods.net

Journal of Geoscience Education articles online: nagt.org/nagt/jge (Join NAGT and subscribe!)

Journal of College Science Teaching articles online: nsta.org/college (Join NSTA and subscribe!)

Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. Thousand Oaks, CA: Sage Publications.

Angelo, T. A., & Cross, K. P. (1993). Classroom assessment techniques: A handbook for college teachers. San Francisco: Jossey-Bass.

Feig, A. P., & Stokes, A. (Eds.) (In press, expected fall 2010). Qualitative research in geoscience education: GSA Special Paper. Boulder, CO: Geological Society of America.

Whitmeyer, S. J., Mogk, D. W., & Pyle, E. J. (Eds.). (2009). Field geology education: Historical perspectives and modern approaches: GSA Special Paper 461. Boulder, CO: Geological Society of America.

Manduca, C. A., & Mogk, D. W. (Eds.). (2006). Earth and mind: How geologists think and learn about the Earth: GSA Special Paper 413. Boulder, CO: Geological Society of America.


1010


Quantitative AnalysisIf you have designed your project well and collected data carefully, analysis should be straightforward.

Data preparation: includes data logging; checking data for accuracy; entering data into the computer; mathematically transforming data as needed; developing appropriate database.

Descriptive statistics: used to describe the basic features of the data in a study. Typically includes univariate analysis (distribution, mean, median, mode, std deviation), and correlation between and among variables (bivariate or multivariate).

Knowing how “well-behaved” your data aremakes the next step possible:

Inferential statistics: statistical methodsapplied to investigate questions, models, and hypotheses.Enables you to make inferences from your sample data.

Various tests for significance: t-test, Analysis of Variance (ANOVA), Analysis of Covariance (ANCOVA),regression analysis, factor analysis, etc. User-friendly software (e.g., SPSS) is readily available. Statistical significance of differences and gains is the gold standard!

Trochim, W., & Donnelly, J. P. (2007). The research methods knowledge base (3rd ed.).Mason, OH: Atomic Dog Publishing. Online version at http://socialresearchmethods.net

1111

Qualitative AnalysisTranscriptionData are typically analyzed in textual form. Audio- and video-recorded observations and interviews must first be transcribed.

CodingInterpreting the data in order to organize them and identify patterns, themes, and relationships. The qualitative analyst reads the data and demarcates various segments (single words to entire sentences or phrases) according to meanings. Each data segment is labeled with a code that allows it to be categorized for further interpretation.

e.g., Coding of responses by Petrified Forest National Park visitors to the question: How do you think the landscape you see [here] came to look the way it does now? (Bueno Watts, 2007).

(F) Erosion of water and wind.(M) 200 million years of time.(F) You can see the rocks falling already still.(M) I think it was a long process. You can see the layers.(F) uhh… You can see the water.(M) [C]learly the amount of sun and the lack of rain had a lot to do with it.

A priori coding: Coded segments are classified under predetermined categories.Emergent theme coding: Categories “emerge” as the data are coded.Typically, there is more than one coder, and they compare their interpretations to ensure reliability.


Bueno Watts, N. (2007). Visitor preconceptions and meaning-making at Petrified Forest National Park. Master’s thesis, School of Earth and Space Exploration, Arizona State University.

1212

Qualitative Analysis (continued)

Codingcan also be done mechanically, using computer programs (e.g., nVivo) that identify and count words, phrases, or sentences.

InterpretationWhen coding is complete, the nature and distribution of coded data are studied and interpreted to uncover meaningful patterns and relationships such as:

• student attitudes about a subject before and after a class or an entire course• behaviors observed during a lecture versus behaviors observed during a hands-on

lab• student preconceptions, misconceptions, and conceptions about subject matter

Note also that all coded qualitative data can be quantified, and analyzed mathematically and statistically as quantitative data (e.g., Chi, 1997).


Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. Journal of the Learning Sciences, 6, 271-315.

Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. Thousand Oaks, CA: Sage Publications.

Trochim, W., & Donnelly, J. P. (2007). The research methods knowledge base (3rd ed.). Mason, OH: Atomic Dog Publishing. Online version at http://socialresearchmethods.net

1313

5. What permission do I need to do this?

Any college or university that receives Federal research funding is legally required to maintain an Institutional Review Board (IRB) to approve and oversee all uses of humans as research subjects.

To work with students you will need IRB approval, even if your study is deemed exempt from subsequent IRB oversight.

The IRB will typically require:

•All investigators to have human subjects research certification (online; e.g., citiprogram.org);

•Informed consent from and ensured confidentiality of students;

•No suggestion of coercion (e.g., grades or recommendations);

•The ability for students to opt out of participation at any time.

hhs.gov/ohrp

1414

6. How do I disseminate my findings?

Presentations and abstracts:

• GSA/NAGT Annual Meeting; GSA Sectional Meetings; AGU Fall and Spring meetings; AAPG Annual Meetings all host numerous geoscience-education sessions and activities.

• National Association for Research in Science Teaching (NARST.org) and American Educational Research Association (AERA.net) hold interdisciplinary research conferences every Spring.

Peer-reviewed journal articles:

• Journal of Geoscience Education (NAGT.org/nagt/JGE), published by NAGT

• Journal of College Science Teaching (NSTA.org/college), published by National Science Teachers Association. NSTA also publishes other journals for secondary, middle, and elementary science education.

• Journal of Research in Science Teaching (www3.interscience.wiley.com/journal/31817/home), published by NARST.

• Science Education (www3.interscience.wiley.com/journal/32122/home); International Journal of Science Education (tandf.co.uk/journals/tf/09500693.html); Electronic Journal of Science Education (EJSE.southwestern.edu); the list goes on.

Peer-reviewed book chapters:

• GSA has published two education-focused Special Papers, and a third is in press.

• Academic publishers such as Springer publish numerous refereed volumes of contributed papers.

Community online resources:

• Science Education Resource Center (SERC.carleton.edu)…this one you should already be very familiar with!

• Digital Library for Earth System Education (DLESE.org)

1515

7. What else should I consider in doing learning research?

Make the purpose and process of your research clear to your students (through informed consent, but also by means of explanation).

Give students as much feedback as you can from your findings.

Focus your work on what interests you most.

Find out if there are institutional resources you can draw on.

Consider collaborating with faculty in your College of Education.

Consider collaborating with geoscientists and geoscience educators outside of your institution.

Consider funding graduate or undergraduate research assistants to work with you (in or out of your academic unit).

Be sure you know how your Department, School, or College views this research.

If you are tenure-track, determine if this research (including potential publications and grants) will help your tenure case.

Cross, P. K., & Steadman, M. H. (1996). Classroom research: Implementing the scholarship of teaching. San Francisco: Jossey-Bass.

1616

Examples of methods and tools applied in research studies

Example: Geoscience ConcepTests (Quantitative; Cognitive domain)ConcepTests are short multiple-choice conceptual questions designed for rapid quantitative assessment of student understanding (Mazur 1997).

Examine the map and answer the question: How many plates are present?

a. 3 (26% , 0%)b. 4 (19%, 18%)c. 5 (44%, 75%)d. 6 (11%, 7%)

Initial Responsesresponses following

discussion

When physical models were used in teachingcorrect responses increased to (56%, 84%)(McConnell et al. 2006)

Geoscience ConcepTest bank at SERC Cutting Edge: serc.carleton.edu/introgeo/interactive/conctest.htmlMazur, E. (1997). Peer instruction: A user’s manual. Upper Saddle River, NJ: Prentice-Hall.McConnell, D. A., Steer, D. N., Owens, K. D., Knott, J. R., et al. (2006). Using conceptests to assess and improve student conceptual

understanding in introductory geoscience courses. Journal of Geoscience Education, 54, 61-68.

1717

Example: Geoscience Concept Inventory (GCI) (Quantitative; Cognitive domain)The GCI (Libarkin & Anderson, 2005) is a valid and reliable multiple-choice instrument widely used to assess undergraduate-level geoscience content knowledge, and compare student learning across classes, institutions, regions, etc.The user designs a 15-question subtest from items in an online test bank.

Geoscience educators arealso invited to create newitems for the test bank:

GCI WebCentergci.lite.msu.edu

Libarkin, J. C., & Anderson, S. W. (2005). Assessment of learning in entry-level geoscience courses: Results from the Geoscience Concept Inventory. Journal of Geoscience Education, 53, 394-401.


1818

Example: Geoscience Concept Inventory (GCI)e.g., GCI used to measure geoscience content knowledge of 63 undergraduate participants in an entirely field-based introductory geology course (Elkins & Elkins, 2007).

•Students demonstrated statistically significant improvements in geoscience content knowledge.

•Students showed significantly greater improvement in knowledge as compared to students from other introductory geologycourses across the USA.

•Findings support the efficacy of geoscience teaching and learning in the field.

Elkins, J. T., & Elkins, N. M. L. (2007). Teaching geology in the field: Significant geoscience concept gains in entirely field-basedintroductory geology courses. Journal of Geoscience Education, 55, 126-132.


1919

Example: Motivated Strategies for Learning Questionnaire (MSLQ) (Quantitative; Affective and cognitive domains)

The MSLQ (Pintrich et al., 1991) assesses student motivation and strategies for learning. It is an 81-item instrument with six subcategories and no right or wrong answers (students respond using a Likert scale).It is valid, reliable, and used in many different educational settings.

e.g., GARNET Project (McConnell et al., 2009): Researchers administered the MSLQ to 340 physical -geology students in diverse colleges and universities.•No major differences in students’ initial motivation and learning strategies among different institutions were found.•Over the course of the semester, students lost test anxiety but also become less academically self-confident, and more likely to seek help from instructors or peers.

McConnell, D., Jones, M. H., Budd, D. A., Bykerk-Kauffman, A., Gilbert, L. A., Knight, K., Kraft, K. J., Nyman, M., Stempien, J. A., VislovaT., & Wirth, K. R. (2009). Baseline data on motivation and learning strategies of students in physical geology courses at multipleinstitutions: GARNET Part 1, Overview [Abstract]. Geological Society of America Abstracts with Programs, 41, 603.


2020

Example: Questionnaire for Semi-Structured Interviews (Qualitative; Affective and cognitive domains)“Semi-structured” means that while the respondent is prompted with a set of questions, he or she is free to extend the discussion in any direction.

This questionnaire was used to query students (mostly K-12 teachers) in a place-based Earth science course (Williams & Semken, in press).

•Items elicited cognitive and affective responses to curriculum, pedagogy, and student’s sense of the places studied.

•Interview transcripts were analyzed to uncover concepts, patterns, and relationships that were linked into thematic categories. •Positive responses to the place-based approach were reported by most students: enhanced place attachment and meaning, enhanced science comprehension, and enhanced teaching.

Williams, D., & Semken, S. (in press). Ethnographic methods in analysis of place-based geoscience curriculum and pedagogy. In A. P. Feig & A. Stokes (Eds.), Qualitative research in geoscience education: Geological Society of America Special Paper.


2121

Example: Script for “Think-Aloud” Experiment (Qualitative; Cognitive domain)“Semi-structured” means that while the respondent is prompted with a set of questions, he or she is free to extend the discussion in any direction.

Off-site study of the Trail of Time (Semken et al., 2009): a 4.5-km walking timeline trail and geological interpretative exhibition at Grand Canyon National Park (Karlstrom et al., 2008).

•A scaled simulation was used to investigate whether visitors could understand and navigate a geologic timeline trail in the Park.

•Specific tasks were posed to respondents as they walked.

•They were asked to “think aloud” as they performed the tasks.

•Findings showed that visitors can make sense of the time scalesand will be able to understand and learn from the actual Trail of Timeat Grand Canyon.

Karlstrom, K., Semken, S., Crossey, L., Perry, D., Gyllenhaal, E. D., Dodick, J., Williams, M., Hellmich-Bryan, J., Crow, R., Bueno Watts, N., & Ault, C. (2008). Informal geoscience education on a grand scale: the Trail of Time exhibition at Grand Canyon. Journal of Geoscience Education, 56, 354-361.

Semken, S., Dodick, J., Ben-David, O., Pineda, M., Bueno Watts, N., & Karlstrom, K. (2009). Timeline and time scale cognition experiments for a geological interpretative exhibit at Grand Canyon. Proceedings of the National Association for Research in Science Teaching, Garden Grove, California.


2222

Example: Concept Sketches (Qualitative; Cognitive and affective domains)

A concept sketch (Johnson & Reynolds, 2005) is a student- or instructor-generated sketch intended to illustrate the important or meaningful aspects of a concept or system.

Student-generated concept sketches reveal level of understanding of complex concepts, and can be coded as artifacts of learning.


Johnson, J. K., & Reynolds, S. J. (2005). Concept sketches—Using student- and instructor-generated, annotated sketches forlearning, teaching, and assessment in geology courses. Journal of Geoscience Education, 53, 85-95.

researching student learning in the geosciences steve semken this session is intended for those who...

Documents