integrated course assessment in introductory psychology how can we measure what our students are...
TRANSCRIPT
Integrated Course Assessment in
Introductory Psychology
How Can We Measure What Our Students Are
Learning?
Jeff NevidSt. John’s University
To contact: [email protected]
© 2011Jeffrey S. Nevid
All Rights Reserved.
3 / 2
The Challenges
What do we want our students to learn?How do we know what they are learning?
Why We Became Teachers
It certainly wasn’t for the money. . .
Source: college_prof_football_coach_salaries.gifGraph comparing college professors' and college football coaches' average ...joeydevilla.com
AND it certainly wasn’t for this. . .
Which can lead to this. . .
Or maybe this (well, maybe not). But grading papers does feel like a prison at times.
There’s got to be an easier way to make course evaluation yield more gain and less pain.
•Source: Black Hills State University
Why measure learning outcomes?
To determine what our students are learning. Are our students acquiring the knowledge and skills we expect them to learn?
To improve learning. Knowledge from course assessments can inform how we teach and how we help our students acquire knowledge we expect them to acquire.
To meet accreditation standards. Course assessments provide data we can provide to accrediting bodies, such as state education departments, regional accrediting agencies, and professional organizations such as APA.
– We live in an age of accountability in which regular assessment of learning outcomes is an important tool in the accreditation process.
Background Reading
3 / 11
Using Psychology to Teach Psychology: The Four E’s of Effective Learning
1. Engage student interest
2. Encode important information
3. Elaborate meaning
4. Evaluate progress
3 / 12
The Four E’s of Effective Learning:A Useful Heuristic for Teaching
Bloom’s Taxonomy (Cognitive domain: original and revised)
Source: Edtechvision.org (left), http://newadventuresatwilkes.blogspot.com/2010/05/digital-learners-real-learning.html (right)
Bloom’s Taxonomy: Original and Revised
Source: Anderson & Krathwohl, 2001
Source: Royal Fireworks Press
Bloom’s Taxonomy (Revised):Underlying Dimensions
Revised Bloom Taxonomy
The Knowledge Dimension
The Cognitive Process Dimension
Remember Understand Apply Analyze Evaluate Create
Factual Knowledge List Summarize Classify Order Rank Combine
Conceptual Knowledge Describe Interpret Experiment Explain Assess Plan
Procedural Knowledge Tabulate Predict Calculate Differentiate Conclude Compose
Meta-Cognitive Knowledge Appropriate Use Execute Construct Achieve Action Actualize
Copyright (c) 2005 Extended Campus -- Oregon State Source:
University http://oregonstate.edu/instruct/coursedev/models/id/taxonomy/#tableDesigner/Developer - Dianna Fisher
Source: Center for Teaching and Learning, Brigham Young University
360 Action Learning Verbs: Find the Verbs that Match Your Learning Goals
The IDEA MODEL. . . A Simpler Model Based on Four Key Skills:
Identify . . .Key figures in psychology and parts of the body
Describe or Define. . Key terms and concepts
Evaluate evidence and Explain underlying processes and mechanisms
Apply. . . Concepts to examples
Integrated Course Assessment
Guided by three key principles. . .
Make it seamless Make it sensible Make it work (without requiring a lot of extra
work)
APA Guidelines for the Undergraduate Psychology Major
Undergraduate Psychology Learning Goals and Outcomes: Approved by APA Council, August 2006
The Guidelines for the Undergraduate Psychology Major provides details for 10 suggested goals and related learning outcomes for the psychology major, grouped into two major categories:
I. Knowledge, Skills, and Values Consistent with the Science and Application of Psychology
II. Knowledge, Skills, and Values Consistent With Liberal Education That Are Further Developed in Psychology
Each of these categories contains five goals:
APA Learning Goals 1-5
I. Knowledge, Skills, and Values Consistent with the Science and Application of Psychology
– Goal 1: Knowledge Base of Psychology – Students will demonstrate familiarity with the major concepts, theoretical
perspectives, empirical findings, and historical trends in psychology. – Goal 2: Research Methods in Psychology – Students will understand and apply basic research methods in psychology, including
research design, data analysis, and interpretation. – Goal 3: Critical Thinking Skills in Psychology – Students will respect and use critical and creative thinking, skeptical inquiry, and,
when possible, the scientific approach to solve problems related to behavior and mental processes.
– Goal 4: Application of Psychology – Students will understand and apply psychological principles to personal, social, and
organizational issues. – Goal 5: Values in Psychology
APA Learning Goals 6-10 II. Knowledge, Skills, and Values Consistent With Liberal Education That
Are Further Developed in Psychology
– Goal 6: Information and Technological Literacy – Students will demonstrate information competence and the ability to use computers
and other technology for many purposes. – Goal 7: Communication Skills – Students will be able to communicate effectively in a variety of formats. – Goal 8: Sociocultural and International Awareness – Students will recognize, understand, and respect the complexity of sociocultural and
international diversity. – Goal 9: Personal Development – Students will develop insight into their own and others’ behavior and mental
processes and apply effective strategies for self‐management and self‐improvement. – Goal 10: Career Planning and Development
What’s the NEW IDEA in Course Assessment?
The IDEA Model of Course Assessment™
Identify. . key figures in psychology, parts of the nervous system, etc.
Define or Describe. . . key concepts, key features of psychological theories, etc.
Evaluate or Explain. . . underlying processes and mechanisms of behavior and mental processes.
Apply. . . psychological concepts to real-world examples.
To learn how to apply the IDEA Model in class, contact me at: [email protected]
Bloom’s Taxonomy
Hey, what’s the new IDEA in integrated course assessment?
The IDEA Model. . .
Maps APA learning goals and outcomes to learning objectives Uses action verbs to measure learning objectives and acquired skills Integrates Bloom’s taxonomy with APA learning goals and outcomesKeys exam questions to learning outcomes and acquired skills Provides feedback about student mastery of learning objectives and types of acquired skills
THE IDEA MODEL™ OF COURSE ASSESSMENT
Integrating APA Learning Goals with Bloom’s Taxonomy
Skill Level in Bloom Taxonomy
(Original/Revised Model)
Learning Objectives
Active Learning Verbs used to Measure Skills
Acquired
APA Learning Goal 1: Knowledge Base of
Psychology
Examples: Memory Knowledge/Remembering Comprehension/Understanding
Define key terms Describe key concepts Identify parts of the nervous system, key figures in psychology, etc.
Define/Describe
Identify
Describe basic processes and stages of memory.
Describe types of long-term memory.
Identify methods of measuring memory.
Identify key brain structures involved in memory.
Application/Applying Apply concepts to examples Apply
Apply knowledge of how memory works to powering up your memory.
Analysis/Analyzing Synthesis/Evaluating Evaluation/Creating
Evaluate theoretical concepts Explain underlying mechanisms or processes Evaluate scientific evidence
Evaluate/Explain Evaluate the reliability of
eyewitness testimony. Explain the difference
between maintenance rehearsal and elaborative rehearsal.
Explain why the concept of recovered memory is controversial.
The IDEA Model™ Identify. . . Define or Describe. . . Evaluate or Explain. . . Apply . . . knowledge of psychology
The IDEA Model™ of Course Assessment. . .
Maps learning objectives to APA learning goals and Bloom’s taxonomy Uses active learning verbs to measure learning outcomes Keys test items to learning outcomes
APA Learning Goal 1. Knowledge Base of Psychology: Memory
SubGoals Learning Objectives Keyed Test Items
1. Demonstrate knowledge of processes and stages of memory
Describe the basic processes and stages of memory. 1, 2, 3, 6, 7, 9, 11, 13
Describe the different types of long-term memory. 4, 10, 12, 14, 15
Explain the roles of the semantic network model and levels-of-processing theory in memory.
17, 18, 19, 20
Explain the difference between maintenance rehearsal and elaborative rehearsal.
23, 26, 28, 30
Apply constructionist theory to explain memory distortions. 37, 38, 41
Evaluate the reliability of eyewitness testimony. 45, 46, 47
Explain why the concept of recovered memory is controversial.
49, 50, 51
2. Demonstrate knowledge of theories of forgetting, methods of measuring memory, and types of amnesia
Describe the major theories and factors in forgetting. 57, 59, 61, 62, 63, 64, 66, 68
Identify different methods of measuring memory. 71, 72, 74
Describe the features of two major types of amnesia. 79, 80, 83,84
Examples of Coded Test Items in IDEA Model™Foundations of Modern Psychology
Identify. . . Wundt is to ______ as James is to ______.
A) structuralism; Gestalt B) structuralism; functionalism C) behaviorism; Gestalt D) behaviorism; functionalism E) functionalism; psychoanalysis
Define or Describe. . . Psychology is best described as a science that studies
A) The role of the mind in explaining behavior B) How the mind controls our behavior C) Observable behavior only D) Mental processes only E) Behavior and mental processes
Evaluate or Explain . . . . Psychology is a scientific discipline in that it focuses on
A) the pursuit of truth, not simply opinion. B) testing opinions and assumptions in the light of evidence. C) systematically building theories to explain phenomena. D) behavioral, as opposed to mental, processes. E) accumulated wisdom of scholars.
Apply . . . Animal trainer Bob Jeffers uses rewards to teach his animals to perform circus tricks. Jeffers's techniques
are based on principles from which school of psychology? A) Behaviorism B) Structuralism C) Psychodynamic D) Functionalism E) Humanism
Need to Ground Pedagogy in Psychological Science Traditionally, pedagogical features in college textbooks development was driven by marketing concerns, not scientific research
To date, little research has been conducted to evaluate effectiveness of pedagogical techniques used in class and psychology textbooks
Textbook developers and instructor need to draw upon knowledge base in cognitive psychology
St. John’s Pedagogy Research ProgramTextbook Modularization Study
Participants: 96 introductory psychology students
Design : A randomized, counterbalanced design in which students read two text passages, one presented in a modularized format and the other presented in a narrative format.
Outcome measure: A 20-item multiple choice quiz measuring content acquisition.
Student Preference Measure: Students indicated which format they preferred overall and which presented information more clearly.
Results: No overall differences in test performance between the modular and traditional formats among students not expressing a preference. But students who preferred the modular approach performed significantly better (p < .05) better on the accompanying exam when they had read the material in their preferred format .
Take away message: Students who prefer a particular approach are likely to become more engaged in reading the material, which might translate into improved acquisition and retention.
Citation: Nevid, J. S., & Carmony, T. M. (2002). Traditional versus modular format in presenting textual material in introductory psychology. Teaching of Psychology, 29, 237-238.
Key concepts, not just key terms, are signaled (highlighted) in the margins of the text.
– Question: Does concept signaling help students encode and retain key concepts?
Participants: 80 introductory psychology students
Design : A randomized, counterbalanced design in which students read two text passages, one with concept signaling and the other without.
Outcome measure: A 20-item multiple choice quiz measuring content acquisition.
Major Findings: Students in the concept signaling condition performed significantly better on test items measuring knowledge of key concepts. No differences were found for non-signaled (surrounding) material.
Take-away message: Marginal inserts used as signaling devices may help address concerns that students often encode facts and definitions but have difficulty extracting key concepts from text material
Citation: Nevid, J. S., & Lampmann, J. L. (2003). Effects on content acquisition of signaling key concepts in text material Teaching of Psychology, 30, 227-229.
Concept Signaling Study
Three Common Problems Instructors Often Face
3 / 35
Poor Attendance
3 / 36
Late for Class
3 / 37
Inattention
3 / 38
Mastery Quizzing Study
What is Mastery Quizzing?
– Ten, pre-post quizzes during the semester of specific concepts discussed during class
– Students have two chances to get the right answer and earn credit toward final grade—at the very beginning of class and at the end
– Mastery quizzing provides incentives for attendance, punctuality, and attention
Participants: Introductory psychology class comprising 61 students, 44 women, 17 men, mostly freshmen
Method: Analyzed student performance on course exams, disaggregated by signaled concepts (concepts tested in mastery quizzes), related concepts (other concepts discussed on mastery quiz days), and non-signaled concepts (control concepts discussed on other days).
Mastery Quizzing Study
Results:– Students showed significant improvement in knowledge of mastery quiz content as
assessed by pre-post lecture comparisons.
– Credits earned on mastery quizzes predicted performance on course examination questions measuring signaled concepts and other concepts from lectures on days mastery quizzes were administered, but not on unrelated concepts (control concepts)
Take-Away Message: – Mastery quizzing cues students to attend to important concepts discussed in class,
and provides incentives for coming to class, coming on time, and paying attention.
Citation: Nevid, J. S., & Mahon, K. (2009). Mastery quizzing as a signaling device to cue attention to lecture material. Teaching of Psychology, 36, 1-4.
Current Study: Item Analysis of IDEA Model
STUDY DESIGN:
Participants: Two introductory psychology courses at St. John’s University, comprising 144 students, 62 males and 82 female, mostly freshmen
Method: Item analysis based on student performance on three non-cumulative multiple choice exams, with test items coded by type of acquired skill (Identify, Define or Describe, Apply, and Evaluate). Items drawn from textbook test-item file.
Outcome measure: Student performance on each item type aggregated across the three exams.
Interrater Reliability: 90% concordance in blind interrater agreement study.
Item Analysis: Computation of item type difficulty level and discriminability
Results: Internal Consistency
– Kuder-Richardson 20 coefficients: Range from .75 to .86 for item types (I, D, E, and A type
questions), demonstrating adequate internal consistency for all item types.
Also, intercorrelations among item types were high (rs = .69 to .83), which is suggestive of an underlying factor of general cognitive ability.
Results: Item Difficulty
– Consistent with the Bloom taxonomy, “E” questions (Evaluate or Explain) proved to be more difficult than “I “(Identify), “D” (Define or Describe) , and “A“ (Apply) questions.
– Also consistent with Bloom’s taxonomy, “A” questions were more difficult than “D” questions.
– “I” questions were more difficult than “D” questions.
– Average difficulty levels for the four item types fell within a moderate range of difficulty (range = .53 to .64).
Item Difficulty
0 0.2 0.4 0.6 0.8
Item Type
Difficulty Level
Identify
Define/Describe
Evaluate/Explain
Apply
The higher the discrimination index, the better the items are in discriminating between poorer and better students > .20 for marginally good discriminability
> .30 for reasonably good discriminability
“Evaluate” and “explain” questions were not only the most difficult items, but were also the best discriminating items:
– Highest item discrimination index overall (.33)– Highest percentage of items (61%) reaching .30 criterion of
reasonably good discrimination
Results: Discrimination Index
Discrimination Index
0 0.1 0.2 0.3 0.4
Item Type
Discrimination Index
Identify
Define/Describe
Evaluate/Explain
Apply
Item Difficulty and Item Discrimination by Item Type in IDEA Model
Item Difficulty
Item Type M SD
Identify .60 .14
Define/Describe .64 .15
Evaluate/Explain .53 .15
Apply .60 .12
Item Discrimination Index
Item Type M SD
Identify .29 .17
Define/Describe .32 .14
Evaluate/Explain .33 .15
Apply .27 .17
Note: Difficulty is based on the proportion of students answering items correctly, which is averaged by question type. The item discrimination index represents the difference between the proportion of students answering an item correctly in the top 27% of the class versus the bottom 27% of the class, averaged by question type.
Item Analysis (contd.)
Summary:
The action verbs used in the present study comprised skills relating to identifying, defining or describing, evaluating or explaining, and applying knowledge of basic concepts in psychology.
Exam performance demonstrated that higher level cognitive skills represented by action verbs evaluating and explaining were the most difficult for students to acquire and also the best discriminating items.
Take-Away Message:
– The IDEA model allows instructors to integrate APA learning goals with course examinations by coding test items that measure acquired skills consistent with learning objectives.
– The IDEA Model provides a heuristic framework for evaluating areas of relative strength and weakness in acquired skills in college coursework organized within a hierarchical model of learning goals (Bloom’s taxonomy).
– By identifying skills deficiencies, instructors can tailor teaching strategies to strengthen these types of learning outcomes.
Before we end. . .
Four Key Steps for Mapping Active Learning Verbs to Learning Outcomes
Step 1. . .
Determine the skill set you would like your students to acquire in a particular course
• Specify learning objectives for each text chapter or instructional unit
Step 2 . . .
Select learning verbs to measure skills you want to assess
• Use action verbs to frame learning objectives
Step 3 . . .
Key course exams to learning objectives and action verbs
– Write items for each type of action verb– Use textbook test-item file (you may need to
retrofit key items to particular learning verbs)
Step 4 . . .
Measure student performance on exams to evaluate student competencies
• Use machine-scored (Scantron) forms to break down performance on each acquired skill (e.g., Identify . . . Define or Describe . . . Evaluate or Explain . . Apply)
• Basic Scantron system can be set up to provide four subtest scores (I,D,E,A), two on front and two on back of standard Scantron form
• Advanced Scantron system connected to computer can generate any combination of subtest scores
Step 5. . .
Use course outcome data to measure student progress and identify areas requiring further improvement
• Rinse and repeat each semester or academic year
3 / 55
Sample References from the St. John’s University Pedagogy Research Program
Nevid, J. S., & Carmony, T. M. (2002). Traditional versus modular format in presenting textual material in introductory psychology. Teaching of Psychology, 29, 237 – 238.
Nevid, J. S. (2003, September). Helping students get the point: Concept signaling as a pedagogical aid. Paper presented at the conference, Taking Off: Best Practices in Teaching Introductory Psychology, Atlanta, GA.
Nevid, J. S., & Lampmann, J. L. (2003). Effects on content acquisition of signaling key concepts in text material Teaching of Psychology, 30, 227-229
Nevid, J. S. (2004, January). Graphing psychology: The effective use of graphs and figures in teaching introductory psychology. Invited address at the presented at the 26th Annual National Institute on the Teaching of Psychology, St. Petersburg, FL.
Nevid, J. S. (2004, February). Evidence-based pedagogy: Using research to find new ways to help students learn. Invited closing address presented at the 11th Midwest Institute for Students and Teachers of Psychology (MISTOP), Glen Ellyn, IL.
Sample References (contd.) Nevid, J. S., & Forlenza, N. (2005). Graphing psychology: An analysis of the most commonly
used graphs in introductory textbooks. Teaching of Psychology, 32, 253-256.
Nevid, J. S. (2006, February). In pursuit of the “perfect lecture.” APS Observer, Teaching Tips, Vol. 19(2).
Nevid, J. S., & Blitzer, J. R. (2006, August). Educational benefits of mastery quizzes as signaling devices. Poster presented at the annual meeting of the American Psychological Association, New Orleans, LA.
Nevid, J. S., & Mahon, K. (2009). Mastery quizzing as a signaling device to cue attention to lecture material. Teaching of Psychology, 36, 29-32.
Nevid, J. S. (2009/2010, Winter). Reaching and teaching millennial students. Psychology Teacher Network, 19 (4) pp. 1, 3, 4.
Nevid, J. S. (2011). Teaching the millennials. APS Observer, Teaching Tips, in press.
Nevid, J.S., McClelland, N., & Pastava, A. (2011, August). Using action verbs as learning outcomes in introductory psychology. Poster to be presented at the meeting of the American Psychological Association, Washington, DC.
