cengage learning webinar, course redesign, assessing learning & cost in course redesign
DESCRIPTION
Building on the overview of the course redesign provided in our webinar in April entitled "Improving Outcomes and Reducing Costs: The Case for Redesign," Dr. Carolyn Jarmon of the National Center for Academic Transformation described the methods institutions have used to measure the impact of course redesign on student learning and instructional costs. Providing specific examples from institutions that have used these methods to prove their success, she also shared the tools that these institutions have used to demonstrate increased learning and reduced costs, so that other institutions seeking to validate their success with course redesign are equipped to do so.TRANSCRIPT
Assessing Learning and Cost in Course Redesign
with Dr. Carolyn Jarmon, Vice President for The National Center for Academic Transformation
REDESIGNING STUDENT LEARNING ENVIRONMENTS:
Measuring Cost and Quality
POLL #1
What stage is course redesign at your institution?
See the poll on the right of your screen…
WHAT IS THE MOST IMPORTANT THING THAT WE HAVE LEARNED ABOUT QUALITY AND COST?
The factors that lead to increased student learning and increased student retention are the same as those that lead to reduced instructional costs!
FIVE PRINCIPLES OF SUCCESSFUL COURSE REDESIGN
#1: Redesign the whole course– Quality: Eliminate “course drift”; greater
course coherence and quality control– Cost: Eliminate duplicate effort; create
opportunities for alternate staffing
FIVE PRINCIPLES OF SUCCESSFUL COURSE REDESIGN
#1: Redesign the whole course– Quality: Eliminate “course drift”; greater
course coherence and quality control– Cost: Eliminate duplicate effort; create
opportunities for alternate staffing
FIVE PRINCIPLES OF SUCCESSFUL COURSE REDESIGN
#2: Encourage active learning– Quality: “Learning is not a spectator
sport.”– Cost: Reduce faculty preparation and
presentation time; reduce grading time(e.g., interactive software, peer learning teams)
FIVE PRINCIPLES OF SUCCESSFUL COURSE REDESIGN
#3: Provide students with individualized assistance– Quality: Students get help when they are
“stuck” and stay on task rather than giving up: software tutorials, F2F in labs or help rooms, “beep a tutor,” SMARTHINKING
– Cost: Apply the right level of human intervention: peer tutors, course assistants
FIVE PRINCIPLES OF SUCCESSFUL COURSE REDESIGN
#4: Build in ongoing assessment and prompt (automated) feedback– Quality: Enables practice, diagnostic
feedback, focused time on task– Cost: Good pedagogy with large numbers of
students; individual and group assessment; faculty spend time on what students don’t understand
FIVE PRINCIPLES OF SUCCESSFUL COURSE REDESIGN
#5: Ensure sufficient time on task and monitor student progress– Quality: Self-pacing vs. milestones for
completion; points for engagement– Cost: Course management systems can
reduce costs while increasing oversight
READINESS CRITERIA• What does it mean to be “ready” to do a major course redesign?• Is your institution ready?• Which courses are “ready”—i.e., are good candidates for a comprehensive
redesign?• What decisions need to be made?
READINESS CRITERION #1Course Choice
• What impact would redesigning the course have on the curriculum, on students and on the institution—i.e., why do you want to redesign this course?
POLL #2
Using the chat function, what course (or courses) is your institution considering for
an upcoming redesign?
Write the course title in the chat box.
FACTORS TO CONSIDER WHEN THINKING ABOUT HIGH IMPACT
• High drop-failure-withdrawal rates• Student performance in subsequent courses • Students on waiting lists• Student complaints• Other departmental complaints• Lack of consistency in multiple sections• Difficulty finding qualified adjuncts
READINESS CRITERION #2Redesign Model
• Which redesign model do you think would be most appropriate for your redesign? Why?
• What aspects fit your particular discipline and your particular students?
REDESIGN MODELS• Supplemental – Add to the current structure and/or change
the content • Replacement – Blend face-to-face with online
activities• Emporium – Move all classes to a lab setting
Fully online – Conduct all (most) learning activities online• Buffet – Mix and match according
to student preferences• Linked Workshop – JIT workshops
linked to college level course
READINESS CRITERION #3Assessment Plan
• Which assessment model do you think would be most appropriate for your redesign? Why?
ASSESSMENT GOAL
To establish the degree to which
improved learning has been achieved as a result of the course redesign.
ASSESSMENT PLANNING
Step 1. Establish the method of obtaining data.
Step 2. Choose the measurement method.
ESTABLISH THE METHOD OF OBTAINING DATA
• Baseline “Before” (traditional) and “After” (redesign)
• Parallel Sections – Compare traditional sections and redesigned sections
CHOOSE THE MEASUREMENT METHOD: FOUR MODELS
A. Comparisons of Final ExamsB. Comparisons of Common Content
Items Selected from ExamsC. Comparisons of Pre- and Post- TestsD. Comparisons of Student Work using
Common Rubrics
COMMON EXAMSUniversity of Massachusetts
In spite of more difficult questions, student scores on exams in a redesigned biology course averaged 73% vs. 61% in the traditional course.
COMMON ITEMSCarnegie Mellon University
• Redesign students showed significant improvement based on comparing exam questions.
• In solving problems, redesign students made fewer than one error per problem, whereas traditional students made about six errors per problem.
PRE/POST TESTSUniversity of Central Florida
Students in the traditional format posted a 1.6-point improvement on a content exam, whereas at 2.9, the mean change for students in the redesigned course was almost double that amount.
COMMON RUBLICSTallahassee Community College
Students in a redesigned English composition course scored significantly higher on final essays, with an average score of 8.34 compared to 7.33 for traditional students.
More Is Not Better!
• Performance in follow-on courses• Student attitude toward subject matter• Student interest in pursuing further
coursework in the discipline• Differences in performance among student
subpopulations• Deep vs. superficial learning
GRADES ARE NOT A SUFFICIENT MEASURE OF STUDENT LEARNING
• Lack of consistency• Different coverage• Different tests and
exams• Curving • Inflation
Use only for course completion!
READINESS CRITERION #4Cost Savings Plan
• Which cost savings strategy do you think would be most appropriate for your redesign? Why?
COST SAVINGS GOALCreate cost savings that can
be used to sustain ongoing redesign, to fund future operations and to
free up resources for program and/or
institutional priorities.
STEP 1: WHAT’S YOUR ENROLLMENT SITUATION?
• Is your enrollment growing or projected to grow?
• Is your enrollment stable or declining?
STEP 2: CHOOSE LABOR-SAVING TACTICS
• Coordinated development and shared instructional tasks for individual development and delivery.
• Software for face-to-face. • Automated grading of homework, quizzes, exams
for hand grading. • Course management software for human
monitoring and course administration. • Interaction with other personnel for one-to-one
faculty/student interaction.
STEP 3: CHOOSE A COST REDUCTION STRATEGY
• Each instructor carries more students by– Increasing section size.– Increasing the number of sections carried.
• Change the mix of personnel teaching the course.
• Do both simultaneously.
COST REDUCTION EXAMPLE
Traditional• Instructor load = 4
sections • 150 minutes/week
3 50-minute or 2 75-minute meetings
• Time spent = 600 minutes per week
Redesign• Instructor load = 6
sections • 150 minutes/week
2 50-minute meetings• Time spent = 600 minutes
per week • Option: Reduce # of sections
from 12 to 8 per semester, saving an additional $32,000
COST REDUCTION EXAMPLE
Traditional• Each instructor
teaches 1 section • Section size = 25• Time spent = 200
hours
Redesign• Time spent = 100
hours• Options:
– Each instructor = 2 sections of 25
– Each instructor = 1 section of 50
GENERAL PSYCHOLOGYFrostburg State University
Traditional• 2 meetings/week• Sections of 50• 9 FT and 9 PT
faculty• $89 cost-per-
student
Redesign• 1 meeting/week• Sections of 150• 1 FT and 3 PT faculty;
ULAs in lab• Required lab time• $26 cost-per-student
ENGINEERING PHYSICSAuburn University
Traditional• 750 students annually• Lecture, lab, recitation
structure• 14 GTAs• High DFW rates• Passively unengaged
students• $390 cost-per-student
• Redesign• Online activities completed
before class• Physics activity session
(combined lab and recitation)• 9 GTAs• Online: pre-tests, post-tests,
problems, conceptual inventories
• $219 cost-per-student
STATICSMississippi State University
Traditional • 3 hours of lecture• Students do homework
and problem-solving outside of class (the toughest part!)
• 5 FT and 5 PT faculty teach 11 sections of ~35 students
• $299 cost-per-student
Redesign• 3-hour laptop lab &
hands-on experiments• Peer-to-peer and GTA
assistance • Lectures streamed for
students to watch and review on their own
• One instructor teaches all sections supported by GTAs and ULAs
• $252 cost-per-student
FAIRFIELD UNIVERSITYGeneral Biology
Traditional• 7 sections (~35)• 7 faculty• 100% wet labs• $131,610• $506 cost-per-student
Redesign• 2 sections (~140)• 4 faculty• 50% wet, 50% virtual• $98,033• $350 cost-per-student
COURSE PLANNING TOOL
A decision-making tool that enables
institutions to compare the
“before” activities and costs (the
traditional course) and the “after”
activities and costs (the redesigned
course)
Instructional Costs per Hour
TRADITIONAL COURSETraditional Course Tenure-Track Non-Tenure- Adjunct/
Faculty Track Faculty TAs/GAs PT FacultyHourly rate = Hourly rate = Hourly rate = Hourly rate =
# of Hours Total Cost# of Hours Total Cost# of HoursTotal Cost# of Hours Total CostI. Course Preparation
A. Curriculum Development $0 $0 $0 $0B. Materials Acquisition $0 $0 $0 $0C. Materials Development 1. Lectures/presentations $0 $0 $0 $0 2. Learning materials/software $0 $0 $0 $0 3. Diagnostic assessments $0 $0 $0 $0 4. Assignments $0 $0 $0 $0 5. Tests/evaluations $0 $0 $0 $0Sub-Total 0 $0 0 $0 0 $0 0 $0
D. Faculty/TA Development/Training 1. Orientation $0 $0 $0 $0 2. Staff meetings $0 $0 $0 $0 3. Attend lectures $0 $0 $0 $0Sub-Total 0 $0 0 $0 0 $0 0 $0Total Preparation 0 0 0 0 0 0 0 0
II. Course Delivery
A. Instruction 1. Diagnose skill/knowledge $0 $0 $0 $0 2. Presentation $0 $0 $0 $0 3. Interaction $0 $0 $0 $0 4. Progress monitoring $0 $0 $0 $0Sub-Total 0 $0 0 $0 0 $0 0 $0
B. Evaluation 1. Test proctoring $0 $0 $0 $0 2. Tests/evaluation $0 $0 $0 $0Sub-Total 0 $0 0 $0 0 $0 0 $0Total Delivery 0 $0 0 $0 0 $0 0 $0
TOTAL 0 $0 0 $0 0 $0 0 $0
CPT TASKS• Calculations you have completed
– All personnel costs expressed as an hourly rate (page 1)– The specific tasks associated with offering the
traditional course and how much time each person spends on each of the tasks (page 2)
• Calculations you need to complete– Tasks and hours in the redesign (page 3)– Total instructional costs for the traditional course (page
4) – Total instructional costs for the redesign (page 4)
REDESIGNED COURSERedesigned Course Tenure-Track Non-Tenure- TAs/GAs Adjunct/
Faculty Track Faculty PT Faculty
# of Hours # of Hours # of Hours # of Hours
I. Course Preparation
A. Curriculum DevelopmentB. Materials AcquisitionC. Materials Development 1. Lectures/presentations 2. Learning materials/software 3. Diagnostic assessments 4. Assignments 5. Tests/evaluationsSub-Total 0 0 0 0
D. Faculty/TA Development/Training 1. Orientation 2. Staff meetings 3. Attend lecturesSub-Total 0 0 0 0Total Preparation 0 0 0 0
II. Course Delivery
A. Instruction 1. Diagnose skill/knowledge 2. Presentation 3. Interaction 4. Progress monitoringSub-Total 0 0 0 0
B. Evaluation 1. Test proctoring 2. Tests/evaluationSub-Total 0 0 0 0Total Delivery 0 0 0 0
TOTAL 0
Annual Cost Comparison
Institution: NOTE If you insert or delete columns, you must adjust the formulas that calculate totals.
Course:
Tenure-Track Non-Tenure- Adjunct/ Faculty Track Faculty TAs/GAs PT FacultySections Sections Sections Sections
ANNUAL COST OF THE TRADITIONAL COURSE
# of sections taught in fall and spring Cost of one sectionCost of instruction by type #VALUE! #VALUE! #VALUE! #VALUE! #VALUE! Total cost of direct instruction
Total cost of course coordinationGRAND TOTAL #VALUE! Total cost of other personnelTotal # of studentsCost per student #VALUE!
ANNUAL COST OF THE REDESIGNED COURSE
# of sections taught in fall and spring Cost of one sectionCost of instruction by type #VALUE! #VALUE! #VALUE! #VALUE! #VALUE! Total cost of direct instruction
Total cost of course coordinationGRAND TOTAL #VALUE! Total cost of other personnelTotal # of students Cost per student #VALUE!
THE CPT – WHY DO IT?
• Provides a structure for the team to think about activities and costs
• Allows consideration of changes in specific instructional tasks
• Permits visualization of duplication and waste
• Enables cost/benefit analysis re: type of personnel per task
TEAM!!!
READINESS CRITERION #5Learning Materials
• Are the faculty able and willing to incorporate existing curricular materials in order to focus work on redesign issues rather than materials creation?
READINESS CRITERION #6Active Learning
• Do the faculty members have an understanding of and some experience with integrating elements of computer-based instruction into existing courses?
READINESS CRITERION #7Collective Commitment
• Describe the members of your team, the skills they bring to the project and what their roles will be in both the planning and implementation phases of the project.
WHY INSTITUTIONAL TEAMS?
• Faculty experts• Administrators• Technology
professionals• Assessment experts
READINESS CRITERIA
1. Course Choice2. Redesign Model3. Assessment Plan4. Cost Savings Plan5. Learning Materials6. Active Learning7. Collective Commitment
NCAT’S REDESIGN METHODOLOGY“A Menu of Redesign Options”
• Six Models for Course Redesign • Five Principles of Successful Course
Redesign• Four Models for Assessing Student
Learning• Cost Reduction Strategies• Readiness Criteria • Course Planning Tool• Five Critical Implementation Issues• Planning Checklist
REDESIGNING STUDENT LEARNING ENVIRONMENTS : Measuring Cost and Quality
Carolyn Jarmon, [email protected]
www.theNCAT.org
Consultation and Training • Provide an orientation to course redesign• Help you develop a set of learning outcomes• Help you determine which method of course redesign will help you achieve your
desired outcomes
Faculty Development• Help faculty develop a deeper understanding of the selected redesign model• Train faculty on strategies for implementing the redesigned curriculum• Work with faculty to determine the key assignments • Help construct an assessment plan around both your desired outcomes and key
assignments
Ongoing Consulting• Ongoing telephone and web-based consulting for the duration of the redesign
Additional Services
Data Collection and Analysis Review and evaluation of existing course including
syllabus, course description and learning outcomes. Map Cengage Learning print and/or digital content to
your specific standards or learning outcomes. Development of supporting course documents Development of media assets and assessment tools