assessment of student problem solving processes jennifer l. docktor ken heller physics education...

Assessment of Student Problem Solving Processes

Jennifer L. DocktorJennifer L. DocktorKen Heller

Physics Education Research & Development Grouphttp://groups.physics.umn.edu/physed

DUE-0715615

3/18/2009 Jennifer Docktor, University of Minnesota 2

Problem Solving Measure Problem solving is an important part of learning

physics. Unfortunately, there is no standard way to measure

problem solving so that student progress can be assessed.

The goal is to develop a robust instrument to assess students’ written solutions to physics problems, and obtain evidence for reliability, validity, and utility of scores.

The instrument should be general

not specific to instructor practices or techniques

applicable to a range of problem topics and types


Reliability, Validity, & Utility

Reliability – score agreement Validity evidence from multiple sources

Content Response processes Internal & external structure Generalizability Consequences of testing

Utility - usefulness of scores

AERA, APA, NCME (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

Messick, S. (1995). Validity of psychological assessment. American Psychologist, 50(9), 741-749.


Overview of Study

1. Drafting the instrument (rubric)

2. Preliminary tests with two raters (final exams and instructor solutions)

3. Training exercise with graduate students

4. Analysis of tests from an introductory mechanics course

5. Student problem-solving interviews (in progress)


What is problem solving?

“Problem solvingProblem solving is the process of moving toward a goal when the path to that goal is uncertain” (Martinez, 1998, p. 605)

What is a problemproblem for one person might not be a problem for another person.

Problem solving involves decision-making.

If the steps to reach a solution are immediately known, this is an exerciseexercise for the solver.

Martinez, M. E. (1998). What is Problem Solving? Phi Delta Kappan, 79, 605-609.Hayes, J.R. (1989). The complete problem solver (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.Schoenfeld, A.H. (1985). Mathematical problem solving. Orlando, FL: Academic Press, Inc.


Problem Solving ProcessUnderstand / Describe

the Problem

Devise a Plan

Carry Out the Plan

Look Back

•Organize problem information

•Introduce symbolic notation

•Identify key concepts

•Use concepts to relate target to known information

•appropriate math procedures

•check answer

Pόlya, G. (1957). How to solve it (2nd ed.). Princeton, NJ: Princeton University Press.

Reif, F. & Heller, J.I. (1982). Knowledge structure and problem solving in physics. Educational Psychologist, 17(2), 102-127.


Problem Solver Characteristics

Inexperienced solversInexperienced solvers::

Knowledge disconnected

Little representation (jump to equations)

Inefficient approaches (formula-seeking & solution pattern matching)

Early number crunching

Do not evaluate solution

Chi, M. T., Feltovich, P. J., & Glaser, R. (1980). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5, 121-152.

Larkin, J., McDermott, J., Simon, D.P., & Simon, H.A. (1980). Expert and novice performance in solving physics problems. Science, 208(4450), 1335-1342.

Experienced solversExperienced solvers::

Hierarchical knowledge organization or chunkschunks

Low-detail overview / description of the problem before equations

qualitative analysisqualitative analysis

Principle-based approaches

Solve in symbols first

Monitor progress, evaluate the solution


Instrument at a glance (Rubric)

5 4 3 2 1 0 NA (P)

NA (S)

Physics Approach

Specific Application

Math Procedures

Logical Progression

Useful Description

SCORE

CATEGORY:(based on literature)

Minimum number of categories that include relevant aspects of problem solving Minimum number of scores that give enough information to improve instruction

Want


Rubric Category Descriptions

Useful DescriptionUseful Description organize information from the problem statement

symbolically, visually, and/or in writing. Physics ApproachPhysics Approach

select appropriate physics concepts and principles Specific Application of PhysicsSpecific Application of Physics

apply physics approach to the specific conditions in problem

Mathematical ProceduresMathematical Procedures follow appropriate & correct math rules/procedures

Logical ProgressionLogical Progression (overall) solution progresses logically; it is coherent,

focused toward a goal, and consistent


Rubric Scores (in general)

5 4 3 2 1 0

Complete & appro-

priate

Minor omissionor errors

Parts missing and/or contain errors

Most missing and/or contain errors

All inappro-

priate

No evidence

of category

NA - Problem NA - SolverNot necessary for this

problem

(i.e. visualization or physics principles given)

Not necessary for this solver (i.e. able to solve without

explicit statement)

NOT APPLICABLE (NA):


Early Tests of the Rubric

Preliminary testing (two raters)

Distinguishes instructor & student solutions

Score agreement between two raters – good

Training Exercise (8 Graduate Students)

Half scored a mechanics problem, half E&M

Scored 8 student solutions with the rubric, received example scores & rationale for first 3, then re-scored 5 and scored 5 new solutions

Answered survey questions about the rubric


Written Training Exercise

Minimal written training was insufficient

confusion about NA scores (want more examples)

perfect score agreement was 34% before training and improved only slightly with training to 44% (agreement within one score 77% 80%)

difficulty distinguishing physics approach & application

Math & Logical progression most affected by training

multi-part problems more difficult to score

Grad students influenced by traditional grading experience


Grad Student Comments Influenced by traditional grading experiences

Unwilling to score math & logic if physics incorrect

Desire to weight categories

“I don't think credit should be given for a clear, focused, consistent solution with correct math that uses a totally wrong physics approach” (GS#1)

“[The student] didn't do any math that was wrong, but it seems like too many points for such simple math…I would weigh the points for math depending on how difficult it was. In this problem the math was very simple” (GS#8)


Grad Student Comments

Difficulty distinguishing categories Physics approach & application

Description & logical progression

“Specific application of physics was most difficult. I find this difficult to untangle from physics approach. Also, how should I score it when the approach is wrong?” (GS#1)

“I think description & organization are in some respect very correlated, & could perhaps be combined” (GS#5)


Analysis of Tests

Calculus-based introductory physics course for Science & Engineering students (mechanics) Fall >900 students split into 4 lecture sections

4 Tests during the semester

Problems graded in the usual way by teaching assistants

After they were graded, I used the rubric to evaluate 8 problems spaced throughout the semester Approximately 300 student solutions per problem

(copies made by TAs from 2 sections)


Exam 3 QuestionShow all work! The system of three blocks shown is released from rest. The connecting strings are massless, the pulleys ideal and massless, and there is no friction between the 3 kg block and the table.

(A) At the instant M3 is moving at speed v, how far (d) has it moved from the point where it was released from rest? (answer in terms of M1, M2, M3, g and v.) [10 points]

(B) At the instant the 3 kg block is moving with a speed of 0.8 m/s, how far, d, has it moved from the point where it was released from rest? [5 pts]

(C)….(D)….

SYMBOLIC CUES ON MASS 3

How would you solve part A?


Grader Scores

Excludes part c) multiple choice question.

Average score the same (9 points or ~ half).


Rubric Scores

•Useful Description: Free-body diagram (not necessary for energy approach)

•Physics Approach: Deciding to use Newton’s 2nd Law or Energy Conservation

•Specific Application: Correctly using Newton’s 2nd Law or Energy Cons.

•Math Procedures: solving for target

•Logical Progression: clear, focused, consistent


Common Responses

Statements in red suggest students focused on M3, which was cued in the problem statement


Example Student Solution



Only consider kinetic energy of mass M3.

? Was cued in problem statement.



(E1=E2=E3)


Example Student Solutions

Considers forces on M3, and uses T=mg (incorrect)


Answer is correct, but reasoning for “F” unclear



Findings

The rubric indicates areas of student difficulty for a given problem i.e. the most common difficulty is specific

application of physics whereas other categories are adequate

Focus instruction to coach physics, math, clear and logical reasoning processes, etc.

The rubric responds to different problem features For example, in this problem visualization skills

were not generally measured. Modify problems to elicit / practice processes


Problem Characteristics that could Bias Problem Solving

Description:

Picture given

Familiarity of context

Prompts symbols for quantities

Prompt procedures (i.e. Draw a FBD)

Physics:

Prompts physics

Cue focuses on a specific objects

Math:

Symbolic vs. numeric question

Mathematics too simple (i.e. one-step problem)

Excessively lengthy or detailed math


Summary A rubric has been developed from descriptions of

problem solving process, expert-novice research studies, and past studies at UMN Focus on written solutions to physics problems

Training revised to improve score agreement Rubric provides useful information that can be

used for research & instruction Rubric works for standard range of physics

topics in an introductory mechanics course There are some problem characteristics that make

score interpretation difficult

Interviews will provide information about response processes

Additional Slides(if time permits)

[email protected]

http://groups.physics.umn.edu/physed


Exam 2 Question (Different)

A block of known mass m and a block of unknown mass M are connected by a massless rope over a frictionless pulley, as shown. The kinetic frictional coefficient between the block m and the inclined plane is μk. The acceleration, a, of the block M points downward.

(A) If the block M drops by a distance h, how much work, W, is done on the block m by the tension in the rope? Answer in terms of known quantities. [15 points]

A block of mass m = 3 kg and a block of unknown mass M are connected by a massless rope over a frictionless pulley, as shown below. The kinetic frictional coefficient between the block m and the inclined plane is μk = 0.17. The plane makes an angle 30o with horizontal. The acceleration, a, of the block M is 1 m/s2 downward.

(A) Draw free-body diagrams for both masses. [5 points](B) Find the tension in the rope. [5 points](C) If the block M drops by 0.5 m, how much work, W,is done on the block m by the tension in the rope? [15 points]

NUMERIC

SYMBOLIC


Grader Scores

Symbolic:

Fewer students could follow through to get the correct answer.

Numeric, prompted:

Several people received the full number of points, some about half.

AVERAGE: 15 points

AVERAGE: 16 points


Rubric Scores•Useful Description: Free-body diagram

•Physics Approach: Deciding to use Newton’s 2nd Law

•Specific Application: Correctly using Newton’s 2nd Law

•Math Procedures: solving for target

•Logical Progression: clear, focused, consistent

prompted


Solution Examples

(numeric question w/FBD prompted)

Could draw FBD, but didn’t seem to use it to solve the problem


Solution Example


NOTE: received full credit from the graderNUMBERS


Symbolic form of question


Symbolic form of question

Left answer in terms of unknown mass “M”


Findings about the Problem Statement

Both questions exhibited similar problem solving characteristics shown by the rubric.

However prompting appears to mask a student’s inclination to draw

a free-body diagram

the symbolic problem statement might interfere with the student’s ability to construct a logical path to a solution

the numerical problem statement might interfere with the student’s ability to correctly apply Newton’s second law

In addition, the numerical problem statement causes students to manipulate numbers rather than symbols


Findings about the Rubric

The rubric provides significantly more information than grading that can be used for coaching students Focus instruction on physics, math, clear

and logical reasoning processes, etc.

The rubric provides instructors information about how the problem statement affects students’ problem solving performance Could be used to modify problems


References

P. Heller, R. Keith, and S. Anderson, “Teaching problem solving through cooperative grouping. Part 1: Group versus individual problem solving,” Am. J. Phys., 60(7), 627-636 (1992).

J.M. Blue, Sex differences in physics learning and evaluations in an introductory course. Unpublished doctoral dissertation, University of Minnesota, Twin Cities (1997).

T. Foster, The development of students' problem-solving skills from instruction emphasizing qualitative problem-solving. Unpublished doctoral dissertation, University of Minnesota, Twin Cities (2000).

J.H. Larkin, J. McDermott, D.P. Simon, and H.A. Simon, “Expert and novice performance in solving physics problems,” Science 208 (4450), 1335-1342.

F. Reif and J.I. Heller, “Knowledge structure and problem solving in physics,” Educational Psychologist, 17(2), 102-127 (1982).

http://groups.physics.umn.edu/[email protected]

Additional Slides


Independent scoring of student solutions by a PER graduate student and a high school physics teacher (N=160)

Category % agree (exact)

% agree (within 1)

Cohen’s kappa

Physics Approach 71.3 97.1 0.62

Useful Description 75.0 99.2 0.63

Specific Application 61.3 96.9 0.48

Math Procedures 65.6 99.4 0.51

Logical Progression 63.1 96.9 0.49

OVERALL 67.3 98.5 0.55

Inter-rater Reliability

KappaKappa::

<0 No agreement

0-0.19 Poor

0.20-0.39 Fair

0.40-0.59 Moderate

0.60-0.79 Substantial

0.80-1 Almost perfect


Inter-rater Agreement

BEFORE

TRAINING

AFTER

TRAININGPerfect

AgreementAgreement Within One

Perfect Agreement

Agreement Within One

Useful Description 38% 75% 38% 80%

Physics Approach 37% 82% 47% 90%

Specific Application 45% 95% 48% 93%

Math Procedures 20% 63% 39% 76%

Logical Progression 28% 70% 50% 88%

OVERALL 34% 77% 44% 85%

Weighted kappa 0.27±0.03 0.42±0.03

Fair Fair agreementagreement

Moderate Moderate agreementagreement


All Training in Writing: Example

CATEGORY SCORE RATIONALE Training includes the actual student solution


Exam 1 Question 1 A block of mass m=2.5 kg starts from rest and slides down a

frictionless ramp that makes an angle of θ=25o with respect to the horizontal floor. The block slides a distance d down the ramp to reach the bottom. At the bottom of the ramp, the speed of the block is measured to be v=12 m/s.

a) Draw a diagram, labeling θ and d. [5 points]

b) What is the acceleration of the block, in terms of g? [5 points]

c) What is the distance, d, in meters? [15 points]

INSTRUCTOR SOLUTION


Grader Scores

>40% of students received the full points on this question

Was this an exercise or a problem?


Rubric Scores

Scores shifted to high end (5’s) or NA


Problem Solving Process

1. Identify & define the problem

2. Analyze the situation

3. Generate possible solutions/approaches

4. Select approach & devise a plan

5. Carry out the plan

6. Evaluate the solution

http://www.hc-sc.gc.ca/fniah-spnia/images/fnihb-dgspni/pubs/services/toolbox-outils/78-eng.gif

1 2 3

4 5 6


Developing & Testing the Rubric

Spring 2007

Summer 2007

Fall 2007

Spring 2008

Spring 2009

1. Draft instrument based on literature & archived exam data

3. Pilot with graduate students (brief training)

2. Test with two raters (consistency of scores)

5. Revise rubric and training materials. Retest.

8. Final data analysis & reporting

6. Collect & score exam problems from fall semester of 1301 course.

Fall 2008

Summer 2008

4. Analyze pilot data (feedback & scores)

7. (Interviews) Video & audio recordings of students solving problems.

Summer 2009

assessment of student problem solving processes jennifer l. docktor ken heller physics education...

Documents

mathematical problem

measureproblem solving

complete problem solver

assessment of student

range of problem topics

university of minnesotawhat

princeton university

progressjennifer docktor