primary trait analysis: anchoring assessment in the

JGE: THE JOURNAL OF GENERAL EDUCATION, Vol. 49, No. 4, 2000.Copyright © 2000 The Pennsylvania State University, University Park, PA.

PRIMARY TRAIT ANALYSIS:ANCHORING ASSESSMENT IN THE CLASSROOM

Ruth BenanderJanice DentonDeborah PageCharlotte Skinner

Introduction

The assessment of learning and teaching is a valuable endeavor,however onerous or ominous. Unfortunately, the difficulty of as-sessment often prevents potential assessors from beginning theproject, even though the necessity of public accountability forcesaction. The assessment of a general education should be designed,implemented, and evaluated by faculty in colleges where depart-ments have considerable autonomy, but must still cooperate at theinstitutional level. The ownership involves faculty designing,implementing, and evaluating student learning. It is our positionthat three issues are central to assessment: First, general educa-tion must be defined appropriately in the context of the students’academic programs. Second, assessment must have a direct rela-tionship with teaching and learning. Finally, and most important,the process must be centered where the actual teaching and learn-ing take place, that is, in the classroom. Only when these threeissues are addressed will the students, faculty, and administratorsfind an assessment process that all can support. We propose Pri-mary Trait Analysis (PTA) as the process that unites the assess-ment goals of students, faculty, and administrators and anchorsgeneral education assessment in the classroom.

Institutions frequently realize general education as a set of corecourses that include the humanities, the social sciences, and thenatural sciences (North Central Association of Colleges andSchools Commission on Institutions of Higher Education [NCA],1997). However, many 2-year institutions have transient studentpopulations whose professional and transfer programs cannot ac-

BENANDER ET AL.280

commodate the luxury of a separate core curriculum. The value ofa general education has not waned, so in order to retain the idealsof general education and continue to accommodate a nontraditionalstudent body, we must expand our definition of how a general edu-cation is realized. At Raymond Walters College, our mission state-ment reflects our educational philosophy:

The college strives to provide a general education which pro-motes tolerance, lifelong learning and a devotion to free in-quiry and free expression, to assure its graduates are indi-viduals of character more sensitive to the needs of commu-nity, more competent to contribute to society, and more civilin habits of thought, speech and action. (“Functional Mis-sion Statement,” 1998)

The goal of general education at Raymond Walters College is toprovide students, regardless of specialization, with a broad learn-ing experience that enables them to become informed, responsible,thoughtful citizens. While general education has often been struc-tured as a core curriculum, the assessment guidelines of the NorthCentral Association of Colleges and Schools Commission on In-stitutions of Higher Education (NCA) suggest that a general edu-cation may also be articulated as a set of specific skills studentsacquire throughout their studies (NCA, 1997). If a general educa-tion is a set of skills, then instructors identify where each particu-lar skill is taught. Students encounter many skill lessons as theyprogress through college.

Regardless of program, general education skills must be assessedappropriately. Because academic assessment measures studentlearning outcomes, where, when, and how the measurement takesplace is critical to the usefulness of the assessment. If the out-comes measured by a testing instrument do not match an instruct-or’s goals, the usefulness of the measure is limited, for the instructoris unable to use the results to improve teaching and learning(Walvoord & Anderson, 1998, p. 169; Lopez, 1996). Outcomesmeasured by national standardized tests inform test administra-tors of general trends, but this information is rarely used to im-prove student learning, unless the students are learning the skills

PRIMARY TRAIT ANALYSIS 281

of the standardized test (Walvoord, Bardes & Denton, 1998). Whilethe scores of the Graduate Record Exam or the Test of English asa Foreign Language clearly inform the specialized courses de-signed to prepare students for these tests, they do not inform gradu-ate school curricula or English curricula, and, in fact, they werenever designed to do so. And while the scores of an English writ-ing test given to the entire student body will give an aggregatescore indicating how well students can place commas or identifyparallelism, that information may not be very helpful in identify-ing how well the chemistry students can write clear laboratoryreports, the counseling students informative case studies, or thelanguage students insightful essays. It is clear that effective writ-ing is a discipline-specific skill, and we believe the same is trueof the more elusive skills of critical thinking and quantitative rea-soning.

Description, analysis, synthesis, and mathematical reasoningare critical thinking skills valued across the academy. Yet, eventhough all disciplines value these skills, each one realizes themdifferently (Farmer, 1993). If a general education is realized aseffective writing, critical thinking, and quantitative reasoning, thenthe assessment of these skills must be discipline-specific if it is tobe meaningful for teaching and learning. If general education con-sists of a set of discipline-specific skills, assessment must be gen-erated from and applied to the site where the teaching and learningof these skills are taking place. We believe, therefore, that theassessment process must be rooted in the classroom.

It is our position that assessment scores are useful only if thosescores can directly inform teaching and learning in the classroom.Individual class scores can be used to guide changes in the teach-ing and learning process and improve faculty-student communi-cation, which is the ultimate goal of any effective classroomassessment program (Angelo, 1991).

Locating assessment measurement directly in the classroompromotes faculty and student investment in the assessment. Fac-ulty members are able to create tests that reflect their teachinginterests. Student learning outcomes can be clearly related to spe-cific teaching, and changes in teaching methods should be reflected

BENANDER ET AL.282

in the outcomes. Students can see the relationship between whatthey must do and how they are evaluated.

Class grades are the traditional and principle form of classroomassessment. Faculty tend to be comfortable both with the tradi-tional grading process as learning assessment and in their abilityto know what A, B, and C work looks like. Students, faculty, andadministrators all seem to support traditional grading, and the prac-tice endures. Unfortunately, however, the criteria for grades arerarely explicit. A more specific method of measurement is neces-sary for useful assessment of teaching and learning that can achievethe same level of support as traditional grading.

Raymond Walters College, a two-year branch of the Universityof Cincinnati, is currently struggling through the thicket of gen-eral education assessment (Benander, Denton & Walvoord, 1997;Denton, Walvoord & O’Meara, 1998). The students of RaymondWalters College are a shifting population, and defining a core setof general education courses could expand 2-year professionalprograms into 4-year programs. Transfer students’ checkerboardtranscripts could become too complex. Students who completetheir education in short bursts between jobs and family demandscould spend their entire careers effectively majoring in generaleducation.

In keeping with the goals set out in our mission statement, thefaculty of Raymond Walters College defined a general educationas learning to write effectively and think critically/reason quanti-tatively. The faculty chose to combine critical thinking and quan-titative reasoning because the science and math faculty believethat critical thinking in their disciplines is best described as quan-titative reasoning. These two skills of effective writing and criti-cal thinking/quantitative reasoning are taught in most courses;however, since they are discipline-specific, each skill must bedefined in terms of the course in which it is being taught. It thenbecomes the responsibility of faculty members to define whatcomprises effective writing and critical thinking/quantitative rea-soning for their disciplines.

Traditional grading is not adequate for this task. BarbaraWalvoord, with her colleagues, has refined a method of assess-ment called Primary Trait Analysis (PTA), which builds on thestrength of the grading process and addresses its weaknesses


(Lloyd-Jones, 1977; Walvoord & Anderson, 1998). PTA evalu-ates student work based on explicit criteria related to an assign-ment. The faculty member creates criteria based on theperformance expected for the assignment. Students are evaluatedexplicitly by what the faculty member considers effective writingand critical thinking/quantitative reasoning. Since information isshared with students, success is not an educated guess and failureis not a mystery of fate. For the faculty member, student successmeans attention can be turned to other skills. Failure means atten-tion can be given to the specific skill that students have not mas-tered. The assessment process occurs throughout the course, notafter it is over. A student’s successes or failures in a class becomeopportunities for learning.

Since, in PTA, the definition of each general education skill, aswell as the criteria for achievement, is explicit for each discipline,these analyses can be shared at the department and college level.They can inform lesson plans and program curricula. Sharing thecriteria and assessment results can also increase collegiality amongfull-time and adjunct faculty.

The majority of the faculty at Raymond Walters College usesPTA to assess general education skills. The faculty support is strongbecause each has some control over the process: each generatesthe form of assessment from his or her own teaching and directlyapplies it to his or her own pedagogical choices. Without the in-centive of control, gaining sincere support from such a diverseand fiercely autonomous group would be impossible. Even withthe personal control and clear application that PTA allows, somefaculty members have yet to involve themselves in the process.Some even argue that the process is merely another administra-tive hoop. However, if the faculty has initiated the process and isable to retain control of it, then it should not be viewed as anadministrative hoop.

Nonetheless, PTA is a different process from the traditionalactivities of the classroom, and learning to construct the assess-ment criteria for an assignment takes time. Keeping these prob-lems in mind, we offer two arguments to promote the use of PTAfor assessment. First, like it or not, accrediting agencies requireassessment (“Criteria for Accreditation,” 1998). If faculty mem-bers do not invest in a process that supports teaching and learning

BENANDER ET AL.284

in their own classrooms, a process will be imposed from without.An external process will not have intimate relevance to individualclassroom content, and the feedback from the measurement willnot likely be pertinent to improving teaching and learning out-comes. Faculty serve students through positive action, not byavoiding a problem and later complaining about an imposed solu-tion. Second, and more positively, PTA is essentially what facultyalready do when they assign grades. The only difference is thatthe criteria for evaluation are explicitly stated in a form that canbe shared.

Application of PTA in the Classroom

Since we have proposed PTA as the tool for basing general educa-tion assessment in the classroom, we would like to present in-stances of how PTA has been applied in various disciplines. Fourfaculty members demonstrate how they use PTA in their coursesto measure critical thinking/quantitative reasoning, one of the gen-eral education skills identified by Raymond Walters College. Eachreports on the assignment, PTA scale, and student achievementassessed by the scale. In addition, each outlines how she uses theinformation to improve students’ critical thinking/quantitative rea-soning skills by modifying teaching and learning strategies.

Course 1

Course Description: Topics in Literature, Books and FilmAssignment: Electronic Posting of Movie ReviewsFor each film you will write a review of the assigned film. Includean appraisal of the director’s choices in presenting the literary textyou are now familiar with, the dominant tone the director seemsto be trying to achieve, and your reactions to the film. You willpost these summaries on the electronic bulletin board for that filmlocated on the class Website.


Table 1. Scale: Movie Review for Topics in Literature

Criteria ScoreAccurate plot review 4Analysis of text beyond literal interpretationSupport with specific details from text/filmPersonal evaluation based on analysis

Accurate plot review 3Analysis of text beyond literal interpretationWeak support with specific details from text/filmPersonal evaluation

Minor inaccuracies of plot 2Analysis of text includes literal interpretationLittle personal evaluationFew specific details as support

Glaring plot inaccuracies 1Literal analysisNo personal evaluationNo specific details as support

No posting 0

Table 2. Results: Scores on Each Review

ReviewScore #1 #2 #3 #4 #5 #6 #7

4 10 11 10 12 13 6 93 7 7 8 5 5 8 72 3 3 3 5 3 6 71 0 0 0 0 0 0 00 1 0 1 1 0 1 5

BENANDER ET AL.286

Analysis: Over the course of the quarter, students wrote sevenmovie reviews, all of which were assessed with the primary traitscale. I handed out the trait scale to students on the first day ofclass, but I am not sure that they consulted it; upon my inquiringwhether they had a copy near the end of the course, few studentswere able to locate it in their notebooks. This assessment activitytaught me that I should refer to the scale more explicitly in classso that it remains in the students’ consciousness.

Judging from the scores, student performance on the electronicmovie reviews appears to be steady. My perception of the indi-vidual performance is more dynamic, however. I found that par-ticular students struggled throughout the class to achieve a 4 onthe scale and moved from a 3 to a 4. The step that involved mov-ing away from literal interpretation to more abstract interpreta-tion was very difficult. Those who were able to move from steady3s to 4s made significant progress in that critical step. In addition,what the table does not show is that movement of individualswithin the scores was dynamic. Although the numbers appearsteady, the individuals representing those numbers changed overtime. Some students stopped working hard on the reviews or didnot put much work into reviews of movies they did not like. Otherstudents worked hard to move from one grade level to another.Thus, I look at the distribution, and I am satisfied that the major-ity of the students met the criteria of accomplishing some level ofanalysis beyond recitation of plot. Nonetheless, I am not satisfiedwith the steady state of the numbers over time. I would like to seemore consistent improvement over the course of practicing thesame skills seven times.

More significantly, I have decided to revise the criteria to moreaccurately reflect the difficulty of skills. I anticipated that it wouldbe easy for students to give an analysis but difficult for them toidentify concrete support for their ideas. However, I discoveredthat students found it easier to point to specific places in the mov-ies that gave them ideas than to articulate those ideas. Therefore,I will revise the scale for the next course to reflect the relativechallenges of these skills; the scale will revolve around movingfrom literal interpretation to abstract interpretation rather thanaround the amount of support provided from the text.


Course 2

Course Description: First-Year Chemistry LaboratoryAssignment: Density DeterminationDetermine the mass and volume of several samples of a solid sub-stance. Use the data to prepare a graph whose slope represents theaverage density of the substance, and then use the mathematicalrelationship between the variables to do additional calculations.

Table 3. Scale: Density Determination

Trait Criteria Score

Drawing a Graph Graph has a title, labels on the axes, appropriate 3scale, best fit line, and the correct variableson the axes.

One of the above characteristics is missing. 2More than one of the characteristics is missing. 1

Determining the Has selected 2 points on the best fit line and 3Equation of Line correctly calculated the value of the slope, in-

cluding units. Has extrapolated the line to deter-mine the value of the y intercept when x = 0.

Has correctly determined either the slope or the 2intercept, but not both.

Has incorrectly calculated both. 1

Application of Has manipulated the equation of the straight line 4the Equation and used it to calculate the x variable given y,

and vice versa. The answer includes units.Same as above but omits units. 3Only correctly calculates the x variable given y; 2

units may or may not be included.Only correctly calculates the y variable given x;

units may or may not be included. 1

BENANDER ET AL.288

Table 4. Results: Scores for Each Trait, 1996 and 1997

Graph Equation Application 1996 1997 1996 1997 1996 1997

N 49 58 49 58 49 58 Trait Score

4 N/A N/A N/A N/A 33 413 21 31 8 35 6 22 27 22 25 16 2 01 1 5 16 7 8 15

M 2.4 2.4 1.8 2.5 3.3 3.2

Analysis: First-Year Chemistry Laboratory is taken concurrentlywith chemistry lecture by students intending to be science majors(engineers, chemists, pharmacists, etc.). It is the first college chem-istry class for these students and the expectation is that they havecompleted a high school chemistry course and had complemen-tary math preparation.

For a number of years, students performed a density determi-nation as their first laboratory experiment. They measured the massand volume of a number of samples, plotted the data, determinedthe slope—which is the average density—and used the algebraicrelationship between the variables to perform other calculations.Typically, students performed poorly on this first lab. First, theydid not know how to draw graphs; that is, they did not know theessential elements of a graph. Second, they claimed never to haveheard of a best fit line (even though my math colleagues insistthey must have). Third, they did not understand the algebraic re-lationship between the variables. After concluding that this prob-lem was not confined to one particular year of students, and thatmy frustration was not helping the students to learn, I decided toteach the skills as part of the chemistry course and to explicitlymeasure student learning. I restructured the lab syllabus so thatthe first 3-hour lab period was devoted to hands-on graphing ex-periences and discussion of the process. I then developed a pri-mary trait scale, an explicit statement of my criteria and standards


for the graphing assignment, which I gave to students in the formof a check-list.

In 1996, the first set of results indicated that students couldgrasp what elements are essential to producing a good and usefulgraph. Being able to determine the slope and the intercept was aproblem: only 16% of the class managed to find both. Once thestudents had an equation to work with, whether it was numeri-cally correct or not, they were able to use the relationship to an-swer additional questions. I concluded that the restructured labcontent had improved student learning when it came to two of thethree areas; the primary trait scale allowed me to identify the factthat I still needed to find a way to help students learn how to de-termine the slope and the intercept. The following year, 1997, Igave the students more opportunities to practice slope and inter-cept calculations, not just in lab, but also in lecture and recitation.Student learning in this area improved noticeably: 60% of the stu-dents correctly determined the slope and intercept. However, 28%of the students incorrectly calculated either the slope or the inter-cept, and I could not determine from my scale if one calculationwas causing more problems than the other. So, in 1998, I modi-fied the scale to allow me to get this piece of information, and Iam now in the process of evaluating that data. Clearly, being ex-plicit about my criteria and standards had rewards for both thestudents and me.

BENANDER ET AL.290

Course 3

Course Description: First Year German Language ClassAssignment: Write a paragraph in German on a specified topic.

Table 5. Scale: Writing a Paragraph in German

Criteria ScoreYou write at least 6, original, complete sentences for each entry. 9–10You write the sentences as a paragraph, with a topic sentence and

other sentences that relate to it.The date of the entry is in the German fashion.You use the grammar and vocabulary in the chapter being studied.You use German word order appropriately and correctly.There are very few errors, none of which affect the meaning or comprehen-

sion of your writing.You write neatly and legibly.You keep the entries in chronological order in a 3-prong folder.You spell and capitalize all words correctly.

One or two of the criteria above are not met. 7–8

Three of the criteria above are not met. 6

Your work is unsatisfactory, with four or more criteria not met. 1–5

You do not hand in your work on time. 0

Table 6. Results: Percentage of Grades on Assignments by Aca-demic Year

Grade 1995/96 1996/97 1997/98

A (9–10) 36 40 58B (8) 24 32 16C (7) 9 4 8D (6) 5 1 2F (<6) 26 23 16


Analysis: Speaking, listening comprehension, reading, writing,and culture are the five areas taught in Raymond Walters College’sforeign language classrooms. I designed the writing assignmentpresented here for the first quarter of German language. Once stu-dents have covered simple conversation with basic vocabulary,the nominative and accusative cases, and the present tense of verbs,it is time for them to begin to use the language to express theirown thoughts.

The journal assignment is an opportunity for a student to ownthe language, albeit on a very elementary level. Because studentstypically encounter many problems when writing in a foreign lan-guage, I chose to construct a primary trait scale that would ad-dress basic paragraph construction, as well as the appropriate useof German language skills. I assign very general topics, such as“Using food vocabulary, write about your likes and dislikes” and“Describe everything (un)interesting you did last week.” At theend of the 3-quarter sequence, each student has a portfolio show-ing the progress of his or her ability to express him- or herself inwritten German.

Our department has found that the use and analysis of primarytrait scales is a powerful tool for improved communication withinthe department. We have shared ideas on the assignment in de-partment meetings, and each faculty member has found the mostcomfortable manner of addressing the writing component of ourcurriculum and of supplementing the college’s general educationgoal of effective writing skills.

During 1995/96, I was very pleased with the results, as therewas great improvement over my expectations of a C/D averagegrade. I decided to continue with the same scale for the next aca-demic year.

For 1996/97, the results were very similar to the previous years,with a slight improvement. However, I continued to wonder whyalmost one-fourth of the assigned work was not turned in. Whilesome students just do not turn in work, I felt the problem mightbe related to either the wordiness of the scale or the possibilitythat students had lost the scale and not used it as reference whilecompleting each writing assignment.

BENANDER ET AL.292

I redesigned the scale for 1997/98 to include an abbreviated setof instructions at the bottom of the page on which students wereto complete the assignment. I also included a German “Fraktur”font for visual appeal, as well as date and chapter lines. The qual-ity of the writing samples improved significantly, although I feelthat the class, as a whole, is no better than those of previous years.I now feel that I am addressing the problem of students not turn-ing in assigned work. While still concerned about the number ofassignments not turned in, I believe this scale has allowed the great-est student success to date, and so I will continue to use it nextyear. During the next year, I will look at other statistics for theclass (possibly class attrition rates and final grades) to see howthey may relate to the results in Table 6. I will then reevaluate thescale and make changes as the need for them becomes apparent.

Course 4

Course Description: Finite MathematicsAssignment: Linear Programming Group ProblemThe group is given demographic information for two fictitiousmagazines. It must determine how much advertisement space topurchase in each magazine to maximize the exposure of an adver-tisement to a particular age group. At the same time, the groupmust stay within budget and guarantee that the ad is seen by aspecified number of readers of a particular income level.


Table 7. Scale: Modeling the Problem Situation

Subtraits Criteria Score

Defining Variables All variables are clearly defined. 3Definitions are given, but are either vague 2 or irrelevant.Variable definitions are missing. 1

Stating the Objective and Both the objective and objective function 3Objective Function are given and are correct.

One or the other is vague or missing. 2Each of the objective and objective function 1 is either missing or vague.

Formulating Constraints All the constraints are correct and the 4 non-negativity constraints (if applicable) are included.Only one error or a consistent incorrect 3 inequality sign error occurs.Two or more errors occur. 2No understanding of the problem is 1 demonstrated.

Table 8. Scale: Constructing the Feasible Set


Graphing the Boundaries Each boundary is graphed and labeled 2 successfully.At least one error is made in graphing 1 and labeling the boundaries.

Indicating the The feasible set (shaded area only) is 2Feasible Set correct for the given constraints.

The feasible set contradicts at least one of 1 the given constraints.

Determining the All the corner points are correctly 4Corner Points determined and clearly identified.

No more than one error occurs OR the 3 corner points are correctly determined from the given constraints, but are not identified as such.Two or more errors occur. 2No attempt is made to find the corner points. 1

BENANDER ET AL.294

Table 9. Scale: Determining the Optimal Point and Value


Applying the Corner The objective function is evaluated at each 4Point Method corner point successfully, and the optimal

point and value are correctly identified.An attempt is made to evaluate the objective 3 function at each corner point and identify the optimal point and value, but one error occurs.A solution is given, but two or more errors 2 occur OR the objective function is not evaluated at every corner point.Solution is not given. 1

Sketching the Isolines Equations of at least 3 isolines are given, are 4 graphed successfully, and are clearly labeled with the appropriate value of the objective function.As above, but the lines are not clearly labeled. 3Either the equations are missing or not 2 graphed correctly, but not both.Less than 3 isoline equations are given and 1 graphed correctly.

Identifying the Optimal point and value correctly identified 3Optimal Point from given isolines.and Value Solution does not follow from given isolines 2

or is not clearly stated.No solution is given. 1

Table 10. Scale: Interpreting the Solution


Determining a The appropriate solution is correctly 3Reasonable Solution identified and justified.

An attempt is made to round the solution, but 2 the result is not optimal or clearly stated OR the correct solution is given without justification.Solution is incorrect and without justification. 1

Applying the Solution For each of the two follow-up questions,Answer is justified, clearly stated and 2 follows from previous work.At least one of the above is not met. 1


Table 11. Results: Class Average, as a Percent, by Quarter forEach Primary Trait

Fall Spring FallTrait (Total Points) 1997 1998 1998

Number of Groups 9 6 10

Modeling the Problem Situation (10) 96.67 91.67 92.00Constructing the Feasible Set (8) 97.22 95.83 91.25Determining the Optimal Point and Value (11) 84.85 75.76 79.09Interpreting the Solution (7) 90.48 83.33 87.14Grade (36) 91.96 86.11 86.94

Analysis: Finite mathematics is the typical lower-level collegecourse designed for liberal arts majors. The emphasis is on real-world applications, with theory motivated by application. A yearof precalculus is a prerequisite. Approximately 50% of my courseis devoted to linear programming, the objective of which is tooptimize a linear function subject to certain restrictions. We ex-amine both geometric and algebraic methods for solving linearprogramming problems. This assignment, worth approximately10% of the course grade, is given after we have finished discuss-ing geometric methods. Its purpose is to give students a taste of arealistic linear programming problem with large data and a non-trivial solution, unlike traditional textbook exercises.

The first time I used the scale to grade the assignment, in fall1997, I was startled to see weakness in an area that I had assumedwould be strong, namely, drawing a conclusion! Most groups hadpresented all the necessary work, but had failed to state the solu-tion clearly. This observation forced me to reflect on and subse-quently revise my teaching techniques to give this particular skillfuture emphasis. When presenting new material, I write solutionsin sentences and emphasize the point that a solution is useless to awider audience if it is not justified and communicated clearly.

Since then, I have given that assignment twice. After the sec-ond round, I revised the scale to include student responses thatpreviously did not clearly meet any of the criteria. I now presentthe scale to students while we are still discussing new material sothat they can see exactly what is expected for each score on aparticular trait. I keep the group’s score on each subtrait and pri-

BENANDER ET AL.296

mary trait, as well as the class average for each, for my own records.The class averages for the primary traits are shown in the tableabove (Table 11). Although the aggregate scores vary from classto class, reflecting the overall skill level of the group of studentsin each, the same strengths and weaknesses appear consistently.This tells me that my scale is a valid assessment tool.

The class averages were highest for the first two traits, model-ing the problem situation and constructing the feasible set. How-ever, the high scores on the first trait are not a true indication ofhow well my students can translate a problem into mathematicallanguage. By students’ own admissions, the most difficult part ofsolving a “word problem” is the modeling process. The high scoreson this first trait can be explained by the fact that the groups workedin class, and I was available to steer them in the right direction ifthey drifted hopelessly astray. Prior to this assignment, studentshad considerable practice solving systems of linear inequalities,the main skill addressed by the second trait, constructing the fea-sible set. I was not surprised to see the high scores here. An areaof weakness is clearly the third trait, determining the optimal pointand value. The subtrait results reveal that students have troubleidentifying the optimal value from the trend of the isolines. Uponreflection, I realized that this was understandable. The isolinemethod is more theoretical than the easier-to-apply corner pointmethod. (In fact, it provides proof that the corner point method isvalid.) Our current text gives the isoline method little emphasis,and I illustrated it in detail in class only once. Students were givena few suggested homework exercises, but none were graded; thestudents had little incentive to complete them and were given nofeedback. How could I possibly expect students to perform wellon this trait? In the future, I must give my students an opportunityto digest and practice the isoline method if I expect those scoresto improve.

I continue to be amazed at how PTA simplifies grading and pro-vides me with useful information about my students’ learning.Circling groups’ scores on a copy of the scale saves me time. Itreduces the amount of comments I need to write to explain whypoints were lost, and it prevents me from digging through the stackof previously graded papers to find how many points I had de-


ducted for a particular error. Another advantage is that portionsof the scale easily adapt to other linear programming assignments.In fall 1998, I began using the scale to grade test and final examquestions in order to monitor student progress within a class. Iam hopeful that the results will show improvement in students’skills.

Application of PTA Beyond the Classroom

In each of the above examples, the instructor has explicitly statedhow critical thinking/quantitative reasoning is realized in her dis-cipline for the given assignment. Student performance guides eachfaculty member toward identifying ways to improve student learn-ing. Nevertheless, while the assessment of general education mustbe located in the classroom, the results of that assessment cannotremain locked within the classroom walls. We must include twovery important internal constituents, the department and the in-stitution, and one ever-present external constituent, the accredit-ing agency, and thus the public at large.

Before involving departments and institutions in the businessof assessing student learning outcomes, however, the role of eachmust be clearly defined. At Raymond Walters College, where de-partments have considerable autonomy, the role of the departmentis to ensure and document that good assessment is taking place inthe classroom. To fulfill this role, the college’s Academic Assess-ment Committee (the faculty committee that defined the academicassessment process) asked members of each department to meetannually to discuss their critical thinking/quantitative reasoningassignment, primary trait scale, and aggregate student scores. Atthis meeting, faculty members also discuss their analyses of thescores and strategies being considered to help students improvetheir critical thinking/quantitative reasoning skills. At the end ofthe meeting, the department summarizes the discussion on a stan-dardized report form, which is sent to the Academic AssessmentCommittee. The standardized process facilitates department-widefaculty involvement and allows the Academic Assessment Com-mittee to review and summarize the data easily.

BENANDER ET AL.298

A number of interesting things have happened as a consequenceof these assessment meetings. First, there was a lively discussionof pedagogy and much self-reflection. Faculty discussed teach-ing strategies that had been successful, or not so successful, inhelping students learn more effectively. Second, faculty foundproblems that had roots in departmental curricular structure andwere able to investigate the problems and devise appropriate so-lutions. Third, faculty members who were still struggling to seethe value of PTA were able to hear from their colleagues the use-fulness of the process to both faculty and students.

Departments reported that students demonstrated effective criti-cal thinking/quantitative reasoning skills in the areas of problemsolving, application of principles, logical interpretation of texts,establishment of connections, evaluation of solutions, and com-munication. Departments further reported that students neededhelp to improve skills in the areas of evaluating arguments, ap-plying theory, synthesizing information, and interpreting and pre-senting data. Various faculty members intend to increase in-classcritical thinking/quantitative reasoning activities, revise the con-tent of critical thinking/quantitative reasoning assignments, statecritical thinking/quantitative reasoning goals more explicitly, ex-pand the use of methods that encourage critical thinking/quanti-tative reasoning, and increase their guidance of students as theywork on assignments.

Also, as a result of this process, departments intend to helpfaculty achieve those goals by undertaking the following: encour-age sharing of critical thinking/quantitative reasoning experiences,examine curricula to determine where skills are taught, consultteaching and learning experts about teaching strategies, providementoring, offer workshops, share articles on teaching strategies,and explore the availability of funding to support these efforts.

The final internal audience is the institutional administration.The two key players at this level are the Academic AssessmentCommittee and the Dean. The role of the committee is to collectthe department reports and summarize them into an institutionaldocument that is shared college-wide and with external constitu-ents, mainly the accrediting agency. The report documents thatassessment of student learning is taking place in the classroom


and that changes are being made to enhance student learning. Italso describes actions that departments are taking to improve stu-dent learning.

From this ground-up, highly decentralized approach to assess-ment, the role of our Dean has evolved, rather than been prescribedfrom the beginning. The Dean, who is the chief academic officerof the college, is an ex-officio member of the Academic Assess-ment Committee. She sees all the committee reports and has ac-cess to the committee’s files on request, but her role, and it is avery important one, is to provide support to individual facultyand departments who want to make changes to the curriculum asa result of their assessment findings. The Dean also encouragesfaculty members to present their assessment research at local andnational meetings by providing additional travel support, over andabove the amount allotted to each faculty member for professionaldevelopment activities.

What has just been described is the link between budget andassessment. In 1993, at the beginning of the Raymond WaltersCollege assessment process, it was unclear how the two would,or could, be linked. What has happened is that classroom researchhas identified areas that need attention, faculty and departmentsuse the data to make requests for support from the college, andthe Dean responds to the requests. It is indeed a very powerfulconnection.

Is this a useful form of assessment, beginning in the classroomand moving through the departments before arriving at institu-tional involvement? We believe so. As of 1998, our second yearof collecting data, 80% of the full-time faculty and 25% of theadjunct faculty (a group not yet targeted for involvement) use PTAto measure students’ critical thinking/quantitative reasoning skills.Of that group, 46% agree that PTA provides very useful feedbackabout students’ critical thinking/quantitative reasoning skills, and50% find that it provides somewhat useful feedback. In addition,46% agree that PTA is a very effective tool for assessing studentlearning, and 49% report that it is a somewhat effective tool. Al-though a full complement of the full-time faculty does not usePTA to measure students’ critical thinking/quantitative reasoningskills, all full-time faculty members did participate in departmen-

BENANDER ET AL.300

tal assessment meetings. Anecdotal evidence suggests that non-participating instructors are employing similar teaching strategiesin the classroom, and many of them are making changes as a re-sult of the departmental pedagogy discussions, although they arenot collecting data to support their actions. Further, our assess-ment program meets many of the American Association of HigherEducation’s principles of good assessment (AAHE 1992).

Conclusion

Primary Trait Analysis is a simple process of making learningexpectations explicit and taking action based on how those ex-pectations are met:

Identify expectation.

Specify outcome of expectation.

Measure Student Learning.

Act on results.

Four faculty members have demonstrated how they have imple-mented this process in their courses. Their results were later dis-cussed in department meetings with other faculty members whohad employed similar processes. All of the scales and results arearchived for a summary of critical thinking/quantitative reason-ing assessment at the institutional level (Academic AssessmentCommittee, 1998).

We believe that PTA is a form of assessment that meets ourvaried needs. It is controlled by the faculty and localized in theclassroom, where teaching and learning take place. PTA is flex-ible enough to allow a diverse faculty to use it to assess studentlearning in general education and to find common ground to talk

➸➸

➸


about teaching and learning. Because of the explicit nature of thistool, we are able to make changes in teaching, in program curricu-lum, and in institutional policy. Another benefit of the adaptabil-ity of PTA is that it is useful, not only at the local level of individualteaching and learning, but also at the much more global level ofinstitutional assessment.

The example of Raymond Walters College serves to show howone particular institution is incorporating the PTA method of as-sessment at all levels. Individual faculty members assess particu-lar assignments, departments discuss the results, and administratorscollect summaries of action. As a result, teaching strategies maychange, department decisions about curricula may change, andinstitutional support may be directed to where it is needed most.Faculty benefit by having a research tool to measure and discussstudent outcomes; thereby, we become better teachers. Ultimately,the winners are the students, who are more carefully attended to,provided for, and supported.

References

Academic Assessment Committee. (1998, December). Report. [Internet] Available: http://www.rwc.uc.edu/phillips/index_assess.htm.

American Association for Higher Education. (1992). Principles of good practice for as-sessing student learning. Washington, DC: AAHE.

Angelo, T. A. (1991). Introduction and overview: From classroom assessment to class-room research. In T. A. Angelo (Ed.), Classroom research: Early lessons fromsuccess (pp. 7–15). New Directions for Teaching and Learning, 46. San Francisco,CA: Jossey-Bass.

Benander, R., Denton, J., & Walvoord, B. (1997). Integrating the assessment of generaleducation into the classroom: A two-year college model. In S. E. Van Kollenburg(Ed.), A collection of papers on self-study and institutional improvement (pp. 152–154). Chicago, IL: NCA.

Criteria for accreditation. (1998, December). [Internet] Available: http://www.ncacihe.org.Denton, J., Walvoord, B. E., & O’Meara, D. (1998). Closing the loop: From classroom

assessment to institutional assessment. In S. E. Kollenburg (Ed.), A collection ofpapers on self-study and institutional improvement (pp. 141–142). Chicago, IL:NCA.

Farmer, D. W. (1993). Course-embedded assessment: A teaching strategy to improve stu-dent learning. Assessment UPdate: Progress, Trends and Practices in Higher Edu-cation 5(1), 10–11.

Functional mission statement. (1998, December). [Internet] Available: http://www.rwc.uc.edu/phillips/index_assess.htm.

BENANDER ET AL.302

Lloyd-Jones, R. (1977). Primary trait scoring. In C. Cooper and L. Odell (Eds.), Evaluat-ing writing: Describing, measuring, judging (pp. 33–66). Urbana, IL: NationalCouncil of Teachers of English.

Lopez, C. L. (1996). Opportunities for improvement: Advice from consultant-evaluatorson programs to assess student learning. Chicago, IL: NCA.

North Central Association of Colleges and Schools Commission on Institutions of HigherEducation. (1997). Handbook of accreditation. Chicago, IL: NCA.

Walvoord, B., & Anderson, V. (1998). Effective grading: A tool for learning and assess-ment. San Francisco, CA: Jossey-Bass.

Walvoord, B., Bardes, B., & Denton, J. (1998). Closing the feedback loop in classroom-based assessment. Assessment UPdate: Progress, Trends and Practices in HigherEducation 10(5), 1–11.

primary trait analysis: anchoring assessment in the

Documents