meeting the challenge of interdisciplinary...
TRANSCRIPT
11 science education and civic engagement 6:2 summer 2014
Introduction to Interdisciplinary Education
“As the pace of scientific discovery and innovation accelerates, there is an urgent cultural need to reflect thoughtfully about these epic changes and chal-lenges. The challenges of the twenty-first century require new interdisciplinary collaboration, which place questions of meanings and values on the agenda. We need to put questions about the uni-verse and the universal back at the heart of university.”
– William Grassie (2013)
As the world becomes more complex, given the rapid ex-pansion of technology, the changing nature of warfare, rising energy, and environmental crises, the value of an interdisciplinary education is increasingly obvious. Social, political, economic, and scientific issues are so thoroughly interconnected that they cannot be explored productively, either by experts or students, within clear-cut disciplinary boundaries.
Despite this fact, several problems arise when institu-tions try to incorporate interdisciplinary education into their programs. Boix Mansilla (2005) noted that the as-sessment of interdisciplinary work by students is of great concern. She explains that because faculty are often disci-pline-specific experts, they are unfamiliar with disciplines outside their realm of expertise and have difficulty defin-ing interdisciplinary work. She goes on to explain that, as a consequence, “the issue [of standards] is marred by contro-versies over the purposes, methods, and most importantly, the content of proposed assessments” (2005, 16).
This paper offers one solution to this dilemma. The fol-lowing analysis explores the current state of interdisciplin-ary education, both in academia broadly, and specifically, at West Point through its interdisciplinary Core Program. The sections that follow will highlight the current issues inherent in interdisciplinary education, define interdis-ciplinary education objectives, and finally, explain the adaptable, multi-functional, interdisciplinary rubric be-ing implemented at the United States Military Academy
RESEARCH
PAPER
Meeting the Challenge of Interdisciplinary Assessment
Elizabeth Olcese United States Military Academy
Joseph Shannon South Seattle College
Gerald Kobylski United States Military Academy
Charles Elliott United States Military Academy
Olcese, et. al.: Meeting the Challenge of Interdisciplinary Assessment 12 science education and civic engagement 6:2 summer 2014
(USMA), a rubric designed to resolve many of the issues in-terdisciplinary educators encounter.
The Current State of Interdisciplinary Education in Academia
“The demand is clear. Whether we try to take a stance on the stem cell research controversy, to interpret a work of art in a new medium, or to assess the recon-struction of Iraq, a deep understanding of contempo-rary life requires knowledge and thinking skills that transcend the traditional disciplines. Such under-standing demands that we draw on multiple sources of expertise to capture multi-dimensional phenom-ena, to produce complex explanations, or to solve intricate problems. The educational corollary of this condition is that preparing young adults to be full par-ticipants in contemporary society demands that we foster their capacity to draw on multiple sources of knowledge to build deep understanding.”
– Veronica Boix Mansilla (2005, 14)
There are currently several studies, including evaluation mea-sures, defining the essence of interdisciplinary education. The above quote from Boix Mansilla’s “Assessing Student Work at Disciplinary Crossroads” highlights the challenge educators are experiencing in preparing students to meet today’s most pressing problems. This paper will not attempt to address the structure of interdisciplinary education as an institu-tional convention, but only to define the essential skills and capacities that a student with interdisciplinary understanding would demonstrate. These definitions are essential to under-standing and creating a framework for interdisciplinary learn-ing, which is arguably the first step in adequately integrating it into educational programs. Interdisciplinarity is a difficult construct to quantify, and many educators have been unable to frame a definition of it or to assess it in student work. As a consequence of these and other challenges, only a limited number of colleges or universities have implemented formal interdisciplinary programs at the institutional level.
Several analyses (Boix Mansilla 2005; Boix Mansilla and Dawes Duraising 2007; Rhoten et al. 2008; Stowe and Eder 2002) address the key issues surrounding interdisciplinary learning in higher education and offer proposals on how to
address them, starting with the definition of the term “in-terdisciplinary.” One definition of interdisciplinary under-standing is “the capacity to integrate knowledge and modes of thinking drawn from two or more disciplines to produce a cognitive advancement—for example, explaining a phenom-enon, solving a problem, creating a product, or raising a new question—in ways that would have been unlikely through single disciplinary means” (Boix Mansilla 2005, 16; Boix Mansilla and Dawes Duraising 2007, 216). A definition is particularly important because “a clear articulation of what counts as quality interdisciplinary work, and how such qual-ity might be measured, is needed if academic institutions are to foster in students deep understanding of complex prob-lems and evaluate the impact of interdisciplinary education initiatives” (Boix Mansilla 2005, 16). An agreed-upon defini-tion is currently lacking in academia, and this has resulted in inconsistent grading, teaching, and learning in interdisciplin-ary education.
One study of well regarded and established interdisci-plinary programs in the U.S., which included Bioethics at the University of Pennsylvania, Interpretation Theory at Swarthmore College, Human Biology at Stanford University, and the NEXA Program at San Francisco State University, involved “69 interviews, 10 classroom observations, 40 sam-ples of student work, and assorted program documentation” (Boix Mansilla and Dawes Duraising 2007, 4). The data were gathered in one-hour to 90-minute semi-structured inter-views with faculty and students inquiring about the manner of assessment used in their respective programs. Next, exam-ples of student work were used to give examples of what the institution viewed as meeting the definition of interdisciplin-arity. From the interviews and student examples, the authors concluded that there are three core dimensions to student interdisciplinary work: disciplinary grounding, advancement through integration, and critical awareness (Boix Mansilla, 2005, Boix Mansilla and Dawes Duraising 2007). These core elements are represented graphically in Figure 1.
The first core element in Figure 1, disciplinary grounding, calls for strong base knowledge in individual disciplines. Dur-ing the interviews, 75 percent of the interviewed faculty felt that strong subject-area knowledge was necessary for inter-disciplinary education that did not sacrifice depth in exchange for breadth. However, the authors noted that the key to suc-cessful disciplinary grounding also included the thoughtful selection of which disciplines to use and how to use them.
Olcese, et. al.: Meeting the Challenge of Interdisciplinary Assessment 13 science education and civic engagement 6:2 summer 2014
Advancement through integration, the second principle, is universal in all student work in the sense that students are supposed to learn from the work they do; however, what sets it apart in interdisciplinary education is that “students advance their understanding by moving to a new conceptual model, explanation, insight, or solution” (Boix Mansilla and Dawes Duraising 2007, 225). In the study, sixty-eight per-cent of faculty identified advancement through integration as a necessary element in interdisciplinary understanding and as the quintessential element for the advancement of student understanding. However, various programs and their students interpret this core element differently. For example, some students in in the NEXA Program at San Francisco State University strive for complex explanations, which evaluate the extent to which disciplines are interwo-ven to create a broad picture of how interconnected dif-ferent disciplines are on a given topic. Other students in the same program prefer to use aesthetic reinterpretations to connect the literary, historical, and social elements of a given topic. Other students, such as those in the Bioethics
program at the University of Pennsylvania, choose to focus on the development of practical solutions based on of the use of multi-disciplined ideas. The final principle from Fig-ure 1, critical awareness, refers to student work being able to withstand examination and criticism and explicitly calls for evidence of student reflection in their work. Student work needs to “exhibit clarity of purpose and offer evidence of reflective self-critique” (Boix-Mansilla and Dawes Durais-ing 2007, 228).
Rhoten et al. (2008) also conducted a study focused on the similarities and differences between the learning outcomes of liberal arts and interdisciplinary programs. For this particular study, the researchers used student and faculty surveys, interviews, and tests to gather data for their analysis. The authors explain that most liberal arts programs “must develop student capacities to integrate or synthesize disciplinary knowledge and modes of think-ing,” which is very similar to the type of synthesis that is expected from an interdisciplinary curriculum (Rhoten et al. 2008, 3–4). The main purpose behind this study was to
FIGURE 1. Three Interrelated Criteria for Assessing Students’ Interdisciplinary Work (Boix Mansilla and Dawes Duraising 2007, 223)
Olcese, et. al.: Meeting the Challenge of Interdisciplinary Assessment 14 science education and civic engagement 6:2 summer 2014
identify the parallels between interdisciplinary and liberal arts programs, in order to show how a program can be made more interdisciplinary without changing its structure or con-tent. Table 1 shows a summary of several parallels between a liberal arts education and an interdisciplinary education.
Rhoten et al. (2008) also analyzed empirical data to draw out trends on the “222 institutions considered ‘Bacca-laureate College-Liberal Arts institutions’ under the 2000 Carnegie Classification system,” whether the interdisciplin-ary programs offered were majors, minors, optional courses, or required courses (Rhoten et al. 2008, 5). In general, “in-terdisciplinary programs are still ‘personally driven,’ whereas departments are ‘self-perpetuating’” (Rhoten et al. 2008, 6). “Personally driven” simply means that if students want to broaden their subject-area exposure they must do so on their own. “Self-perpetuating” refers to fact that departments within an institution need to act in their own self-interest in order to survive and thrive; therefore they tend to avoid interdisciplinary efforts. Interdisciplinary education does not support the mission of individual departments, and if students seek it, they must do so on their own initiative. One would therefore conclude that the only way to truly incor-porate interdisciplinary education into schools is by making it institutionally mandated, at least for the core curriculum that all students are required to take.
Schools should strive to integrate interdisciplinary ef-forts into their institutions because “interdisciplinarity breeds innovation” (Rhoten et al. 2008, 12). Although such innovation carries tremendous benefits, the difficulty of measuring student and educator success was again identi-fied as a barrier. Most schools that are already making efforts towards interdisciplinarity believe that they are somewhat successful according to Rhoten et al. (2008). However, in or-der to mark and measure success, and to continually improve interdisciplinary programs in schools, the authors propose a value-added assessment, which is intended to provide an
“assessment regime that measures growth that has occurred as a result of participation in the institution or academic pro-gram” (Rhoten et al. 2008, 14). Moreover, some cross-cutting goals that are embedded especially in interdisciplinary stud-ies, such as life-long learning, curiosity, creative thinking, synthesis, and integration, have acquired the reputation of being ineffable and, correspondingly, unassessable” (Rhoten et al. 2008, 83). This common problem was addressed by Stowe and Eder (2002) who identified several assessment measures that are placed on a continuum, as seen in Figure 2. These measures can also be used to better define interdis-ciplinary standards by providing a multi-tiered adjustable scale that can help to quantify the assessment of student work based on an instructor’s desired outcomes.
TABLE 1. Comparison of Liberal Arts Education and Interdisciplinary Education Objectives (Rhoten et al. 2008)
Olcese, et. al.: Meeting the Challenge of Interdisciplinary Assessment 15 science education and civic engagement 6:2 summer 2014
Stowe and Eder (2002) state that using a rubric to define and measure interdisciplinary work would improve the “appar-ently subjective nature” of interdisciplinary assessment. They further recommend the rubric as a “visible standard—a scor-ing guide—that allows the assessor and the public, for that matter, to recognize expectations and make increasingly fine distinctions about the quantity and quality of student learning” (96). They expand on their recommendation by noting that assessment must be focused on both improving interdisciplin-ary learning and “improving student learning,” and should be
“embedded within larger systems… and create linkages and enhance coherence within and across the curriculum” (80). Without cooperation across different programs, it is impos-sible to foster an interdisciplinary learning environment.
An example of such cooperation can be seen at USMA, where several academic departments have moved towards a cooperative environment focused on interdisciplinary learn-ing (Elliott et al. 2013). This paper will focus on the educa-tion of the USMA Class of 2016 from their freshman year, when the plan to use energy conservation and the NetZero 1project was adopted to infuse interdisciplinary themes into their core courses. The five Student Learning Outcomes from this effort include four individual discipline-focused outcomes as well as a fifth, which aims to “develop an interdisciplinary perspective that supports knowledge transfer across disciplin-ary boundaries and supports innovative solutions to complex
1 NetZero is an energy initiative by the Department of the Army on several Army posts, including West Point, to produce as much energy as it consumes by the year 2020.
energy problems/projects” (Elliott et al. 2013, 33). In a larger sense, this objective illustrates that interdisciplinary educa-tion addresses the mission of USMA and the Army’s focus on the “development of adaptive leaders who are comfortable operating in ambiguity and complexity will increasingly be our competitive advantage against future threats to our Nation,” as outlined by General Martin Dempsey, Chairman of the U.S. Joint Chiefs of Staff (Elliiott et al. 2013, 30).
Framing the ProblemThe Academy produces graduates who can think dynami-cally in the ever-changing world described in the quotes from Grassie and Boix Mansilla at the beginning of this article. At West Point, this is accomplished by taking not only a multi-disciplinary approach to education, but also an interdisciplin-ary one. The Academy’s Core Curriculum describes the re-quired classes that all cadets must complete or validate. The Core Curriculum does not include any classes required for a cadet’s major. Other non-academic requirements include three tactics courses and seven physical education courses. The interdisciplinary aspect is a new addition to the curricu-lum. In recent years, several committees have recommended promoting interdisciplinary approaches to better meet both the Academy’s and the Army’s goals as outlined in Elliott et al. (2013).
To achieve these goals, several academic departments in-volved in the Core Curriculum developed an interdisciplinary
FIGURE 2. Perspectives on Assessment (Stowe and Eder 2002, 84)
program for the entering plebe2 class, the Class of 2016. Dur-ing the first week of classes, freshmen wrote an essay in their Introduction to Mathematical Modeling course, or MA103, about how they would use concepts from different courses to tackle the challenges that NetZero and the alarming problem of energy consumption in the Army pose to West Point. After thirty instruction sessions (approximately thir-teen weeks), the freshman revised these essays in their Com-position course EN101. This time they used the knowledge acquired throughout the semester in the English course and in the other courses they were taking. Faculty from the De-partment of Mathematical Sciences and the Department of English and Philosophy evaluated these revised essays from different perspectives to emphasize the importance and rel-evance of multiple disciplines. This led to the realization that it was impossible to adequately compare the essays, since the assignments, rubrics, and faculty were not consistent and there was no common rubric to standardize the grading ap-proach. To mitigate this challenge, the essays were compared in our study using the Flesch-Kincaid3 test and a comparison of the final grades for the various essays. Scores for a sample of three essays for 25 students, a total of 75 essays, were used to compare improvement in a measureable, quantitative man-ner. The test consisted of a null hypothesis that there was no significant difference among the ratings, indicating neither improvement nor deterioration of scores from the different assignments throughout the semester, and an alternative hy-pothesis that there actually was a difference between scores. A two-tailed t-test yielded p-values ranging between 0.3 and 0.6. This indicated that the Flesch-Kincaid results were in-
2 “Plebe” is a term referring to the freshman class at West Point.3 A Flesch-Kincaid readability test is a formula designed to evaluate
the difficulty and complexity of technical writing. It consists of two readings: grade level and reading ease. The former uses the formula
0.39 ( total words ) + 11.8 ( total syllables ) – 15.59
to calculate a level of understanding score rated 0-100 that generally correlates with the U.S. grade standards of level of education (i.e. a score of 8.2 means that paper is at an 8th grade level). The latter produces a score of 0-infinity (as it does not have a theoretical upper bound) using the formula
206.835 – 1.015 ( total words ) – 84.6 ( total syllables ) to indicate the difficulty level of reading a paper. For this reading, a lower score indicates a more difficult paper. The Department of Defense uses this test to regulate its documents and manuals; most are required to be between a 6th and 10th grade level rating.
conclusive, meaning that neither the null nor the alternative hypothesis could be rejected.
Despite the inconclusive results of the Flesch-Kincaid test, there was a demonstrated improvement in student work, albeit an improvement that was perceived on the basis of a subjective analysis of the essays. Therefore, a new rubric was developed to re-grade all of the essays in a standardized fash-ion against the desired elements for that particular set of as-signments. To facilitate a comparison, this new and straight-forward rubric aimed at grading each assignment from the different departments on the same scale. The grades were on a 1–10 scale, and the rubric can be seen in Table 2. The essays were then re-graded according to the same rubric and the results were compared again using a two-tailed t-test.
The challenge in evaluating interdisciplinary work is that the term “interdisciplinary” is not well-defined or broadly understood. This became even clearer after the Chemistry faculty conducted an interdisciplinary group capstone in the General Chemistry course with the Class of 2016 dur-ing the second semester of their freshman year. The cap-stone presented the students a complex and challenging en-ergy problem that was both current and militarily relevant to their future roles as Army officers. This project required groups of students to write a memorandum summarizing their findings on an experimental, portable, and green bat-tery recharger for soldiers in the field, and then to provide a presentation of their results to their commander. Cadets conducted an experiment on the battery recharger to test its efficiency, to compare it to current recharging methods, and to address the social and leadership challenges that would occur when this new equipment was integrated into a unit. In addition, the capstone leveraged the students’ various courses and experiences to scaffold understanding of key concepts and technology necessary to engage the problem. The fresh-man cadets were expected to utilize what they learned from math modeling, information technology, general psychology, and general chemistry courses in formulating their solution.
The rubric used to grade these capstones was developed by the Chemistry faculty with input from all the participat-ing courses, and then later utilized by the Chemistry faculty in assessing the capstones. The collaborative rubric identi-fied numerous concepts in each course, and as a result, it was several pages long. Perhaps most significantly, it did not de-fine the term “interdisciplinary” for the faculty and the stu-dents in the capstone, nor did it make clear the associated
total sentences total words
total sentences total words
Olcese, et. al.: Meeting the Challenge of Interdisciplinary Assessment 16 science education and civic engagement 6:2 summer 2014
expectations. At the conclusion of the rubric, faculty were asked to rate on a 1–10 scale how interdisciplinary their stu-dents’ submissions were. The results, displayed in Figure 3, had a standard deviation of .186 and were inconsistent in both the average instructor rating and the range of different ratings faculty assigned. This indicated that the faculty did not share the same understanding of “interdisciplinary” in assessing student work.
Boix Mansilla and Dawes Duraising (2007) state that student interdisciplinary work should
“be well-grounded in the disciplines” “show criti-cal awareness,” and “advance student under-standing” (223). These criteria both define the basic learning objectives of an interdisciplinary education and address the need for baseline knowledge in the subjects being addressed in student work. While these criteria may not be included in a rubric or other grading mechanism, they provide more of a defined objective regard-ing interdisciplinary student work.
Although the idea of graduating interdisci-plinary-minded students is appealing to many programs, the challenge of measuring the suc-cess of interdisciplinary curriculums in produc-ing these “multi-disciplined” graduates has yet to
be addressed. The problem of scaling and measuring inter-disciplinary education is itself interdisciplinary in nature and, consequently, an abstract idea for many (Boix-Mansilla and Dawes Duraising 2007, 218). Interdisciplinary education eval-uation currently lacks a “sound framework” for assessment
TABLE 2. Table 2. Rubric Used to Evaluate the Population Sample of NetZero Essays from Fall 2012.
FIGURE 3. Chemistry Instructor Evaluation of Interdisciplinary Synergy in Capstone Projects during Spring 2013. Courtesy of the United States Military Academy Department of Chemistry and Life Sciences.
Olcese, et. al.: Meeting the Challenge of Interdisciplinary Assessment 17 science education and civic engagement 6:2 summer 2014
since the effects of interdisciplinary efforts on student learning are neither well-defined nor proven (Boix Mansilla 2005, 18). As seen in Figure 2 (Stowe and Eder 2002), the assessment of interdisciplinary work is a non-static scale where the balance between the perspectives and entities is never quite the same from project to project, or from class to class. Stowe and Eder (2002) offer a flexible scale for assessment that allows each in-terdisciplinary quality to be judged according to faculty expec-tations: how discovery-oriented versus objective-orientated do they want student assignments to be? Rhoten et al. (2008) do correlate several common learning outcomes of a liberal arts education with their interdisciplinary counterparts as seen in Table 1. Although useful for demonstrating extensive possible outcomes and correlations, the linkages are broadly defined and do not specify objectives; this exemplifies the issues of scale, definition, and the non-quantified nature of interdisci-plinary education that currently prevail in academia.
All of the aforementioned problems can be traced to a lack of clarity on standards (Boix-Mansilla 2005, 16). Stowe (2002) explicitly calls for a standard for grading, collecting data, and creating a shared understanding, which he suggests could be found in a rubric. A standardized rubric, which is adaptable to several mediums and is general enough to be applicable to several disciplines, is desperately needed for evaluating and assessing interdisciplinary work. Such a rubric needs to clearly define the necessary elements of an interdisciplinary product and be sufficiently adaptable to align with project require-ments; this would resolve several of the problems we have identified. In addition, Stowe and Eder (2002) call for the in-clusion of very specific elements in a rubric, so that it can ad-dress current problems and properly evaluate interdisciplinary work. Among these requirements are assessing complex intel-lectual processes, promoting objectivity, reliability, and validity in assessment, clearly defining learning objectives for students, and being flexible and adjustable for course or curriculum pro-gression (96). Although we conducted a thorough search, we failed to find a rubric that adequately fulfills this need.
Interdisciplinary Rubric Development The goal of the rubric developed at USMA is to create a grad-ing mechanism that can be used in multiple project mediums across multiple disciplines. Simultaneously this rubric main-tains the integrity of the interdisciplinary goals by creating a
more defined standard with which to grade interdisciplinary student work. The rubric also contains open areas for point allotment as well as weighting for each category, which allows faculty to allot points and focus where they see fit. Developing such a rubric required several steps: defining the term inter-disciplinary, identifying the elements that student work needs to demonstrate in order to illustrate interdisciplinary thinking, creating a model that visually represents the interconnectivity of these elements, and then using the defined elements and model to arrive at the rubric categories.
The first step in the rubric development process was to define the term interdisciplinary:
Interdisciplinary: The seamless integration of multi-di-mensional, multi-faceted ideas into a clearly demonstrated understanding of an issue’s breadth and depth, with sound judgment and dynamic thinking.
Boix Mansilla’s definition of interdisciplinary understand-ing4 provided the starting point for the development of the rubric. Additionally, material from the research discussed above identified missing elements from Boix Mansilla’s defi-nition. For example, the best students’ interdisciplinary work included ideas from multiple disciplines that were integrated to demonstrate the level of understanding that the student has attained.
The second step in the rubric development process was to expand the definition of interdisciplinary, in order to create a shared understanding between students, faculty, and those evaluating the interdisciplinary work. To this end, the feed-back and lessons learned from previous student work were used to identify the elements common to successful interdis-ciplinary work. These principles include: discipline specific knowledge, multi-perspective understanding, integration, practical integrated solutions, reflection, and clarity of pur-pose. To illustrate the interconnectivity of these principles, a conceptual model of the characteristics was created. Initially, the intention was to create a linear model to represent the core principles. However, several issues, such as missing connec-tions and limited complexity, led to the immediate conclusion
4 Boix Mansilla defines interdisciplinary understanding as “the capacity to integrate knowledge and modes of thinking drawn from two or more disciplines to produce a cognitive advancement… in ways that would have been unlikely through single disciplinary means” (2005, 16).
Olcese, et. al.: Meeting the Challenge of Interdisciplinary Assessment 18 science education and civic engagement 6:2 summer 2014
that a linear model could not completely describe complex nonlinear problem solving. The resulting model, which il-lustrates a cyclical thinking process, is shown in Figure 4.
The model begins with the framing and scoping of the problem before the application of discipline-specific knowl-edge, which as we have seen is an essential starting point for interdisciplinary work. The core principle of the integration of ideas was partitioned into multi-perspective un-derstanding, integration, and practical integrated solutions. Multi-perspective understanding and discipline-specific knowledge are connected by an addition sign, which symbolizes understanding a topic from multiple perspectives. This illustrates that students must be able to use discipline-spe-cific knowledge to make this essential connection. The arrow labeled “integration” in the lower part of the model represents the synthesis of disci-pline-specific knowledge and multi-perspective understanding into practical integrated solutions. Practical integrated solutions are then connected to reflection via a multiplication sign to show that reflection has a multiplicative effect on interdis-ciplinary understanding. The arrow labeled “clar-ity of purpose” represents the cyclical process and shows the compilation of all the previous elements back into discipline-specific knowledge. The knowledge gained from the various parts of
the cycle can be used in the further learning of other applicable disciplines. This model’s goal is not to ex-plain the rubric, but to illustrate how interdisciplinary education is cyclical in nature, how the characteristics of interdisciplinary understanding are relevant to in-terdisciplinary education, and how student learning should continue to build.
Next, the core principles of what makes student work interdisciplinary were established, defined, and examined. The elements in Figure 1 above, taken from Boix Mansilla and Dawes Duraising (2007), were used as a starting point for the development of this rubric’s core principles: be well grounded in the disci-plines, show critical awareness, and advance student leaning through understanding (223). For the pur-pose of this rubric, some elements were modified and expanded to create six core principles. A list of the six core principles that were incorporated into the rubric, along with their definitions, appear in Table 3.
Problem framing and scope are derived from the idea that interdisciplinary work should show critical awareness. The definition used in the rubric is very flexible, so that edu-cators can adapt it for different project mediums and fac-ulty, departments, and/or university requirements. Critical awareness, as defined by Boix Mansilla (2007), includes the
FIGURE 4. The Cyclical Model of the Key Interdisciplinary Characteristics. This model demonstrates the interconnectivity of the defined interdisciplinary elements.
Core Principle DefinitionFraming and Scope Cadets are able to clearly define
a stated purpose/thesis with the appropriate knowledge.
Discipline Knowledge Strong demonstrated subject knowledge applied correctly.
Integration of Ideas Multi-dimensional feasible, practical solution with multi-faceted and seamlessly connected ideas.
Clarity of Purpose Demonstrated clear understanding of the topic’s breadth and depth with a defined purpose of investigation.
Reflection Connection of ideas indicating reflection of the interconnectivity of disciplines and importance of the issue at large.
Appropriate Presentation Information is presented in the appropriate medium with proper tone, word choice, spelling, grammar, etc.
TABLE 3. Basic Definitions of Mastery in Each of the Six Core Principles of the Interdisciplinary Rubric
Olcese, et. al.: Meeting the Challenge of Interdisciplinary Assessment 19 science education and civic engagement 6:2 summer 2014
definition of purpose as well as the integration of ideas. The definition used for problem framing and scope in the ru-bric requires that the student’s work have a clearly defined purpose. This was created as a separate category because we had observed a clear trend of misunderstanding among faculty regarding the level of complexity that they expected. This is an important aspect of student interdisciplinary understanding; it allows the faculty to scale assignments according to the expected level of student understand-ing and allows the student to recognize just how complex and multi-disciplined a product the instructor is seeking. For example, if students were assigned a project on how to effectively stock a warehouse, an instructor would not have the same expectations of a freshman who has taken only introductory courses in mathematical modeling and economics as of a senior who had taken nonlinear opti-mization, supply chain management, and microeconomics courses. Having this requirement in the rubric makes clear the expectation that students will properly identify what they want to address, and also allows the instructor to have a frame of reference in a project.
The rubric’s second core principle, discipline knowledge is well grounded in the disciplines and is intentionally more open-ended, so that it can be readily adapted to different departments, projects, and situations (Boix Mansilla 2007). Identifying theories, examples, findings, methods, etc. may not be relevant or necessary in a given problem. Therefore, although the evaluator is given an area in the rubric that calls for disciplinary knowledge, the rubric does not explic-itly indicate how that knowledge is to be graded. For ex-ample, in our warehouse stocking project, a freshman might be expected to mathematically model the effects of chang-ing employee wages on productivity. A university senior, on the other hand, might be expected to produce a business recommendation to stakeholders by addressing the intrica-cies of supply chain management on warehouse profits as well as its psychological implications for employees. The discipline knowledge area of the rubric enables the evalua-tor to determine how much knowledge and understanding students are expected to demonstrate, while ensuring that the importance of disciplinary understanding is not lost on an interdisciplinary project.
The integration of ideas principle is really the quintes-sential element for the interdisciplinarity of this rubric. All six core principles are important interdisciplinary factors,
but if this element were removed, the rubric could be used for a project that is not interdisciplinary. Integration of ideas derives its meaning from the critical awareness and advanced student understanding pieces identified above in Figure 1. This rubric defines integration of ideas as multi-dimensional, feasible, practical solutions with multi-faceted and seamlessly connected ideas. It is important to note the difference between being integrated and being seamlessly integrated. The seamless integration of ideas, which can take on different meanings depending on the assignment, is an indicator of true multi-dimensional, multi-faceted understanding. Seamless integration. We define the term seamlessly integrated to mean that ideas are not simply laundry-listed, but instead are connected in an intelligent and logical fashion. The definitional elements of multi-di-mensional and multi-faceted identify the need for complex-ity in student work. It is multi-dimensional when students make use of multiple dimensions of their education or, in other words, use multiple disciplines, in their work. Multi-faceted means that students are able to use evidence and knowledge to back up their multi-dimensional claims. The most important component is that students be able to dem-onstrate a clear understanding of what they are presenting. This also relates to a student’s ability to demonstrate the span of an issue’s breadth and depth. In other words, stu-dents should be able to apply disciplines to an issue or topic with an appropriate understanding of the level of each of the disciplines. The use of extraneous disciplines merely for the sake of incorporating more disciplines does not neces-sarily make student work interdisciplinary. In fact, it contra-dicts the idea of advancing the complexity of the student’s thought process.. Students who apply the appropriate level of discipline breadth and depth indicate their ability to use sound judgment or logic, as well as their ability to think dynamically.
The next two core principles, clarity of purpose and re-flection, were added to address the students’ failure to in-ternalize what they were learning and understanding; this was revealed during the analysis of the USMA interdis-ciplinary program. The main challenge was that students did not fully grasp why a given project was interdisciplin-ary, or why that was important. To alleviate this, the core principle clarity of purpose was added to the rubric to help students understand the “why”; the intent was to motivate them to define the purpose of their investigation and to
Olcese, et. al.: Meeting the Challenge of Interdisciplinary Assessment 20 science education and civic engagement 6:2 summer 2014
take an in-depth approach to the problem. This is dif-ferent from problem framing and scope in a very impor-tant way: problem framing and scope focuses on a well-defined thesis statement or purpose statement, whereas clarity of purpose focuses on the content of student work. In other words, problem framing and scope ask whether students have a clearly stated framework for their project, while clarity of purpose asks whether they demonstrate their personal interdisciplinary understanding and then explain it well to their audience. Similarly, the next prin-ciple, reflection, calls for a clear and delineated connection of ideas and an indication that students have reflected on the interconnectivity and importance of their areas of study. These two core principles are drivers of inter-nalization and cognitive advancement in interdisciplinary learning. They are particularly important because often students do not reflect on what they have learned. The reflection piece is intended to facilitate a deeper under-standing of what they are learning and to encourage stu-dents to consider how the material fits into the greater scheme of their education.
The final element of the rubric shown in Table 3 is the presentation principle. This principle calls for informa-tion that is presented in a suitable medium with proper tone, word choice, spelling, grammar, etc. In short, did the students address the audience correctly and present their knowledge intelligently while doing so? This section can be adapted to the type of project and course for which the rubric is being used. For example, English faculty would probably expand this section because of its importance to their learning outcomes, whereas chemistry faculty may place more emphasis on the discipline-knowledge portion.
The newly developed rubric was presented to the Math course leaders for use on the freshman’s “mini” 5 capstone exercise in December 2013. The rubric was sent to the faculty with minimal guidance. The feedback from the course director made it clear that the students and faculty did not fully grasp the intention or expectations behind the rubric. A few factors contributed to this: sixty-six percent of the faculty were new to the department; the interdisciplinary expectations were not fully explained to the faculty; although everyone received the rubric, each
5 The term “mini” is used to differentiate this capstone from the larger and more encompassing Chemistry capstone encoun-tered by the cadets at the conclusion of freshman year.
instructor created his or her own rubric for the mini-capstone; and the students who took the mini-capstone and the faculty who graded their work were under sig-nificant time pressure. The mini-capstone in its creation, execution, and grading was not given adequate time due to end of semester requirements at USMA during the November-December time period. An important con-clusion from this feedback was that the faculty needed to have a common understanding of what is expected on an interdisciplinary project. To achieve this for the General Chemistry capstone project in the spring of 2014, a grad-ing calibration exercise was conducted. This calibration included good and poor examples of interdisciplinary work from the previous year’s chemistry capstone, and showed faculty how to distinguish between good and poor work and how to use the rubric in assigning a grade.
Implementing the Interdisciplinary Rubric
The first step in implementing the rubric was calibra-tion with the faculty. With such an exercise, the faculty should take away a common understanding of what ex-actly interdisciplinarity is as well as the knowledge of what constitutes a good final project. The plan for the calibration exercise developed for USMA faculty who would be grading the CH102 General Chemistry cap-stone in the spring of 2014 was an hour-long presentation and discussion. Prior to the presentation, faculty received a packet of examples of cadet work in each of the major portions of the previous year’s capstone project. The ex-amples included “A” work as well as examples of common integration errors students make: the “laundry list,” the
“tacked on at the end,” and the “no real knowledge” inte-gration errors. The “laundry list” is an example of how a student may mention and be knowledgeable in multiple disciplines but does not integrate them, providing instead a “laundry list” of the different disciplines and explaining the relevance of each individually. The “tacked on at the end” error (or whatever we may call it) exemplifies how a student may go in-depth in one discipline, particularly in the discipline for which the assignment was given, then tack on a sentence or two at the end mentioning other disciplines in order to call the project interdisciplinary. The “no real knowledge” example presents a plethora of
Olcese, et. al.: Meeting the Challenge of Interdisciplinary Assessment 21 science education and civic engagement 6:2 summer 2014
ideas but does not demonstrate that the student learned or integrated disciplines and/or ideas. With these ex-amples, faculty became more familiar with what correct and incorrect work looked like. The “A” level example was not meant to illustrate the perfect or only solution; it was merely one example. Faculty evaluated each example us-ing the standard A, B, C, D, F grading scale based on how interdisciplinary they felt each project was.
At the start of the presentation portion of the rubric calibration, faculty were introduced to the interdisciplin-ary characteristics and model from Figure 4. This ensured understanding of interdisciplinary characteristics prior to the introduction to the rubric itself. After the characteris-tics were covered, the results from the exercise, which the faculty just had completed, were discussed. This clarified any misunderstandings that faculty had about the inter-disciplinary characteristics, while the examples of chem-istry capstones from the previous year provided a frame of reference. Next the rubric was thoroughly explained, showing how it was scalable, expandable, and concise to meet instructor needs for interdisciplinary student projects.
The General Chemistry capstone rubric for 2014 dif-fers from its 2013 predecessor in two very important ways. First, it is significantly shorter; its two pages (compared to seven pages) emphasize quality over quantity. Instead of listing every detail of the project, the new capstone rubric has five categories that address the math model-ing, leadership, information security, oral communication, and the required submission components of the project, all without specific details. This allows the students to be more creative in their answers to the given problem.
The 2013 rubric was not based on any interdisciplinary principles or examples. Instead, it listed specific require-ments from the disciplines the students were supposed to integrate. The result was quite the opposite: the 2013 capstone projects tended to be disjointed because of the slew of specific requirements. This year’s capstone rubric incorporates the interdisciplinary principles described in Table 3. Problem framing and scope is addressed in the Proj-ect Summary section with the requirement for a bottom line up front (BLUF), or thesis. Discipline knowledge is asked for in the Discrete Dynamic Modeling, Persuasion and Conformity in a Leadership Environment, and Infor-mation Security sections. Although the course-specific
requirements must be addressed, Integration of ideas is as-sessed in the Oral Communication and Project Summary sections, which requires that fluid transitions and logi-cally ordered and related ideas be integrated. Appropriate presentation is also adequately addressed in these sections, as the rubric lays out clear expectations of the written and oral presentations for students, including their tone, body language, and level of professionalism. Clarity of purpose and reflection are asked for in the Project Summary sec-tion, which calls for contingency plans and thoroughly explained analysis of the total problem.
Initial instructor feedback on the use of this rubric is that it better defined expectations for the students’ interdisciplinary work, for both the instructor and the students. After using the rubric in the calibration exer-cise, instructors stated that they felt more confident and prepared than they had in 2013 when there was no such exercise and assessment tool available; this year they un-derstood what was asked of them and of the students. Ini-tial comparisons of the interdisciplinary assessments of the students’ work from 2013 and 2014 are quite positive. On a scale of 0–10, the average interdisciplinary score given by instructors was 5.69 in 2013, with zero being the least interdisciplinary and 10 the most. (See Figure 3 for these data.) In 2014 this improved to 7.79 (actually 15.5/20). There was also less variability between instruc-tors. For example, in 2013 the standard deviation of the mean scores assigned by each of the instructors was 1.86 (Figure 3). In 2014, the standard deviation between the instructors’ mean scores was .98 (1.96/20), a decrease of over 47%.
Future Work and ConclusionNow that the General Chemistry capstone for USMA
Class of 2017 has concluded, several analyses must be completed to evaluate the progress of interdisciplinary education at USMA. At a minimum, an analysis of the grades and feedback from the students and faculty needs to be conducted. The analysis of the grades should in-clude a distribution of grades compared with their ex-pected distribution, as well as a quantitative and a quali-tative analysis of the capstones compared to the previous years’ capstones. This could be done using the methods previously employed, including the use of Flesch-Kincaid,
Olcese, et. al.: Meeting the Challenge of Interdisciplinary Assessment 22 science education and civic engagement 6:2 summer 2014
paired t-test, the distribution of the faculty’s interdisci-plinary rating similar to Figure 3, and/or a cross-course sample of projects re-graded by the course director.
The discussion and research that have taken place at West Point since the first General Chemistry cap-stone project in 2013 indicate that the results of this year’s changes should be positive. Although there is as yet no statistical evidence to demonstrate improvement, the gen-eral understanding of how interdisciplinarity looks, how to produce it, and how to assess it is much more expan-sive now than in 2013. The reason for this might be that faculty and students at USMA are now experienced with interdisciplinary work and have a clearer understanding of interdisciplinary assessment and its importance over the course of a year.
The world is a complex and rapidly changing place that requires its future scientists, scholars, engineers, teachers, and leaders to think dynamically and across disciplines.. Interdisciplinary assessment is necessary for the future of education, particularly at West Point where we recognize that “adaptive leaders who are comfortable operating in ambiguity and complexity will increasingly be our competitive advantage against future threats to our Nation” (Elliott et al. 2013, 30). Only time will tell whether this interdisciplinary rubric has met its goal of creating a grading mechanism that can be used in mul-tiple project mediums across multiple disciplines. Given the extensive research and analysis done at West Point to create this much- needed and useful tool, the prospects for future interdisciplinary education are promising.
About the AuthorsElizabeth Olcese graduated with a Bachelor of Science degree in Operations Research from the United States Military Academy at West Point, NY in 2014. She served as a student researcher for West Point’s Core
Interdisciplinary Team focused on enhancing opportuni-ties for interdisciplinary learning in West Point’s core aca-demic curriculum. Upon completion of the Quartermas-ter Basic Officer Leader Course at the Army Logistics School in Fort Lee, VA, she will serve as a second lieuten-ant for the 25th Infantry Division at Schofield, Hawaii.
Joseph C. Shannon graduated with a doc-torate in Curriculum and Instruction with a focus in Science Education from the College of Education at the University of Washington, WA. He is a former mem-
ber of West Point’s Core Interdisciplinary Team that was focused on enhancing opportunities for interdisciplinary learning in West Point’s core academic curriculum. He is a former Academy Professor at the United States Mili-tary Academy and Program Director for the General Chemistry Program in the Department of Chemistry and Life Science. He is currently the Dean of Academic Pro-grams at South Seattle College in West Seattle, Washington.
Gerald Kobylski graduated with a doctor-ate in interdisciplinary studies (Systems Engineering and Mathematics) from Ste-vens Institute of Technology, NJ. He cur-rently is co-leading a thrust to infuse in-
terdisciplinary education into West Point’s core academic curriculum. He is also deeply involved with pedagogy, faculty development, and assessment. Jerry is an Acad-emy Professor at the United States Military Academy, a Professor of Mathematical Sciences, and a Commissioner for t Middle States on Higher Education.
Lieutenant Colonel Charles (Chip) Elliott graduated with a doctorate in Geography and Environmental Engineering from Johns Hopkins University in Baltimore, MD and is a registered professional engi-
neer in Virginia. He is currently the General Chemistry Program Director and the Plebe (Freshman) Director for the Core Interdisciplinary Team at the United States Military Academy. He has previously taught CH101/102 General Chemistry, EV394 Hydrogeology, EV488 Solid & Hazardous Waste Treatment and Remediation, EV401 Physical & Chemical Treatment, and EV203 Physical Geography. He is currently an Assistant Professor in the Department of Chemistry and Life Science.
ReferencesBoix Mansilla, V. Interdisciplinary Understanding: What Counts
as Quality Work?” Interdisciplinary Studies Project, Harvard Graduate School of Education.
Olcese, et. al.: Meeting the Challenge of Interdisciplinary Assessment 23 science education and civic engagement 6:2 summer 2014
Boix Mansilla, V. 2005. “Assessing Student Work at Disciplinary Cross-roads.” Change 37 (1): 14–21.
Boix Mansilla, V., and E. Dawes Duraising. 2007. “Targeted Assessment of Students’ Interdisciplinary Work: An Empirically Grounded Framework Proposed. The Journal of Higher Education 78 (2): 216–237.
Elliott, C., G. Kobylski, P. Molin, C.D. Morrow, D.M. Ryan, S.K. Schwartz, J.C. Shannon, and C. Weld. 2013. “Putting the Backbone into Interdisciplinary Learning: An Initial Report.” Manuscript submitted for publication, United States Military Academy, West Point, NY.
Grassie, W. (n.d.). “Interdisciplinary Quotes.” Thinkexist.com. http://thinkexist.com/quotation/as-the-pace-of-scientific-discovery-and/1457907.html (accessed November 16, 2013).
Ivanitskaya, L., D. Clark, G. Montgomery, and R. Primeau. 2002. “Inter-disciplinary Learning: Process and Outcomes.” Innovative Higher Education, 27 (2): 95–111.
Newell, W.H. 2006. “Interdisciplinary Integration by Undergraduates.” Issues in Integrative Studies 24: 89–111.
Repko, A.F. 2007. “Interdisciplinary Curriculum Design.” Academic Exchange Quarterly 11 (1): 130–137.
Repko, A.F. 2008. “Assessing Interdisciplinary Learning Outcomes. Aca-demic Exchange Quarterly 12 (3): 171–178.
Rhoten, D., V. Boix Mansilla, M. Chun, and J. Thompson Klein. 2008. “Interdisciplinary Education at Liberal Arts Institutions.” Teagle Foundation White Paper.
Stowe, D.E., and D.J. Eder. 2002. “Interdisciplinary Program Assessment.” Issues in Integrative Studies 20: 77–101.
Olcese, et. al.: Meeting the Challenge of Interdisciplinary Assessment 24 science education and civic engagement 6:2 summer 2014
Olcese, et. al.: Meeting the Challenge of Interdisciplinary Assessment 25 science education and civic engagement 6:2 summer 2014
Appendix
Encl
osur
e 1. Ca
tego
ryBa
sic-
F (_
_Poi
nts)
Nov
ice-
C (_
_Poi
nts)
Inge
niou
s-B
(__P
oint
s)M
aste
ry-A
(__P
oint
s)
1.1
Proj
ect p
urpo
se/t
hesis
is n
ot p
rese
nt o
r ext
rem
ely
uncl
ear t
o th
e po
int w
here
the
audi
ence
doe
s not
un
ders
tand
the
prob
lem
.
1.1
Proj
ect h
as a
pur
pose
/the
sis th
at is
miss
ing
info
rmat
ion
or co
ntai
ns to
o m
uch
info
rmat
ion.
The
aud
ienc
e ha
s a
gene
ral u
nder
stan
ding
of t
he is
sue
at h
and
but d
oes n
ot
clea
rly u
nder
stan
d.
1.1
Proj
ect h
as a
wel
l-sta
ted
purp
ose/
thes
is w
hich
ena
bles
au
dien
ce to
hav
e a
good
und
erst
andi
ng o
f iss
ue a
t han
d bu
t is m
issin
g m
inor
ess
entia
l ele
men
ts o
r con
tain
s ex
tran
eous
info
rmat
ion.
1.1
Proj
ect h
as a
clea
rly st
ated
pur
pose
or t
hesis
whi
ch
enab
les t
he a
udie
nce
to cl
early
und
erst
and
the
issue
at
hand
.
1.2
Disc
iplin
es u
sed
do n
ot a
pply
dire
ctly
to is
sue
at h
and.
1.2
Disc
iplin
e us
e is
faul
ty a
nd co
ntai
ns 3
+ ex
tran
eous
su
bjec
t use
.1.
2 Ap
prop
riate
disc
iplin
es/s
ubje
cts u
sed.
1-2
ext
rane
ous
topi
cs th
at d
o no
t tie
into
pre
sent
atio
n w
ell.
1.2
Appr
opria
te d
iscip
lines
/sub
ject
s use
d; n
o ex
tran
eous
to
pics
use
d.
1.3
Idea
s and
issu
es v
ery
uncl
ear.
1.3
idea
s and
issu
es u
ncle
ar.
1.3
Idea
s and
issu
es d
elin
eate
d w
ell.
1.3
Idea
s and
issu
es cl
early
del
inea
ted
2.1
Faul
ty b
ase
know
ledg
e. B
asic
und
erst
andi
ng in
chos
en
topi
cs is
not
pre
sent
.2.
1 Ba
selin
e su
bjec
t kno
wle
dge
and
unde
rsta
ndin
g in
ch
osen
topi
cs.
2.1
Good
bas
e su
bjec
t kno
wle
dge
and
dem
onst
rate
d ba
sic
unde
rsta
ndin
g in
chos
en to
pics
.2.
1 St
rong
bas
e su
bjec
t kno
wle
dge.
Dem
onst
rate
d de
pth
of
unde
rsta
ndin
g in
chos
en to
pics
.2.
2 In
appr
opria
te u
se o
f kno
wle
dge.
Sev
ere
erro
rs in
ap
plic
atio
n of
disc
iplin
e kn
owle
dge.
2.2
Appr
opria
te u
se o
f kno
wle
dge.
Sev
ere
erro
rs in
ap
plic
atio
n of
disc
iplin
e kn
owle
dge
to is
sue
at h
and.
2.2
Appr
opria
te u
se o
f kno
wle
dge.
Slig
ht e
rror
s in
disc
iplin
e ap
plic
atio
n to
issu
e at
han
d.2.
2 Ap
prop
riate
use
of k
now
ledg
e. C
orre
ctly
util
izes
disc
iplin
e kn
owle
dge
in a
pplic
atio
n to
issu
e at
han
d.
3.1
Prov
ides
mul
ti-di
men
siona
l sol
utio
ns th
at a
re n
ot
feas
ible
and
/or p
ract
ical
due
to li
ttle
und
erst
andi
ng o
f the
iss
ue a
t lan
d. C
onne
ctio
n of
idea
s is f
aulty
.
3.1
Prov
ides
mul
ti-di
men
siona
l, pr
actic
al co
nclu
sions
. Ide
as
may
not
be
com
plet
ely
feas
ible
bec
ause
of e
rror
s or
misu
nder
stan
ding
of t
he is
sue
at h
and.
Idea
s are
co
nnec
ted,
but
not
seam
less
ly.
3.1
Prov
ides
mul
ti-di
men
siona
l, fe
asib
le, p
ract
ical
co
nclu
sions
. Ide
as a
re co
nnec
ted,
but
not
seam
less
ly.
3.1
Prov
ides
mul
ti-di
men
siona
l, fe
asib
le, p
ract
ical
co
nclu
sions
with
mul
ti-fa
cete
d an
d se
amle
ssly
conn
ecte
d id
eas.
3.2
Inte
grat
ion
not p
rese
nt o
r is i
rrat
iona
l and
/or
inef
fect
ive.
Im
bala
nce
of d
iscip
line
detr
acts
from
inte
ntio
n of
pro
ject
.
3.2
Inte
grat
ion
is pr
esen
t but
is ir
ratio
nal o
r ine
ffect
iven
ess.
Disc
iplin
es a
re in
tegr
ated
but
ver
y un
bala
nced
.3.
2 In
tegr
atio
n is
ratio
nale
and
for t
he m
ost p
art e
ffect
ive.
Di
scip
lines
are
inte
grat
ed b
ut a
re i
mba
lanc
ed.
3.2
Inte
grat
ion
is ra
tiona
le a
nd e
ffect
ive.
Disc
iplin
es a
re
both
inte
grat
ed a
nd b
alan
ced.
3.3
Uncl
ear o
r non
-pre
sent
find
ings
, con
clus
ions
, re
com
men
datio
ns, a
nd/o
r exa
mpl
es. O
r sai
d to
pics
not
gr
ound
ed in
disc
iplin
e(s')
kno
wle
dge.
3.3
Dece
nt fi
ndin
gs, c
oncl
usio
ns, r
ecom
men
datio
ns, a
nd/o
r ex
ampl
es w
ith co
nnec
tions
to d
iscip
line
know
ledg
e.3.
3 So
und
findi
ngs,
conc
lusio
ns, r
ecom
men
datio
ns, a
nd/o
r ex
ampl
es g
roun
ded
in d
iscip
line
know
ledg
e.3.
3 Q
ualit
y fin
ding
s, co
nclu
sions
, rec
omm
enda
tions
, and
ex
ampl
es fr
om g
roun
ded
disc
iplin
e kn
owle
dge.
3.4
Idea
s are
laun
dry-
liste
d an
d ar
e no
t wel
l exp
lain
ed
and/
or in
tegr
ated
.
3.4
Basic
rang
e of
subj
ect i
nteg
ratio
n pr
esen
t. Id
eas
som
ewha
t lau
ndry
-list
ed a
nd d
o no
t con
tain
exp
lana
tions
or
inte
grat
ion.
3.4
Good
rang
e of
subj
ect i
nteg
ratio
n op
port
uniti
es a
re
expl
oite
d.3.
4 Fu
ll ra
nge
of su
bjec
t int
egra
tion
oppo
rtun
ities
hav
e be
en so
ught
out
and
exp
loite
d.
4.1
Does
not
gra
sp th
e br
eadt
h an
d de
pth
of th
e iss
ue in
qu
estio
n to
an
acce
ptab
le le
vel.
4.1
Unde
rsta
nds t
he sc
ale
of th
e iss
ue in
que
stio
n bu
t doe
s no
t effe
ctiv
ely
com
mun
icat
e it.
4.1
Dem
onst
rate
s a g
ener
al u
nder
stan
ding
of t
he is
sue
in
ques
tion'
s bre
adth
or d
epth
.4.
1 De
mon
stra
tes c
lear
und
erst
andi
ng o
f the
issu
e in
qu
estio
n's b
read
th a
nd d
epth
.
4.2
A ge
nera
l pur
pose
of i
nves
tigat
ion
is pr
esen
t but
not
to
a sa
tisfa
ctor
y le
vel o
r is n
ot p
rese
nt.
4.2
Defin
es th
e pu
rpos
e of
inve
stig
atio
n to
a sa
tisfa
ctor
y le
vel,
but c
onta
ins e
xtra
neou
s mat
ter o
r loo
se e
nds.
4.2
Defin
ed p
urpo
se o
f inv
estig
atio
n is
pres
ent.
4.2
Clea
rly d
efin
es th
e pu
rpos
e of
inve
stig
atio
n.
5.1
Idea
s in
proj
ect a
re co
nnec
ted
and
dem
onst
rate
m
inim
al re
flect
ion
on th
e im
port
ance
of t
he is
sue
at la
rge.
5.1
Conn
ectio
ns o
f ide
as d
emon
stra
tes s
ome
refle
ctio
n on
di
scip
linar
y in
terc
onne
ctiv
ity a
nd th
e im
port
ance
of t
he
issue
at l
arge
.
5.1
Conn
ectio
n of
idea
s ind
icat
es st
uden
t has
refle
cted
on
the
inte
rcon
nect
ivity
and
impo
rtan
ce o
f the
issu
e at
larg
e.
5.1
Clea
r and
del
inea
ted
conn
ectio
n of
idea
s ind
icat
es
stud
ent h
as re
flect
ed o
n th
e in
terc
onne
ctiv
ity a
nd
impo
rtan
ce o
f the
issu
e at
larg
e.
5.2
Cour
se S
peci
fic R
efle
ctio
n (if
des
ired)
5.2
Cour
se S
peci
fic R
efle
ctio
n (if
des
ired)
5.2
Cour
se S
peci
fic R
efle
ctio
n (if
des
ired)
5.2
Cour
se S
peci
fic R
efle
ctio
n (if
des
ired)
6.1
Info
rmat
ion
is co
nvey
ed in
such
a w
ay th
at d
etra
cts
from
the
unde
rsta
ndin
g of
the
prob
lem
at h
and.
6.1
Info
rmat
ion
is co
nvey
ed in
such
a w
ay th
at th
e au
dien
ce h
as a
gen
eral
und
erst
andi
ng o
f the
pro
blem
at
hand
. Ins
tanc
es o
f lau
ndry
-list
ing
are
pres
ent a
nd sl
ight
ly
detr
act f
rom
the
cohe
renc
e of
the
proj
ect.
6.1
Info
rmat
ion
is co
nvey
ed in
such
a w
ay th
at th
e au
dien
ce u
nder
stan
ds th
e sc
ope
of th
e pr
oble
m a
nd id
eas
are
wel
l-con
veye
d. M
inor
inst
ance
s of l
aund
ry li
stin
g th
at
do n
ot d
etra
ct fr
om th
e co
here
ncy
of th
e pr
ojec
t.
6.1
Info
rmat
ion
conv
eyed
in su
ch a
way
that
aud
ienc
e un
ders
tand
s the
scop
e of
the
prob
lem
cohe
rent
ly co
nvey
ed
idea
s (no
t lau
ndry
list
ed su
bjec
ts).
6.2
Pres
enta
tion
orde
r is i
llogi
cal a
nd/o
r has
disc
repa
ncie
s th
at d
etra
ct fr
om th
e lo
gic,
art
icul
atio
n, fl
uidi
ty, o
r the
pr
esen
tatio
n of
idea
s. Ap
prop
riate
tone
and
/or d
ept n
ot
pres
ent.
6.2
Pres
enta
tion
orde
r has
disc
repa
ncie
s in
logi
c or f
luid
ity.
Idea
s are
not
wel
l art
icul
ated
. App
ropr
iate
tone
and
dep
th
pres
ent,
but i
s not
ove
rall
effe
ctiv
e.
6.2
Pres
enta
tion
orde
r is l
ogic
al a
nd fl
uid
with
min
or
disc
repa
ncie
s. Id
eas a
re a
rtic
ulat
ed a
nd co
nvey
ed w
ell.
Effe
ctiv
e to
ne a
nd d
epth
pre
sent
.
6.2
Pres
enta
tion
orde
r is l
ogic
al a
nd fl
uid.
Idea
s are
ar
ticul
ated
and
conv
eyed
effe
ctiv
ely.
App
ropr
iate
and
ef
fect
ive
tone
and
dep
th p
rese
nt.
6.3
Term
s not
def
ined
app
ropr
iate
ly o
r pro
ject
pre
sent
ed
with
term
inol
ogy
in su
ch a
way
that
it d
etra
cts f
rom
the
audi
ence
's un
ders
tand
ing.
And
/or u
se o
f ina
ppro
pria
te
term
inol
ogy
or la
ngua
ge.
6.3
Appr
opria
te te
rms u
sed
but n
ot d
efin
ed fu
lly fo
r au
dien
ce/s
ettin
g co
ntex
t.
6.3
Term
s app
ropr
iate
ly d
efin
ed a
nd u
sed
as n
eede
d fo
r au
dien
ce/s
ettin
g; m
inor
disc
repa
ncie
s in
use
or
unde
rsta
ndin
g of
term
inol
ogy
or la
ngua
ge.
6.3
Term
s app
ropr
iate
ly d
efin
e an
d us
ed a
s nee
ded
for
audi
ence
/set
ting;
app
ropr
iate
use
of t
erm
inol
ogy
and
lang
uage
.
6.4
Erro
rs in
gra
mm
ar, p
unct
uatio
n, sp
ellin
g, a
nd fo
rmat
di
stra
ct fr
om th
e pr
ojec
t con
tent
. Pro
ject
form
at is
ge
nera
lly in
corr
ect.
6.4
Gram
mar
, pun
ctua
tion,
spel
ling,
and
form
at co
ntai
n sig
nific
ant e
rror
s tha
t det
ract
from
the
proj
ect c
onte
nt
sligh
tly. P
roje
ct fo
rmat
is st
ill g
ener
ally
corr
ect.
6.4
Prop
er g
ram
mar
, pun
ctua
tion,
spel
ling,
and
form
at u
sed
thro
ugho
ut w
ith m
inim
al e
rror
s. Pr
ojec
t for
mat
use
s ge
nera
l cor
rect
ness
.
6.4
Prop
er g
ram
mar
, pun
ctua
tion,
spel
ling,
and
form
at u
sed
thro
ugho
ut w
ith n
o er
rors
.
6. A
ppro
pria
te P
rese
ntat
ion
Wei
ght _
__%
4. C
larit
y of
Pur
pose
Wei
ght _
__%
3. In
tegr
atio
n of
Idea
s Wei
ght
___%
2. D
isci
plin
e Kn
owle
dge
Wei
ght
___%
1. P
robl
em F
ram
ing
& S
cope
W
eigh
t ___
%
5. R
efle
ctio
n
W
eigh
t ___
%