the research behind the carnegie learning math series

THE RESEARCH BEHINDTHE CARNEGIE LEARNING® MATH SERIES

Dr. Steve Ritter, Ph.D.Co-Founder and Chief Scientist, Carnegie Learning, Inc.

The Research Behind the Carnegie Learning Math Series

www.carnegielearning.com 1

TABLE OF CONTENTS

Executive Summary ..........................................................................................................................2

Introduction ...................................................................................................................................4

Motivating Students to Learn

TheInfluenceofBeliefs,Goals,andExpectations...............................................................4

MotivationintheCarnegieLearningMathSeries...............................................................8

ACT-R and How People Learn

Instructionalcomponents....................................................................................................10

Conclusion......................................................................................................................................17

About the Author.............................................................................................................................17

References......................................................................................................................................18

2


www.carnegielearning.com

EXECUTIVE SUMMARY

The Carnegie Learning® Math Series (CLMS) is a blended solution to mathematics instruction for students in grades 6-8. Both our MATHia™ software and our text build on the proven effectiveness of our Cognitive Tutor programs and have been strongly influenced by research into how students learn and how to best motivate students to

succeed academically. The research addressed in the product includes:

Motivational Research

As students enter middle school, many start to lose interest in academic success. This loss of interest can be attributed to students’ alienation, resulting from a feeling that the school environment is unwelcoming to them. Recent research identifies several elements of this alienation, as well as practices that can re-focus students on academic achievement.

• Students who believe that they can get smarter will work harder. Teaching students about the way that the brain changes as they learn has been shown to encourage students to believe that they have the capability to learn. This results in better academic performance. Within CLMS, we praise effort above innate ability. We also provide messages about how learning leads to brain growth.

• Students who approach a task with the intention of succeeding (rather than avoiding failure) are more likely to excel. Those students who consider long-term learning to be their goal may learn more flexibly. In CLMS, we provide feedback focused on success and use badges to reward effective learning behaviors.

• Expectations about performance that reflect stereotypes (particularly about race and gender) can lead to anxiety, reduced mental capacity, and diminished performance. CLMS includes assurances that the instruction is appropriate for all students and messages counteracting negative stereotypes help mitigate against negative effects.

Learning Research

Carnegie Learning does not just read the research on how people learn; we are active participants in this research and frequently publish results in refereed journals and at conferences. CLMS incorporates a wide variety of activities and approaches that represent the result of decades of this research. Some of the major tasks components

of our instructional approach include:

ActiveLearning

The research makes clear that students need to actively engage with content if they are to benefit from it. All of our activities within MATHia and the classroom environment encourage students to be thoughtful about their work, to consider hypotheses and conclusions from different perspectives, and to build deep understanding of mathematics.



CognitiveTutorTasks

Cognitive Tutor Tasks have long been the heart of the research-based approach behind Carnegie Learning’s software. These tasks emphasize problem solving and build a cognitive model of each student’s abilities in order to provide each student with appropriate pacing and tasks. CLMS includes Cognitive Tutor tasks for all topics.

WorkedExamples

Research shows that learning is best supported with a mix of problem solving, as in the Cognitive Tutor tasks and worked examples. MATHia software includes extensive worked examples. Our text includes worked examples as well as multiple examples of student work to encourage comparison and self-explanation.

FluencyTasks

Success in mathematics builds upon the ability to fluently recognize mathematical relationships. Students need to immediately “see” that 1⁄2 and 0.5 are equal and to immediately “know” that 1⁄2 is greater than 1⁄5. Beyond the goal of fluency, research has shown that time pressure can encourage students to adapt more efficient and flexible approaches to problem solving. CLMS includes game-like fluency tasks in order to help

students build representational and procedural fluency.

4



Introduction

Carnegie Learning was founded by cognitive psychologists, and we have always based our instruction on the best available research about how people learn.

The Carnegie Learning Math Series (CLMS) is a three-year sequence providing a complete set of educational materials that will help students master middle school mathematics. Our software, called MATHia, provides a continuous formative assessment of student abilities and builds a cognitive model of student knowledge giving each student a personalized experience. Our text materials encourage active engagement and deep understanding of mathematics. The classroom model inspires students to learn with and from each other, and provides enough flexibility to accommodate many different learning environments. Carnegie Learning provides a full complement of teacher professional development to ensure that implementations are successful.

In developing the Carnegie Learning Math Series (CLMS), we have revisited and expanded our approach to instruction based on a review of the research about cognition and motivation. This paper describes the major research contributions to the series.

MOTIVATING STUDENTS TO LEARN

In the United States, students often lose interest in academic achievement in middle school (Wang and Pomerantz, 2009; Lepper, Sethi, Dialdin and Drake, 1997). In developing CLMS, we have focused on motivation research that helps explain this decrease in interest so that we can maintain student engagement and achievement. Our view of the research is that middle school students tend to view the school environment as out of their control and unresponsive to their interests. As a result, they set unproductive learning goals, which focus on satisfying others (e.g., teachers, guardians, peers) or avoiding failure, rather than on gaining personal satisfaction from academic achievement. We counter this lack of control by allowing students to follow their interests and personalize their environment. We focus on emphasizing the internal rewards inherent to succeeding in a difficult task that leads to improved academic performance.

The Influence of Beliefs, Goals, and Expectations

Recent research on academic achievement shows that students’ beliefs about the nature of intelligence, their goals within a learning task, and their perception of expectations about them have strong effects on their academic performance. This research has four related foci:

• Theory of intelligence: Students who believe that intelligence can increase with effort perform better academically.

• Learning orientation: Some students approach a task with the immediate desire to succeed, while others focus on improving their abilities for the long term, with less emphasis on immediate performance. Similarly, some students

We counter this lack of control by allowing students to follow their interests and personalize their environment. We focus on emphasizing the internal rewards inherent to succeeding in a difficult task that leads to improved academic performance.



strive to excel, while others focus on avoiding failure. These orientations towards learning tasks strongly affect outcomes.

• Stereotype threat: Girls and minorities are particularly susceptible to “stereotype threat,” a phenomenon where students believe that others have low expectations for them as a result of their group status (e.g., girls, minorities). The effect is to put more pressure on these students, resulting in lower performance.

• Intrinsic vs. extrinsic motivation: People are motivated both by internal rewards (because the task makes them feel good) and by external rewards (such as money or recognition by others). While both types of motivation are important, in the long term, students who are intrinsically motivated to learn tend to work harder and achieve at higher levels.

These different influences on learning are strongly related. Students who believe that their intelligence is fixed usually do not approach tasks with a goal of long-term learning; they focus on immediate performance. This orientation makes them less likely to accept a difficult challenge, because they focus on the possibility of failure, rather than the potential learning benefits of struggling with a difficult task. Students in stereotyped groups often believe that their success is determined by external and unchangeable factors, like their race or gender, so interventions that emphasize their ability to learn tend to be especially effective with them.

TheoriesofIntelligence

Dweck (1999) describes two theories that students might have about intelligence. Some students have an “entity” theory of intelligence, a belief that intelligence is a relatively fixed trait. Other students have an “incremental” view of intelligence, a belief that, with effort, they can become smarter. Students with an “entity” theory of intelligence tend to try to place themselves in situations where they will achieve success. They view easy tasks as an opportunity to show off their (fixed) intelligence, and difficult tasks as a risk that they may appear stupid. Students with an incremental view of intelligence are much more open to academic challenge, because they accept that effort can lead to long-term learning, even if it results in immediate failure.

Perhaps the most surprising thing about the research on theories of intelligence is that students can change their theories, and that these changes can lead to improved academic performance. Blackwell, Trzesniewski and Dweck (2007) focused on the long-term results of theories of intelligence on academic performance. They looked at 12-year-olds’ math performance over two years. Students who initially believed that their intelligence could change performed significantly better in mathematics two years later than those who initially believed that intelligence was a fixed trait, even though the two groups of students started with similar math achievement scores. A subgroup of students who were taught about the expandability of intelligence showed higher math scores two years later than students who did not receive such training.

6



LearningOrientation

Some research (e.g., Elliott, 1999; Elliott & McGregor, 2001) has emphasized how students’ goals in a learning task affect their performance. Goals can be described along two dimensions: orientation (emphasizing either immediate performance or long-term learning) and valence (either a focus on doing well or a focus on avoiding failure). This leads to four orientations, as shown in Table 1.

ORIENTATION

LEARNING PERFORMANCE

Valence

ApproachI’m using this at-bat to work on hitting to the opposite field

I want to hit a home run

AvoidanceI need to keep playing so my skills don’t fade

I just don’t want to strike out

Table1:Illustrationoflearningorientationandvalence,asappliedtobaseball

The research clearly shows that a “performance avoidance” attitude results in poorer performance. That is, students who are focused on avoiding immediate failure are more likely to fail. A particular problem with the “performance avoidance” orientation is that it is self-reinforcing. If students are focused on failure and, because of that focus, experience failure, it reinforces their anxiety. This anxiety then affects their performance in their next task, which, again, sets them up for failure.

There are mixed results about the relative effects of learning vs. performance, within the “approach” valence. In some cases, a performance approach can give students a clear focus, which results in better immediate performance. However, a learning approach orientation can help students focus on the big picture, which can lead to bigger gains in the long term. These students, who are self-directed in their learning, seem to learn more flexibly and with more understanding (e.g., Grolnick & Ryan, 1987). Some researchers believe that a “performance approach” orientation can become a “performance avoidance” orientation in high challenge situations, so they advocate emphasizing learning orientation (e.g., Grant & Dweck, 2003).

StereotypeThreat

Much recent research has shown that student performance is strongly influenced by stereotypes that result in low expectations for students (Schmader, 2010). This work has helped to describe the connection between social and cognitive factors involved in performance. Schmader and Johns (2003), for example, showed that students who are reminded about stereotypes have lower working memory capacity. Students act as if their fear of confirming the negative stereotype saps enough of their mental energy that they are less able to perform the main task. Some researchers propose that stereotype threat comprises a large component of group differences found on standardized tests (Walton & Spencer, 2009).



Good, Aaronson and Inzlicht (2003) demonstrated how stereotype threat can be countered by focusing on students’ theories of intelligence. They developed a program in which high school students acted as mentors for seventh graders. In addition to talking to the seventh graders about school adjustment issues, the mentors worked with students to create web sites. Some of the students created web sites that focused on how the brain forms new connections when people learn, along with other scientific information about how the brain changes with learning. Other students created web sites directly focused on helping students understand that middle school is a transition period and that it is normal to struggle at first. This view can encourage students to attribute their failures to external rather than internal factors and so may be particularly beneficial to students who believe that success in school is out of their control. A third group of students addressed both messages, and a fourth group created a web site with an anti-drug theme. This last condition was considered a control condition, since it did not directly focus on academic achievement, but it allowed students to spend an equal amount of time with their mentors.

Although none of the mentoring conditions focused on mathematics, the researchers looked at results on the Texas state exam (TAAS). Students in all three experimental conditions outperformed those in the control condition. This effect was particularly profound for girls. Although boys outperformed girls in the control condition, there were no significant gender differences in any of the other conditions. Boys in the “incremental intelligence” condition outperformed boys in the control condition. This research demonstrates that engaging students with the idea that their brain changes as they learn can actually have effects on seemingly unrelated academic tests.

The relationship between stereotype threat and theories of intelligence is most apparent in studies that look at how student performance on high stakes exams is affected by the way that the exams are described. Steele and Aronson (1995) found that African-American students performed worse on an exam if they were told that the exam measured their inherent abilities than if they were told that the exam was simply a measure of problem solving. Good, Aronson & Harder (2008) found that women who were told “this mathematics test has not shown any gender differences in performance or mathematics ability” before taking a calculus exam performed better than men on the exam, despite the fact that the exam was framed as a test of ability (and thus would otherwise encourage stereotype threat). Women who were not given that statement did no better than men.

Summarizing the research, Halpern et al. (2007) provide strong recommendations for encouraging girls in math and science. These recommendations include emphasizing that academic abilities are expandable, providing students with clear feedback and including activities that emphasize students’ interests and career goals.

IntrinsicMotivationandPersonalization

The discussion about learning orientation relates to the form of motivation. In general, people can be motivated by extrinsic factors, like getting paid or receiving praise from others or from intrinsic factors, like being satisfied or enjoying the activity itself (Deci & Ryan, 1985). Intrinsically motivated students are more likely to focus on the internal rewards associated with learning new material, and thus adapt a learning orientation

8



to the task. (Lepper, 1988). Intrinsic motivation also leads to persistence in the face of failure.

While extrinsic motivators and rewards can have strong positive effects, we tend to focus on using such rewards as a path to building students’ intrinsic motivation. We want students to get to the point where they find solving a complex math problem to be a rewarding experience in itself. One of the ways to build intrinsic motivation is to relate tasks to students’ existing interests.

Schraw and Lehman (2001) emphasize the difference between personal and situational interest. Personal interests are ones that reflect students’ long-term goals and experiences. Situational interest has to do with immediate reactions and tend to be more transitory. It is believed that situational interest, if maintained over time, can develop into personal interest, which is at the heart of long-term intrinsic motivation (Hidi and Renninger, 2006).

Several researchers have found that personalizing problems for students leads to strong improvements in performance in mathematics. Such personalization has included customizing word problems to include the student’s name (or the student’s friends’ names) and areas of interest (Anand and Ross, 1987; Cordova and Lepper, 1996; Ku and Sullivan, 2002). Baker et al. (2009) have shown that problems within the Cognitive Tutor that capture student interests are more likely to be taken seriously, and Walkington (in preparation) conducted a study looking at the influence of taking account of student preferences in Cognitive Tutor word problems.

Motivation in the Carnegie Learning Mathematics Series

As summarized above, the research literature on academic motivation is extensive, and it can seem fairly complex. However, at a high level, the research literature confirms many teachers’ and parents’ experiences with middle school students. Middle school students feel alienated, and this alienation can be particularly pronounced in school. Middle school students feel that their schoolwork is not relevant to their lives, that they have little choice in what they choose to study, and that classroom-provided materials and tasks have not been developed with them in mind. They focus on short-term performance goals because they fail to see the relevance of schoolwork to their long-term success.

Middle school students often take statements of criticism or concern to be about them personally, rather than about their performance or behavior in a particular situation. One summary of the “stereotype threat” literature is that stereotype threat results from a feeling that students are being put in a situation where they are being evaluated for who they are, and that the results are stacked against them. To this extent, students who believe that academic performance reflects their inherent and unchangeable intelligence will be more likely to emphasize avoidance of failure and to avoid challenges at which they may not succeed.

Throughout CLMS, we have included elements that guide students towards appropriate and effective attitudes towards learning. Table 2 summarizes how various features of CLMS reflect research on academic motivation.

Several researchers have found that personalizing problems for students leads to strong improvements in performance in mathematics. Such personalization has included customizing word problems to include the student’s name (or the student’s friends’ names) and areas of interest

(Anand and Ross, 1987; Cordova and Lepper, 1996; Ku and Sullivan, 2002)



Table 2: Academic Motivation and CLMS Features

MOTIVATIONAL GOAL CLMS FEATURES

Emphasize Malleability of Intelligence • Messages of the day include facts about how the brain changes and grows as students learn.

• Messages of praise emphasize effort and learning, not inherent ability.

Reduce Stereotype Threat • Messages of the day present role models and other statements that counter negative stereotypes that students may possess.

Emphasize Learning Orientation • Feedback is private to each student, providing a safe environment where failure is seen as an opportunity to learn, rather than a judgment of inherent ability.

• Problem content focuses students on long-term academic and career goals.

• Badges emphasize processes and behaviors that lead to learning, in addition to some rewards that focus on immediate performance.

Increase Intrinsic Motivation • Clear feedback about errors helps students persist in and succeed in solving complex problems.

• Cognitive Tutor tasks closely monitor students’ level of knowledge and present challenges at the appropriate level for each student.

Combat Academic Alienation • Humorous problems maintain student interest and speak to students in their own voice.

• Student names and friends’ names are used within some word problems.

• Word problems are chosen based on students’ interests.

Provide Students with Autonomy and Control over their Learning

• Students are given choices of activities and some control over their learning path.

• Students can choose a level of challenge in fluency tasks.

• Students can choose characters, colors, patterns and other aspects of the software environment.

10



ACT-R AND HOW PEOPLE LEARN

John Anderson, one of Carnegie Learning’s founders, has devoted his career to building a complete model of human cognition and performance. This model, called ACT-R (Adaptive Control of Thought – Rational: Anderson, 1990, 1993; Anderson & Lebiere, 1998; Anderson, Bothell, Byrne, Douglass, Lebiere & Qin, 2004), is now recognized as the most comprehensive description of how the mind works, and has been the basis of thousands of publications (see http://act-r.psy.cmu.edu/publications/index.php). The ACT-R theory is regularly used to model and predict important characteristics of human behavior, including error patterns and response times in studies of a variety of cognitive tasks. The Cognitive Tutors represent an effort to apply this knowledge of how people learn to mathematics (Ritter et al., 2007).

A full explanation of ACT-R is beyond the scope of this paper, but some of the tenets important to education (Anderson, 2002; Koedinger, Corbett & Perfetti, 2010) include:

• There are two basic types of knowledge: procedural and declarative. Declarative knowledge includes facts, images, and sounds. Procedural knowledge is an understanding of how to do things. All tasks involve a combination of the two types of knowledge.

• As students learn, they generally start out with declarative knowledge, which becomes proceduralized through practice. Procedural knowledge tends to be more fluent and automatic. Declarative knowledge tends to be more flexible and usable in a wider range of contexts.

• The knowledge required to accomplish complex tasks can be described as the set of declarative and procedural knowledge components relevant to the task.

• Knowledge becomes strengthened with use. Strong knowledge can be remembered and called to attention rapidly and with some certainty. Weak knowledge may be slow, effortful, or impossible to retrieve. Different knowledge components may represent different strategies or methods for accomplishing a task (including incorrect strategies or methods). The relative strength of these components helps determine which strategy is used.

• Learning involves the development and strengthening of correct, efficient, and appropriate knowledge components.

• There are strong limits on students’ ability to reason. These limits are referred to as “working memory capacity.” As knowledge becomes more proceduralized, it takes up less working memory.

It is important to understand that the terminology used here differs somewhat from the same terms as they are often used in an educational context. For example, a “procedure” in ACT-R is simply a component of knowledge that can produce other knowledge components and/or lead to external behavior. On the other hand, mathematics educators might refer to the “procedure” of solving a linear equation. An ACT-R model of that task would consist of many procedures and facts. Even a simple task like adding integers may consist of many knowledge components, including ones

The Franklin Institute presented the 2011 Benjamin Franklin Medal in Computer and Cognitive Science to Carnegie Learning co-founder John R. Anderson, Ph.D., for the development of the first large-scale theory of the process by which humans perceive, learn and reason, and its application to computer tutoring systems.



associated with recalling arithmetic facts, executing counting actions, etc. (c.f. Lebiere, 1999).

ACT-R views learning as a process of encoding, strengthening and proceduralizing knowledge. This process happens gradually. New knowledge will be forgotten (or remain weak enough to stay unused) if it is not practiced, and elements of knowledge compete to be used, based on their strength (Siegler & Shipley, 1995). Because the ability to perform a task relies on the individual knowledge components required for that task, education is most efficient when it focuses students most directly on the individual knowledge components that are weakest in their brains.

The interaction between declarative and procedural knowledge leads to an emphasis on active engagement with the conceptual underpinnings of procedures, so that students appropriately generalize this knowledge.

Instructional Components

CognitiveTutorTasks

Cognitive Tutor tasks emphasize active engagement with the conceptual basis of mathematical procedures. This follows from ACT-R’s description of how declarative and procedural knowledge interact. If procedures are formed based on shallow conceptual understanding, the procedures may be fragile and students will not be able to generalize their knowledge to new situations. Forming general procedures requires executing those procedures in a way that emphasizes conceptual knowledge (Rittle-Johnson & Siegler, 1998; Rittle-Johnson, Siegler & Alibali, 2001; Rittle-Johnson & Koedinger, 2002; 2005).

When multiple strategies are available for solving a particular problem, the Cognitive Tutor software uses a process called “model tracing” to identify the particular strategy that a student is pursuing. Complex tasks consist of simple knowledge components, which the Cognitive Tutor displays in the “skillometer.” The position of the skill dial represents the system’s estimate of the student’s knowledge of each knowledge component. The process of picking tasks so that they address appropriate knowledge components (or skills) for each student is called “knowledge tracing.”

Procedural knowledge can be thought of as a collection of possible actions that can be taken at any given time. Choosing the best one of the possible actions to take depends on context. For this reason, educational activities need to be structured so that students can practice procedures within an appropriate range of contexts. Cognitive Tutor tasks represent these different contexts with different skills, resulting in a system that ensures that students encounter problems in all appropriate contexts.

Cognitive Tutor tasks are structured to minimize the amount of working memory required to be devoted to elements of the task that are not the target of learning. In part, this is done through the design of the tasks themselves. Supports provided during problem solving within the Cognitive Tutor software act to reduce the amount of working memory required. Such supports include the use of immediate feedback, directed hints, and worked examples (Sweller and Cooper, 1985).

ACT-R views learning as a process

of encoding, strengthening and

proceduralizing knowledge.

12



The combined focus on design, mastery learning, and a detailed analysis of the components of student knowledge results in tasks that dramatically improve student learning, that are relative to traditional approaches (Corbett, 2001). Several high-quality field studies attest to the effectiveness of the approach (Ritter et al., 2007).

Fluency

The National Math Panel, the National Council of Teachers of Mathematics’ (NCTM) Focal Points, and the Common Core State Standards all emphasize the importance of fluency as well as conceptual understanding. ACT-R provides us with a way to think about how we build fluency, and how fluency and conceptual understanding are related (Anderson & Schunn, 2000). Fluency results from three related mental processes:

• For declarative information, repeated retrieval strengthens facts and makes them easier to retrieve.

• For procedural information, practice strengthens procedures and makes them more likely to be executed and less error-prone.

• Through repeated practice, declarative memory is proceduralized (translated into procedural memory), which allows tasks to be executed more quickly and with less conscious effort.

A real-world example may help to understand these processes. When a person first learns to drive, full concentration is required. New drivers cannot talk while they are driving, and it is not a good idea for them to even turn on the radio. This is because new drivers have not fully proceduralized the process of driving. They need to explicitly recall basic declarative information like the fact that the gas pedal is on the right and the brake is on the left, and consciously remember to check the rear view mirror when changing lanes. As they practice driving, these actions become proceduralized and require less mental effort. It almost seems as the driver’s foot “knows” to go to the left to push the brake pedal, even though they are not consciously aware of thinking about the location of the pedal. This is a hallmark of proceduralization. Since procedures require less working memory, drivers may feel more comfortable talking or listening to the radio while they drive.

Proceduralization and practice have always been part of the basis for activities within the Cognitive Tutor. When Carnegie Learning developed the MATHia software in the CLMS, we focused on two additional functions of fluency:

• Fluency reduces mental effort:

To the extent that basic mathematics facts and procedures are well practiced, they will take less “working memory,” which allows students to focus on higher level problem solving. Haverty (1999) showed that practice with multiplication facts enabled students to better induce mathematical functions. Arroyo et al. (2010) showed that practice with basic mathematics facts resulted in better performance on both skill recall and on problem solving.

The combined focus on design, mastery learning, and a detailed analysis of the components of student knowledge results in tasks that dramatically improve student learning, that are relative to traditional approaches (Corbett, 2001).



• The requirement for fluent performance emphasizes alternative procedures:

Much of what we think of as “number sense” is the rapid choice and application of one of a set of procedures for thinking about mathematical relationships. For example, consider the question:

Which is greater, 1⁄4 or 1⁄17?

Many students, if asked such a question without any time pressure, will use a mathematically valid but slow strategy: convert to a common denominator. However, if students think of 1⁄4 as taking one piece of a thing divided into 4 equal pieces and of 1⁄17 as taking one piece of 17, then the answer can be seen very quickly. Thinking about the meaning of the fractions leads to more rapid response. Siegler (1988) showed that different students use different strategies, and that they also make different choices about whether they are satisfied with a slow but correct strategy. Siegler and Lemaire (1997) showed that the imposition of time pressure can force students to consider alternative strategies. Educators often think of fluency as emphasizing rote memorization. However, in a task like this, the imposition of time pressure actually forces students to use conceptual knowledge more than they might otherwise.

This is the essence of “number sense” (Tsao, 2004). Number sense is the ability to rapidly recognize mathematical patterns and relationships and to flexibly apply appropriate strategies to solve problems (Rittle-Johnson & Star, 2007). This aspect of fluency is often termed “procedural flexibility” (Baroody &Dowker, 2003; Star&Seifert, 2006; Star & Rittle-Johnson, 2009).

Taken together, recall from memory is one of a number of strategies that students might use in solving problems. Recall can be very rapid, but it is also limited in its applicability. When we developed fluency tasks for CLMS, we started by reviewing the research on the strategies that students apply in various tasks. For example, consider the problems:




14



Answering each requires applying conceptual knowledge of fractions in slightly different ways. Several researchers have identified a set of strategies that students use in comparing fractions and conditions under which each strategy may be most efficient (Behr et al., 1984a; Behr et al., 1984b, Clarke & Roche, 2009). For example, if fractions have common numerators, then understanding that larger denominators indicate smaller parts can lead to more rapid response than identifying common denominators. When the numerator is one less than the denominator, a “complement” strategy can be used: 16⁄17 is larger than 3⁄4 because 16⁄17 is only one small piece less than a whole, while 3⁄4 is a larger piece less than a whole. Our designs for fluency tasks help students learn such strategies, recognize when they are appropriate, and then rapidly execute the comparison.

One consequence of the proceduralization of knowledge and development of fluency is that cognition becomes perception. Chase and Simon (1973) provided a model describing how chess masters can rapidly perceive complex patterns in chessboards. This is an early example of how fluent perception aids problem solving. Proficiency in mathematics similarly depends on the ability to rapidly perceive the deep structure of mathematical expressions (Kellman et al., 2008; Kellman, Massey & Son, 2009). Fluency with multiples is crucial to students’ ability to see, for example, 3⁄9 as an unreduced fraction. Similarly, to be competent in mathematics, students need to rapidly perceive x+3⁄5 as a division of one term by another. Learning to “read” mathematics fluently is similar, in many ways, to learning how to read English fluently. The fluency tasks in CLMS are designed to ensure that students understand mathematical syntax and relations fluently enough to derive meaning from mathematical expressions without imposing much working memory load.

TextandClassroomActivities

A good summary of the research on educational materials can be found in Pashler et al. (2007). These recommendations reference much of the same research summarized above and build on our understanding of learning provided by ACT-R and other models. Our classroom activities emphasize active learning and making sense of the mathematics, and we ask deep questions that require students to thoroughly understand the material. The text includes many worked examples, to ensure that students’ initial exposure to materials imposes a relatively small working memory load.

ActiveLearning

There is substantial research demonstrating that active student engagement in the classroom leads to better educational outcomes. For example, Coleman, Brown and Rivkin (1997) tested how various task parameters affected learning. Students who were asked to explain scientific content learned more deeply than those who were asked to simply summarize it. In addition, students who were given the goal of explaining the content to another student performed better than students who were asked to study with the goal of explaining the content to an expert.

Chi (2009) synthesized these and other results into a model called ICAP: Interactive, Constructive, Active, Passive. A large number of studies show that, as instruction moves up the scale from passive to interactive, learning becomes more robust. Interactive

The fluency tasks in CLMS are designed to ensure that students understand mathematical syntax and relations fluently enough to derive meaning from mathematical expressions without imposing much working memory load.



instruction requires students to communicate and come to a shared understanding of the content. Constructive instruction does not require interaction, but it does require students to produce work products that go beyond the inputs that they are given. Active instruction involves producing work products, but such tasks may not require students to go beyond the content given. For example, producing a summary of a chapter would be considered active, but producing a synthesis or explanation of material in the chapter would be constructive.

Chi’s work provides a crucial balance between the cognitive psychology of learning and the demands of classroom instruction. Studies show that students can learn from passive instruction (such as reading to themselves), but only if they are mentally engaged with the content. One particularly effective form of such engagement is self-explanation (Roy & Chi, 2005). The demands of the classroom, however, dictate that students produce work products (a constructive instructional approach) to ensure both that students are actively processing the material, and that the teacher understands the student’s level of knowledge.

Consistent with Chi’s recommendations for classroom instruction, activities in the Cognitive Tutor text are designed to support implementation at any of the four levels. Practical considerations such as timing and school environment dictate that not all instruction is interactive, but we aim to provide both activities and support to help teachers become comfortable as they move their instruction to more effective models. The format of the student text, as a consumable workbook, encourages active instruction.

StraighttotheMath

Our instructional materials have little extraneous material, and we do not use illustrations, unless they are essential to helping students understand the material. This approach follows from research showing that “seductive details” used to spice up the presentation of material often have a negative effect on student learning (Mayer et al., 2001; Harp & Meyer, 1998). Students may not know which elements of an instructional presentation are essential and which are intended simply to provide visual interest, so it is often best to focus on the essential material. While we strive to make our educational materials attractive and engaging to students, research shows that only engagement based on the mathematical content lead to learning.

WorkedExamplesandSelf-explanation

In all aspects of CLMS, we provide worked examples. Sweller and Cooper (1985) argue that worked examples are educationally efficient because they reduce working memory load. Ward and Sweller (1990) found that alternating between problem solving and viewing worked examples led to the best learning.

Subsequent research effort has been devoted to understanding the conditions under which students benefit the most from worked examples. Students often read worked examples with the intent to confirm that they understand the individual steps. However, the educational value of the worked example often lies in thinking about how the steps connect to each other and how particular steps might be added, omitted or changed, depending on context. Chi et al. (1994) found that prompting students to self-explain

Our classroom activities emphasize active learning and

making sense of the mathematics, and we ask deep

questions that require students to thoroughly

understand the material. The text

includes many worked examples, to ensure that students’ initial

exposure to materials imposes a relatively

small working memory load.

16



while reading examples led to better learning. Hausmann and VanLehn (2007) found that embedding self-explanation prompts within learning materials can encourage students to more thoughtfully self-explain. Based on this research, we emphasize self-explanation and making sense of examples throughout our text materials.

One concern with self-explanation is that students may erroneously generalize from an example. There are often multiple explanations for how one step might follow from another, so how are we to correct misconceptions that students may have? Several recent studies have found that asking students to find errors in erroneous examples or asking students to determine which of several solution methods is correct can lead to better generalizations (Durkin & Rittle-Johnson, 2009; Rittle-Johnson & Star, 2007; Siegler, 2002; Booth, Koedinger & Siegler, 2007). Following from this research, some exercises in our text ask students to identify errors in incorrect solutions or to identify which of two solution methods is correct.

Research on reading for understanding provides us with guidelines for developing text that promotes self-explanation. Simply embedding prompts within text that remind students to stop and think can promote productive reflection and result in deeper learning (Davis, 2003; McNamara, 2004). We also use graphic organizers within the text, and ask students to summarize their knowledge.

Collaboration

Collaborative problem solving encourages an interactive instructional mode, and we have looked to research to provide us with practical guidance for making collaboration work. Chi et al. (2008) show that, in some cases, simply allowing students to work collaboratively can encourage sufficient depth of collaboration. However, structured collaboration, including scripted dialogs (Rummel, & Spada, 2005; Weinberger, Ertl, Fischer, & Mandl, 2005) and routine prompts (Soller, Linton, Goodman, & Lesgold, 1999) are often recommended to ensure that collaborators encourage each other to engage deeply with the content.

The collaborative tasks within the CLMS classroom are designed to encourage active dialog, centered on structured activities. One particularly effective structure for collaboration is “think-pair-share” (Lyman, 1981), in which students think about the goals and initial approach to solving the problem, pair up for interactive problem-solving and then share their results with the rest of the class.

Open-endedProblemSolving

A fundamental concern in instruction involves when to provide assistance to students and when to withhold such assistance, and allow students to struggle (Koedinger & Aleven, 2007). There is strong evidence that withholding instruction can be counterproductive in many cases (Kirschner, Sweller & Clark, 2006). On the other hand, there is also evidence that withholding scaffolding and allowing students to self-test (even without feedback) can be beneficial to learning (Roediger & Karpicke, 2006). Cognitive Tutor exercises are tests, in this sense, because they ask students to demonstrate their knowledge. In general, cases where allowing students to struggle produces more robust and transferrable learning have been referred to as “desirable difficulties” (Bjork, 1994).

The collaborative tasks within the CLMS classroom are designed to encourage active dialog, centered on structured activities



The resolution to the question of when, and when not to guide students, follows from our understanding of cognitive psychology. New learning requires minimum extraneous working memory load, so assisting students makes sense. On the other hand, the deep thought and wide range of experiences that lead to robust learning and generalization often require students to struggle with material. In many classroom activities, we encourage students to build this kind of robust and general knowledge. While we design these activities to encourage students to explore problems deeply, we incorporate peer, teacher or technology-based support to ensure that students do not spend too much time in unproductive exploration. The right amount of scaffolding within these complex challenges can help students adopt a learning orientation to the task.

Conclusion

There is a large body of research on mathematics learning, and this paper only touches on some of the most important strands that we have considered in developing CLMS. Carnegie Learning’s commitment is to doing what works, rather than sticking to a narrow ideology. Part of that commitment is continually testing and refining our products in order to improve their effectiveness. Carnegie Learning is widely recognized for our extensive field testing and for using data to improve instruction (Ritter et al., 2009). As more teachers and students gain experience with CLMS, we will collect more data and discover new ways to improve the product. If you would like to participate in some of our research, please contact us.

ABOUT THE AUTHOR

Steve Ritter, Co-Founder and Chief Scientist at Carnegie Learning, has been developing and evaluating educational systems for over 10 years. He earned his Ph.D. in Cognitive Psychology at Carnegie Mellon University in 1992 and helped to found Carnegie Learning in 1998. As a postdoctoral associate and research scientist at Carnegie Mellon, Dr. Ritter was instrumental in the development and evaluation of the Cognitive Tutors for mathematics. He is the author

of numerous papers on the design, architecture and evaluation of Intelligent Tutoring Systems and served as chairman of the IEEE Learning Technology Standards Committee working group on tool/agent communication.

While we design these activities

to encourage students to explore

problems deeply, we incorporate

peer, teacher or technology-based support to ensure

that students do not spend too much time in unproductive

exploration.

18



REFERENCES

Anand, P.G., & Ross, S.M. (1987). Using computer-assisted instruction to personalize arithmetic materials for elementary school children. Journal of Educational Psychology, 79(1), 72–78.

Anderson, J.R. (1990). The Adaptive Character of Thought. Hillsdale, NJ: Erlbaum.

Anderson, J. R. (1993). Rules of the Mind. Hillsdale, N. J.: Erlbaum.

Anderson, J. R. (2002). Spanning seven orders of magnitude: A challenge for cognitive modeling. Cognitive Science, 26, 85-112.

Anderson, J. R., Bothell, D., Byrne, M. D., Douglass, S., Lebiere, C., & Qin, Y . (2004). An integrated theory of the mind. Psychological Review, 111, (4). 1036-1060.

Anderson, J. R. & Lebière, C. (1998). The atomic components of thought. Mahwah, NJ: Erlbaum.

Anderson, J. R., & Schunn, C.D. (2000). Implications of the ACT-R learning theory: No magic bullets. In R. Glaser (Ed.), Advances in instructional psychology (Vol. 5). Mahwah, NJ: Erlbaum.

Arroyo, I., Woolf, B. P., Royer, J. M., Tai, M. & English, S. (2010). Improving Math Learning through Intelligent Tutoring and Basic Skills Training. In Aleven, Kay and Mostow (Eds.)Intelligent Tutoring Systems 2010, Part I, Lecture Notes in Computer Science 6094, pp. 423–432.

Baker, R.S.J.d., de Carvalho, A.M.J.A., Raspat, J., Aleven, V., Corbett, A.T., Koedinger, K.R. (2009) Educational Software Features that Encourage and Discourage “Gaming the System”. Proceedings of the 14th International Conference on Artificial Intelligence in Education, 475-482.

Baroody, A. J., & Dowker, A. (Eds.). (2003). The development of arithmetic concepts and skills: Constructing adaptive expertise. Mahwah, NJ: Lawrence Erlbaum.

Behr, M., Wachsmuth, I., & Post, T. (1984a). Tasks to Assess Children’s Perception of the Size of a Fraction. In A. Bell, B. Low & J. Kilpatrick (Eds.), Theory, Research and Practice in Mathematical Education (pp. 179-18). Fifth International Congress on Mathematical Education, South Australia: Shell Centre for Mathematics Education.

Behr, M., Wachsmuth, I., Post T., & Lesh R. (1984b). Order and Equivalence of Rational Numbers: A Clinical Teaching Experiment. Journal for Research in Mathematics Education, 15(5), 323-341.

Bjork, R.A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe and A. Shimamura (Eds.) Metacognition: Knowing about knowing. (pp.185-205). Cambridge, MA: MIT Press.



Blackwell, L. S., Trzesniewski, K.H. and Dweck, C. S. (2007). Implicit Theories of Intelligence Predict Achievement Across an Adolescent Transition: A Longitudinal Study and an Intervention. Child Development, 78(1), 246-263.

Booth, J.L., Koedinger, K.R., & Siegler, R.S. (2007, October). The effect of corrective and typical self-explanation on algebraic problem solving. Poster presented at the Science of Learning Centers Awardee’s Meeting in Washington, DC.

Chase, W.G. and Simon, H.A. (1973). Perception in chess. Cognitive Psychology 4 (1): 55–81.

Chi, M. T. H. (2009). Active-Constructive-Interactive: A Conceptual Framework for Differentiating Learning Activities.Topics in Cognitive Science, 1, 73–105.

Chi, M. T. H., DeLeeuw, N., Chiu, M.-H., & LaVancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science, 18, 439-477.

Chi, M. T. H., Roy, M., & Hausmann, R. G. M. (2008). Observing tutorial dialogues collaboratively: Insightsabout human tutoring effectiveness from vicarious learning. Cognitive Science, 32, 301–341.

Clarke, D. M. and Roche, A. (2009). Students’ fraction comparison strategies as a window into robust understanding and possible pointers for instruction. Educational Studies in Mathematics, 72(1), 127-138.

Coleman, E. B., Brown, A. L., & Rivkin, I. D. (1997). The effect of instructional explanations on learning from scientific texts. The Journal of the Learning Sciences, 6, 347–365.

Corbett, A.T. (2001). Cognitive computer tutors: Solving the two-sigma problem. User Modeling: Proceedings of the Eighth International Conference, UM 2001, 137-147.

Cordova, D. I. & Lepper, M. R. (1996). Intrinsic Motivation and the Process of Learning: Beneficial Effects of Contextualization, Personalization, and Choice. Journal of Educational Psychology. 88(4) 715-730.

Davis, E. (2003). Prompting middle school science students for productive reflection: generic and directed prompts. The Journal of the Learning Sciences, 12(1), 91–142.

Deci, E. L. and R. M. Ryan (1985). Intrinsic Motivation and Self Determination in Human Behavior. NY, Plenum Press.

Durkin, K., & Rittle-Johnson, B. (2009, June 7). Comparing incorrect and correct examples when learning about decimals and the effects on explanation quality. Poster session presented at the Institute of Education Sciences Research Conference.

Dweck, C. S. (1999). Self-theories: Their role in motivation, personality and development.Philadelphia : The Psychology Press.

Elliot, A. J. (1999). Approach and avoidance motivation and achievement goals. Educational Psychologist, 34, 169–189.

20



Elliot, A. J., & McGregor, H. A. (2001). A 2 X 2 achievement goal framework. Journal of Personality and Social Psychology, 80, 501–519.

Good, C., Aronson, J., & Harder, J. A. (2008). Problems in the pipeline: Stereotype threat and women’s achievement in high-level math courses. Journal of Applied Developmental Psychology, 29, 17-28.

Good, C., Aronson, J., & Inzlicht, M. (2003). Improving Adolescents’ Standardized Test Performance: An Intervention to Reduce the Effects of Stereotype Threat. Journal of Applied Developmental Psychology, 24, 645-662.

Grolnick, W. S. & Ryan, R. M. (1987) Autonomy in children’s learning: An experimental and individual difference investigation. Journal of Personality and Social Psychology, Vol 52(5), 890-898.

Halpern, D., Aronson, J., Reimer, N., Simpkins, S., Star, J., and Wentzel, K. (2007). Encouraging Girls in Math and Science (NCER 2007-2003). Washington, DC: National Center for Education Research, Institute of Education Sciences, U.S. Department of Education.

Harp, S. F. & Meyer, R. E. (1998). How seductive details do their damage: A theory of cognitive interest in science learning. Journal of Educational Psychology, 90, 414-434.

Hausmann, R. G. M., & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (Vol. 158, pp. 417-424). Amsterdam: IOS Press.

Haverty, L. (1999). The importance of basic number knowledge to advanced mathematical problem solving. Unpublished doctoral dissertation. Carnegie Mellon University.

Hidi, S. and Renninger, K. A. (2006). The four-phase model of interest development. Educational Psychologist, 41(2), 111-127.

Kellman, P. J., Massey, C. M., Roth, Z., Burke, T., Zucker, J., Saw, A., Aguero, K., & Wise, J. (2008).Perceptual learning and the technology of expertise: Studies in fraction learning and algebra. Pragmatics & Cognition, 16(2), 356–405.

Kellman, P. J., Massey, C. M. and Son, J. Y. (2010), Perceptual Learning Modules in Mathematics: Enhancing Students’ Pattern Recognition, Structure Extraction, and Fluency. Topics in Cognitive Science, 2: 285–305.

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based experiential and inquiry-based teaching. Educational Psychologist, 41(2), 75-86.



Koedinger, K. R., & Aleven V. (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Educational Psychology Review, 19(3), 239-264.

Koedinger, K. R., Corbett, A. T. and Perfetti, C. (2010). The Knowledge-Learning-Instruction (KLI) Framework: Toward Bridging the Science-Practice Chasm to Enhance Robust Student Learning. Technical Report, Carnegie Mellon University department of Human Computer Interaction, CMU-HCII-10-102, http://www.learnlab.org/documents/KLI-Framework-Tech-Report.pdf

Ku, H and Sullivan, H. J. (2002). Student performance and attitudes using personalized mathematics instruction. Educational Technology Research and Development. 50(1), 21-34.

Lebiere, C. (1999). The dynamics of cognition: An ACT-R model of cognitive arithmetic. Kognitionswissenschaft., 8 (1), pp. 5-19.

Lepper, M. R. (1988). Motivational considerations in the study of instruction. Cognition and Instruction, 5, 289-310.

Lepper, M.R., Sethi, S., Dialdin, D., & Drake, M. (1997). Intrinsic and extrinsic motivation: A developmental perspective. In S. S. Luthar, J. A. Burack, D. Cicchetti & J. R. Weisz (Eds.), Developmental psychology: Perspectives on adjustment, risk, and disorder (pp. 23-50). New York: Cambridge University Press.

Lyman, F. T. (1981). The responsive classroom discussion: The inclusion of all students. In A. Anderson (Ed.), Mainstreaming Digest (pp. 109-113). College Park: University of Maryland Press.

Mayer, R. E., Heiser, J. and Lonn, S. (2001). Cognitive constraints on multimedia learning: When presenting more material results in less understanding. Journal of Educational Psychology, 93, 187-198.

McNamara, D. S. (2004). SERT: Self-explanation reading training. Discourse Processes, 38(1), 1–30.

Pashler, H., Bain, P., Bottge, B., Graesser, A., Koedinger, K., McDaniel, M., and Metcalfe, J. (2007) Organizing Instruction and Study to Improve Student Learning (NCER 2007-2004). Washington, DC: National Center for Education Research, Institute of Education Sciences, U.S. Department of Education.

Ritter, S., Anderson, J.R., Koedinger, K.R., & Corbett, A. (2007) The Cognitive Tutor: Applied research in mathematics education. Psychonomics Bulletin & Review, 14(2), pp. 249-255.

Ritter, S., Kulikowich, J., Lei, P., McGuire, C.L. & Morgan, P. (2007). What evidence matters? A randomized field trial of Cognitive Tutor Algebra I. In T. Hirashima, U. Hoppe & S. S. Young (Eds.), Supporting Learning Flow through Integrative Technologies (Vol. 162, pp. 13-20). Amsterdam: IOS Press.

22



Ritter, S., Harris, T. H., Nixon, T., Dickison, D., Murray, R. C. and Towle, B. (2009). Reducing the knowledge tracing space. In Barnes, T., Desmarais, M., Romero, C., & Ventura, S. (Eds.) Educational Data Mining 2009: 2nd International Conference on Educational Data Mining, ‎Proceedings. Cordoba, Spain

Rittle-Johnson, B. and Koedinger, K. R. (2002). Comparing instructional strategies for integrating conceptual and procedural knowledge. Meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education (Athens, GA). October, 2002.

Rittle-Johnson, B. and Koedinger, K. R. (2005). Designing better learning environments: Knowledge scaffolding supports mathematical problem solving. Cognition and Instruction. 23(3), 313-349.

Rittle-Johnson, B. & Siegler, R.S. (1998). The relation between conceptual and procedural knowledge in learning mathematics: A review. In C. Donlan (Ed.), The development of mathematical skill. (pp. 75-110). Hove, UK: Psychology Press.

Rittle-Johnson, B., Siegler, R.S. & Alibali, M.W. (2001). Developing conceptual understanding and procedural skill in mathematics: An iterative process. Journal of Educational Psychology, 93, 346-362.

Rittle-Johnson, B., & Star, J. R. (2007). Does comparing solution methods facilitate conceptual and procedural knowledge? An experimental studyon learning to solve equations. Journal of Educational Psychology, 99(3).

Roediger, H. L. & Karpicke, J. D. (2006). The power of testing memory: Basic research and implications for educational practice. Perspectives on Psychological Science, 1, 181-210.

Roy, M. & Chi, M.T.H. (2005). Self-explanation in a multi-media context. In R. Mayer (Ed.), Cambridge Handbook of Multimedia Learning (pp. 271-286). Cambridge Press.

Rummel, N., & Spada, H. (2005). Learning to collaborate: An instructional approach to promoting collaborate problem solving in computer-medicated settings. Journal of the Learning Sciences, 14 (2), 201–241.

Schmader, T. (2010). Stereotype threat deconstructed. Current Directions in Psychological Science, 19(4), 14-18.

Schmader, T., & Johns, M. (2003). Converging evidence that stereotype threat reduces working memory capacity. Journal of Personality and Social Psychology, 85, 440-452.

Schraw, G. and Lehman, S. 2001. Situational Interest: A Review of the Literature and Directions for Future Research. Educational Psychology Review.

Siegler, R. S. (1999). Individual differences in strategy choices: Good students, not-so-good students and perfectionists. Child Development, 59, 833-851.



Siegler, R. S. (2002). Microgenetic studies of self-explanations. In N. Granott & J. Parziale (Eds.), Microdevelopment: Transition processes in development and learning (pp. 31-58). New York: Cambridge University.

Siegler, R. S. and Lemaire, P. (1997). Older and younger adults’ strategy choices in multiplication: Testing predictions of ASCM using the choice/no-choice method. Journal of Experimental Psychology: General, 126(1), 71-92.

Siegler, R. S., and Shipley, C. (1995). Variation, selection, and cognitive change. In T. Simon & G. Halford (Eds.), Developing cognitive competence: New approaches to process modeling (pp. 31-76). Hillsdale, NJ: Erlbaum.

Soller, A., Linton, F., Goodman, B., & Lesgold, A. (1999). Toward intelligent analysis and support of collabarative learning interaction. Proceeding of the Ninth International Conference on Artificial Intelligence and Education (pp. 75–82). France: Le Mans.

Star, J. R. &Rittle-Johnson, B. (2009). It pays to compare: An experimental study on computational estimation. Journal of Experimental Child Psychology, 101, 408-426

Star, J. R., & Seifert, C. (2006). The development of flexibility in equation solving. Contemporary Educational Psychology, 31(280-300).

Steele, C.M., & Aronson, J. (1995). Stereotype Threat and the intellectual test-performance of African-Americans. Journal of personality and Social Psychology, 69 (5): 797-811.

Sweller, J. & Cooper, G.A. (1985) The use of worked examples as a substitute for problem solving in learning algebra, Cognition and Instruction 2: 59–89.

Tsao, Y. (2004). Exploring The Connections Among Number Sense, Mental Computation Performance, And The Written Computation Performance Of Elementary Preservice School Teachers. Journal of College Teaching & Learning, 1:12, 71-90.

Walkington, C. (in preparation). Robust Learning in Culturally and Personally Relevant Algebra Problem Scenarios. Unpublished doctoral dissertation. University of Texas, Austin.

Walton, G. M., & Spencer, S. J. (2009). Latent ability: Grades and test scores systematically underestimate the intellectual ability of negatively stereotyped students. Psychological Science, 20, 1132–1139.

Wang, Q & Pomerantz, E. M. The Motivational Landscape of Early Adolescence in the United States and China: A Longitudinal Investigation. Child Development, 80(4).

Ward, M., & Sweller, J. (1990). Structuring effective worked examples. Cognition and Instruction, 7, 1-39.

Weinberger, A., Ertl, B., Fischer, F., & Mandl, H. (2005). Epistemic and social scripts in computer-supportedcollaborative learning. Instructional Science, 33, 1–30.

Frick Building20th Floor437 Grant StreetPittsburgh, PA 15219

Phone (Toll Free): 888.851.7094Local: 412.690.6284Fax: [email protected]

the research behind the carnegie learning math series

Documents