instructional sensitivity and transfer of learning at · pdf fileinstructional sensitivity and...

Li et al. Close, Proximal, and Distal Items

1

Instructional Sensitivity and Transfer of Learning at Different Distances:

Close, Proximal and Distal Assessment Items

Min Li

1, Maria Araceli Ruiz-Primo

2, Michael Giamellaro

2, Kellie Wills

1,

Hillary Mason

2, & Ming-Chih Lan

1

1 University of Washington at Seattle 2 University of Colorado Denver

Paper Presented Paper Set at the AERA Annual Meeting Vancouver, Canada

April, 2012

The work reported herein was supported by National Science Foundation (DRL-0816123). The findings and opinions expressed in the paper do not reflect the position or policies of the National Science Foundation, but the authors.


2

Instructional Sensitivity and Transfer of Learning at Different Distances:

Close, Proximal and Distal Assessment Items

It is well documented that students perform differentially on assessments if they are more or less sensitive to what the students have experienced through instruction (Airasian & Madaus, 1983; Madaus, Airasian, & Kellaghan, 1980; Ruiz-Primo, Shavelson, Hamilton, & Klein, 2002). This concern has been repeatedly raised by researchers who observed that student scores on large scale tests reflect factors other than the quality of instruction or educational programs implemented (Burstein, Aschbacher, Chen, & Lin, 1990; Commission on Instructionally Supportive Assessment, 2001; Leinhardt, 1983; Popham, 2006, 2007a, 2007b; Wiliam, 2007). Most research done on instructional sensitivity focuses on evaluating assessments already developed and administered as “after the fact,” but is silent on how to construct instructionally sensitive assessments. This paper aims to test the assessment development approach we proposed (Ruiz-Primo & Li, 2008) to systematically manipulate item characteristics to construct items varying in instructional sensitivity. The question that has guided the test of the approach is, what module characteristics can be systematically manipulated to develop items at different distances (close and proximal) that prove to be instructionally sensitive? Both judgmental and empirical evidence is provided to examine the appropriateness of this assessment development approach.

INTRODUCTION

Defining Instructional Sensitivity

In the literature, the term instructional sensitivity has been considered and used interchangeably with another term, instructional validity, both of which largely overlap with a few other commonly used terms, curricular validity and content validity. The intended meaning has included the curriculum content that was taught (McClure as cited by Linn, 1983a; Schmidt et al., 1983) as well as the nature and quality of the teaching of the content (Burstein, 1989; Burstein et al., 1990; Popham, 2007b; Yoon & Resnick, 1998). For example, Yoon and Resnick (1998) defined instructional sensitivity as “the extent to which an assessment is systematically sensitive to the nature of instruction offered” (p. 2). Their definition is aligned with more recent conceptions of instructional sensitivity including the degree to which student performance on an assessment accurately reflects the quality of the instruction provided to promote students’ learning (Popham, 2006, 2007a, b). In other words, an assessment can be claimed to be sensitive to instruction only if instruction changes a student’s score; “If instruction does not change a student’s score on a test very much, then that test is insensitive to instruction” (Wiliam, 2007, p. 5).1 Similarly, the concept of instructional sensitivity can be applied at the assessment item level. Haladyna and Roid (1981) defined the term as the tendency for an item to vary in difficulty as a function of instruction; when instruction is reasonably effective, items should be easier when administered to instructed students and difficult when administered to uninstructed students. 1 Admittedly, if the instructional sensitivity of the assessment has been established, then the lack of change in students’ scores signals poor or misaligned instruction.


3

In the Development and Evaluation of Instructionally Sensitive Assessments (DEISA) project, we adapted and extended the definition of instructional sensitivity as referring to the extent to which the assessments or assessment items: (1) represent the enacted curriculum (the material covered by the test has actually been taught), (2) reflect the quality of the enacted curriculum (the quality of instruction); and (3) have formative value; if items are sensitive to instruction, teachers and/or students must be able to use the assessment information to adjust instruction. Consistent with this definition, we conceptualize the instructional sensitivity as a continuum from more sensitive to less sensitive. Assessments then can appear as immediate, close, proximal, and distal (Ruiz-Primo & Li, 2012). Immediate assessments are artifacts from the enacted curriculum such as notebooks, worksheets, or tasks for group work, which are often the learning tasks planned by teachers that still provide assessment information about student achievement. At a close level, assessments are curriculum-sensitive because they are similar to the content and activities as described in the curriculum and implemented in the classroom. At a proximal level, assessments consider the knowledge and skills relevant to the curriculum, but context (e.g., scenarios) differs from what is studied in the module. At a distal level, assessments are based on state or national standards in a particular domain, and thus are highly probable to be only minimally related to what students learned in the module.

DEISA Approach to Developing Instructionally Sensitive Assessments

As explained in the introductory paper, the DEISA approach to developing instructionally sensitive assessments and evaluating their technical quality is guided by three theoretical underpinnings: transfer of learning, big ideas, and type of knowledge (Ruiz-Primo & Li, 2012). These foundations are used to define the construct, operationalize instructional sensitivity, generate the item prototypes, and derive the interpretative and validity arguments. We have experimented and refined this approach over three tryouts. On each tryout, we improved the approach based on what we found from the previous tryout and applied it to different science curricula to test for robustness. In tryouts 1 and 2 we developed items for the same three science modules (i.e., FOSS-Environments, FOSS-Landforms, and BSCS-Heat and Change) and in the third tryout we included two new science modules (FOSS-Mixtures and Solutions and STC-Land and Water).

The DEISA approach has four critical steps:

1. Mapping. Construct definition and articulation of sources of instructional sensitivity (SOIS). Define the learning goals of the science modules at hand, identify the critical concepts, principles, procedures and explanation models to be assessed to determine whether the critical (original) learning had occurred, and pinpoint the corresponding learning experience described in the curriculum to achieve these learning goals.

2. Big Ideas. Identify the big ideas (principled understanding) from which the DEISA assessment items needed to be developed. These two to three big ideas align with the identified learning goals from the curriculum maps. They are developed after repeated reviews of the module curriculum and then are further delineated into more specific statements that represent levels of understanding within each Big Idea.

3. Item development. Develop the DEISA items by starting with the close items that tap the critical concepts, principles, procedures and explanation models of the science module at hand and then expanding to the proximal items with systematic manipulations of item characteristics.


4

4. Validation. Collect data to test both judgmentally and empirically the claims on the instructional validity of the score interpretations of the DEISA items.

In what follows, we describe in detail the two steps related to assessment development, Step 1 on articulating the sources of item manipulation and Step 3 on item development (for a complete description of all the steps, see Ruiz-Primo & Li, 2012).

Defining the Construct and Articulating the Sources of Instructional Sensitivity. We developed a strategy, called mapping the intended curriculum, to gather information about the critical characteristics of a science unit or module. The purpose of mapping is twofold: (a) to deeply understand the learning goal(s) to be achieved by the teacher and the students at the end of a unit/module; and (b) to understand how these learning goals are to be achieved by the teacher and students.

For the first purpose of understanding the learning goals of the unit, we proposed two activities during the mapping: (1) identify the scientific knowledge and scientific practices students should achieve if the science unit or module is taught as intended by curriculum developers. These learning goals should be the focus of the instructionally sensitive assessment. And (2) classify the scientific knowledge and scientific practices by types of knowledge - declarative, procedural or schematic knowledge. The critical question asked for each lesson of the unit or modules was “What are the critical concepts, procedures, processes, explanations, or principles to which teachers and students need to pay attention?”

The second purpose focuses on identifying the different aspects of the science unit or module that should be considered potential manipulators during the development process of the items. We hypothesized that some characteristics of any science modules could be potential sources of manipulation of instructional sensitivity. In the curriculum mapping, we capture such information in five columns which represent the elements of the original learning context which should be included in the items varying in instructional sensitivity to assess near and far transfer (Thorndike, 1913 as mentioned in Bransford, Brown, & Cocking, 1999).

In sum, the module maps track seven aspects of each lesson within a unit or module (Figure 1): (1) the learning targets for the lesson in terms of scientific knowledge and scientific processes; (2) the type of knowledge being tapped or engaged in through the lesson (i.e., declarative, procedural, and schematic knowledge); (3) the activities that are critical to achieve the learning targets for the lesson; (4) the documentation required of students; (5) the materials used; (6) the graphical representations; and (7) the vocabulary involved in this lesson/activity/investigation. The first two aspects are relevant to defining the construct to be measured. Further, all seven aspects were considered Sources of Instructional Sensitivity (SOIS) that could potentially be manipulated in order to adequately assess the transfer of learning.

Figure 1. Module map template.


5

The first two main columns to be mapped aim to identify the scientific knowledge and practices targeted by each lesson within a module and clarify them into types of knowledge. For these identified scientific knowledge and practices, we also evaluate how explicitly they appear and are connected with the learning scaffolds in the curricular materials. Sometimes the concepts, procedures, and explanations that are essential for teachers and students to pay attention to are clear and explicitly stated in both the teacher guides and the student materials. However, sometimes these elements may be only clearly stated and explained in the teacher guide and student materials or they may be implicitly mentioned a few times in the teacher guide. This information helps to determine how clear and consistent the curriculum materials are in pointing out the critical concepts, procedures, and explanations to teachers and students.

The last five columns to be mapped aim to pinpoint the different aspects of the learning context in which the targeted scientific knowledge and practices are learned, including:

(1) The activities in which teachers and students are engaged to achieve the learning goals. We only include activities that contribute to a learning target in some meaningful way. Activities should be described succinctly, and include information about: (a) the organization of students during the activity; (b) the actual task in which the students are engaged, like discussion, activities, writing in journals, etc.; (c) and the content of that task. Identifying the activities helps us to manipulate the context of the assessment items so that both familiar and unfamiliar contexts can be included in the items.

(2) The characteristics of the forms provided by curriculum developers to document the students’ activity reports and/or responses (e.g., handouts or class poster). Three pieces of information are tracked for this characteristic: (a) the type of documentation (i.e. student sheets, chart paper, journaling, group presentation, etc.); (b) the content of the documentation (the prompts of what students were asked to do); over the three tryouts, we also developed a summary document to list all the exact prompts that the curriculum recommends or suggests that teachers need to use; and (c) some indication of the importance of this documentation, or what it does and how it is linked to the learning target.

(3) The materials used during the implementation of the activities. By critical we mean those materials that are of essential importance or indispensable for students to achieve the learning target. Unlike many lesson planning approaches, the materials we identify should be things that directly affect the students’ learning, not necessarily materials needed because of the specific logistics of the activity.

(4) The graphical representations used in the unit or module. Graphical representation refers to a table, a diagram, a schema, or a drawing used to present the needed information. As an example think about a concept definition such as “the alveoli are tiny air sacs in the lungs.” A graphical representation that accompanies the definition (e.g., imagine a figure with the alveoli in the lungs) provides students a better idea of what the alveoli are. Graphic representations also include the picture of the set-up of a controlled experiment or the investigation data that students need to interpret.

(5) The critical vocabulary used in the unit or module. By critical vocabulary, we refer to those words or phrases that are of essential importance or indispensable for students to know and be aware of (such as, “heat transfer” for Heat and Change, “range of tolerance” for the Environments module, or “erosion” for the Landforms module). Based on what we learned


6

from the teacher and student interviews in Tryout 1, we were aware that teachers might choose different words in their classroom discourse. Therefore when applying the refined approach in Tryouts 2 and 3, we considered the information from this column only as terms that must be used in order to accurately represent the big ideas to be tapped and to successfully implement the curriculum. The terms should then be used in all the assessment items developed when appropriate (e.g., variable, heat transfer, range of tolerance, erosion, ).

Teachers of the modules were invited to map the science module in the summer prior to the year when they would be teaching the module and administering the DEISA assessment. The similar mapping process was carried out by researchers to produce the research version of the curriculum maps. For each tested module, a collective map was created by converging the map developed by the teachers with the researcher map. This collective map helped to guide the discussion about the big ideas. State and national standards were checked for alignment with the connections between the concepts, principles and explanation models targeted by the science modules.

Developing Items: Manipulating Item Proximity. In the project we have mainly focused on multiple-choice items as they are still largely used at the classroom, district, state (e.g., Colorado Student Assessment Program, CSAP), national (e.g., NAEP), and international (e.g., TIMSS, PISA) levels. We believed that the DEISA work was most likely to have a higher impact if this was the focus.

We developed multiple-choice items in what we named “bundles of triads.” Each triad has one Close (C) item which was developed first and two types of proximal items: Proximal 1 (P1 as near proximal) and Proximal 2 (P2 as far proximal). The proximal items were developed by changing the item characteristics of the close item. The intent with the triad was to: (1) establish, with performance on the close item, that learning of the concept, principle, or explanation model took place after instruction, (2) manipulate different contexts with the two types of proximal items in a way that could provide some evidence about how far students were able to transfer. The difference between the three types of items was the magnitude or the extent of the change applied to the items based on the module, often minimal to no changes for Close, small changes for P1 and big changes for P2. Presumably, the triads should ask exactly the same or very similar question to the students, with the appropriate adaptations when needed across P1 and P2.

By applying the information captured in the curriculum map, we focused on five dimensions for the manipulation of the items based on the intended curriculum:

1. Characteristics of the question. On each triad we focused on two aspects: (1) familiarity of the question in relation to the module - how familiar the question (multiple-choice prompt) asked to the student was in relation to the instructional activities experienced in the science module. We asked whether the question asked should or should not be a complete surprise to the student based on what they did in the module as recorded in the column of documentation in the map and the summary document of all the prompts. And (2) difference of the questions asked in P1 and P2 items to the question asked in the C item – how different the questions were to the question asked in the close item (i.e., no change meaning the same question, small change such as some words added or omitted, and big change such as a different question was asked in the triad).


7

2. Exposure of the students to the big idea in the module. In the exposure dimension we focus on the content and activities in the module that could support the achievement of the module big ideas rather than on the amount of instructional time received by the students (more of a fidelity of implementation issue). Two aspects were considered in the exposure: (1) a general sense of the extent of the opportunities (none, few, multiple times) that students had to experience and make the connections expressed in the big ideas. We do this by examining the curriculum maps with respect to how frequently the scientific knowledge and practices that underlie these big ideas are sufficiently addressed across lessons. And (2) explicitness, the implicit or explicit manner with which the intended big idea was reflected in the instructional activities of the module to go beyond the surface characteristics of the activities (e.g., an implicit manner refers to a case when the guiding discussion questions focused around the different ways a model can be used but the discussion questions were not worded clearly for this focus). The explicitness of the big ideas can be summarized from the information recorded in the explicitness column in the curriculum maps.

3. Cognitive demands. This dimension focuses on the overlap between the expected strategies to be used to respond to the item and the strategies used during the instructional experiences that the students experience in the module. We focus on two aspects: (1) whether the item required students to go beyond what was studied in the module, extended understanding; and (2) whether the reasoning required was in some way different from the pattern of reasoning introduced in the module (e.g., all instructional activities ask students to explore factors that lead to increased erosion but the item is about the reduction of erosion). The mapping process enables us to have a precise idea about the cognitive demands involved in the instructional experiences for the big ideas to be achieved. Then during the item development, deviations from these original cognitive demands for the proximal items can be proposed and tracked.

4. Setting of the item. In this dimension we manipulate the differences in the contexts used in the module. This dimension focuses on the difference between the what and how from the module critical activities (familiar contexts to students) and the what and how presented in novel contexts in the items. We manipulated the following setting aspects: (1) type of targeted organisms and objects as the focused organisms or objects that the assessment item is about (e.g., brine shrimp, beetles, mountain, park, coffee, ice, etc.); (2) process, referring to the phenomenon explicitly or implicitly described in an item which often is observable; process is connected to and explained by a scientific theory; (3) scenario or system, which includes information about a set-up or system in which organisms or objects and the processes are situated, such as presenting an experiment set up to investigate how volume of water impacts the amount of erosion or describing a specific ecosystem to ask students the relationship between one environmental factor and an organism; and (4) graphic representation in the item, referring to all different types of non-textual information included to supplement what is presented in the texts or to provide unique meaning.

5. Experimental setting. When items focus on experimental settings, two aspects were manipulated according to what students experienced in the module: the independent variable and dependent variable involved in the experiment. We chose not to


8

manipulate the controlling variables due to the lack of emphasis on this concept in the modules.

For most of the five dimensions, we judge the manipulation based on a three-scale code: no change, small change, and big change from what is presented in the module, plus a “not applicable” code. We also captured information about the multiple-choice component regarding the location of the item manipulation: stem of the item, options, or both. This information can help to learn more about which component of the item is more effective to manipulate based on the item statistics obtained.

Figure 2 provides an example of one item bundle for the Environments module with no distal item. Figure 3 provides the five dimensions and an example of the manipulations done to this bundle across the three proximities. Not all aspects within each dimension manipulated are presented in the figure. In Dimension 1, question characteristics, we focus on the characteristics of the questions asked. The skeleton of the question, what is the optimum condition for a particular organism, is regularly asked in three out of the six lessons in the Environments module. This should be a familiar question to students. The wording of the question in P1 resembles that of the close item except that the necessary changes of the organism and the environmental factor are also mentioned (these manipulations are captured in the dimension of setting). However, the wording of the question in P2 differs slightly regarding the order of the organism and the environmental factor from that in C and P1. This is then coded as a small change.

In Dimension 2, Exposure of the Big Idea in the module, we focused on extent of opportunities and explicitness. The big idea tapped in this bundle, “students understand the operationalized definition of optimum condition,” is explicitly taught in three lessons of the module, and is therefore coded as explicit and multiple times. This judgment is applied to the whole bundle.

In Dimension 3, Cognitive demands, we focus on the differences between the pattern of reasoning reflected in the module and in the bundle items. The three items require exactly the same reasoning process with the same level of understanding of the big ideas as what students experience in the module. Therefore, a code of no change is assigned to all three items.

In Dimension 4, Setting of the Item, we only focus on the first three aspects since no graphic representations are included. The degree of change progresses as the items become more distal to the characteristics of the curriculum, from no change in the C item (the same investigation that students conduct with brine shrimp hatching), to small change in the P1 item (an investigation about eggs hatching with a different organism, the same process, but in a scientist’s lab), and to big change in the P2 item (an investigation about a taxonomically different organism like a mushroom, a different process, and a different set up done by the mushroom grower).

In Dimension 5, we focus on the dependent and independent variables involved in the experimental settings in the three items. The module investigation (i.e., the brine shrimp hatching) uses the amount of salt as the independent variable and the amount of hatched shrimp eggs as the dependent variable. In the C and P1 items, the dependent variable is kept the same, but the independent variable is presented with reference to the volume of water as the salinity level of the water, therefore coded as small change. In the P2 item, both the dependent and independent variables are completely different from those in the module


9

investigation, thus coded as big change. Lastly, in this bundle all the proposed manipulations were done only to the stem.

Figure 2. Example of an Environment bundle: Close (C), Proximal 1 (P1), and Proximal 2 (P2).


10

Close* Proximal P2

Distal P1

1 Question Characteristics NC NC SC 2 Exposure to the Big Idea EM EM EM 3 Cognitive Demands NC NC NC 4 Setting NC SC BC 5 Experimental Setting SC SC BC

* NA = Not applicable; NC = No Change; SC = Small change; BC = Big change; EM = Explicit and multiple times

Figure 3. Example of the degree of manipulation in the bundle presented in Figure 2; no distal item is included.

As shown in the above example, manipulations should vary systematically from item to item within the same bundle, with more manipulations with big changes when item proximity moves away from the module. The first three dimensions, Questions, Exposure of Big Ideas, and Cognitive Demands are expected as manipulations for all the items as they are directly related to the construct to be measured. The other two dimensions are dependent on whether a context or an experimental setting is included in the items or not.

As explained in the introduction paper, in the second tryout and third tryout of the project, we continuously revised the DEISA approach to advance our understanding of the “messy middle” we previously defined as proximal items. We purported to develop items that could be more distinguishable in terms of the five aspects of manipulation, which then can be reasonably located on the sensitivity continuum rather than being too close to the two extremes.

In this paper, we empirically examined these item manipulations we used with the assessment items in Tryout 2. We applied the manipulation framework to analyze the item characteristics and quantify the extent of manipulation involved with items. We perform regression analysis using the manipulation scores to ask what module characteristics can be systematically manipulated to develop items at different distances (close and proximal) that prove to be instructionally sensitive? We provide the judgmental evidence by examining the association between the item manipulation scores and the pre-classified item proximity from the item developers and researchers, and the empirical evidence by examining the association between the item manipulation scores and the PPDI as the index of item proximity.

METHODOLOGY

Participants

The second tryout study was carried out in three school districts (an urban, a suburban, and a rural) with six or seven teachers per district. Characteristics of the districts were described in the introductory paper. Table 1 provides sample size information of teachers and students about the participating classes in the second tryout.


11

Table 1. Information by District and Teacher in Tryout 2

School District 1 School District 2 School District 3

Locale Suburban Urban Rural Curriculum Heat and Change Landforms Environments n n n

Teacher 1 38* 21 17 Teacher 2 23 22 18 Teacher 3 23* 21 15 Teacher 4 42* 23 23 Teacher 5 27* 22 21 Teacher 6 24 16 16 Teacher 7 ‐ ‐ 15

* Teacher taught two groups.

‐ There is no seventh teacher.

The order of the modules presented in later tables and figures thereafter will be Landforms, Environments, and Heat and Change since the first two modules are the focus of this paper.

Item Development Process

Puzzled at the messy middle after analyzing the item manipulation codes and effect sizes from Tryout 1 in which we only developed items as close items and proximal items, we made four revisions of the item development process, hoping to make the process more systematic so that we could test our hypothesis around item manipulations. The revisions were motivated by identifying two purposes: looking for evidence that could help us delineate where “close” ends and “proximal” begins, and defining better how item characteristics could be manipulated. First of all, the item development is carried out on the basis of “bundles of triads” all of which assessed the same construct expressed as levels of big ideas. For each bundle, one close item (C) is developed first and then two proximal items (Proximal 1 and Proximal 2) are developed using the close item as reference. Second, we made a close match between the curriculum map (tracked as critical activities, graphic representations, materials, and vocabulary), and the source of item sensitivity (SOIS) for the item manipulations after simplifying and streamlining these SOIS to make the manipulation more straightforward and consistent across the bundles. Third, rather than developing items for each learning target as was done in Tryout 1, learning targets for each module were revised and organized around levels of big ideas so that we could develop and sample more items on the same construct to be measured. Lastly, the items used in the second tryout were developed by the research team instead of experienced educators and assessment specialists from school districts. Two members of the research team, also co-authors of this paper, with content expertise in biology and physics/geology were in charge of the item development.

Due to time constraints, the team was only able to test the new approach with two of the three modules: Landforms and Environments, by developing new items and revising some of the tested items from Tryout 1. Therefore, for the Heat and Change module, the same items developed for Tryout 1 were used with the necessary revisions to the items that did not have the proper psychometric characteristics as a replication study. The items developed were


12

reviewed by three experts: two context experts (a biologist and a physicist/geologist) and a sociolinguist. Table 3 provides information about the number of items developed for each module using the improved approach.

Table 3. Number of Items Developed for Tryout 2

Modules Close Proximal 1 Proximal 2 Total

Landforms 21 19 19 59 Environments 16 19 15 50 Heat and Change a 29 19 48

Total 66 38 34 157

a. Items from Tryout 1, levels of proximity were not considered.

Among all revised and newly developed items, we selected and organized them into booklets for each module, four booklets for Landforms and Environments, and only two booklets for Heat and Change. Each booklet includes 30 items, mostly as multiple-choice and only one or two as open-ended responses. Based on the item development approach these multiple-choice items varied in instructional sensitivity, ranging from Close to P2. Table 4 reports the number of items we examined in this paper. For the analysis of item manipulations and pre-classified item proximity, we included 72 items for Landforms, 51 items for Environments, and 31 items for Heat and Change. When analyzing the connection between item manipulation scores and PPDI as the index of the item proximity, we only included items which were administered to students – 54 items for Landforms, 46 items for Environments, and 31 items for Heat and Change. For these items, we were able to use student test scores to calculate the item statistics and the number of items by proximity was reported in parentheses for each module (see Table 4).

Table 4. Number of Items Analyzed b

Modules Close Proximal 1 Proximal 2 Total

Landforms 24 (19) 24 (18) 24 (17) 72 (54) Environments 17 (16) 17 (15) 17 (15) 51 (46) Heat and Change a 17 (17) 14 (14) 31 (31)

Total 58 (52) 41 (33) 41 (32) 154 (131)

a. Items from Tryout 1, levels of proximity were not considered.

b. Numbers of items with PPDI information were reported in parentheses .

Sources of Data and Analysis

We used three sources of data:

1. The item manipulation codes. To track the manipulation done to each item, the five item-manipulation dimensions previously described were applied in a consensus coding with four researchers including two item developers. Codes varied from no change, small change, big change, and not applicable. In the Landforms and Environments modules, this coding


13

discussion was conducted based on the bundle of items so that the item manipulations could be easily compared and recognized when referring to the close item in each bundle. For the Heat and Change module, the coding was conducted with stand-alone items. Within each module, the bundles or items were randomly ordered and presented with the final form of items as they appeared in the booklets and with the correct answer to avoid any possible bias for the coding decisions. In this paper, we considered the codes as interval variables and used manipulation scores for each of the dimensions in the data analysis.

2. The item classifications designated by the item developers. We consider the pre-classifications of items as the hypothesized item proximity. By conducting ordered logit regressions using the item manipulation scores as predictors on the pre-classified proximity codes, we were able to test whether the items were manipulated as proposed in the manipulation framework.

3. The pre-to-post difference index (PPDI). The PPDI, introduced by Cox and Vargas in 1966 as a measure of item instructional sensitivity based on the test scores, has been considered a strong statistic of instructional sensitivity (Polikoff, 2010). PPDI is calculated by subtracting the percentage of correct responses in the pretest from that in the posttest. Therefore, it refers to the gain of student performance from the pretest to the posttest. A higher PPDI indicates a more sensitive item, assuming that patterns of student performance should correspond to the item sensitivity. We conducted a regression with the manipulation scores as predictors on PPDI in order to examine what aspects of item manipulation were more influential on the item proximity.

PRELIMINARY RESULTS

To respond our research question, What module characteristics can be systematically manipulated to develop items at different distances (close and proximal) that prove to be instructionally sensitive?, we provide both judgmental and empirical evidence across the three modules, including the two modules in which the improved DEISA approach was implemented (Environments and Landforms), and the module with the original item development approach (Heat and Change). In what follows, we first present the judgmental evidence on the item development and then report the empirical evidence by analyzing the PPDI and manipulation scores.

Judgmental Evidence: SOIS Manipulated

The manipulated items were tracked across the two or three item proximities, depending on the science modules. Appendix A provides information about the percentage of times that each type of change category was manipulated across the different types of proximal items. Figure 4 provides a summary of all the codes for each of the modules.


14

Landforms Environments

(a) (b)

Heat and Change

(c)

Figure 4. Patterns of changes across the proximities and the three science modules: Landforms (a), Environments (b), and Heat and Change (c).

0

10

20

30

40

50

60

70

80

90

Nochange

SmallChange

BigChange

N/A

Close

Prox1

Prox2

0

10

20

30

40

50

60

70

80

90

Nochange

SmallChange

BigChange

N/A

Close

Prox1

Prox2

0102030405060708090

Nochange

SmallChange

BigChange

N/A

Close

Proximal

Percentage

Percentage

Percentage

Type of Code

Type of Code

Type of Code


15

Patterns of changes across the three units were not exactly the same. However, the trend was similar: Close items show higher percentage in “no changes” and Proximal 2 items showed higher percentages of “big changes,” which was aligned with the item development rules for manipulating the SOIS. As it can be seen in Appendix A and Figure 1, big changes were done in approximately one third to one half of the Close items and no changes were done in roughly one quarter of Proximal 2, especially in the Landforms module (see Figure 1a). More consistent patterns of the item manipulation can be observed in the Landforms and the Environments modules than in the Heat and Change modules, as indicated by the chi-square test ( 2

Landformsx =90.07, df=6, p<.01; 2tsEnvironmenx =83.45, df=6, p<.01; 2

_ ChangeHeatx =6.01, df=3,

p=.11). This finding was not a surprise since the revised DEISA approach was only used with the item development in Landforms and Environments modules instead of the Heat and Change module. The chi-square results indicate that overall the items developed in Tryout 2 did follow the item development rules described in the manipulation framework.

We took a closer look at each aspect of item manipulation in a series of ordered logit regressions. Instead of reporting the regression coefficients, we calculated the probabilities that an item is close, P1, or P2, under three conditions based on the regression equations from the ordered logit regressions: no manipulation when the manipulation score is 0; average manipulation when the manipulation score is the closest integer to the mean for the module; and maximum manipulation when the manipulation score reaches the maximum for the module. Table 5a reports the calculated probabilities based on the total manipulation score and 5b to 5f report the probabilities for each of the five dimensions of item manipulation.

Table 5a. Probabilities of Item Proximity Classification for the Total Manipulation Score by Module

No manipulation Average manipulation Maximum manipulation

Module Close P1 P2 Close P1 P2 Close P1 P2

LF 0.722 0.227 0.050 0.295 0.457 0.248 0.007 0.040 0.953

EN 0.859 0.127 0.014 0.183 0.539 0.278 0 0.001 0.999

HC 0.834 0.166 0.587 0.413 0.195 0.805

Table 5b. Probabilities of Item Proximity Classification for the Manipulation Score of Questions by Module



LF 0.643 0.286 0.071 0.212 0.448 0.340 0.038 0.186 0.776

EN 0.435 0.405 0.160 0.148 0.394 0.458 0.002 0.011 0.987

HC 0.542 0.458 Only two values in data set (no middle value).

0.571 0.429


16

Table 5c. Probabilities of Item Proximity Classification for the Manipulation Score of Exposure by Module



LF 0.322 0.333 0.345 0.333 0.333 0.333 0.356 0.333 0.311

EN 0.308 0.334 0.359 0.349 0.334 0.317 0.393 0.330 0.278

HC 0.442 0.558 0.635 0.365 0.792 0.208

Table 5d. Probabilities of Item Proximity Classification for the Manipulation Score of Cognitive Demands by Module



LF 0.445 0.344 0.211 0.321 0.367 0.312 0.089 0.224 0.687

EN 0.418 0.337 0.246 0.330 0.348 0.322 0.137 0.268 0.595

HC 0.600 0.400 Only two values in data set (no middle value).

0.455 0.545

Table 5e. Probabilities of Item Proximity Classification for the Manipulation Score of Setting by Module



LF 0.678 0.289 0.033 0.111 0.522 0.367 0.000 0.006 0.994

EN 0.949 0.051 0 0.042 0.888 0.071 0 0 1

HC 0.836 0.164 0.655 0.345 0.089 0.911

Table 5f. Probabilities of Item Proximity Classification for the Manipulation Score of Experimental Setting by Module



LF Regression could not be performed because this was coded for only two items.

EN 0.551 0.323 0.127 0.270 0.406 0.324 0.010 0.044 0.946

HC Not enough variation in this variable to run the regression.


17

The probabilities calculated based on the results of the ordered logistic regressions confirm the manipulation patterns presented in Figure 3. Items in the Environments module and Landforms module were more aligned with the manipulation rules compared to the items in the Heat and Change module. Between these two modules for which the revised DEISA approach was employed, the items in the Environment module align better with the manipulation rules than those in the Landforms module. When comparing the results across the five dimensions, we found the manipulations of item characteristics were consistent with the manipulation rules for some dimensions, such as Questions and Setting. One possible explanation is that apparently some categories are easier to manipulate than others (see Appendix A), thus resulting in more variation in the manipulation scores. For example, type of organism or object used in the module has more codes as small change or big change. Other dimensions are more difficult, for example, graphic representation is the category with highest percentage in the “non applicable” across different proximities in the three modules, ranging from 50% to 71%. Interestingly, the dimension of experimental setting was coded for only 2 out of 72 items in the Landforms module and approximately 30% of the items in the Heat and Change module; only in the Environment modules, more than 70% of the items involved the experimental setting. These differences suggest that some manipulation dimensions are more frequently used than others because of the greater applicability to the content of one module over another.

Empirical Evidence: SOIS Contributing to Item Proximity Based on Test Scores

The differences in the patterns across manipulation dimensions provide evidence that it is possible to manipulate, in a more or less consistent manner, the SOIS as we hypothesized. The issue is whether these manipulations are supported by empirical evidence. To this end, we conducted a series of regression analyses to identify which SOIS contributed most significantly to the item proximity expressed as PPDI. This analysis was replicated with similar results when using the pretest scores as the co-variant in the regression modeling.

Across the five dimensions, only the “Setting” was consistently found to be a statistically significant factor that influences the PPDI . Table 6 reports the regression coefficients for the manipulation scores of the Setting dimension. The negative sign of the regression coefficients confirm that the larger changes made in terms of the targeted organisms or objects, process, setup, and graphic representation, the large distance from the enacted curriculum indicated by the lower PPDI statistics. This relationship was found at a significance level of .01 for the Landforms items and only at a significance level of .10 for the Environments items. Although non-significant, it is important to notice that all the regression coefficients, except the dimension of “Exposure to Big Ideas,” yielded the appropriate direction across the three modules.

Table 6. Regression Result with Setting as the Predictor by Module

Module Intercept (standard error) β (standard error) Residual error estimate

LF 0.181 (0.024) ** ‐0.028 (0.010) ** 0.141

EN 0.193 (0.026) ** ‐0.018 (0.010) + 0.119

HC 0.203 (0.036) ** ‐0.011 (0.014) 0.133

Note: + indicates p<.10; * indicates p<.05; ** indicates p <.01.


18

Considering that the graphic representation was one dimension with the highest proportion of not applicable codes, we re-ran the regression analysis after excluding this from the Setting manipulation score. The results in Table 7 show that across the Landforms and Environments modules, the three aspects of the Setting are influential to the item proximity indicated by students gain from the pretest to the posttest.

Table 7. Regression Result with Setting as the Predictor by Module after Excluding the Graphic Representation

Module Intercept (standard error) β (standard error) Residual error estimate

LF 0.185 (0.026) ** ‐0.031 (0.012) * 0.145

EN 0.198 (0.025) ** ‐0.023 (0.010) * 0.118

HC 0.212 (0.032) ** ‐0.022 (0.016) 0.130

Note 1: *Indicates p<.05; ** indicates p <.01.

Note 2. The manipulation score then only consisted of type of organism, process, and setup or system.

The regression coefficients for the dimensions of questions, cognitive demands, and experimental setting were negative but not statistically significant. The summarized frequency of the codes in Appendix A provided some possible explanations for this finding. The lack of statistical significance was mainly due to the small variation of the manipulation scores and high percentage of the not applicable codes, especially for the dimension of experimental setting. As discussed in the previous section, some dimensions were easier to be included in the items or more straightforward to be manipulated. The efforts in Tryout 3 then should be directed toward exploring ways to ensure the item manipulation stated by the rules occur systematically at the planned distance and maximize the item manipulation for the P2 items.

CONCLUSIONS

This paper presents information about the second tryout of an approach to developing and evaluating instructionally sensitive assessments (DEISA) in the context of science education. By analyzing the item manipulation scores, the pre-classifications of item proximity by the item developers, and the PPDI of student test scores, the study addressed the questions, what module characteristics can be systematically manipulated to develop items at different distances (close and proximal) that prove to be instructionally sensitive?

This question is at the core of the validity assumptions for instructional sensitivity as we hypothesized that the SOIS – Sources of Instructional Sensitivity presented in the curriculum map, if systematically handled in the item development with respect to the item proximity, should lead to increasing distance from the enacted curriculum when a higher proportion of big change codes is involved. We thus applied the revised DEISA approach in Tryout 2 with the Landforms and Environments modules. The improvements of the item development compared to the previous one used with the Heat and Change module include: developing items in the unit of an item bundle by starting with the close item as the reference and


19

extending to P1 and P2 items; re-organizing and simplifying the manipulation dimensions to be more consistent with the mapped curriculum characteristics and more straightforward to change across items; using the levels of big ideas as the construct definitions instead of the learning targets so that more items can be developed and tested for each construct; and involving the researchers as the item developers instead of practitioners to ensure the quality of items and increase the efficiency of item development, review, and revision. Importantly, all these revisions aimed to clarify the item manipulation in relation to the proximity so that we could learn more about the messy middle between the close-distal continuum as well as the manipulation of the SOIS.

In the paper we have described specific characteristics about the SOIS resulting from the mapping process and the manipulation framework and how they can be manipulated for items varying in proximity. Using the consensus coding, we analyzed the item characteristics in relation to the intended curriculum based on the curriculum map and conducted the regression analysis to examine the relation between the item manipulation scores and the pre-classified proximity and PPDI. Although the patterns of manipulation were slightly inconsistent across modules, overall, it is telling that a higher frequency of big changes was associated with the P2 items and a higher frequency of no changes was associated with the Close items. This pattern was also confirmed by the ordered logit regression in which the results reveal that items with no manipulation most likely are close, items with the averaged level of manipulation most likely appear to be P1, and items with the maximum manipulation most likely are P2. All these findings are supportive that the DEISA approach was well executed in the process of item development by researchers as the item manipulation scores were aligned neatly with the manipulation rules.

In contrast to the judgmental evidence, the empirical evidence from the regression analysis of the manipulation scores and PPDI is less encouraging. Among the five manipulation dimensions, only the Setting dimension was negatively associated with the pattern of student performance between the pretest and posttest with an acceptable statistical significance in the Landforms and Environments modules. Whereas this finding suggests that revised approach is promising as the items in the Heat and Change module were based on the previous approach, we noticed a few limitations in the implementation of this approach which to some degree could contribute to the lack of significance for other manipulation dimensions. First, some dimensions had a lot of non-applicable codes which may be due to the fact that for items with certain characteristics these manipulations were either impossible or very difficult to incorporate. We need to gain a greater clarity about the reasons for these non-applicable codes and develop more effective statistical methods to analyze the data instead of treating them as missing values. Second, we notice that some dimensions had less variation compared to other dimensions. This reminds us to be more mindful in the item development process. We need to adhere carefully to the manipulation rules by maximizing the manipulation for the P2 items and reducing the use of big changes in the Close items. Lastly, we should also attend to the inter-relation among these five manipulation dimensions. For example, the correlation patterns suggested that the dimension of questions was highly correlated with some other dimensions. Therefore, it is interesting to conduct the factor analysis to explore the underlying relationship between the coded manipulation aspects and to take the collinearity and the interaction between dimensions into account in the regression modeling. All these analyses may lead to new ways of organizing and collapsing the manipulation aspects and identifying the most effective manipulation.


20

REFERENCES

Airasian, P. W., & Madaus, G. F. (1983). Linking testing and instruction. Journal of Educational Measurement, 20, 103-118.

Bransford, J. D., Brown, A. L., & Cocking, R. R. (1999). How people learn: Brain, mind, and school. National Research Council. Washington, DC: National Academies Press.

Burstein, L. (1989). Conceptual considerations in instructionally sensitive assessments. CSE Technical Report 333. Center for Research on Evaluation, Standards, and Student Testing. Los Angeles, CA: University of California, Los Angeles.

Burstein, L., Aschbacher, P., Chen, Z., & Lin, L. (1990). Establishing the content validity of tests designed to serve multiple purposes: Bridging secondary-postsecondary mathematics. CSE Technical Report 313. Center for Research on Evaluation, Standards, and Student Testing. Los Angeles, CA: University of California, Los Angeles.

Commission on Instructionally Supportive Assessment. (2001). Building tests to support instruction and accountability: A guide for policymakers (James Popham, Commission Chair). Washington, DC: Author. Retrieved December 23, 2007, from http://www.testaccountability.org/.

Cox, R. C., & Vargas, J. S. (1966). A comparison of item selection techniques for norm referenced and criterion referenced tests. Internal Manuscript: University of Pittsburg.

Haladyna, T., & Roid, G. (1981). The role of instructional sensitivity in the empirical review of criterion-referenced test items. Journal of Educational Measurement, 18(1), 39-53.

Leinhardt, G. (1983). Overlap: Testing whether it is taught. In G. F. Madaus (Ed.). The courts, validity, and minimum competency testing (pp. 153-170). Boston, MA: Kluwer-Nijhoff Publishing.

Linn, R. L. (1983). Curricular validity: Convincing the courts that it was taught without precluding the possibility of measuring it. In G. F. Madaus (Ed.). The courts, validity, and minimum competency testing (pp. 115-132). Boston, MA: Kluwer-Nijhoff Publishing.

Madaus, G. F., Airasian, P. W., & Kellaghan, T. (1980). School effectiveness: a reassessment of the evidence. New York: McGraw-Hill.

Polikoff, M. S. (2010). Instructional sensitivity as a psychometric property of assessments, Educational Measurement: Issues and Practices, 29(4), 3-14.

Popham, W. J. (2006). Diagnostic assessment a measurement mirage? Educational Leadership, 64(2), 90-91.

Popham, W. J. (2007a). Conducting instructional-sensitivity reviews of educational accountability tests. Unpublished paper. Los Angeles, CA: University of California, Los Angeles.

Popham, W. J. (2007b). Instructional sensitivity of tests: Accountability’s dire drawback. Phi Delta Kappan, 89(2), 146-150, 155.


21

Ruiz-Primo, M. A., & Li, M. (2012). Assessing transfer of learning: Instructionally sensitive assessments, curriculum, and instruction. Paper to be presented at the AERA annual meeting, Vancouver, Canada.

Ruiz-Primo, M. A., Shavelson, R. J., Hamilton, L. & Klein, S. (2002). On the evaluation of systemic education reform: Searching for instructional sensitivity. Journal of Research in Science Teaching, 39(5), 369-393.

Schmidt, W. H., Porter, A. C., Schwille, J. R., Floden, R. E., & Freeman, D. J. (1983). Validity as a variable: Can the same certification test be valid for all students? In G. F. Madaus (Ed.). The courts, validity, and minimum competency testing (pp. 133-151). Boston, MA: Kluwer-Nijhoff Publishing.

Wiliam D. (2007, September). Sensitivity to instruction: The missing ingredient in large-scale assessment systems? Paper presented at the Annual Meeting of the International Association for Educational Assessment. Baku, Azerbaijan.

Yoon, B., & Resnick, L. B. (1998). Instructional validity, opportunity to learn and equity: New standards examinations for California Mathematics Renaissance. CSE Technical Report 484. National Center for Research on Evaluation, Standards, and Student Testing (CRESST) University of California, Los Angeles.


22

APPENDIX A. ITEM MANIPULATION CODES

Dimension of Questions: Question in Relation to Module

Module Proximity 0(Not A Surprise)

2(A Surprise)

Environments Close 17 0 P1 16 1 P2 12 5

Heat and Change Close 13 4 Proximal 11 3

Landforms Close 19 5 P1 17 7 P2 12 12

Dimension of Questions: Question in Relation to Close Item

Module Proximity 0(No Change)

1(Small Change)

2(Big Change)

9(Not Applicable)

Environments Close 0 0 0 17 P1 15 1 1 0 P2 4 5 8 0

Heat and Change Close Proximal

Landforms Close 0 0 0 24 P1 5 7 12 0 P2 1 7 16 0

Dimension of Exposure to Big Ideas: Primary Big Ideas Tapped

Module Proximity 0(Explicit and Multiple)

1(Implicit or Few)

2(Implicit and Few)

Environments Close 8 7 2 P1 8 7 2 P2 10 5 2


Landforms Close 8 14 2 P1 8 14 2 P2 9 13 2


23

Dimension of Exposure to Big Ideas: Secondary Big Ideas Tapped


1(Implicit or Few)

2(Implicit and Few)

9(No Secondary BI)


Heat and Change Close 1 5 11 Proximal 3 0 11


Dimension of Exposure to Big Ideas: Combination of Primary and Secondary Big Ideas


1(Implicit or Few)

2(Implicit and Few)

9(No Secondary BI)




Dimension of Cognitive Demands: Extension of Reasoning


2(Big Change)

9(Not Applicable)



Landforms Close 1 23 P1 2 22 P2 1 12 11


24

Dimension of Cognitive Demands: Pattern of Reasoning


2(Big Change)




Dimension of Setting: Type of Targeted Organisms or Objects


1(Small Change)

2(Big Change)

Environments Close 17 0 0 P1 4 10 3 P2 1 6 10



Dimension of Setting: Process


1(Small Change)

2(Big Change)

9(Not Applicable)

Environments Close 15 2 P1 15 2 P2 13 1 2 1

Heat and Change Close 16 1 Proximal 14

Landforms Close 14 10 P1 15 1 8 P2 11 1 2 10

Dimension of Setting: Setup or System


1(Small Change)

2(Big Change)

9(Not Applicable)


Heat and Change Close 6 2 2 7 Proximal 3 4 7 0



25

Dimension of Setting: Graphic Representation


1(Small Change)

2(Big Change)

9(Not Applicable)




Dimension of Experimental Setting: Independent Variable


1(Small Change)

2(Big Change)

9(Not Applicable)




Dimension of Experimental Setting: Dependent Variable


1(Small Change)

2(Big Change)

9(Not Applicable)




Location of Item Manipulation

Module Proximity 1(Stem mainly)

2(Options Mainly)

3(Both Stem and

Options)

9(Not Applicable)





26

instructional sensitivity and transfer of learning at · pdf fileinstructional sensitivity and...

Documents