applying the rasch model in keep - 1 - - university of
TRANSCRIPT
Applying the Rasch Model in KEEP - 1 -
Running head: Applying the Rasch Model in KEEP
Applying the Rasch Model to Evaluate a High School Science Implementation of the Kentucky
Electronics Education Project (KEEP)
Presenter: Weijia Ren (University of Kentucky)
Co-presenters: Kelly D. Bradley (University of Kentucky)
Janet K. Lumpp (University of Kentucky)
1
Applying the Rasch Model in KEEP - 2 -
Abstract
Kentucky Electronics Education Project (KEEP) uses the multidisciplinary nature of
microelectronics as a theme to connect real world content with K-12 classroom science
education. Science teachers are trained in a series of circuit building activities through summer
workshops and implement them in their classes. Students are given a survey to evaluate their
perspectives after completing the activities. The sample includes 61 students from a high school
in Lexington, Kentucky. Data are fitted into the Rasch rating scale model. Findings indicate a
high reliability and validity of the survey instrument and a high satisfaction of the project among
students. The result shows that most students think the project is easy. However, most
misunderstandings appear to occur in item 3 “Etching the copper”. The least favorite item and
the most difficult item are both item 2 “Ironing the pattern onto the circuit board”. Possible
reasons and solutions for improving the steps in the project and corresponding items are
described at the end.
Keywords: Rasch model, KEEP, circuit building, program evaluation
2
Applying the Rasch Model in KEEP - 3 -
Introduction
With the rising expectations of U.S. students to be the first among students all over the
world in math and science achievement and the widening gap between the United States and
some other countries, such as India and China, in educating and graduating students with the
engineering degree, the Kentucky Electronics Education Project (KEEP) was initiated and
developed to engage K-12 students in math and science education with a hands-on circuit
building activity. This real world example is beyond the traditional “bulbs and batteries” example
and provides students with the basic science knowledge about electricity, which is also aligned
with the national science education standards.
Although nowadays some reform proposals have been posted to improve the K-12
science education, there has never been a specific curriculum or program to connect students’
classroom learning with the real world scientific activities. Therefore, it is the effort and goal of
KEEP to help produce a set of science education curricula to apply real life science activities to
the classroom education. Thus it is necessary and important to get instant feedbacks from
teachers and students and based on them, to improve this “pioneer” project in K-12 science
education.
The Kentucky Electronics Education Project began in 1997. It involves curriculum
development, teacher workshops and classroom activities initiated by the educational outreach
aspects of two National Science Foundation research grants (Lumpp et al., 2003). This project
was initiated by Dr. Janet Lumpp, a professor in the College of Engineering at the University of
Kentucky. The specific objectives of KEEP are to educate teachers regarding electronic assembly
3
Applying the Rasch Model in KEEP - 4 -
technologies and properties of electronic materials; to develop curriculum materials and kits; to
organize hands-on projects and fieldtrips; and to solicit education/industry partnerships (Lumpp
et al.). This circuit building activity incorporates integrated curricula and performance
assessment, and moreover, provides a real world activity that is a foundation for math and
science application to the K-12 science education classroom.
Science Education is an indispensable element of the education system. Students can only
build a comprehensive knowledge background with a competitive science curriculum. In order to
develop such science curriculum, real world scientific inquiries, technologies and examples are
indispensable. Incorporating true scientific inquiry into today’s classrooms, especially K-12
classrooms, is a major focus of current science education reform and is critical to scientific
literacy (Rutherford et al., 1990). In a meta-analysis of 20 studies that examined the effects of
technology use on students’ cognitive, affective, and behavioral learning outcomes, a modest
positive effect was found between technology and other learning outcomes (Waxman et al.,
2002). Also, the report to the nation by the National Commission identifies the need to facilitate
teachers in learning how to integrate technology into science and mathematics education
(USDOE, 2000). Therefore, implementing engineering design is a powerful strategy for the
integration of science, mathematics, technology, and for engaging a broad population of students.
Although engineering design could be used to provide relevant and standards-based
content for integrating engineering into K-12 science classes, nowadays most popular science
textbooks for grades 4-12 incorporate little if any engineering content or activities (Cantrell et al.,
2002). Not long ago, national reports had indicated that science education in America was falling
4
Applying the Rasch Model in KEEP - 5 -
short of its goal for achieving scientific literacy for its youth. Moreover, “experts have also been
warning for a few years now that the United States is at risk of losing its lead position in the
education of science and engineering Ph.D.’s” (Jaschik, 2005). In order to attract more students
to study science or pursue an engineering degree in college, teachers have to begin to cultivate
children’s interest in science when they are in the elementary school. The long-term goal of
KEEP is to enhance the quality of teacher instruction, support gains in student achievement, and
increase students’ interest in STEM curricula and careers (Lumpp et al., 2006).
In KEEP, circuit building is used as an activity to bring the real world microelectronics
models into classroom education. During the past 10 years, the circuit building activity is given
to the students at the end of each spring semester, typically in May. There are altogether seven
steps in the circuit building project, including copper cleaning, ironing the pattern onto the circuit
board, etching the copper, drilling holes, removing toner, inserting components, and soldering.
These steps are related to the following topics in microelectronics: copper etching and plating,
semiconductors, insulators, conductors, energy conversion, heat transfer, materials properties,
dimensions, routing, soldering, circuit symbols, and component types. These topics tie directly
with national standards on math skills and concepts, physical science, science and technology,
environmental issues, and science as inquiry (Lumpp et al., 2003).
This manuscript focuses on the attitudes of students who have participated in the circuit
building activity. Students’ attitudes towards the project are reflected by two sections: students’
satisfaction of each step in the project and the overall project; and students’ perspectives on the
difficulty level of each step and the overall project. Students’ attitudes are very important in order
5
Applying the Rasch Model in KEEP - 6 -
to evaluate teachers’ performance as well as the whole KEEP project. Based on students’
feedbacks, the project could be properly improved, and steps in this activity could be adequately
designed and ordered. Specific steps which most students are confused will be improved and
restructure as well. In addition to the improvement of the project, the application of the survey
instrument will also be improved. The content and order of the survey will be analyzed and
whether it is necessary to add or remove any questions from the current survey will be discussed.
In a word, the purpose of this manuscript is to analyze students’ feedbacks to this project and
improve both the project and the survey instrument.
Evaluation is a tool to help teachers judge whether a curriculum or instructional approach
is being implemented as planned, and to assess the extent to which stated goals and objectives
are being achieved (Fleischman & Williams, 1996). This study combines both process evaluation
and outcome evaluation which determines the effect of the program, and also how the program
produced that effect and how to improve the program. Data collection and record is an important
part of the design in the evaluation, which may include record, keeping forms, questionnaires,
interview guides, tests, or other assessment measures (Fleischman & Williams). In this study,
questionnaire is chosen to be the effective and reliable method to collect data fast and without
causing major inconveniences to the class. To analyze the collected data, Item Response Theory
is often utilized. Item Response Theory is based on assumptions concerning the mathematical
relationship between abilities and item responses (Rudner, 2001). However, traditional IRT
model may not fit in all situations. Rasch model, as a subset of IRT model, works better and
more accurate when analyzing data from assessment to measure things such as abilities, attitudes,
6
Applying the Rasch Model in KEEP - 7 -
and personality traits. In the case of this study, because students’ attitudes are the major factor,
Rasch model will be a better fit.
Method
Participants
The participants to this survey were the 61 students who took part in the KEEP activity in
May 2006 from one high school in Lexington, Kentucky. There were 38 males and 23 females.
The participants include 49 seniors, 11 juniors and one sophomore. Among them, 41 students had
completed similar KEEP project before, while the other 20 students had not taken any similar
courses before. During the analysis, each participant was assigned a person code including
information about their current grade level, gender, and the typical grades they receive in classes,
whether or not they want to do a similar project again and whether or not they have participated
in a similar project before. Currently, students who participate in KEEP are middle school and
high school students in Lexington, KY. Therefore, the population of interest in this study is
middle school and high school students who have participated in the circuit building project. And
the 61 students in the sample are fitted in the population of interest.
Instrumentation
This survey was designed by experts at the University of Kentucky as an onsite paper
questionnaire. Students were asked to complete this pencil-and-paper questionnaire in class. The
project representatives brought the questionnaires to this high school and students were required
to finish the survey within five to ten minutes at the end of the circuit building activity. This
survey utilized a 4-point Likert-type scale to rate students’ satisfaction of each step of the project,
7
Applying the Rasch Model in KEEP - 8 -
including copper cleaning, ironing the pattern onto the board, etching the copper, etc. (1=very
unsatisfied, 2=unsatisfied, 3=satisfied, 4=very satisfied), their perspectives on the difficulty level
of each step (1=very difficult, 2=difficult, 3=easy, 4=very easy), and their perspectives on the
classroom environment (see appendix for details). In addition to the three tables, six close-ended
questions were also designed to gather the demographic information of the students, such as their
grade level, gender, etc. Four open-ended questions were designed to gather information about
what students learned from the project, their favorite and least favorite parts in this project, etc.
(see appendix for details).
Data Analysis
In this manuscript, students’ satisfaction of the project and students’ perspectives on the
difficulty level of the project are highlighted. There were no missing data for these two sections
among the collected responses. Data were analyzed in Winsteps, employing the rating scale
Rasch model. Rasch is mathematically identical to the most basic Item Response Theory (IRT),
however it is a comparatively more viable proposition for practical testing since it can be applied
in the experimental context in which persons interact with assessment questions or items. In this
model, data must fit the model, and no extreme item or person will be analyzed. The rating-scale
Rasch model is an extension of the dichotomous model and a simplified version of the partial
credit model. It specifies that the whole set of items share the same rating scale structure. The
mathematic expression of rating-scale Rasch model is )]([exp)]([expjin
jinnixτδβτδβ+−ΣΣ+−Σ
=Π where
x= 0, 1,…, m. πnix shows the probability of student n responding in category x to item i (Wright
8
Applying the Rasch Model in KEEP - 9 -
et al., 1982).
Throughout the analysis, several tables and figures are used to describe and analyze both
students’ satisfaction and the difficulty level of the project. A statistical summary table is
produced to describe the separation rate and reliability of the students and the items. Separation
is the number of statistically different performance strata that the test can identify in the sample.
The reliability rate indicates the validity and reliability of the instrument, and whether or not the
test discriminates the sample into enough levels for the research purposes. In Rasch model,
separation rate and reliability rate are highly correlated. Then, a construct keymap will be
provided to further test the validity of the categories of the four choices as well as the whole
instrument.
In addition, an item/person misfit table is analyzed. The misfit data is described and
explained. The statistics show how well the data fit the model. Fit implies meeting requirements
or matching intentions. It specifies an indication of the match between a group of persons and a
set of items. If some persons or some items did not perform as expected, they were flagged as
persons or items differing from expectation and therefore misfitting (Shaw, 2006).
Following, a variable map is constructed to illustrate the empirical hierarchy of items in
the survey, as connected to the students’ level of willingness to endorse each item. The variable
map visually reveals the hierarchy and the order of the items as well as the gap between every
two items. It also displays the logit of every student and every item. A logit (log-odds unit) is a
unit of interval measurement which is well-defined within the context of a single homogeneous
test (Winsteps Help, 2007).
9
Applying the Rasch Model in KEEP - 10 -
Lastly, a correlation matrix is used to analyze the correlation coefficient between the
seven steps and the overall project. The matrix is run by Minitab. From the analysis, how these
seven steps correlated with each other as well as with the overall project will be discovered.
Results and Discussion
The results are divided into two sections and discussed separately. Section 1 is students’
satisfaction of the project and section 2 is students’ perspectives on the difficulty level of the
project.
Students’ Satisfaction of the Project
Reliability and separation for students’ satisfaction.
In order to have an overall view of the reliability and validity of the instrument and the
students, the statistical summary tables of both the student (Table 1) and the item (Table 2) are
analyzed at first.
Table 1 shows that the person reliability in the model is 0.77, and the person separation is
1.83. Table 2 shows that the item reliability is 0.89 and the item separation is 2.91. The higher
the reliability is, the more reliable the instrument is. In this case, reliability which is higher than
0.7 is acceptable, thus the reliabilities in both tables (0.77 and 0.89) are acceptable.
In table 1, the person separation of “1.83” indicates that 1.83 levels of performance were
consistently identified by the test for the tested sample. That means students were roughly
separated into 2 groups. These two separated groups include those who were satisfied with the
project and dissatisfied with the project.
10
Applying the Rasch Model in KEEP - 11 -
The person separation is slightly lower than 2, which indicates a narrow range of person
measures or a small number of items (8 items) in the survey. The student reliability in table 1 is
0.77 out of 1, which is quite a good reliability and indicates the model is reliable for the 61
students. If the designers want to increase the reliability, the survey could be lengthened, and
more questions could be added in the survey. Increasing the sample size and testing people with
more extreme attitudes (high or low) may also help.
The reliability and separation rate in table 2 are more acceptable and indicate the survey
is reliable for the eight items and these items are better separated than the students. The item
separation rate “2.91” indicates that the eight items are generally separated into 3 groups: items
that students were satisfied, items that students thought to be “OK”, and items that students were
unsatisfied. The item reliability is 0.89 out of 1, which is higher than the student reliability above,
shows that the eight items as well as the whole survey instrument are reliable.
Construct keymap for students’ satisfaction.
Construct keymap is a figure which shows reliability and validity of the categories in the
instrument visually. In this plot, items are ordered from the least favorite-item 2 “Ironing” to the
most favorite-item 8 “Overall project”. If the category order “1, 2, 3, 4” of one item is not
consistent, that item might cause misunderstanding or unexpected answers. In figure 1, all items
have a consistent category order; therefore, the instrument and scales are given evidence as to
their validity. The survey instrument is thus proved to be both reliable and valid. For items 1, 4, 5,
6, 7and 8, since category 1 does not appear, “very unsatisfied” is not a likely response regardless
of the respondents’ abilities or attitudes.
11
Applying the Rasch Model in KEEP - 12 -
Item/person misfit order table for students’ satisfaction.
In addition to the statistical summary tables, the person and item misfit order tables are
also helpful to test assumptions. The analysis tests whether this instrument was a valid tool for
data collection. Table 3 reflects the misfit statistics of the item.
In table 3, the outfit mean square statistic (MNSQ=1.49) for item 3 is outside the selected
range of item outfit “Mean+S.D.” (0.97 + 0.24). The high outfit value indicates this item
received the most unexpected responses and implies that students’ responses to this item were
out of the project designer’s expectation.
The possible reason for students’ misunderstanding maybe the teachers usually etch the
copper by themselves rather than teaching students to etch because etching will take a long time.
This item is typically carried out by the teacher outside of class time. While the circuit boards
soak in the etching solution, the students would have to wait to perform the next step. Perhaps
the students rated this item inconsistently because they did not personally perform the step. In
order to solve the misunderstanding, teachers may show students a video about how to etch the
copper. Although students may not be able to do it by themselves due to the limited time, they
will at least know how to etch after watching the video.
As for the misfit students, seven misfit students are found to respond with more
unexpected answers than the other 54 respondents. Since the range of student outfit is 0.45-1.49
(Mean+S.D.), person 12 SFBCYY (see Appendix for details) has an outfit figure of MNSQ=2.35,
which is much higher than the upper bound and the highest among all responds. That indicates
she endorsed the most unexpected answers among all respondents. Similarly, student 10
12
Applying the Rasch Model in KEEP - 13 -
SMABYN, student 26 SMABYY, student 18 SMAAYY, student 52 JFAAYN, student 14
JFANYY, and student 41 SMABYY also endorsed more unexpected answers than the rest of
students.
The most unexpected response that the seven persons endorsed is item 3 “Etching the
copper”, which is also shown as the most misfit item in the previous item misfit table. Other top
unexpected items include item 6 “Inserting components” and item 7 “Soldering”. Most of these
misfit persons have a high measure score, which means they are generally satisfied with the
overall project, while they may not as satisfied with items 3, 6, and 7.
Variable map for students’ satisfaction.
The variable map is another important figure to provide visual information about the
relative scales of person and item. In this map, from top to bottom, items are ranked from the
least favorite item to the most favorite item. Respondents are ranked from students who are most
satisfied with the project to those who are least satisfied with the project. The variable map is
shown in figure 2.
In figure 2, students are most satisfied with item 8 “Overall project”, and least satisfied
with item 2 “Ironing the pattern”. It is very interesting that students endorse the overall project as
the most satisfied item. It indicates students are satisfied with the whole project but each single
step in this project may have some weaknesses with which they are not satisfied. Other elements
may also contribute to students’ satisfaction, such as the classroom environment, teachers’
teaching methods, etc.
From the variable map, the item “Overall project” does not reveal any additional useful
13
Applying the Rasch Model in KEEP - 14 -
information than other items. Therefore, this item may be revised or deleted. In addition, since
the mean of students’ satisfaction is almost two standard deviations higher than the mean of item;
therefore, most students are satisfied with most parts of this project.
The significant gap between item 2 “Ironing the pattern” and other items indicates the
need for a better scale coverage. Item 2 is also the most difficult to endorse, indicating students
are more dissatisfied with this item than others. It is possible that students think this step is
difficult, boring and not as important as other steps. The step “Ironing on the pattern” includes
ironing the toner onto the board, and then repairing any missing lines with a permanent match. If
the pattern is beyond repair, the students must repeat both of the cleaning and ironing steps. It
may be the case that the students rate “Ironing” as difficult because of the repair process. From
students’ comments, “ironing” is regarded as difficult because it is hard to control the melting
temperature of the toner and if they make any small mistake, they have to redo the whole process
again. In order to identify the specific difficulty level of ironing, this step and its corresponding
question in the survey could be separated into two— “Ironing” and “Repairing the pattern”.
Teachers may give students more detailed instructions for ironing and help student finish ironing
more quickly. The large gap at the top indicates that most students were satisfied with most of
the project, which again implies students’ satisfaction with this project. The smaller gap between
item “Soldering” and item “Inserting components”, together with other two gaps, divide the eight
items into three groups by students’ satisfaction degree. This is also verified in the previous item
separation table.
Correlation matrix for students’ satisfaction.
14
Applying the Rasch Model in KEEP - 15 -
In this correlation matrix, the correlation coefficients as well as the p-value between each
step and other steps are revealed and analyzed. Since α is set at 0.05, any correlation coefficient
that has a p-value higher than 0.05 is not significant. Therefore, the correlation coefficient
between “Ironing” and “Etching” is not significant with a p-value at 0.176>α=0.05, so it is hard
to predict students’ satisfaction of ironing by their satisfaction of etching, and vise versa.
Similarly, the correlation coefficients between “Copper Cleaning” and “Inserting Components”
(p=0.471), “Copper Cleaning” and “Soldering” (p=0.069) are not significant. And all these three
correlation coefficients are relatively weak. “Inserting Components” has the highest correlation
coefficient (0.528) with “Overall Project” among all the seven steps, and “Drilling” has the
lowest (0.328). It implies if students are satisfied with the step “inserting components”, they will
more likely to be satisfied with the overall project. And students may be satisfied with the overall
project no matter whether they are satisfied with drilling or not.
Students’ Perspectives on the Difficulty Level of the Project
Similar to the analysis above, statistical summary table, construct keymap, item/person
misfit tables, the variable map and item correlation matrix are used to analyze students’ ability
and the difficulty level of the project.
Reliability and separation for the difficulty level.
Table 4 and table 5 show that the item reliability (0.95) and separation (4.23) are higher
than the student reliability (0.77) and separation (1.84) for students’ perspectives on the difficulty
level. The results indicate that both the student sample and the items in the survey are reliable,
15
Applying the Rasch Model in KEEP - 16 -
especially the item reliability. Students are successfully separated into 2 groups: one thinks the
project is difficult and the other thinks it’s easy. The 8 items are separated into 5 difficulty levels,
rating from the easiest to the most difficult. This is also visually displayed in the variable map in
the following figure 4.
Construct keymap for the difficulty level.
In figure 3, for item 3 “Etching the Copper”, there is a disorder between category 1 and 2.
It indicates that for this item, students with less ability are more likely to endorse category 2
“difficult” than category 1 “very difficult”. This also implies that item 3 has the highest misfit
rate. In addition, since category 1 is missing for Item 1, 4, 5 and 8, the category “1=very
difficult” is not the most likely response in this survey, however, if more students with less
ability are included in the sample, this category may be useful for these items. In other words,
these four items are regarded to be easy by all students. In general, most of the categories are in
order, which implies this survey instrument is valid.
Item/person misfit order table for the difficulty level.
Surprisingly, table 6 also indicates that item 3 “Etching the copper” is the most misfit
item since it is outside the upper bound of outfit—“Mean+S.D.” (1.00+0.32). This item is also
the misfit item in students’ satisfaction. Similarly, students’ concepts about this item may be
different from the designed concepts. Since item 3 is the misfit item in both sections, it indicates
a significant mismatch between some students and this item. The wording may be ambiguous or
misleading. As mentioned above, since the step “Etching the copper” is understood at different
levels among students, teachers may want to describe in detail or show students how this step
16
Applying the Rasch Model in KEEP - 17 -
processes via video or other instructional methods.
Student whose mean square is outside the range of Mean + S.D. (1.00+0.59) is
considered to be misfit. Person 52 JFAAYN has a highest mean square of 3.83 among all
students, indicating she’s the most misfit student. Also, student 42 JMABYY, student 58
SMABYY, student 46 SFABNN, student 17 SMABYY, student 33 SMABNY and student 47
SMAAYN all endorsed more unexpected answers than the rest of students.
The most unexpected answers they endorse are “1=very difficult” and “2=difficult” for
item 2 “Ironing” and item 3 “Etching”. Most of the 7 persons have positive measure value, which
means they think this project is easy. Therefore, their perspectives on item 2 and item 3 may be
the reason why the two items become the misfits.
Variable map for the difficulty level.
Figure 4 illustrates the variable map for student perspectives on the difficulty level of the
project, from top to bottom, the 8 items are ranked from the most difficult item to the easiest item,
while the persons are ranked from the most able students to the least able students. The variable
map is shown in figure 4.
In this map, “Ironing the pattern” is regarded as the most difficult item, while the
“Copper Cleaning” is the easiest item. Since “Copper Cleaning” is the first step, students may
generally view it easier than all other following steps. The difficulty level of the overall project is
in the middle of all steps, which implies some steps are easier and others are more difficult. Also,
the students’ mean ability is higher than the items’ mean difficulty level, thus most students
thought the project was easy. The range of the items is almost 4 standard deviations, and there is
17
Applying the Rasch Model in KEEP - 18 -
an obvious gap between “Ironing” and “Soldering”. It implies that students think “Ironing” is
much more difficult than other steps.
Together with the last section, “Ironing” is regarded as the step which is least favorite and
most difficult. It may be concluded that because “Ironing” is the most difficult step, most
students are not satisfied with it. As mentioned above, designers may want to separate this step
into “Ironing” and “Repairing the pattern”. Altogether, there are five visible gaps which divide
the eight items into 5 difficulty levels, which verifies the item separation rate in table 5 as well.
Correlation matrix for the difficulty level.
In the correlation matrix of the difficulty levels of the eight items, the correlation
coefficients between “Copper Cleaning” and other steps are almost all insignificant
(p-value>α=0.05). Similarly, the correlation coefficients between “Ironing” and “Drilling”
(p=0.744), “Ironing” and “Removing the toner” (p=0.569), “Ironing” and “Inserting
Components” (p=0.091) are not significant. Since “Copper Cleaning” and “Ironing” are the first
two steps in this project, students may hardly define their difficulty levels, therefore, students’
answers to these two steps are not related tightly with other steps, that is to say, it is hard to
estimate other steps’ difficulty levels based on students’ answers to these two questions.
Item 2 “Ironing” has a lower correlation coefficient with item 8 “Overall Project” (0.260)
than all other items, which means students who think the overall project is easy may think the
item 2 “Ironing” is either difficulty or easy. Item 4 “Drilling” has a higher correlation coefficient
with the overall project (0.499) than other steps, which indicates that students who think the
overall project is easy tend to think drilling is easy too, and vise versa. In figure 4, the same
18
Applying the Rasch Model in KEEP - 19 -
conclusion was drawn. There is a big gap between “Overall project” and “Ironing”, but not
between “Overall project” and “Drilling”.
Conclusion
Based on the analysis above, most students are satisfied with the circuit building activity,
and think this overall project is easy. Though Rasch model, students’ ability and the
corresponding item are clearly stated and compared along one scale and therefore properly
analyzed. The survey instrument is reliable and can well separate both the sample students and
items. The four categories of the choice are also valid in this instrument. Most students
understand all items and steps well without confusions or misunderstandings.
Item 3 “Etching the copper” is the most misfitting step in both section 1 “students’
satisfactions of the project” and section 2 “students’ perspectives on the difficulty level of the
project”. The possible reason may be that teachers did this step by themselves instead of teaching
students to do it; therefore, teachers may want to give more details about this step and show
students a video about the etching process to make students to be aware of the step.
In addition, item 2 “Ironing the pattern” is regarded as both the least favorite and most
difficult item in this project. Many students put “Ironing” as the most difficult step in their
answers to one of the open-ended questions. Their comments imply that because the toner does
not melt at the expected temperature, it needs several tries to make it work and if any mistake is
made, the whole pattern should be repaired. Therefore this step may need to be separated into
two steps –“Ironing” and “Repairing the pattern” in order to test the specific difficulty level of
19
Applying the Rasch Model in KEEP - 20 -
ironing and teachers may give students more details about the toner’s temperature.
In addition, the results from this survey may be biased since most of the respondents are
students in good standing and they all come from one school. To improve it, more students with
extreme grades may be included. With “good standings”, the students may be interested in all
studies or subjects, consequently it may not be concluded that this project is successful and liked
by all students if all students in the sample like science and engineering. Thus a close-ended
question may be raised to ask students to choose their favorite subject.
This preliminary study evaluates both the project and the survey instrument. Different
from many other studies, this study provides both outcome evaluation and improvement
suggestions to this ongoing project. But when testing students’ satisfaction to this project, some
other aspects should also be taken into consideration, such as the classroom environment, the
tools, and teachers’ instructions. In addition to students’ attitude evaluation, teachers’ attitude
evaluation is also indispensable. Therefore, in future studies, the analysis of students’
achievement, teachers’ instructions, and teachers’ attitudes may also be helpful to develop this
project. Because KEEP is more like a short-term workshop than a long-term curriculum, some
other alternative assessments such students’ portfolio may not fit in this case. With effort of
engaging KEEP in school curriculum, more assessments and evaluations will be conducted to
improve the development of the project.
20
Applying the Rasch Model in KEEP - 21 -
References
Cantrell, P., & Robinson, M. (2002). How do 4th through 12th grade science textbooks address applications in engineering and technology? Bulletin of Science, Technology & Society, 22, 31-41. Fleischman, H.L., & Williams, L. (1996). An Introduction to Program Evaluation for Classroom Teachers, Development Associates, Inc, Arlington, VA. Retrieved on August 22, 2007 from http://teacherpathfinder.org/School/Assess/assess.html Frechtling, J., & Sharp, L., & Carey, N., & Vaden-Kiernan, N. (1995). Teacher enhancement programs: A perspective on the last four decades, National Science Foundation, Washington, DC. Jaschik, S. (2005). Lost dominance in Ph.D. production. Inside Higher Ed. Retrieved October 29th, 2006 from http://www.insidehighered.com/news/2005/07/15/science Lumpp, J.K., & Bradley, K.D. (2003). “Development and Dissemination of KEEP- Kentucky Electronics Education Project”. Proceedings of the Elec. Comp. & Tech. Conference. New Orleans, LA. Lumpp, J.K., & Bradley, K.D. (2006). “Math and Science across the Board: Connecting Professional Development to Classroom Practices via an Embedded Research Initiative”. American Society for Engineering Education Annual Conference Program & Proceedings. Chicago, IL. Rudner, L. M. (2001). Item Response Theory, Retrieved on August 22, 2007 from http://edres.org/irt Rutherford, F. J., & Ahlgren, A. (1990). Science for all Americans, Oxford University Press, New York. Shaw, F. (2006). Fits about “Misfit”, Retrieve November 14th, 2006 from http://rasch.org/rmt/rmt51g.htm USDOE. (2000). “Before It’s Too Late: A Report To The Nation From The National Commission on Mathematics and Science Teaching for the 21st Century”. Washington, D. C: United States Department of Education. Waxman, H.C., & Connell, M.L., & Gray, J. (2002). A quantitative synthesis of recent research on the effects of teaching and learning with technology on student outcomes. Naperville, IL:
21
Applying the Rasch Model in KEEP - 22 -
North Central Regional Education Laboratory. Winsteps Help (2007). Logit and Probit: What are they? Retrieved March 25th, 2007, from http://www.winsteps.com/winman/whatisalogit.htm Wright, B. D., & Masters, G. N. (1982). Rating scale analysis: Rasch measurement. Chicago: Institute for Objective Measurement, Inc.
22
Applying the Rasch Model in KEEP - 23 -
Appendix Survey
I. Students’ Satisfaction of the Project.
1. Copper Cleaning-Preparing the surface
2. Ironing the Pattern onto the Circuit Board
3. Etching the Copper-Creating Black Patterned Lines
4. Drilling Holes
5. Removing Toner
6. Inserting Components
7. Soldering
8. Overall Project
(A: Very unsatisfied, B: Unsatisfied, C: Satisfied, D: Very satisfied)
II. Difficulty Level of the Project.
1. Copper Cleaning-Preparing the surface
2. Ironing the Pattern onto the Circuit Board
3. Etching the Copper-Creating Black Patterned Lines
4. Drilling Holes
5. Removing Toner
6. Inserting Components
7. Soldering
8. Overall Project
(A: Very difficult, B: Difficult, C: Easy, D: Very easy)
23
Applying the Rasch Model in KEEP - 24 -
III. Person Code:
Example:
Person 12 SFBCYY:
S-senior; F-female; BC-the grades she typically receives in classes is B’s and C’s; Y-she would
like to do a project like this again; Y-she has done a project like this before.
24
Applying the Rasch Model in KEEP - 25 -
Tables and Figures
Table 1. Summary of 61 measured students for students’ satisfaction
____________________________________________ _______
Real RMSE 1.01 Separation 1.70 Student reliability 0.74
Model RMSE 0.95 Separation 1.83 Student reliability 0.77
__________________________________________________________________________
Table 2. Summary of 8 measured items for students’ satisfaction
__________________________________________ _________
Real RMSE 0.29 Separation 2.79 Item reliability 0.89
Model RMSE 0.28 Separation 2.91 Item reliability 0.89
____________________________________________________________________________
Figure 1. Construct keymap for students’ satisfaction.
25
Applying the Rasch Model in KEEP - 26 -
Table 3. Item statistics in order of misfit for students’ satisfaction
______________________________________________ ______
Infit Outfit
Item Model measure MNSQ MNSQ Item name
3 0.20 1.56 1.49 Etching the copper_________
Mean 1.00 0.97
S.D. 0.25 0.24
______________________________________________________________________________
26
Applying the Rasch Model in KEEP - 27 -
Figure 2. Variable map of students’ satisfaction of the project (M=mean, S=one standard
deviation, T=two standard deviation).
27
Applying the Rasch Model in KEEP - 28 -
Table 4. Summary of 61 measured students for the difficulty level
____________________________________________________
Real RMSE 0.75 Separation 1.67 Student reliability 0.74
Model RMSE 0.70 Separation 1.84 Student reliability 0.77
____________________________________________________________________________
Table 5. Summary of 8 measured items for the difficulty level
______________________________________ ___________
Real RMSE 0.24 Separation 3.98 Item reliability 0.94
Model RMSE 0.22 Separation 4.23 Item reliability 0.95
_____________________________________________________________________________
Figure 3. Construct keymap for the difficulty level.
28
Applying the Rasch Model in KEEP - 29 -
Table 6. Item statistics in order of misfit for the difficulty level
_____________________________________________ _______
Infit Outfit
Item Model measure MNSQ MNSQ Item name
3 -0.65 1.51 1.42 Etching the copper________
Mean 0.99 1.00
S.D. 0.34 0.32
______________________________________________________________________________
29