Students’ Performance on Engineering Mathematics Applying Rasch Measurement Model

Azrilah Abdul Aziz Infrastructure University Kuala Lumpur, Faculty of

Information & Computing Technology, Unipark Suria, Jalan Ikram-Uniten, 43000 Kajang, Selangor, Malaysia

azrilah@iukl.edu.my

Azami Zaharim Centre for Engineering Education,

Faculty of Engineering & Built Environment, 43600 UKM Bangi, Malaysia azami.zaharim@gmail.com

Norain Farhana Ahmad Fuaad, Zulkifli Mohd Nopiah The Fundamental Studies of Engineering Unit, Faculty of Engineering & Built Environment,

43600 UKM Bangi, Malaysia norain.farhana@gmail.com, zmn1993@gmail.com

Abstract— This paper explores the students’ performance on their basic engineering mathematics. It is essential for the students to meet the expectation of the engineering program in ensuring that the students be able to follow thru the whole program smoothly and achieved the expected learning outcome at the end of the program. Meaningful performance measurement is critical in identifying the performance of the students, and on the other hand identifies the weakness of the students. This study applies an entry test to new students for the engineering program in a local Malaysian higher learning institution. The test will enable students profiling according to their ability in mathematics, and would guide the lecturers to provide necessary remedial actions for the respective students. The Rasch measurement model is used to measure the students’ ability against the engineering mathematics on the logit ability difficulty ruler. Rasch measurement model converts the test result from the test into ration type data and sorts the ability according to Rasch-Guttman matrix. This is the basis of Rasch measurement model in measuring the ability of the students by considering the difficulty of all the items involved in the test, and not just based on raw scores of the test. The correlation between all the students and all the items are displayed on a person-item distribution map for easy reading.

Keywords— students’ performance; mathematics; entry test; engineering; Rasch model

I. INTRODUCTION The subject of mathematics is the basis for all engineering

programs. Engineers use mathematical models, such as sets of equations, to analyze the behavior of physical systems. Therefore it is essential to equip engineering students with basic knowledge of mathematics and statistics for them to be able to apply it in their field. This unit is set up to assist the faculty in planning and making decisions towards ensuring students managed proper and necessary mathematics background before and after embarking into the engineering program.

Mathematics is a key element supporting a large number of engineering courses and subsequently, it is vital for engineering students to embrace a strong mathematics knowledge that can keep their motivation for reasonable progress of their engineering programs. The objective of teaching mathematics to engineering students is to find the right balance between practical applications of mathematical equations and in-depth understanding of living situation [1]. On the other hand, the impact of teaching mathematical thinking skills on an engineer will enable them to use mathematics in their practice [2].

In 1995, the Engineering Council has published the results of difficulties in mathematical skills that undergraduate engineers faced in studying mathematics. They are having difficulties in achieving higher grade especially in mathematic courses. This explains the growing challenge of undergraduates being accepted for degree courses with relatively low mathematics qualifications. One of the main reasons is the lacking of fundamental knowledge especially in understanding the theories [3]. The decline of the student performance is futher worsen with the lacking in mathematical skills among the students. Skill requires practice and the right practice in turn requires strong determination from oneself together with the right and skillful educator.

In the effort of improving the performance of engineering mathematics among the students in the University Kebangsaan Malaysia, several mathematic courses in engineering mathematics were identified. One of the them is the Vector Calculus course which is taught for all Engineering students. As a first level engineering mathematics course, the content requires student to understand vector calculus and complex calculus at a level which enables them to discuss their applications in engineering. This course will help the students to be prepared with higher level of engineering mathematics along the way. Vector Calculus is a compulsory 4 credit hour subject for the first year engineering students.

978-1-4799-0086-2/13/$31.00 ©2013 IEEE

Hence, the purpose of this study is to measure the students’ mathematics’ achievement at the end of the program. This study used Rasch measurement model (RM) in analyzing Bloom’s Cognitive Separation for final exam question of Engineering Mathematics 1 (Vector Calculus) course. Ghulman et.al [4] mentioned that Rasch measurement is useful with its predictive feature to overcome missing data.

II. ISSUES The freshmen in the engineering program were selected

based on their high school examination results; however there is a need in ensuring that their knowledge in mathematics meets the expectation of the engineering programs at this university. Even though, the basic mathematics has been taught in high school, but more often than not, the students still find it difficult to be able to grasp the concept and the application of mathematics. The students will find it difficult to catch up with the syllabus and they might not be able to apply the mathematics foundation throughout the whole undergraduate program. This will affect their performance and the quality of the graduating students. The Fundamental Studies of Engineering Unit (UPAK) developed an entry mathematics test for the enrolling students and measure their ability in mathematics before segregating them into groups according to their mastery in basic engineering mathematics.

III. METHODOLOGY The test comprised of various topics in basic engineering

mathematics that is developed according to Bloom’s taxonomy level of mastery. The test is administered to the 214 engineering students during their first session of the first semester in the engineering program. The test consists of 18 items of basic engineering mathematics topics. At the end of the session, the results of the test will be tabulated and the students will be profiled according to their mastery level. The students will then be placed in separate classrooms for different treatment according to their weaknesses. Marks from each student for each question, or referred to as items, will then be tabled accordingly in a spreadsheet format, before running it in the Rasch measurement model software, Winsteps.

The purpose of applying Rasch model is to obtain measurement from categorical response data. In their raw form, categorical data cannot have mathematical operations; addition, summation, and even mean. Therefore meaningful inference cannot be made upon categorical data. It only allows for data summarization using mode and median in reporting of the data [5]. Rasch Model embodies certain criteria to be met in order to obtain measurement [6], [7], [8], [9] and that the data should fit the model [10] and not from the perspective of statistical modeling, model to fit the data.

IV. FINDINGS AND DISCUSSIONS The summary statistics reveal that the Cronbach alpha value

is given at 0.70 indicating acceptable internal consistency of the raw responses pattern. Apart from that RM provides Item reliability value of 0.98 to show the excellent item difficulty spread within the items (the test questions) used in the study [11], [12]. It provides an interpretation of the difficulty

measurement ruler that exists in this study in measuring the students’ ability on their mathematics performance. The maximum item or the highest location of the item on the logit ruler is located at +1.11 logit and the minimum or the lowest item on the ruler is located at -0.67 logit. The summary statistics of the items is shown in Table 1.

TABLE I. SUMMARY STATISTICS FOR ITEM

Subsequently, the summary statistics for person reveals a fair ability spread of the students involved in this study, at 0.73 [11], [12]. The maximum or the highest item on the difficulty logit ruler is at +0.18 logit while the minimum or the lowest item is located at -1.18 logit. The person separation index at 1.63 is able to segregate the students into roughly about two (2) groups; students with average performance and those with poor performance on their mathematics. The summary statistics of the 214 students is shown in Table 2.

TABLE II. SUMMARY STATISTICS FOR PERSON SUMMARY OF 214 MEASURED Person -------------------------------------------------------------------------------| TOTAL MODEL INFIT OUTFIT || SCORE COUNT MEASURE ERROR MNSQ ZSTD MNSQ ZSTD ||-----------------------------------------------------------------------------|| MEAN 52.7 16.0 -.34 .11 1.04 .1 1.13 .1 || S.D. 19.5 .3 .24 .02 .49 1.1 1.17 1.3 || MAX. 101.0 17.0 .18 .26 2.86 3.5 9.90 6.3 || MIN. 7.0 14.0 -1.18 .10 .30 -2.4 .23 -1.8 ||-----------------------------------------------------------------------------|| REAL RMSE .12 TRUE SD .20 SEPARATION 1.63 Person RELIABILITY .73 ||MODEL RMSE .11 TRUE SD .21 SEPARATION 1.86 Person RELIABILITY .78 || S.E. OF Person MEAN = .02 |-------------------------------------------------------------------------------

The dimensionality of the instrument or in this case the test is determined by the standard residual variance or the eigenvalue which indicates the ‘direction’ of the measurement. The eigenvalue provided upon running the data in Winsteps is at 58.5% that in more than minimum 40% mark. Therefore the instrument is within acceptable dimensionality and showed sufficient reliability in measuring the students’ performance in engineering mathematics concept.

V. STUDENT PROFILING Mean person measure is given at -0.34 logits, indicating

that in general this group of students are poor at their engineering mathematics. The Person-Item distribution map or commonly referred to as the Wright map as in Fig. 1, showed that the students’ ability measurement mapped their location just at the mean item of 0.00 logit. Mean item are set to 0.00 logit where Rasch theorized that a person have a chance of 50:50 in succeeding a given task. This serves as the virtual zero (0) for the measurement ruler. As the item gets harder, the odd of success is reduced; getting less chance to success. Therefore in the case of this test, the items are more difficult and the chance of success for the student is low [13].

Those students with average performance are located above the mean item, while the poor students are located below the mean item. The line that separates the two (2) groups of students is marked against the mean item value of 0.00 logit.

There are only seventeen (17) students who managed to have average performance on their engineering mathematical concepts. They are those students located above mean item 0.00 logit. They find that the test items are a little bit challenging to them. At the same time, they find particularly difficult for those items that have difficulty logit measures above than their ability logit measures. In other words, those items located above 0.00 logit on the measurement ruler are difficult for those average students to solve [14]. Refer to Table 3 for detail list of items with their respective logit measurement.

TABLE III. ITEM MEASURE TABLE -------------------------------------------------------------------------------------------|ENTRY TOTAL TOTAL MODEL| INFIT | OUTFIT |PT-MEASURE |EXACT MATCH| ||NUMBER SCORE COUNT MEASURE S.E. |MNSQ ZSTD|MNSQ ZSTD|CORR. EXP.| OBS% EXP%| Item ||------------------------------------+----------+----------+-----------+-----------+------|| 2 41 214 1.11 .12|3.22 4.9|2.07 2.9| .20 .12| 96.3 87.8| Q2 || 12 53 214 .96 .10|2.34 3.7|2.49 4.0| .10 .14| 90.7 83.1| Q12 || 7 189 214 .36 .05|1.11 .7| .71 -1.6| .41 .29| 40.2 24.1| Q7 || 14 94 101 .30 .07|1.09 .4|1.01 .1| .37 .32| 21.8 23.4| Q14 || 16 182 159 .28 .05| .80 -1.1| .69 -1.7| .40 .33| 24.5 18.1| Q16 || 1 259 214 .24 .04| .78 -1.5| .92 -.4| .29 .34| 16.8 18.8| Q1 || 11 403 214 .07 .03| .78 -1.9| .82 -1.3| .48 .41| 9.8 15.1| Q11 || 10 419 214 .05 .03|1.29 2.3|1.27 1.8| .41 .42| 10.3 14.8| Q10 || 17 379 165 .00 .03| .64 -3.3| .57 -3.4| .46 .42| 29.7 13.8| Q17 || 18 445 165 -.06 .03| .88 -1.1| .82 -1.4| .47 .44| 17.6 13.0| Q18 || 13 281 101 -.13 .04| .59 -3.2| .76 -1.4| .38 .50| 19.8 14.4| Q13 || 15 563 160 -.17 .03|1.56 4.7|1.55 3.9| .45 .51| 5.0 12.1| Q15 || 4 1161 214 -.41 .02|1.28 3.3|1.20 2.0| .62 .55| 2.8 9.8| Q4 || 6 1188 214 -.42 .02| .85 -1.9| .92 -.8| .44 .55| 15.0 10.0| Q6 || 9 1216 214 -.44 .02|1.36 4.1|1.37 3.5| .54 .55| 5.6 10.1| Q9 || 8 1264 214 -.46 .02| .58 -6.1| .56 -5.4| .67 .56| 15.0 10.0| Q8 || 5 1539 214 -.63 .03|1.10 1.1|1.36 2.8| .31 .55| 10.3 9.9| Q5 || 3 1604 214 -.67 .03|1.11 1.1| .95 -.4| .62 .54| 12.1 10.5| Q3 ||------------------------------------+----------+----------+-----------+-----------+------|| MEAN 626.7 189.9 .00 .04|1.19 .3|1.11 .2| | 24.6 22.2| || S.D. 523.5 37.9 .48 .03| .64 3.0| .50 2.6| | 25.9 22.8| |-------------------------------------------------------------------------------------------

There are six (6) items to be of difficult to the students with average performance.

The second group of students, those who are of poor performance, are located below the mean item 0.00 logit. Majority of the students are located here that is 180 from 214, that is about 84%, are within poor performance group. They not only found that six (6) items are difficult but including another three (3) of which have difficulty logit above 0.00. In total they found nine (9) items of difficult to be achieved.

On top of that, there are seventeen (17) students who are located below than the lowest items on the ruler measurement ruler. Those seventeen (17) students find that they have strong difficulties in completing all the items in the test. It is suggested that those students be placed under special program for them to better understand the engineering mathematics and then be able to continue with their engineering program.

VI. CONCLUSION Preliminary findings shown that the person and items

involved in the study have good ability and difficulty spread, with eigenvalue of the raw variance explained by measures is more than 40%. This indicates that the sample and the items are reliable in measuring the students’ ability in mathematics. This group of students found to face difficulties in engineering mathematics with their mean person at -0.34 logit measures. The person-item distribution map reveals the location of the 2 students’ group according to their respective ability logit

measures on mathematics. The groups are of those who achieved average performance and those who are poor in the subject of engineering mathematics.

There are also students needed to undergo special attention for them to be able to achieve the objective of the engineering mathematics. This is to ensure that they can continue their studies in the engineering program and would not face any difficulties in applying the concepts in their engineering field later.

This test managed to reveals the students’ performance and their ability logit measures which enable them to be group according to their performance. By applying RM, the study again, is able to identify which of the test items of difficulty and easy for the students. In other words, this can be used for preparing a proper teaching plan in making sure that the subject is being taught correctly in obtaining optimum performance.

Person - MAP - Item <more>|<rare> 2 + | | | | | | | | | | | | | Q2Appl | 1 + |T Q12Appl | | | | | | |S | | Q7Appl Q14Appl | Q1Appl Q16Appl . | .# T| .## | Q10Appl Q11Appl 0 .# +M Q17Appl ### | Q18Appl .###### S| Q13Appl ###### | Q15Appl .########## | .######## M| ######## | Q4Appl Q6Comp .###### |S Q8Appl Q9Comp .##### | .### S| Q5Comp ## | Q3Appl # | .# T| . | . |T -1 . + . | | . | | | | | | | | | | | | -2 + <less>|<frequ> EACH "#" IS 3. EACH "." IS 1 TO 2

Optional 2

+0.18 logit

-0.41 logit -0.42 logit

-0.34 logit

-0.67 logit

-1.18 logit

Average students

Poor students

0.00 logit

Fig. 1. The location of person and items on the Wright map.

ACKNOWLEDGEMENT

The authors wish to acknowledge the financial support received from the Centre for Engineering Education Research,

University Kebangsaan Malaysia as research grant (PTS-2012-091 and OUP-2012-126) in the effort to improve the quality of teaching and learning in engineering education.

REFERENCES

[1] Sazhin, S.S. 1998. Teaching Mathematics to Engineering Students, International Journal Engineering Education 14(2): 145-152.

[2] Cardella, M.E. 2008. Which Mathematics Should We Teach Engineering Students? An Empirically Grounded Case for a Broad Notion of Mathematical Thinking. Teaching Mathematics and Its Applications 27(3): 150-159.

[3] Sutherland. R., & Pozzi. S. (1995). The changing Mathematical Background of Undergraduate Engineers. Engineering Council London.

[4] Ghulman, H.A., & Masodi, M.S. 2009. Modern Measurement Paradigm in Engineering Education: Easier to read and better analysis using Rasch-based approach. 2009 International Conference on Engineering Education (ICEED 2009), December 7-8, 2009, Kuala Lumpur,Malaysia.

[5] Leedy, P.D. and J. E. Ormond, 2009. Practical Research: Planning and Design. 9th Edn., Pearson, ISBN-13: 9780137152421.

[6] Mok, M. and B. Wright, 2004.Overview of Rasch Model Families. In Intoduction to Rasch Measurement: Theory, Models, and Applications.Jam Press, Minnesota.

[7] Rasch, G., 1961. “On general laws and the meaning of measurement in psychology,” in Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, vol.IV, Berkeley, California: University of California Press, pp. 321-334.

[8] Bond, T.G. and C.M. Fox, 2007.Applying the Rasch Model: fundamental measurement in the Human Sciences. 2nd Edn., Lawrence Erlbaum, Mahwah, New Jersey

[9] Andrich, D., 1988. Rasch models for measurement. Sage Publications, Beverly Hills

[10]Andrich, D., 2004. Controversy and the Rasch model: a characteristic of incompatible paradigms.Medical Care, 42(1): 1-16, 2004.

[11] Linacre, J.M. (2006) A User’s Guide to Winsteps Rasch-Model Computer Programs.

[12] Fisher, W.P. Jr. (2007) Rating Scale Instrument Quality Criteria. Rasch Measurement Transactions, 21:1, pp. 1095.

[13] Wright, B.D. and Stone, M.H. (1979) Best test design Rasch Measurement. Chicago,IL: Mesa Press

[14] Lunz, M.E. (2010) Using The Very Useful Wright Map. Measurement Research Associates Test Insights. Downloaded from http://www.rasch.org/mra/mra-01-10.htm on 18 Feb 2013