outcomes-based assessment of a basic engineering course (statics of rigid bodies)

Proceedings of the Scholarship and Innovation in Learning and Teaching (SILT) Symposium 2014 De La Salle University, Manila, Philippines

March 7, 2014

Outcomes-Based Assessment of a Basic Engineering Course (Statics of Rigid Bodies)

Andres Winston C. Oreta1* and Cheryl Lyne C. Roxas2

1 Professor, Civil Engineering Dept., De La Salle University 2 Asst. Professor, Civil Engineering Dept., De La Salle University

*Corresponding Author: andres.oreta@@dlsu.edu.ph

Abstract: This paper presents the implementation of the Outcomes-Based Education (OBE) framework in a basic undergraduate engineering courseEngineering Mechanics (Statics of Rigid Bodies)from syllabus design to course assessment. The revision of the syllabus using the OBE format was carried out and outcomes-based assessment is conducted to determine if the intended learning outcomes for this course using the revised syllabus are achieved. Two types of assessment methods were applied: direct and indirect methods to assess the attainment of the course learning outcomes. The direct method used the scores in the quizzes and final exams. The indirect method used an end-of-course evaluation by the faculty and students to determine the perception on the achievement of outcomes. Based on the outcomes-based assessment using direct methods, the achievement of the learning outcomes is barely satisfactory while the review of the course syllabus and teaching and learning activities is recommended. Key Words: Outcomes-Based Education, Assessment, Syllabus, Engineering Mechanics, Learning Outcomes

1. INTRODUCTION

Outcomes-Based Education is an educational model in which the curriculum and pedagogy and assessment are all focused on student learning outcomes (Driscoll & Wood 2007 p.4). Outcomes-Based Education is now accepted as a framework in the accreditation of Engineering Programs. The ABET in the US adopts its Engineering Criteria, which basically follows the OBE framework. Similarly, the Washington Accord, which recognizes substantial equivalence in the accreditation of qualifications in professional engineering for the member countries, also adopts similar criteria. As a result, various studies have been conducted by engineering educators on how to effectively implement the OBE framework in engineering schools. Felder and Brent (2003) gave an overview on ABET accreditation process and described the instructional techniques that should effectively prepare students to achieve the program

outcomes. Designing the learning outcomes of a course follows a hierarchy from top to bottom starting from the University Vision-Mission when the expected graduate attributes are defined, then to the Program level when the student or program outcomes are articulated, and down to the course level. The key to OBE is that the course learning outcomes are aligned with the program outcomes and the expected graduate attributes. Moreover, the course content, teaching and learning activities (TLAs), and assessment tasks (ATs) must address the intended course learning outcomes (LO). However, whether the course learning outcomes are achieved or not needs to be assessed.

Assessment methods have been applied by

various universities in the US to satisfy the ABET accreditation criteria. Rogers (2003) emphasized that grades in the courses cannot capture the achievement of the learning outcomes because various factors were included in the computation of the final grade. Specific outcomes-based assessment

Proceedings of the Scholarship and Innovation in Learning and Teaching (SILT) Symposium 2014 De La Salle University, Manila, Philippines

March 7, 2014

tools must be used to directly and indirectly measure the achievement of outcomes. Menhart (2011) discussed direct and indirect student assessment methods, used by the Department of Engineering Technology at the University of Arkansas at Little Rock (UALR). He presented an example for a Digital Systems Design course where exams, quizzes, and reports are used for direct assessment and an end-of-course evaluation was completed by the students for their self-assessment of achievement of outcomes. This self-assessment generates indirect assessment data, which complements the traditional direct assessment data. Gurocak (2009) presented an approach for assessment of course outcomes using direct measures. Specific homework and exams were mapped to the knowledge and skills described by the course outcomes. At the end of the semester, students, aside from receiving their usual letter grades, received a score on the scale of 1 to 5 for every course outcome indicating how well he/she achieved each outcome. The data (scores) coming from each course are used at the program level to assess the program outcomes. Based on these scores and the data, the faculty may make changes in the syllabus, teaching and learning activities, or even revise the learning outcomes to improve the course delivery, thus closing the loop.

This paper presents the results of a research

(Oreta & Roxas, 2013) on the implementation of the OBE framework in a basic undergraduate engineering courseEngineering Mechanics (Statics of Rigid Bodies or STATICS)from syllabus design to course assessment. The syllabus using the OBE format is implemented and outcomes-based assessment is conducted to determine if the intended learning outcomes for this course using the revised syllabus are achieved. Two types of assessment methods were applied: direct and indirect methods to assess the attainment of the course learning outcomes. The direct method used the scores in the quizzes and final exams. The indirect method that uses an end-of-course evaluation by the faculty and students is conducted to determine perceptions on the achievement of outcomes. A pilot study on OBE application at the course level will provide useful information to the faculty and the department on how OBE can be effectively implemented to improve learning and meet accreditation needs.

2. STATICS AND THE LEARNING OUTCOMES

Statics of Rigid Bodies (STATICS) is a

branch of Engineering Mechanics that deals with the

study of forces and interaction of forces that occur in rigid bodies that are in static equilibrium. It is a three unit course required for all engineering programs and a prerequisite and essential to structural or machine design courses. Various knowledge and skills are aimed to be achieved in this course. The key to course design is the formulation of the course learning outcomes. Powers (2008) provided guidelines and examples on how to write learning outcomes. The revised OBE syllabus of STATICS states three intended course learning outcomes (LO) as follows:

At the end of this course, the student must

be able to: LO1. Analyze the properties (components, resultants, and moments) of a force and force systems in 2D & 3D. LO2. Solve equilibrium problems of various types of structures including friction problems using analytical models, rigid bodies, FBDs, and equations of equilibrium. LO3. Solve the properties (centroid, center of gravity, and moment of inertia) of areas, lines, and volumes and apply these properties in equilibrium problems.

To realize the attainment of the outcomes,

the Constructive Alignment Principle (Biggs, 2003) is applied. In the learning plan of the STATICS Syllabus, the LOs being addressed by each lesson or class meeting is indicated and the various TLAs are listed for the instructors guidance. Instructors are expected to use both blackboard and multimedia in their lectures. Aside from traditional lectures, the instructor can design activities that will engage the students to construct learning. TLAs appropriate with the topics of the STATICS course (e.g. video showing, problem solving, lectures, etc.) are indicated in the syllabus to guide the instructor in the delivery of the course during the term.

The assessment tasks (ATs) used during the

term are also aligned with course learning outcomes to help students achieve the LOs. In the present STATICS syllabus, the traditional assessment methods employed are aligned with the LOs as shown in Table 1.

Table 1. Assessment Tasks in STATICS

Learning Outcome

Assessment Tasks

Date

LO1 LO2 Long Quiz #1 5th Week LO1 LO2 Long Quiz #2 8th Week LO1 LO3 Long Quiz #3 13th Week LO1 LO3 Final Exams 14th Week

The raw scores in the Long Quizzes and the

Proceedings of the Scholarship and Innovation in Learning and Teaching (SILT) Symposium 2014 De La Salle University, Manila, Philippines

March 7, 2014

Final Exam are the basis of the final grades in the course.

3. OUTCOMES-BASED ASSESSMENT

It must be noted that grades are not directly used in assessing the achievement of learning outcomes in the course and program level. Because many factors contribute to an assigned grade, it is almost impossible to make inferences about what a student knows or can do by only looking at the grades for a course (Rogers, 2003, p. 8).

There are two ways of assessing the

attainment of outcomes: direct and indirect methods. Direct assessment is based on an analysis of student behaviours or products in which they demonstrate how well they have mastered learning outcomes. Direct assessment methods use quizzes, exams, and reports to measure students performance. Indirect assessment, on the other hand, is based on an analysis of reported perceptions about student mastery of learning outcomes. Examples of indirect assessment methods are surveys, interviews, evaluations, questionnaire, and focus group.

3.1 Direct Assessment Methodology

Rogers (2003) emphasized that grades in the

courses can not capture the achievement of the learning outcomes because various factors were included in the computation of the final grade. However, Rogers stated that, the numeric score that is directly linked to students' performance on a specific performance criteria can be used as evidence of program learning outcomes (Rogers, 2003, p. 9). Hence, direct assessment may used student ratings for long quizzes and the final exam. For each long quiz, the questions or problems that address specific course learning outcomes are identified. Table 2 is a sample spreadsheet of the scores of a sample Long Quiz #1. In the sample Long Quiz No. 1, problem nos. 1 and 2 address LO1, while problem nos. 3 and 4 assess LO2. For each student, the total score for each outcome is obtained and when divided by the maximum score and multiplied by 5 will yield the LO score. In assigning an outcome score (1 to 5 scale) to a student, Gurocak (2008) used the following equation:

=

LOan for PointsMax LOan for Earned Points Total

5 score LO

(Eq. 1)

Hence, the LO1 score for student 1 (S1) is obtained as follows: LO1 Score = 5 x (20 + 15)/50 = 4. Table 2. Direct Assessment Sample Student LO1 (Max: 50 pts) LO2 (Max: 50 Pts)

P1 25%

P2 25%

LO1 Score

P3 30%

P4 20%

LO2 Score

S1 25 15 4 30 20 5 S2 10 5 1.5 20 15 3.5

Average 2.75 4.25 Throughout the term, the instructor keeps

track of the performance of each student on each LO by considering all assessment tasks that are aligned with each LO and the performance for each LO for the class at the end of the term is assessed. Gurocak (2008) suggested that if a course learning outcome scores less than or equal to 3.0, the instructor indicates that outcome and suggests either minor or major changes (actions) to be taken by the program. Gurocak (2008) stated that the major changes are things that would result in modifications of the master syllabus of the course such as adding, rewording or deleting a course outcome, or changes in the list of topics covered. Minor changes are things that can be implemented next time when the course is offered without altering its master syllabus. These could include additional lecture to be spent to cover a particular topic, a change in software, use of supplemental textbooks, etc (p. 5).

3.2 Indirect Assessment Methodology

Indirect assessment, in the form of an end-of-

course-evaluation, is conducted. The indirect assessment task is a questionnaire wherein the students answer questions related to the course, instructor, and learning outcomes using a Likert scale of 1 through 5, where 5 means strongly agree, 4 means agree, 3 means neutral, 2 means disagree, and 1 means strongly disagree. An end-of-course-evaluation almost similar in content with the students form was also prepared for the instructors to fill out. The end-of-course evaluation forms are completed by the students and the faculty during the final examination. The survey can be conducted online (using surveymonkey) or in the class.

Aside from complementing direct

assessment, indirect assessment serves also the purpose of triangulationusing more than one assessment method to measure attainment of learning outcomes. Aside from learning outcomes, questions related to the course, instructor, teaching,

Proceedings of the Scholarship and Innovation in Learning and Teaching (SILT) Symposium 2014 De La Salle University, Manila, Philippines

March 7, 2014

and learning activities were included in the evaluation form to capture more data that may be useful in improving the course design and delivery. Indirect assessment is useful when a negative or a relatively low rating is obtained in a specific question in the survey as this will guide the teacher on a possible weakness in the course design and/or delivery.

4. RESULTS AND DISCUSSION

The study was conducted during Term 1, AY2012-2013 when STATICS is offered as a regular course to majority of the engineering students. Five sections labeled as Sections U, B, D, G, and P under five Civil Engineering teachers were selected for the data gathering. Students enrolled in the various sections in STATICS are usually mixed from different engineering degree programs. The data for the quizzes were gathered from the respective professors. Copies of the questionnaire for each quiz were collected and the scores for each problem per quiz were recorded. The complete data can be obtained from the appendices of the research (Oreta & Roxas, 2013).

4.1 Direct Assessment Results

Table 3 shows the summary of the scores of the three learning outcomes using the quiz scores for direct assessment of the five sections. The average scores of the LOs for the five sections are: LO1 = 3.35, LO2 = 2.77, and LO3 = 3.38. If the cut-off value of 3.0 is used, LO1 and LO3 seems to be satisfactorily achieved and LO2 needs further review. However, further analysis of Tables 3 and 4 would present issues that needs to be addressed. Table 3. Direct Assessment Results (Long Quizzes) Section LO U B G D P Ave LO1 2.85 3.68 3.51 3.05 3.66 3.35 LO2 3.31 2.73 2.59 2.16 3.04 2.77 LO3 3.44 3.34 3.09 3.41 3.61 3.38 Some observations about the results are summarized below: (a) Table 3 shows that the LO1 score ranges from a

low of 2.85 to a high of 3.68. The LO2 score ranges from a low of 2.16 to a high of 3.31. The

LO3 score ranges from a low of 3.09 to a high of 3.61.

Table 4. Number of Problems (Total Points) in Long Quizzes Section LO U B G D P Ave LO1 3

(33) 1

(40) 3

(55) 1

(10) 2

(50) 2

(38) LO2 8

(144) 8

(250) 8

(165) 7

(210) 8

(200) 8

(194) LO3 2

(18) 1

(30) 2

(60) 1

(30) 1

(25) 1

(33) (b) Table 4 shows that the number of problems and

their corresponding maximum points for each LO. It is shown that the assigned number of problems and points to LO1 and LO3 for each section are relatively less compared to LO2. On the average, only three problems with 33 points were allotted for LO1 and only one problem with 33 points for LO3. The scores for LO1 and LO3 are higher in most cases compared to the scores for LO2 except for Section U. It can be observed that some teachers gave only one problem for a specific LO (e.g. LO1 and LO3 for Section B/D and LO3 for Section P), meaning only one assessment task was used for that LO. The reliability is in question when there is only one assessment task or problem used to assess an LO. Triangulation requires at least two assessment tasks or two problems in case of quizzes to effectively assess achievement of a learning outcome. Even if LO1 and LO3 have an average rating greater than 3.0, the rating is barely passing. The results from the final exams may be used to verify the achievement of these two LOs.

(c) LO2 is too broad. This is the reason why there are more problems (average of eight) and more points (average of 194 points) assigned. In fact, all problems in Quiz no. 2 address LO2 only. Moreover, the analysis of properties, specifically the components of forces, is usually incorporated in solving equilibrium problems. There is a need to review LO2 and probably break it into two learning outcomes to capture specific skills and knowledge.

Proceedings of the Scholarship and Innovation in Learning and Teaching (SILT) Symposium 2014 De La Salle University, Manila, Philippines

March 7, 2014

In order to show the distributional characteristics of the LO scores obtained in each STATICS section for the quizzes, box plots were generated. A box plot is a graphical way of displaying the distribution of data through the use of median, quartiles, and extreme values (Barette, Leech & Morgan 2005). In Figure 1, there is a clear indication of variability in the distribution of LO scores. It can also be observed that no faculty obtained the same median and distribution in any of the LO. This can be attributed to the differences in terms of the level of difficulty in the quiz items and also the time spent in discussing the topics. There is also an uneven size of the sections in the box plots, which indicate that students achieved the LOs in different scales, that is, some have the same scores in a certain part of the scale while others have varying scores in other parts of the scale. For example, in the LO1 of P, the scores are very similar towards the 3rd quartile. Notice that some box plots are shorter than the others, which indicate a high level of agreement in the scores. Long box plot specifies difference in the scores.

In the previous section, the scores of the long quizzes were used to directly assess the attainment of the learning outcomes. The scores in the long

quizzes are also affected by the type of problems (which vary in difficulty per section) and the point system used in checking each problem (which depends on the teachers criteria). To determine the achievement of the LOs for an assessment, which is the same for all sections, the final exam is used to verify the direct assessment using long quizzes. The final exam is a 50-item multiple-choice exam, which involves problem solving and objective questions. The questions related to each LO were identified and grouped to determine the LO Scores (using Equation 1). Table 5. Direct Assessment Results (Final Exam) Section LO U B G D P Ave LO1 2.47 2.20 2.54 2.19 2.79 2.44 LO2 2.58 2.30 2.82 2.43 2.85 2.60 LO3 2.19 2.60 2.48 1.75 2.27 2.26 Table 5 shows the average LO scores for each outcome per section. Compared to the LO scores based on Long Quizzes, the values are relatively lower especially for LO1 and LO3. Obviously, the assessment of LO1 and LO3 using quizzes must be reviewed. The scores for LO2 of 2.77 and 2.60 based on Quizzes and Final Exam, respectively, are close enough, which means that the LO2 assessment using quizzes is acceptable. Still, the LO scores are relatively low.

In order to show the distributional characteristics of the LO scores obtained in each STATICS section for the final exam, box plots were also generated. Figure 2 presents a box plot of the LO scores in the final exam per faculty. Same median but different distribution of scores can be observed for some faculty in each LO. For example, in LO1, G and P have the same median and distribution. Although U has the same median, its distribution of scores is higher than that of G and P. This explains the evident variability in the LO scores. Almost the same distribution of scores can be observed in LO2 and LO3, while difference among sections can be seen in the LO1 scores.

Figure 1. Box plot of LO scores in the long quizzes per faculty

Proceedings of the Scholarship and Innovation in Learning and Teaching (SILT) Symposium 2014 De La Salle University, Manila, Philippines

March 7, 2014

Comparing the box plots of quizzes and the

final exam, it can be deduced that there is an evident difference in the LO scores, with lower scores obtained in the final exam. Noticeable variability in the box plots is evident in the quizzes compared to the final exam. This can be explained by the differences in the long quizzes given by each faculty, unlike in the final exam wherein all STATICS students answered the same questions.

4.2 Indirect Assessment Results

The end-of-course evaluation was conducted during the final exams for both the faculty and students. Box No. 1 shows a section of the survey for the students.

Table 6 presents the results of the survey

evaluation by the faculty and students. The five faculty members rated his/her assessment of the class achievement of the LOs. The teachers perception on students achievement of LOs is high.

Table 6. Indirect Assessment Results: Faculty vs Students (in parenthesis) Section LO U B G D P Ave LO1 4

(4.07) 4

(3.82) 4

(3.94) 4

(3.67) 5

(3.95) 4.20

(3.89) LO2 4

(4.17) 4

(4.07) 4

(3.94) 4

(3.73) 5

(3.65) 4.20

(3.91) LO3 4

(4.31) 4

(3.53) 4

(3.69) 4

(3.87) 5

(3.51) 4.20

(3.78)

From a sample of 147 students from five sections, the assessment of the students on achievement of the LOs are shown as average scores (in parenthesis) in Table 6. The student perception of their achievement of learning outcomes is lower than that of their teacher in most cases except for Section U. However, the scores from indirect assessment are relatively higher than the scores from direct assessment. This is the same as the observation by Menhart (2011) that the indirect method results do not fully reflect the reality of what students can actually do in the course. It appears that some students had an inflated view of their own capabilities (p. 5).

Menhart (2011) also noted that positive results in indirect methods must be confirmed by direct assessment methods, because students may overestimate their abilities. Negative responses in indirect methods should be examined carefully, especially if the result represents a significant percentage of the class. This should be taken as an early warning sign to improve the course (p. 6). Aside from this, faculty members may also overestimate their teaching performance.

For the present study, both the faculty and students have a positive perception (agree to

Figure 2. Box plot of LO scores in the final exam f lt

Box No. 1: Survey on Learning Outcomes

For the Student: Rate your over-all assessment per learning outcome using the scale. (5. Strongly Agree 4. Agree 3. Neutral 2. Disagree 1. Strongly Disagree )

______ 1. LO1: I can correctly and completely analyze the properties of a force (components, resultants and moments) and force systems in 2D & 3D, in most cases.

______ 2. LO2: I can correctly and completely solve equilibrium problems of various types of structures including friction problems using analytical models, rigid bodies, FBD and equations of equilibrium, in most cases.

______ 3. LO3: I can correctly and completely solve the properties of sections (centroid, center of gravity and moment of inertia) represented as areas, lines and volumes and apply these properties in equilibrium problems, in most cases.

Proceedings of the Scholarship and Innovation in Learning and Teaching (SILT) Symposium 2014 De La Salle University, Manila, Philippines

March 7, 2014

strongly agree) on the achievement of learning outcomes. The positive perception of the students is a good indication of their satisfaction on the delivery of the course.

5. CONCLUSIONS From the results of outcomes-based assessment

using direct and indirect methods, the following recommendations are proposed to further improve the teaching and learning of STATICS. The course learning outcomes must be revised,

specifically, LO2 must be broken into two or more learning outcomes. This will address subtasks that rated low in the student assessment.

The number of problems per learning outcome must be standardized making sure that there are at least two problems per learning outcome to satisfy the concept of triangulation.

Teaching and learning activities and assessment task must be reviewed and revised.

More student-centered learning activities like problem solving, seatwork, and collaborative work must be adopted to assure mastery of the learning outcomes.

Standardized lecture notes and problem sets, as well as departmental long quizzes, can help reduce the variance among professors handling STATICS. Hence, a better analysis and a more reliable conclusion can be derived. These recommendations were incorporated in

the revised syllabus of STATICS, which was adopted in AY 2013-2014. Among the changes of the revised proposed syllabus are: The number of learning outcomes is increased

from three to five: o LO1. Identify and determine the components

and resultant of forces and force systems in 2D and 3D.

o LO2. Analyze effects of forces on rigid bodies in static equilibrium using free body diagrams and equations of equilibrium.

o LO3. Analyze the external and internal effects of forces on structures such as beams, trusses, frames, and simple machines.

o LO4. Analyze the effects of friction forces on rigid bodies in static equilibrium.

o LO5. Solve the properties (centroid, center of

gravity, and moment of inertia) of areas, lines, and volumes and apply these properties in equilibrium problems.

Problem Sets which addresses specific LOs are now required as assessment tasks: o Problem Set No. 1 (LO1, LO2) Resultants

and Equilibrium of Force Systems in 2D o Problem Set No. 2 (LO3) Analysis of

Trusses and Frames o Problem Set No. 3 (LO4, LO5) Friction,

Forces in 3D, Centroids (Optional) There is a recommendation on distribution of

problems for the long quizzes and problem sets to equally address the various LOs: o Long Quiz No. 1 At least two problems on

LO1 and two problems on LO2 o Long Quiz No. 2 At least three problems

on LO3 o Long Quiz No. 3 At least two problems on

LO4 (Friction) and two problems on LO5. One problem on forces in space (LO1/LO2).

Based on this study of applying outcomes-based assessment in the course level, the following recommendations are suggested to simplify the process: Indicate on the specific problem in the long

quizzes, problem sets, and even final exams which learning outcome is addressed.

For easy recording of scores for each LO, standardize the number of problems per LO as suggested in the proposed revised syllabus.

6. ACKNOWLEDGMENTS

This research was funded by the DLSU University Research Coordination Office (URCO).

7. REFERENCES Barette, K., Leech, N., Morgan, G., (2005). SPSS for

intermediate statistics: Use and interpretation (2nd ed.). New Jersey: Lawrence Earlbaum Associates, Inc.

Biggs, J. (2003). Aligning teaching and assessing to course objectives. Paper presented at the Teaching and Learning in Higher Education: New Trends and Innovations at the University of Aveiro, 13-17 April 2003.

Proceedings of the Scholarship and Innovation in Learning and Teaching (SILT) Symposium 2014 De La Salle University, Manila, Philippines

March 7, 2014

Driscoll, A. & Wood, S. (2007). Outcomes-based assessment for learner-centered education, Stylus Publishing, Inc.

Felder, R., & Brent, R. (2003). Designing and teaching courses to satisfy the ABET engineering criteria. Journal of Engineering Education, 92(1), 725.

Felder, Richard and Brent, Rebecca (2004). The ABCs of Engineering Education: ABET, Blooms Taxonomy, Cooperative Learning, and so on, Proc. 2004 American Society for Engineering Education Annual Conference & Exposition

Gurocak, H. (2009). Direct measures for course outcomes assessment for ABET Accreditation. Paper presented at the 2008 American Society for Engineering Education (ASEE) conference. Washington State University-Vancouver.

Menhart, S. (2011). Direct versus indirect assessment methodologies. Paper presented at the 2011 Midwest Section Conference of the American Society for Engineering Education (ASEE). Retrieved from http://www.asee.org/papers-and-publications/papers/section-proceedings/midwest/2011

Oreta, A. W. C., & Roxas, C. L. C. (2013). Outcomes-based assessment of engineering mechanics (STATICS) using direct and indirect methods. University Research Coordination Office, Project No. 47FU311. DLSU, Manila

Powers, J. (2008). Converting class syllabi to the outcomes based teaching and learning format. Communication Studies Department, Hong Kong Baptist University.

Rogers, G. (2003). Do grades make the grade for program assessment? Retrieved from www.abet.org