Program Outcome Assessment Summary Report 2011/2012 Academic Year

Department of Computer Science Eastern Washington University

TABLE OF CONTENTS

1.  Summary of Actions Taken in Response to AY10/11 Assessment ........................................................ 2 

2.  Summary of Program Outcome Assessment in AY11/12 ..................................................................... 3 

3.  Advancement Programming Exam Scores ............................................................................................ 5 

4.  Ethics Assessment ................................................................................................................................. 7 

4.1  Links between Program Outcomes and GenEd Objectives for PHIL 212 ...................................... 7 

4.2  PHIL 212 Assessment Data ............................................................................................................ 8 

5.  Senior Project Rubric Scores ................................................................................................................. 9 

5.1  Oral Communication ..................................................................................................................... 9 

5.2  Teamwork ................................................................................................................................... 10 

5.3  Technical Performance ............................................................................................................... 10 

6.  Course‐specific Outcome‐related Learning Objectives ...................................................................... 12 

7.  Master’s Oral Communication Rubric Scores ..................................................................................... 13 

8.  PHIL 212 Attachments ......................................................................................................................... 14 

 

2 

 

1. SummaryofActionsTakeninResponsetoAY10/11AssessmentFollowing is a summary of actions taken on recommendations from the AY10/11 assessment summary: 1. The Advancement Programming Guidelines were modified to include explicit statements that (a) the

APE is intended to test implementation skills, as opposed to design and analytical skills, and (b) test instructions should be kept as straightforward as possible.

2. The CS department Chair held preliminary discussions with the (now former) Director of the Philosophy program, who is also an instructor for PHIL 212, Introduction to Ethics. It was agreed that CS would investigate the possibilities of using the University’s General Education assessment for Phil 212. The results of this attempt are detailed in Section 4.

3. Technical Performance and Teamwork rubrics were requested of all Senior Project teams in AY11/12, in addition to the Oral Communication rubrics.

4. The Outcome-Related Learning Objectives (ORLOs) for CSCD 240 were revisited and revised in consultation with Dr. Clark and Mr. Steiner, the two primary instructors for that course.

5. Instructors were asked to ensure that grade scales are clear (possible points) when citing student performance in support of ORLOs.

6. Assessment of binary search trees was removed from CSCD 300. This subject is still assessed in CSCD 320.

7. ORLOs were written for CSCD 327, which replaced CSCD 425 as a required course for all majors.

 

3 

 

2. SummaryofProgramOutcomeAssessmentinAY11/12The following table summarizes the assessment methods applied to Program Outcomes during the 2011/2012 academic year. Details of the assessment data are provided in the sections that follow.

Program Outcome  Assessed Via a) Students will have the ability to use current techniques, skills, and tools necessary for computing practice.

APE, CSCD 240, CSCD 300, CSCD 320, CSCD 327, CSCD 330, CSCD 340, CSCD 350, CSCD 370

b) Students will recognize the need for, and will have the ability to engage in, continuing professional development.

Senior Project Oral Communication Rubric, CSCD 340

c) Students will have the ability to analyze the local and global impact of computing on individuals, organizations, and society.

d) Students will have the ability to communicate effectively with a range of audiences.

Senior Project Oral Communication Rubrics

e) Students will have an understanding of professional, ethical, legal, security, and social issues and responsibilities.

PHIL 212

f) Students will have the ability to function effectively on teams to accomplish a common goal.

Senior Project Teamwork Rubric

g) Students will have the ability to design, implement, and evaluate a computer‐based system, process, component, or program to meet desired needs.

APE, CSCD 260, CSCD 300, CSCD 320, CSCD 327, CSCD 330, CSCD 340, CSCD 350, Senior Project Technical Rubric

h) Students will have the ability to analyze a problem and identify and define the computing requirements appropriate to its solution.

APE, CSCD 240, CSCD 260, CSCD 300, CSCD 320, CSCD 350, Senior Project Technical Rubric

i) Students will have the ability to apply knowledge of computing and mathematics appropriate to the discipline.

APE, CSCD 240, CSCD 260, CSCD 300, CSCD 320, CSCD 330, Senior Project Technical Rubric

j) Students will have the ability to apply mathematical foundations, algorithmic principles, and computer science theory in the modeling and design of computer‐based systems in a way that demonstrates comprehension of the tradeoffs involved in design choices.

CSCD 300, CSCD 320, Senior Project Technical Rubric

k) Students will have the ability to apply design and development principles in the construction of software systems of varying complexity.

CSCD 350, Senior Project Technical Rubric

The following table summarizes assessment findings from each of the major assessment methods that were used in AY11/12. The ORLO category only lists courses where concerns were found. See Section 6 for a summary of the ORLO report findings from each course. Assessment Instrument Result of Assessment APE Exam No concerns. No recommendations. GECC Assessment of Phil 212

There is concern that the PHIL 212 Course Objectives are unable to help cover Program Outcome (c), which was covered by the course that Phil 212 has replaced in the curriculum. Also, while some of the Phil 212 Course Objectives may help with Program Outcomes (d) and (e), as

 

4 

 

detailed in section 4, the GECC schedule for assessment of these objectives is presently inadequate.

Senior Project Oral Presentation Rubric

No concerns. No recommendations.

Senior Project Teamwork Rubric

This rubric was not applied to all senior project teams. It is recommended that data from all senior project sections be collected during AY 12/13.

Senior Project Technical Performance Rubric

This rubric was not applied to all senior project teams. It is recommended that data from all senior project sections be collected during AY 12/13.

Course-specific Outcome-Related Learning Objectives (ORLOs)

All courses met objectives. No recommendations.

Masters Oral Presentation Rubric

No concerns. No recommendations.

The following recommendations are made as a result of AY 11/12 assessments. These recommendations will be reviewed by department faculty during Winter Quarter 2013: 1. It is recommended that the Assessment and Undergraduate Affairs committees collaborate to propose

a new course that will specifically address the societal impacts and ethical issues of computing. This proposal should be communicated to the Director of the Philosophy program, the Dean of the College of Science Health and Engineering, and the Dean of the College of Arts, Letters, and Education. The course proposal should contain assessable outcome-related learning objectives that support program outcomes (c), (d), and (e), with assessment of at least one category each year.

2. It is recommended that ORLOs be created for CSCD 110, 210, and 211, and that those ORLOs include learning objectives that specifically address program outcome (e). These ORLOs should also provide secondary coverage of the program outcomes covered by APE assessment.

3. It is recommended that the ORLOs for CSCD 340 modified to include learning objective(s) that address program outcome (c).

4. It is recommended that the ORLOs for CSCD 350 be modified address program outcomes (c) and (e). New instructor Dan Tappan should also review the entire set of CSCD 350 ORLOs.

5. The Teamwork and Technical Performance rubrics should be applied to all CSCD 490 project teams in the AY12/13 cycle.

6. The Assessment Plan should be updated to reflect curriculum changes: PHIL 212 (or a new course) replacement of CSCD 402, CSCD 327 replacement of CSCD 425, and the group of GUI courses replacing CSCD 370.

 

5 

 

3. AdvancementProgrammingExamScoresAPE scores support Program Outcomes (a), (g), (h), and (i). APE exams were administered during each quarter of the academic year plus summer 2011. An 80% is required to pass the exam. Exam section topics are:

(1) Linked List Manipulation 30% (2) Recursion 20% (3) Data Abstraction and Class Design 30%

(4) General Programming 20% Summary data for AY 11/12 and the preceding two years: Previous Year Last Year This Year Number of Exams Taken 109 144 138 Overall Average Score 75.1% 78.9% 79.0% Overall Pass Rate 57.8% 64.6% 63.0% Failures on 3+ Attempt 3.7% 2.1% 4.0%

Fig 3.1 shows subsection scores along with pass rates for this year alongside the previous two years.

AY 11/12 vs. AY 10/11 and AY 09/10 APE Scores by Subsection

73.8%70.4%

79.6%

74.1%75.3%75.1%

57.8%

84.0%84.2%

73.6%78.9%

64.6%

82.1%82.8%

71.0%

78.4%79.0%

63.0%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Pass Rate Ave Score Linked List Recursion Data Abstractopm General

AY 09/10

AY 10/11

AY 11/12

Fig. 3.1 – Summary APE Data by Exam Section

 

6 

 

The average overall exam score was 79.0%, essentially unchanged since last year and an improvement over the year before. The overall pass rate was slightly lower than last year. The proportion representing failures on a third attempt rose to 4%, which is similar to the level of the year before last. The scores and pass rates are acceptable. Subsection scores were reasonably balanced, with little change in average performance. Gains from last year in the General Programming area, which primarily covers File I/O and Exceptions, were maintained this year. In previous years Java core instructors have expressed a fundamental tradeoff between recursion and linked list performance, dependent on the class time devoted to each topic. This year’s data supports that, with a rise in linked list performance occurring alongside a corresponding drop in recursion performance. The performance in these two areas is reasonably well balanced, so no action is recommended at this time.

 

7 

 

4. EthicsAssessmentIn previous years CSCD 402, Computing Ethics, supported Program Outcomes (c), (d), and (e). The department was unable to continue delivering this course due to a rapid decrease in faculty resources occurring at the same time as a rapid increase in enrollments. As a consequence, the department chose to change the degree requirements to replace that course with PHIL 212, Introduction to Ethics. This is the first assessment cycle on the effects of that replacement. PHIL 212 also satisfies a List 3 Humanities and Fine Arts (HFA) requirement in the University’s general education structure, and is thus subject to University assessment of course goals and objectives for the HFA list. The Computer Science department agreed to analyze the possibility of using the University assessment of that course rather than ask for any special assessment during the first cycle.

4.1 LinksbetweenProgramOutcomesandGenEdObjectivesforPHIL212

Possible mappings between the CS Program Outcomes and General Education Goals and Objectives for HFA courses are: CS Program Outcome (c): Students will have the ability to analyze the local and global impact of computing on individuals, organizations, and society.

This program outcome was supported by the CSCD 402 course, but no Goal or Objective for the PHIL 212 course could be identified that specifically addresses the impact of Computing.

CS Program Outcome (d): Students will have the ability to communicate effectively with a range of audiences.

HFA Goal 3: Students will develop reading skills as well as creative and analytic writing skills appropriate to an introductory level in a discipline.

Objective 2: Students will produce written material consistent with introductory learning levels.

This course goal and objective supports the program outcome with respect to writing skills. Oral communication skills are assessed elsewhere in the Computer Science curriculum (see section 5.1).

CS Program Outcome (e): Students will have an understanding of professional, ethical, legal, security, and social issues and responsibilities.

HFA Goal 1: Students will employ the basic forms of criticism and expression in philosophy, literature, and/or at least one of the arts, and they will understand the social and environmental contexts of the artists and/or writers.

This course goal provides a partial, and somewhat weak, link to the program outcome in the sense that it indicates that social context of the (ethics) philosophers is discussed in the course. It does not specifically cover professional, legal, and security aspects of the program outcome.

HFA Goal 2: Students will understand the methods for interpretation and criticism appropriate to the humanities and fine arts.

 

8 

 

Objective 2: Students will draw competent interpretations based on given reasons and evidence. Objective 3: Students will demonstrate that differing interpretations of the same information are possible and that different conclusions may be reached using the same evidence.

These course objectives support the program outcome in the area of analysis of different ethical models and arguments.

4.2 PHIL212AssessmentData

Two sections of PHIL 212 were assessed according to University general education assessment requirements during Winter quarter 2012. One of these sections was an online course. Both reports (attached) assessed a single HFA objective:

HFA Goal 1: Students will employ the basic forms of criticism and expression in philosophy, literature, and/or at least one of the arts, and they will understand the social and environmental contexts of the artists and/or writers.

As discussed above, this assessment provides partial support of program outcome (e). Some important conclusions of the two instructors from the attached reports are: “… students are decreasingly well prepared to handle the textual analysis of difficult material that this course relies upon.” “ … while the majority of the students (30 out of 39) understood the material well enough to score around or above the 70% mark, there is still more clustering than I would like to see at the top and the bottom of the grade scale.” The assessments from Phil 212 indicate mixed results in support of program outcome (e). The instructor providing the first quote above has also made some personal comments to the CS department chair that he has enjoyed having Computer Science students in his classes. Taken in combination with the quote from the second instructor, we have to wonder if it wouldn’t be useful to ask for a separate assessment of CS students in this course. While there exist some reasonable possibilities linking course objectives to program outcomes (d) and (e), there is no HFA goal or objective that appears to support program outcome (c), which was previously covered by the department’s CSCD 402, Computing Ethics course. The department should therefore identify another assessment opportunity for program outcome (c). An additional difficulty with using the University’s assessment of HFA goals and objectives is that the University’s assessment cycle for HFA has yet to be well-defined. The responsible university committee (GECC) has been discussing ways in which to assess all general education objectives within a single NWCCU accreditation cycle, but at present (AY 12/13), only a single general education category is being asked to assess a single objective each year. It does not appear that there will be any assessment of HFA courses during AY 12/13.

 

9 

 

5. SeniorProjectRubricScoresOral Communication rubrics were applied to senior project teams during Fall, Winter, Spring, and Summer, in sections of both CSCD 488 and CSCD 490. Teamwork and Technical Performance rubrics were applied to the work of CSCD 490 senior project teams during Winter and Summer. These assessments are tied to Program Outcomes as follows:

Oral Communication Rubric Scores: (b), (d) Teamwork Rubric Scores : (f) Technical Rubric Scores: (g), (h), (i), (j), (k)

5.1 OralCommunication

Fig 5.1 shows the average scores for presentations from the Fall and Spring offerings of CSCD 488, and the Winter, Spring, and Summer offerings of CSCD 490. The Professional Development Awareness score relates to Program Outcome (d), and the other scores relate to Program Outcome (b).

Capstone Oral Communication

3.40 3.47 3.48 3.47 3.50 3.493.40

3.253.253.303.34

3.623.503.49

3.26

3.66

3.36

3.69

3.473.403.41

3.64

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

Organiz

ation &

Flow

Response

to Q

uesti

ons

Visuals

Time M

anagement

Engaging A

udience

Mechanics

Eye C

onta

ct

Elocutio

n

Prese

ntation

Prof D

ev Aware

ness

Avera

ge Sco

re

AY10/11 (N=13)

AY11/12 (N=24)

Fig. 5.1 – Project Oral Communication Rubric Scores

 

10 

 

The scores indicate that students’ are performing well against these learning objectives, and there appears to be some improvement over the previous year in a couple of categories. No action based on Oral Communication data is recommended.

5.2 Teamwork

Fig 5.2 shows the average scores for the teamwork performance of 9 senior project teams from the Winter and Summer offerings of CSCD 490. These scores relate to Program Outcome (f).

Teamwork Performance

3.50

3.20

3.50

3.70

3.53 3.49

3.64

3.42

3.623.50

3.61 3.56

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

Involvement Division of Effort Communication Group Dynamics Troubleshooting Average Score

AY 10/11 (N=4)

AY 11/12 (N=9)

 Fig. 5.2 – Project Teamwork Rubric Scores

The scores indicate that students’ are performing well against these learning objectives. No specific action based on Team performance data is recommended.

It is apparent, however, that this rubric was not applied to all senior project teams. It is recommended that data from all senior project sections be collected during AY 12/13.

5.3 TechnicalPerformance

Fig 5.3 shows the average scores for the technical performance of 9 senior project teams from the Winter and Summer offerings of CSCD 490. These scores relate to Program Outcomes (g), (h), (i), (j), (k).

 

11 

 

Capstone Technical Performance

3.55 3.503.71

3.53 3.623.37

3.23

3.503.553.583.65 3.563.523.633.593.79

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

Defini

tion

Analys

is

Desig

n

Imple

men

tatio

n

Perfo

rman

ce

Plannin

g

Docum

enta

tion

Avera

ge

AY 10/11 (N=4)

AY 11/12 (N=9)

Fig. 5.3 – Project Technical Performance Rubric Scores

The scores indicate that students’ are performing well against these learning objectives. No specific action based on Team performance data is recommended.

It is apparent, however, that this rubric was not applied to all senior project teams. It is recommended that data from all senior project sections be collected during AY 12/13.

 

12 

 

6. Course‐specificOutcome‐relatedLearningObjectivesCourse-specific assessments were conducted for the following courses. Here we summarize the findings for each course. Full reports are available in the /faculty_reposit/ABET/ directory for the course.

Course Term Outcomes Result of Assessment CSCD 240

Spring 12

(a), (h), (i) Objectives met.

CSCD 260

Fall 11 (g), (h), (i) Objectives met.

CSCD 300

Fall 11 (a),(g),(h),(i),(j) Objectives met, including some of the binary tree objectives that have been removed from the ORLO. Additional time dedicated to debugging strategies as a result of conversion to 5 hours per week seems to have helped as instructor is experiencing fewer office visits.

CSCD 320

Fall 11 (a),(g),(h),(i),(j) Objectives met, although instructor intends to add examples and perhaps slow down the pace in an attempt to improve student weaknesses in analysis and algorithm design.

CSCD 327

Winter 12

(a),(g) Objectives met. Instructor plans to provide more examples and in-class exercises to help students better digest normalization concepts.

CSCD 330

Spring 12

(a),(g),(i) Objectives met.

CSCD 340

Fall 11 (a),(b),(g) Objectives met.

CSCD 350

Spring 12

(a),(g),(h),(k) Objectives were met. Instructor has difficulty measuring student abilities with use cases in this course and believes the ORLOs should be reviewed with new Software Engineering faculty member that is starting in Fall 12.

CSCD 370

Winter 11

(a) Objectives met.

 

13 

 

7. Master’sOralCommunicationRubricScoresThe Oral Communication rubric was also applied at the defense of 7 Master’s degree candidates during AY 11/12. The rubric was applied by audience members present at the exam. The results are summarized in Fig. 7.1.

3.75 3.77 3.713.83 3.76

3.98

3.773.88

3.75 3.78

3.573.67

3.453.57 3.57

3.81

3.593.74

3.54 3.53

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

Organization & Flow

Response to Questions

Visuals Time Management

Engaging Audience

Mechanics Eye Contact Elocution Presentation Prof Dev Awareness

Master's Oral Communication AY10/11 (N=7)

AY11/12 (N=7)

Fig. 7.1 – Master’s degree Oral Communication Rubric Scores

Although not generally as high as the preceding year, all of the scores are acceptable. No action based on Master’s Oral Communication scores is recommended.

 

14 

 

8. PHIL212Attachments

 

15 

 

 

16 

 

 

17 

 

 

18 

 

 

19 

 

 

20 

 

 

21 

 

 

22 

 

 

23