ASEE CIEC ConferencePhoenix, AZ, February 6-10, 2013Dr. Norali Pernalete, Dr. Cordelia Ontiveros and Dr. Phil Rosenkrantz, California State Polytechnic University, PomonaThomas Cossio, The Boeing Company

Industry Involvement in Developing Soft

Skills for Students in the College of

Engineering at Cal Poly Pomona

Cal Poly Pomona

+

In Memoriam

Tom CossioDirector, Quality IntegrationThe Boeing Company1962-2013

Cal Poly Pomona

2

Cal Poly PomonaOne of the largest engineering programs in California

One of 24 California State University campuses

Over 5000 engineering students, seven departments, 11 undergraduate degrees, 5 masters degrees

Highly ranked among non-research universities

Tournament of Roses Parade float co-built each year with Cal Poly San Luis Obispo

Cal Poly Pomona

3

Outline of Presentation - 1

Senior project symposium used for assessment

Added Symposium Survey in 2008 to get higher level assessment data

First Year Results

Three year initiative for Communications Teamwork Problem Solving

Cal Poly Pomona

4

Outline of Presentation - 2

Second Year Results

Five Year Results and Trends Overall By department By evaluator classification

 Conclusion and Future Plans: What worked and what didn’t work Future plans & Ideas Broader Goals

Cal Poly Pomona

5

Projects Symposium Day

Each Spring approximately 75 industry friends attend Project Symposium Day and evaluate over 200 senior projects.

Historically each program uses their own rubrics and processes for evaluation and assessment.

In 2008 the College of Engineering Assessment Committee conducted a post-symposium survey of industry representatives to assess seven outcomes from a college-wide perspective.

Projects Symposium URL: http://www.csupomona.edu/~engineering/events/symposium/index.htm

Cal Poly Pomona

Excerpt from Typical Project Presentation Assessment Rubric

Cal Poly Pomona

7

Outcomes4 Exemplary

(Professional Level Quality)

3 Proficient (Good Quality for Graduating

Senior)

2 Marginal (Acceptable Level Quality for

Graduating Senior, but room for improvement)

1 Unacceptable (Quality level is not acceptable)

Ability to apply advanced material in the discipline (UD)

Clearly demonstrates highly professional understanding of material included in upper division courses.

Adequately demonstates understanding of upper division course material, but not necessarily at a fully professional level.

Display of professional level understanding of upper division material is somewhat uncertain or raises doubts in the listeners' minds about the authority of the presentation

Presentation fails to demonstrate that the speaker(s) understand or appropriately use knowledge of the discipline

Ability to identify problems and determine their root causes.

Clearly demonstrates a structured and appropriate problem solving approach was used for the project. Creative thinking and solid reasoning was clearly demonstrated.

Adequate approach to problem solving but may have lacked rigor or thoroughness in several aspects.

Problem solving methodology addressed and used, but approach not formulated as well as it could have been.

Problem solving approach or methodology missing or inadequate. Does not reflect a systematic approach to problem solving.

On-line survey sent via email to industry representatives that attended the symposium 43 responses in 2008 44 responses in 2009 48 responses in 2010 50 responses in 2011 32 responses in 2012

Approximately a 2/3 response

Project Symposium Survey

Cal Poly Pomona

8

SurveyMonkey Survey Designed and Sent to Symposium Attendees

Cal Poly Pomona

9

2008 Results - Number of Industry Representative Visits by Department

Aeros

pace

Chem

ical

Civil

Elec

trica

l & C

ompu

ter

Engi

neer

ing

Tech

nolo

gy

Mec

hani

cal

Indu

stria

l & M

anuf

actu

ring

0

2

4

6

8

10

12

14

Cal Poly Pomona

10

Qualities Rated on the Survey

Knowledge - Ability to apply knowledge of math, science, and engineering

Conduct Experiments - Ability to design and conduct experiments, as well as to analyze and interpret data

Design a System - Ability to design a system to meet desired needs

Multidisciplinary Teams - Ability to function on multi-disciplinary teams

Solve Engineering Problems - Ability to identify, formulate, and solve engineering problems

Communications - Ability to communicate effectively

Use Engineering Tools - Ability to use the techniques, skills, and modern engineering tools necessary for engineering practice

ABET a-e, g, k

Cal Poly Pomona

11

Average importance for engineering graduates? (5 = very important)

know

ledg

e

design

exp

erim

ents

design

multi-

disc

iplin

ary te

ams

solve

engine

ering

prob

lem

s

com

mun

icat

ions

use

engine

ering

tools

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

5.00

2008 Results

Cal Poly Pomona

12

Effectiveness of the Cal Poly Pomona educational experience?

knowledge design exper-iments

design multi-disci-plinary teams

solve engi-neering prob-

lems

communi-cations

use engineer-ing tools

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

5.00

2008 Results

Cal Poly Pomona

13

Gap Analysis Comparison(gap = effectiveness – importance)

ap

ply

kn

ow

led

ge

de

sign

exp

erim

en

ts

de

sign

a sy

stem

mu

lti-discip

lina

ry te

am

s

solv

e e

ng

ine

erin

g p

rob

-le

ms

com

mu

nica

te

use

en

gin

ee

ring

too

ls

-1.00

-0.50

0.00

0.50

2008 Gap

Cal Poly Pomona

14

Analysis

Gap analysis showed three areas needing greatest improvement: Communications Ability to solve

engineering problems Multidisciplinary

teamwork

College adopted a three year plan to address these three focus areas

Cal Poly Pomona

Three-Year College Plan2008-2009: Communications2009-2010: Multi-disciplinary Teamwork 2010-2011: Problem Solving

Annual Themes Adopted

in Fall 2008

Cal Poly Pomona

2008-2009: Communications

The ability to communicate orally, visually, and in written form

Includes critical thinking and comprehension skills as well as interpersonal communication skills

Leads up to the teamwork theme in 2009-2010.

Cal Poly Pomona

17

2009-2010: Multi-disciplinary Teamwork

The ability to perform as an effective team leader or team member in multidisciplinary situations.

This theme could align very well with the teamwork approaches found throughout the business world today and embodied in various quality management systems and strategies (e.g., Six Sigma Quality, AS 9100, ISO 9001:2004, etc.).

There are several “High Performance Team” models that could be considered and adopted under this theme.Cal Poly Pomona

18

2010-2011: Problem Solving

Almost every engineer will need problem solving skills.

These skills include: Problem identification Prioritizing Root cause analysis Problem diagnosis Creative and innovative problem solving Implementation

Cal Poly Pomona

19

Communications Program2008-2009

“Professor for a Day” panel discussion plus class speakers.

Engineering Futures presentation on communications by ΤβΠ National Office

Liberal Arts and Science faculty attended Senior Project Presentations. STEM Committee formed to work on joint projects.

Three workshops for faculty led by invited experts from inside and outside the COE.

Pilot project requiring presenters to provide a one-page abstract usable for immediate assessment of writing skills .

Pilot project using English M.A. students to coach students preparing presentations.

Cal Poly Pomona

Second Assessment Cycle 2009

Same assessment process followed at Project Symposium Day in 2009

Program changes: Added Showcase

presentations Change in makeup of

industry visitors

Changes may confound results

Cal Poly Pomona

Number of Industry Representative Visits by Department

Aeros

pace

Chem

ical

Civil

Elec

trical

& C

ompu

ter

Engi

neer

ing

Tech

nolo

gy

Mec

hani

cal

Indu

stria

l & M

anuf

actu

ring

0

2

4

6

8

10

12

14

16

18

08 Count09 Count

Cal Poly Pomona

22

Average importance for engineering graduates? (5 = very important)

know

ledg

e

desig

n ex

perim

ents

desig

n

mul

ti-di

scip

linar

y te

ams

solv

e en

gine

erin

g pr

oble

ms

com

mun

icat

ions

use

engi

neer

ing

tool

s1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

5.00

08 Q3 Avg09 Q3 Avg

Cal Poly Pomona

23

Analysis

Results show consistency in responses from year-to-year

Same three areas most important: Communications Engineering problem

solving Multidisciplinary

teamwork

Cal Poly Pomona

Effectiveness of the Cal Poly Pomona educational experience?

knowledge design exper-iments

design multi-disci-plinary teams

solve engineer-ing problems

communica-tions

use engineer-ing tools

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

5.00

08 Q4 Avg09 Q4 Avg

Cal Poly Pomona

25

Analysis

Averages dropped in four areas: Ability to design

experiments and analyze data

Ability to design a system Communications Ability to use engineering

tools

Averages increased in three areas: Knowledge Multi-disciplinary teams Problem solving

Communications dropped the most of all the outcomes assessed

Cal Poly Pomona

Gap Analysis Comparison

apply

know

ledge

desig

n e

xperim

ents

desig

n a

syste

m

multi-d

isciplin

ary

team

s

solv

e e

ngin

eerin

g p

roble

ms

com

munica

te

use

engin

eerin

g to

ols

-1.20

-1.00

-0.80

-0.60

-0.40

-0.20

0.00

0.20

08 Gap09 Gap

Cal Poly Pomona

27

Analysis

The gaps were smaller in all areas except: Ability to Design

Experiments and Analyze Data

Communications Ability to use engineering

tools

Three focus areas still have largest gap

Communications continues to be one of our areas in need of improvement

Cal Poly Pomona

Q6-How willing would you be to refer someone to the Cal Poly Pomona College of Engineering?

1 Not

Willing 2 3 4 5 6 7 8 9

10 V

ery W

illing

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

08 Response09 Response

Cal Poly Pomona

29

9-10: Promoters – (-12%) People who are selling your programs through word-of-mouth

7-8: Passives – (+10%) People who are satisfied, but have reservations about recommending the program to others. Usually not satisfied with one or more aspects.

1-6: Detractors – (+2%) People who are probably saying negative things about the program.

Cal Poly Pomona

Question 6 Implications30

Analysis

Slight decrease in scores using the scale given

Written comments show strong support and confirm concerns about communications

Overall analysis Communication skills are

growing in importance Valuable program but:

Did not reach enough students or faculty

Need to continue and expand

Cal Poly Pomona

2009-2010 ProgramMulti-disciplinary Teamwork

Initial Observations Team projects are used in many classes but very

few teamwork skills are taught Many students are learning negative teamwork

skills as the result of being on dysfunctional teams and witnessing traits like slacking or dominating.

Lack of resources for instructors and students Difficult to evaluate team member performance FYE is not enough. More reinforcement at

sophomore level.Cal Poly Pomona

32

2009-2010 ProgramMulti-disciplinary Teamwork

Develop a program with the following goals: Develop resources for faculty such as on-line

and face-to-face workshop and on-line resources that can be used in class

Provide resources for students Collaboration with other colleges to help our

students in GE and support courses

Results: Held one workshop for faculty One very well attended presentation by alumni

Cal Poly Pomona

33

2010-2011 ProgramProblem Solving

Faculty were too busy preparing for ABET to put in proper effort on this program

Ultimate goal is to prepare resources for faculty and students

Held one very well attended panel discussion by alumni for students

Resources developed but deployed on a limited basis

Cal Poly Pomona

34

Five Year Overall Results for Importance

Cal Poly Pomona

35

Apply knowledge

Conduct exp

eriments

Design a sy

stem

Teams

Solve problems

Communications

Use engineerin

g tools

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

5.00

20082009201020112012

Overall Five Year Results for Effectiveness

Cal Poly Pomona

36

Apply knowledge

Conduct exp

eriments

Design a sy

stem

Teams

Solve problems

Communications

Use engineerin

g tools

0.00

1.00

2.00

3.00

4.00

5.00

20082009201020112012

Overall Gap Analysis

Cal Poly Pomona

37

Appl

y kn

owle

dge

Cond

uct e

xper

imen

ts

Des

ign

a sy

stem

Team

s

Solv

e pr

oble

ms

Com

mun

icati

ons

Use

eng

inee

ring

tool

s

-1.00

-0.80

-0.60

-0.40

-0.20

0.00

0.20

0.40

20082009201020112012

Question 6 – Five Year Results How willing would you be to refer someone to the Cal Poly Pomona College of Engineering?

Cal Poly Pomona

38

1 Not

Willin

g 2 3 4 5 6 7 8 9

10 V

ery

Willin

g0%

10%

20%

30%

40%

50%

60%

20082009201020112012

Department Level and Position Level Results

Cal Poly Pomona 39

Industrial & Manufacturing Engineering Gap Analysis

Cal Poly Pomona

40

Appl

y kn

owle

dge

Cond

uct e

xper

imen

ts

Des

ign

a sy

stem

Team

s

Solv

e pr

oble

ms

Com

mun

icati

ons

Use

eng

inee

ring

tool

s

-1.40

-1.20

-1.00

-0.80

-0.60

-0.40

-0.20

0.00

0.20

0.40

0.60

20082009201020112012

Interpretation of Industrial & Manufacturing Engineering Results

Gap analysis was presented to the IME Department Industrial Advisory Council Members in November 2012.

They were asked if the result were valid. The unanimous response was that the results were loud and clear…

Communications and teamwork are important for industrial and manufacturing engineers and we need to do a better job.Cal Poly Pomona

Aerospace Engineering Gap Analysis

Cal Poly Pomona

42

Ap

ply

kn

ow

led

ge

Co

nd

uct

exp

eri

me

nts

De

sig

n a

syst

em

Te

am

s

So

lve

pro

ble

ms

Co

mm

un

ica

tio

ns

Use

en

gin

ee

rin

g t

oo

ls

-1.2

-1.0

-0.8

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6

20082009201020112012

Interpretation of Aerospace Engineering Results

Aerospace Engineering Presentations were not traditional senior projects—rather were class design projects which probably confounded the results

No noticeable improvement over time

Communications had the largest overall gap

First three outcomes were large gaps compared to other majors

Cal Poly Pomona

Chemical Engineering Gap Analysis

Cal Poly Pomona

44

Ap

ply

kn

ow

led

ge

Co

nd

uct

exp

eri

me

nts

De

sig

n a

syst

em

Te

am

s

So

lve

pro

ble

ms

Co

mm

un

ica

tio

ns

Use

en

gin

ee

rin

g t

oo

ls

-1.4

-1.2

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

20082009201020112012

Interpretation of Chemical Engineering Results

Problem Solving and Communications were the biggest gaps

All areas showed improvement over time

Probably some of the smaller gaps among all the departments

Cal Poly Pomona

Civil Engineering Gap Analysis

Cal Poly Pomona

46

Ap

ply

kn

ow

led

ge

Co

nd

uct

exp

eri

me

nts

De

sig

n a

syst

em

Te

am

s

So

lve

pro

ble

ms

Co

mm

un

ica

tio

ns

Use

en

gin

ee

rin

g t

oo

ls

-1.5

-1.0

-0.5

0.0

0.5

1.0

20082009201020112012

Interpretation of Civil Engineering Results

Based on Large Team Projects

Communications showed the biggest gap

2011 was an anomaly compared to other years

No real trends in the gap results over time

Civil was very proactive in improving communications -- need to question effectiveness of efforts

Cal Poly Pomona

Electrical & Computer Engineering Gap Analysis

Cal Poly Pomona

48

Ap

ply

kn

ow

led

ge

Co

nd

uct

exp

eri

me

nts

De

sig

n a

syst

em

Te

am

s

So

lve

pro

ble

ms

Co

mm

un

ica

tio

ns

Use

en

gin

ee

rin

g t

oo

ls

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

20082009201020112012

Interpretation of Electrical & Computer Engineering Results

Communications was the largest gap category

Program had some of the smallest gaps in the college

Mixed as to improving or regressing – two categories showed improvement over time and the rest were scattered

Cal Poly Pomona

Mechanical Engineering Gap Analysis

Cal Poly Pomona

50

Ap

ply

kn

ow

led

ge

Co

nd

uct

exp

eri

me

nts

De

sig

n a

syst

em

Te

am

s

So

lve

pro

ble

ms

Co

mm

un

ica

tio

ns

Use

en

gin

ee

rin

g t

oo

ls

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

20082009201020112012

Interpretation of Mechanical Engineering Results

Communications the largest gap area

Some areas seem to have small gaps

2012 showed gaps in areas that had been mostly even or free of gaps

Cal Poly Pomona

Engineering Technology Gap Analysis

Cal Poly Pomona

52

Ap

ply

kn

ow

led

ge

Co

nd

uct

exp

eri

me

nts

De

sig

n a

syst

em

Te

am

s

So

lve

pro

ble

ms

Co

mm

un

ica

tio

ns

Use

en

gin

ee

rin

g t

oo

ls

-1.4

-1.2

-1.0

-0.8

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6

20082009201020112012

Interpretation of Engineering Technology Results

Communications shows the largest gap

No real improvement trends over time

Cal Poly Pomona

Gap Analysis Based on Evaluator Position – Categorized by Evaluator

Cal Poly Pomona

54

En

gin

ee

r

Su

pe

rvis

or

Mid

dle

Up

pe

r

Exe

cuti

ve

Ow

ne

r

-1.5

-1

-0.5

0

0.5

1

Apply knowledge

Conduct experi-mentsDesign a system

Teams

Solve prob-lems

Communica-tions

Use en-gineering tools

Interpretation of Results by Position

Engineers were about half of the respondents overall and showed gaps in all areas with communications, problem solving and teamwork being the largest three

Supervisors were distinctly different than all other categories showing no gaps

Gaps became progressively larger as the position of the management evaluator went up in rank

Expectations vary based on perspective…this is worthy of interpretation by various stakeholders

Cal Poly Pomona

What Worked

Obtained data that was otherwise being ignored

Data was useful at the college and department level

Some Assessment Coordinators used the feedback to guide department improvement efforts for outcomes

Communicated with the College of Letters, Arts & Social Sciences (CLASS) at a meaningful level

Showcase and Alumni Presentations were very effective

Cal Poly Pomona

What Didn’t Work

Results did not reach faculty as effectively as needed for buy-in and continuous improvement

Low participation among faculty in communications workshops

We did not follow through effectively with CLASS faculty to sustain interaction and effect changes

Most departments did not review results with Department Advisory Councils to interpret the results and plan for action

Cal Poly Pomona

Future Plans

Continue assessment program.

Keep providing resources for three focus areas that will attract faculty buy-in. Expand reach of programs for both faculty and students with on-line resources that are reusable.

Tie into First Year Experience (FYE) programs

Strengthen collaboration with Liberal Arts and Science departments

Take a long view and be persistentCal Poly Pomona

58

Ideas for Improving Communications Program

Expand and increase seminars for faculty

Adopt a default “style guide” to use throughout the COE (Default guide would be used unless instructor or department specifies otherwise).

College-wide student competition judged by industry with prizes for the best presentation, written report, and research paper.

Expanded used of the Showcase in FYE courses.Cal Poly Pomona

59

Other IdeasChoose a book about communications to use throughout the curriculum (at the college or department level). Use in engineering classes to help make it real to the students.

Cal Poly Pomona

60

Broader Goals

Establish communications, teamwork, problem solving, and other “soft skill” outcomes threads across undergraduate curriculum: Multi-year Interdepartmental—across the college of

engineering Cross-disciplinary—across the campus

Requires collaboration with general education suppliers

Cal Poly Pomona

61