instructional requirements for practitioners of decision ... · –what subordinate skills are...

Instructional Requirements for Practitioners of Decision

Analysis: A Descriptive Study

Richard M. Justice, Jr.Decision Sciences

Lilly Research Laboratories

Decision Analysis Affinity GroupMay 17 - 19, 2000Calgary, Alberta Canada

DAAG 2000Company ConfidentialCopyright © 2000 Eli Lilly and Company

2

Abstract

The purpose of the present research is a descriptive study

of the skills that are needed by practitioners of decision

analysis. The study is based upon a survey of the literature

and on a pilot-study of practitioners of decision analysis

using an internet-based survey.

Special thanks to Matt Vanauwelaer, without whom a

timely and professional internet tool would not have been

possible.


3

Decision Analysis with Supertree, 2nd Ed. McNamee and Celona

"The real contribution and challenge of decision

analysis occur at the much more general level

of defining the problem and identifying the true

decision points and alternatives. Many decision

analyses never reach the point of requiring

a decision tree."


4

Human Performance: ADDIE Model

Analysis Design Develop

ImplementEvaluate


5

Needs AnalysisSystematic process for:

•Identifying problems/opportunities– Human resource practices– Organizational structure– Management practices– Personnel development– Personnel support

•Prioritizing needs

•Making decisions on how to address the needs


6

Closing the personnel development gap: Front-End Analysis

•Identify instructional goals– What do you want the student to be able to do?

•Conduct an instructional analysis– What subordinate skills are required of the student?

•Identify entry behaviors– What must the student be able to do in order to begin?

•Write performance objectives– What will the learner be able to do when they complete the instruction?


7

Closing the personnel development gap: Design & Develop

•Design criterion-referenced test items– How will you measure the learner's ability to achieve the learning goals?

•Design the instructional strategy– How will you include pre-instructional activities, presentation of

information, practice and feedback, testing, and follow-through activities?

•Develop and/or select instruction– How will you produce/select the learner's manual, instructional

materials, tests and instructor's guide?


8

Closing the personnel development gap: Implement and Evaluate

•Deliver the instruction– Will you use computer-based instruction, self-study, or lecture?

•Conduct the formative evaluation– How will you collect information that are useful in improving the

instruction: 1:1 evaluation, small group evaluation, field evaluation?

•Revise instruction– Based on the formative evaluation, how will you improve the instruction?

•Conduct summative evaluation– Were your instruction efforts worth the investment?


9

Human performance from a DA perspective

H. R. Practices

Org. Structure

PersonnelDevelopment

ManagementPractices

PersonnelSupport

EffectiveAnalyst

Personal

Job-Related

skills

Interpersonal

Skills

Goals

Behaviors

Et cetera


10

Method•Convenience sample of practitioners

– 109 invitations to participate, 40 (37%) surveys returned– anonymous responses

•Survey– internet-based– six demographic questions– 30 technical skills questions– 18 consulting skills questions– 3 open-ended questions at the conclusion– Likert scale response to:

– current importance– future importance– frequency of use

•Responses– transferred to Microsoft Excel (v. 9.0) for descriptive statistical analysis


11

Demographics

0

5

10

15

20

25

30

35

40

45

University On-The-Job

No.

of R

espo

nses

Number of Respondents and Location ofInstruction in DA & Consulting

D.A.

Consulting


12

Demographics

0

2

4

6

8

10

12

14

16

18

MBA Engineering Physical Science Mngt.Science Decision Science

No.

of R

espo

nses

Number of Respondents andEducation Degree


13

Demographics

0

2

4

6

8

10

12

14

16

BA/BS MS MBA EdD/PhD

No.

of R

espo

nses

Number of Respondents andDegree


14

Demographics

0

2

4

6

8

10

12

14

Indust/ Manuf Other Utility/Commun Consult/Contract Health Services Education

No.

of R

espo

nses

Number of Respondents andPlace of Employment


15

Demographics

0

1

2

3

4

5

6

7

8

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

No. of Years in DA

Freq

uenc

y of

Res

pons

e

Frequency of Response andYears in D. A.


16

Demographics

0

5

10

15

20

25

DDP D&RA SIM IDM AHP

Method

Freq

uenc

y

Frequency of Response andD. A. Method


17

Results: Current, Future, Frequency

0.0

1.0

2.0

3.0

4.0

5.0

6.01 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101

106

111

116

121

126

131

136

141

Question Number

Aver

age

Res

pons

e

Average Value of Response andQuestion Number


18

Results: Current Importance

0.0

1.0

2.0

3.0

4.0

5.0

6.0

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47

Question Number

Aver

age

Res

pons

e



19

Results: Future Importance

0.0

1.0

2.0

3.0

4.0

5.0

6.01 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47

Question Number

Aver

age

Res

pons

e



20

Results: Frequency of Use

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.01 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47

Question Number

Aver

age

Res

pons

e



21

Results: Consistency Comparison

48362412

47352311

46342210

4533219

4432208

4331197

4230186

4129175

4028164

3927153

3826142

3725131

FreqFutNowQFreqFutNowQFreqFutNowQFreqFutNowQ

48362412

47352311

46342210

4533219

4432208

4331197

4230186

4129175

4028164

3927153

3826142

3725131

FreqFutNowQFreqFutNowQFreqFutNowQFreqFutNowQ

Highest Importance, Highest Frequency

Lowest Importance, Lowest Frequency

Highest Importance, Highest Frequency

Lowest Importance, Lowest Frequency


22

Implications: Instructional SequenceGroup 1:

Technical Skill Consulting SkillDistill key insights. Being supportive.Use of spreadsheet software. Listening.Recognize proper tool for task. Group or team interaction.Developing an appropriate decision frame. Written communication competency.Conducting a sensitivity analysis. Small group communication.Determining expected value. Knowledge of decision analysis jargon.Developing alternatives.

Group 2:

Technical Skill Consulting SkillKnowledge of modeling/analysis software. Assertiveness.Understanding probability theory. Confrontation.Ability to assess probabilities. Dealing with mixed motivations.Ability to de-bias experts. Dealing with bias.Construct appropriate decision diagrams. Dealing with political climates.Construct issue hierarchy. Large group presentations.Identify useful distinctions. Tolerance for ambiguity.Recognize when decision analysis is needed. Team leadership.Understanding probabilistic dependencies. Project management skills.Understanding/using decision hierarchies. Identify level of commitment to action.


23

Implications: Instructional SequenceGroup 3:

Technical Skills Consulting SkillsAssess value of perfect information. Negotiating.Understanding Real Options. Systems thinking.Translate decision diagram to decision tree.Construct and use strategy tables.Understand and use Monte Carlo simulation.

Group 4:

Technical Skills Consulting SkillsUnderstanding network systems w/ feedback.Ability to use clairvoyance test.Ability to assess risk attitude.Use of analytical hierarchy software.Determining certain equivalent.Determining utility functions.Linear algebra techniques/concepts.Using multi-attribute utility models.


24

Additional Implications

•Great opportunity for university/industrial training to identify/design/develop instruction for DA and Consulting skills

•Opportunity for designing individual training plan

•Overwhelming use of one decision paradigm suggests an opportunity for expanding the analyst's toolbox to include other models

•"Frequency of skill use" may have implications on the organizational structure of decision groups

•Translating diagram into tree is in group 3. Is it because software does it or were McNamee/Celona correct?


25

Open-ended Comments - Current DA•"I am actively promoting the use of DA in the company. It is not widely used. Suggestions on how to get more people to accept DA as a useful tool would be much appreciated."

•"Our work sometimes involves OR optimization models. Sometimes we are needed for our computer skills. We assist in determining Research Strategies which we can then model."

•"Support of actual software tools and models, providing useful tips, avoiding problems."

•"Herding Cats!!!!"


26

Open-ended Comments - Future DA

•"I believe that finance and DA will continue to learn from each other and create powerful valuation/analytic tools. Real options appears to be moving in that direction."

•"Financial/Venture analysis/Deal negotiating."

•"Aligning DA with other Decision-Making-Tools in the company."

•"I believe a practical blend of DA and system dynamics would be a very potent blend."


27

Open-ended Comments - Future DA

•"Value of imperfect information."

•"Internet capabilities."

•"Scenario Planning will become increasingly important."


28

Future Direction for Study

•Pilot is now complete, roll-out to larger audience

•Conduct correlation studies with demographics and survey responses

•Identify/design/develop instructional modules

instructional requirements for practitioners of decision ... · –what subordinate skills are...

Documents