Design Evaluation Methods

Chr i stopher Sa ldana , Ph .D.W o o d r u f f S c h o o l o f M e c h a n i c a l E n g i n e e r i n gG e o r g i a I n s t i t u t e o f Te c h n o l o g yA t l a n t a , G e o r g i a U S A

2

Understand the importance of evaluation in the design process

Identify criteria for evaluation

Utilize three levels of evaluation matrices for product development through concept screening and scoring

Strategies for Visual/Descriptive Communication of Designs

Learning Objectives

3

Design Tools – Current Progress

House of Quality Specification Sheet Function Tree

Morphological ChartConcept Generation

Place Masson Target

Moveto

Target

Navigateto

Target

Brake onTarget

GeneratePower

TransmitPower

HitTarget

Generate Power

Transmit Power

Brake on Target

Move to Target

Navigate to Target

Gravity Mouse Traps

Car Hit by Trap Rip Cord Effect Ramp Catapult

Friction String Break Anchor Rubber Stopper Weighted Skid

Equal Size Wheels Larger Front Wheels

Rolling Sliding Projectile Launch

Problem Understanding

Design Alternatives

3

4

Customer Wants1. Dose metering accuracy2. Portability3. Durability4. Ease of handling5. Readability of settings6. Ease of use7. Ease of manufacture

Example: Insulin PenRedesigning the Insulin Pen

4

5

Example: Insulin PenRedesigning the Insulin Pen

5

6

Example: Insulin PenRedesigning the Insulin Pen

6

7

Benefits• Customer-focused product, competitive design• Better product-process coordination, faster product introduction• Effective group decision-making• Documentation of design process

Challenge• Need to make informed decisions despite lack of information• Selection requires estimation, analysis, prototyping• Identify bad concepts versus picking optimal ones

Structured Evaluation

8

How do we evaluate each of these designs?

What criteria do we use for evaluation?

What is the best design?

Structured Evaluation

9

Multi-voting: team members vote independently, work together to resolve differences and/or average results

Strengths/Weaknesses: list strengths and weaknesses of design concepts, use this to evaluate based on specific opinions

Prototype and Test: build or simulate, use empirical or simulated test data!

Concept Selection Strategies

10

Concept Selection Process

11

Concept screening

• First-level evaluation matrix

Concept scoring

• Second-level evaluation matrix

• Third-level evaluation matrix

Stages and Types of Concept Selection

12

1. Identify the criteria for comparison. 2. Select the alternatives to be compared.

• Alternatives are developed during concept generation. • All concepts should be compared at e same level of abstraction.

3. Generate scores. • Use a design concept as datum, with all the other being compared to it • Evaluate each alternative as better (+), same (S), or worse (-) relative to datum.

4. Compute the total score • Sum the total number of (+)’s, (-)’s, (S)’s • Compute overall score with +1 for (+)’s, -1 for (-)’s, 0 for (S)’s

5. Note: other variations on scoring in the first-level evaluation• Optional scale: +3 if extremely better than datum +2, +1, 0, -1, -2, -3

First Level Evaluation (Pugh Matrix)

13

+ = better than datum; - = worse than datum; S = same as datum

First Level Evaluation (Pugh Matrix)

14

First Level Evaluation Matrix (example)

+ = better than datum; - = worse than datum; S = same as datum

15

• Ranking depends on choice of datum

• Does not factor in how much better a specific alternative is compared to others

• Some criteria may be more important

• Consider: (i) no datum, (ii) numerical rating, (iii) criteria weighting• Second-level evaluation matrix – (i), (ii)

• Third-level evaluation matrix – (i), (ii) and (iii)

First-Level Evaluation Matrix

16

Second Level Evaluation Matrix Common Scale:4 = very good (ideal)3 = good2 = adequate1 = just tolerable0 = unsatisfactory

Alternate Scale:10 = ideal solution9 = solution exceeds requirement8 = very good solution7 = good solution6 = good solution with drawbacks5 = satisfactory solution4 = adequate solution3 = tolerable solution2 = weak solution1 = very inadequate solution0 = useless solution

17

Second Level Evaluation Matrix

18

Third Level Evaluation Matrix

19

Third Level Evaluation Matrix

20

Summary - Evaluation MatrixElements• Designs rated relative to customer

requirements in HOQ

• Level 1: sum +/-/S relative to datum

• Level 2: numerical rating

• Level 3: weighted sum of num. rating

Describing this figure in text• Which design performed best? Why?

• Which performed worst? Why?

• Use numerical information from figure

21

• Not doing it • Running with the first idea • Forgetting the customer • Evaluation matrix doesn't correspond to HOQ • Letting an "experienced" designer make the choices • Going by gut feel • Letting a manager decide • Not buying into the process as a team • Ignoring cost

Concept Selection Pitfalls

22

Design Tools – Complete Process

House of Quality Specification Sheet Function Tree

Morphological Chart Evaluation MatricesConcept Generation Concept Selection

Place Masson Target

Moveto

Target

Navigateto

Target

Brake onTarget

GeneratePower

TransmitPower

HitTarget

Generate Power

Transmit Power

Brake on Target

Move to Target

Navigate to Target

Gravity Mouse Traps

Car Hit by Trap Rip Cord Effect Ramp Catapult

Friction String Break Anchor Rubber Stopper Weighted Skid

Equal Size Wheels Larger Front Wheels

Rolling Sliding Projectile Launch

Problem Understanding

Design Alts. Final Design

23

Detailed Design - Communication23

General organization• Primary systems and subsystems

• Mechanisms, operation/sequencing, construction and materials

• Performance relative to specificationsClarity in written descriptions• Be clear in describing design features. Match

words in the body to label text in figures.• Avoid describing things that are not shown with

evidence or detail. Don’t rely on the reader’s imagination.

223

24

Detailed Design - CAD24

Important CAD elements:

Labels match text explanations

Mechatronics (see ME2110 website)

Common COTS components (fasteners, etc.)

Detail views

Dimensions

Formatting (text, resolution, annotations)

24

25

Understand the importance of evaluation in the design process

Identify criteria for evaluation

Utilize three levels of evaluation matrices for product development (First-level, Second-level, Third-level)

Strategies for Visual/Descriptive Communication of Designs

Evaluation Tools Summary

26

Introductory Project Report

Goal: design a delivery system capable of launching an object onto a target that can be placed between 20-25’ from a specified position. The object of interest is a plastic sphere 1.5” in diameter and 0.095 oz. in weight.

Design tools: HOQ, specification sheet, function tree, morphological chart, alternative generation, concept selection

Detailed design: labels, triggering, mechanisms, sequencing, materials

Constrained design (materials, ...)

27

• Cover Page: See book for example cover page. • Abstract: One-paragraph summary with key findings/results. • Introduction: Restatement of the problem, goal and inherent design challenges in

achieving this goal. • Problem Understanding: Important customer needs, engineering specifications to

address needs, tradeoffs and synergies, functions and concept generation. • Design Overview: Presentation of the preferred alternative, detailed description of

the design including quality figures that shows its parts and how it operates.• Alternative Designs: Presentation of non-selected alternatives and brief description

of how they operate. Discussion of quantitative concept selection results.• Conclusions: Summary with key results and potential next steps and final thoughts.• Appendix: Figures and tables.

Introductory Project Report - Layout