UNCLASSIFIED

Value Focused Metrics

Professor D.F. Davis

David.Davis@africom.mil

ddavis@gmu.edu

Presentation to 28 ISMOR

1 September 2011

UNCLASSIFIED 2

Metrics modeling: Using the metrics framework as the base for analysis.

All assessment/metrics/evaluations normally consist of a set of measurements that are aggregated to some overall meaning. This aggregation technique is what I’m referring to as a metrics model.

Definitions

UNCLASSIFIED 3

Most Frequent

Top down decomposition aggregated by Multi-Attribute Utilities

Most Important

Next Important Next Important Next Important

And so on….

1)(0,1,)()(1

xuwwherexuwxU ii

n

iii

UNCLASSIFIED 4

If and only if the decompositions are collectively exhaustive and mutually exclusive – cover every thing, doesn’t leave anything out, and are independent in definition.

Most common error is that they are nowhere near independent, and are often highly dependent. Of course, knowing that you have included all necessary elements is always risky.

But, they are easy to use and all you have to do is determine the weights.

However

UNCLASSIFIED 5

Bumper Sticker

Metrics need to be organic to the plan, not appliquéd.

UNCLASSIFIED 6

Value Focused Metrics

Use decomposition for only the highest level, fundamental or strategic elements (Fundamental Objectives Hierarchy or FOH) – be careful and don’t expect much.

Use a Means Ends Objectives Network (MEON) to represent the intermediate, dependent objectives and exploit the relationships. This is difficult because it requires that you graph your understanding of the inherent theory of change.

Identify, for some or all of the objectives, metrics that would indicate the level of achievement of that objective.

UNCLASSIFIED 7

Value Focused Metrics - 2

Objectives should be written as something you wish to achieve. If they can be caused to happen directly, they are tasks or actions, not objectives.

Therefore actions are something that you identify and execute with the hope that they will assist in the achievement of the objectives.

The MEON represents how you believe the objectives support one another and propagate the results of actions up to the achievement of the fundamental objectives.

UNCLASSIFIED 8

Fundamental Objective Hierarchy

Means Ends Objective Network

Controls, Actions,Programs, Projects

The System

ProcessMetrics

OutcomeMetrics

PerformanceMetrics

UNCLASSIFIED 9

The Somalia Game

A discovery game was held with multiple agencies of the US Government to look at the impact on US Africa Command’s End States and Objectives of various shocks or scenarios.

Impacts on objectives and actions to be taken were collected from the teams. Impacts were represented by specific expectations on metrics for individual objectives. Prevention and response actions were modeled by applying these actions to one or more objectives and developing likelihood expectations based on player input.

The actual game was held as For Official Use Only and is not currently fully releasable. However, the gist of the process can be seen in the following models and tables where the actual objective is represented by its coding identifier.

UNCLASSIFIED 10

Numbers

The MEON requires that each local model, an objective and it’s contributing (parent) objectives, have a conditional probability model. These models were developed in the Joint Planning Team for Somalia at the Command. Several sessions augmented by modeler subject mater expertise resulted in a fully described model for the FOH, MEON and the Metrics.

During the game, the players identified which objectives their actions would effect and to what degree and directions (negative, positive) that action would impact the objective. This information along with modeler input was used to provide the local models for prevention and response turns.

The model results were not used in the game, but were part of the post game analysis. The Control Group, or White Cell, led all in-game discussions.

UNCLASSIFIED 11

A Model Base Model

Fundamental Objectives Hierarchy

Means Ends Objectives

Network

Developed over six months and three continents: Africa – Nairobi and Djibouti, Europe – Stuttgart, US – Washington

Shows how the overall end states are achieved through a network of intermediate objectives.

UNCLASSIFIED 12

Base Model with Metrics

Concurrent to the development of the Base Model, each objective was addressed by asking the question: How would we know that the objective was achieved?

During the game, the impacts assessed by the player teams were reflected in values of the metrics.

UNCLASSIFIED 13

Base Model with Actions

During the game, there were two turns that addressed needed actions: Prevention and Response. These actions were identified by which objectives they impacted and with what magnitude and direction.

UNCLASSIFIED 14

Base Scenario 1a Impact Scenario 1b Impact Scenario 1 Prevention Scenario 1 Response

Strategic 0.488 0.314 0.399 0.802 0.938

F1 0.473 0.342 0.453 0.836 0.938

F2 0.524 0.248 0.273 0.721 0.939

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

End State Achievment

Strategic

F1

F2

UNCLASSIFIED 15

M1_1 M1_2 M1_3 M1_4 M1_5 M1_6 M2_1 M2_2 M2_3

Base 0.200 0.200 0.067 0.129 0.159 0.173 0.577 0.368 0.381

Scenario 1 - Diplomacy 0.010 0.973 0.042 0.006 0.0002 0.131 0.373 0.014 0.026

Scenario 1 - Others 0.760 0.973 0.040 0.006 0.0002 0.190 0.379 0.014 0.041

Scenario 1 Prevention 0.751 0.963 0.212 0.683 0.673 0.911 0.684 0.831 0.540

Scenario 1 Response 0.984 0.976 0.937 0.985 0.728 0.923 0.896 0.952 0.902

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

Intermediate Objective Achievment

Base

Scenario 1 - Diplomacy

Scenario 1 - Others

Scenario 1 Prevention

Scenario 1 Response

UNCLASSIFIED 16

BaseScenario 1 -Diplomacy Scenario 1 - Others

Scenario 1 Prevention Scenario 1 Response

M1_5 0.1593 0.0002 0.0002 0.6730 0.7277

0.0000

0.1000

0.2000

0.3000

0.4000

0.5000

0.6000

0.7000

0.8000

0.9000

1.0000

Critical Objective

UNCLASSIFIED 17

Questions?