Issues in the Validation of Battle Models

Presented at 19 ISMOR

David Frankis

‘The Barbican’, East Street, Farnham, Surrey GU9 7TB 01252 738500

www.Advantage-Business.co.uk August 2002

Acknowledgements

Dstl

This work was carried out under contract to Dstl by Advantage Technical Consulting

RMCS

Today’s Presentation

Why validation

CLARION

What was done

Issues raised

Questions

Why Validation?

UK Government decision-making must pass the test of independent scrutiny

Making a logical case based on credible information is key to this

OA claims to be able to support this by quantifying key aspects, objectively

The validity of this quantification is therefore crucial

A new version of CLARION required an update to its validation status

CLARION General

A Land-Air campaign model

Object Oriented C++ implementation

Functionality is based on the concept of missions:

Each entity (e.g. a division) has a mission

Subordinate units are tasked with missions based on the superior’s mission

Defined set of mission types

Generally Brigade level and above

CLARION Functionality

Movement and Attrition

Command

Communications

Sensing

Close combat, Arty, Recce, Helo

Some Air aspects

CBW

EW

No logistics in version examined (V3.0)

What is Validation?

The model is realistic?

The representation of internal processes is correct?

Known effects are covered?

Sufficient detail is included?

The results are plausible?

Conclusions drawn are substantiated?

Other Validation Issues

Scope of validation

Model only, or ancillary tools

Status of any comparison

Danger of mutually-supporting invalid models

Validation Activities

Prioritisation of Requirement

Selection of Comparison Method

Generation of Scenario

Comparison Activity

Analysis and Reporting

Prioritisation

CLARION has wide scope of functions and contexts

Key stakeholders were consulted for their views

Formal method used to prioritise

Main outcome: focus on mainstream uses, not functions less used (Air, EW, CBW)

Selection of method

Possible comparison approachesHistorical AnalysisTrials and ExercisesOther modelsMilitary (and analytical) JudgementWargame

These are not mutually exclusive

Wargame was selected as best approach at a workshop

Dstl staff selected most appropriate (commercial) game

Scenario Generation

Workshop held with scientific and military analysts

Fictitious scenario overlaid on a map

Outline scheme of manoeuvre developed

Comparison and Analysis

Scenario entered in CLARION

Adjusted with military input

Then into wargame and played

Further CLARION adjustment to reflect military intentions in wargame

Comparison of outputs

Some practical difficulties arising from wargame limitations

Findings

Validation as part of study process

Data adjustment

User interface issues

Extraneous effects

Validation as Part of Study Process

Ideally, the data and the way the model is used requires (re-)validation on each study

Validation is an iterative process

How much?

What if the iteration doesn’t converge?

In exceptional cases, could have independent teams of analysts

Process Elements

Selection of Scenario

Scheme of manoeuvre

CLARION input

Exercise in CLARION

Interpret outputs

Study conclusions

Wargame

validate

User Interface Issues

If the user interface is unfriendly or unintuitive, analysts will lack confidence

Longer learning curve for new analysts and scrutineers

Resulting loss of confidence in results through uncertainty and reduced effective validation effort

Data Adjustment

In order to capture effects not explicit in the model, analysts adjust the input data

Acceptable as long as analysts doing the adjustment are doing the reporting

Legacy effects

Unpredictable interactions when done more than once

Extraneous Effects

CLARION scenarios are acknowledged to develop much more quickly than reality

As long as all processes (movement, attrition, communication) are accelerated the same for both sides, does not matter for many study purposes

BUT study results are easy to rubbish because they seem to have low credibility

Conclusions: General

Model unlikely to be the limiting factor on confidence in study results

The use of a good model cannot compensate for a poor process or the use of insufficiently skilled analysts

Where studies focus on scenarios, they, and their data, should be validated for that study

Consider use of a wargame tool to support the development of a scheme of manoeuvre in campaign studies

Conclusions: Process Elements

Treat input data collection and refinement as integral to the study, not a necessary evil

Iterate the review of input data, output results, and the use of adjunct tools to converge on a ‘valid enough’ solution

Ensure the military plan remains valid when conducting sensitivity excursions

For major studies, consider some parallel working

Use different experts at different stages to ensure freshness of perspective