analysis of competing hypotheses using … of competing hypotheses using subjective logic (ach-sl) 1...

10TH INTERNATIONAL COMMAND AND CONTROL RESEARCH AND TECHNOLOGY SYMPOSIUM

THE FUTURE OF C2

DECISIONMAKING AND COGNITIVE ANALYSIS

Analysis of Competing Hypothesesusing Subjective Logic

Simon Pope, Audun Jøsang

CRC for Enterprise Distributed Systems Technology (DSTC Pty Ltd)Level 7, General Purpose South

The University of Queensland 4072 AustraliaTel: +61-7-3365-4310 Fax: +61-7-3365-4311

simon.pope,[email protected]

http://www.dstc.edu.au

This page is intentionally left blank.

ANALYSIS OF COMPETING HYPOTHESES USING SUBJECTIVE LOGIC (ACH-SL) 1

Analysis of Competing Hypothesesusing Subjective Logic

Simon Pope, Audun Jøsang

CRC for Enterprise Distributed Systems Technology (DSTC Pty Ltd)Level 7, General Purpose South

The University of Queensland, 4072 AustraliaVoice: +61 7 3365 4310 Fax: +61 7 3365 4311

simon.pope,[email protected]

Abstract

Intelligence analysis is a complicated task that requires a high degree of analytical judgement under conditionsof considerable uncertainty. This judgement is used to fill in the gaps in knowledge and is the analyst’s principalmeans of managing uncertainty.

Much of intelligence analysis includes judging the relevance and the value of evidence to determine thelikelihood of competing hypotheses. The challenge is to create better formal methods of analysis that can be usedunder a wider variety of circumstances and which can handle both empirical data and formally-expressed beliefsas evidence for or against each hypothesis.

The authors have developed a formal approach to the evaluation of competing hypotheses that is based onthe belief calculus known as Subjective Logic. The development of this formal approach allows for integrationof empirical and statistical data, as well as for judgements made by analysts. Lastly, this formal approach makesredundant the separate analysis of “diagnosticity of evidence”. Under this formal approach, diagnosticity is formallyderived from the model and need not be considered as a separate input to the model, except as a means of limitingthe initial set of evidence that should be formally considered.

I. INTRODUCTION

Intelligence deals with all the things which should be knownin advance of initiating a course of action.

– Intelligence Activities [1], in Warner [2]

Intelligence is a difficult term to define precisely, yet its role and importance can be both intuitivelyunderstood and appreciated. From one perspective it may be seen as an end product – ‘information’ thatis used to enhance or aid a decision making process. From yet another perspective, it refers to the processthat is applied to information, in order to transform it into a more useful product [2]. More importantlythan what intelligence is perhaps, is what intelligence does.

Intelligence, both as a product and a process, is a means by which better decisions can be made, basedon an increased understanding of likely courses of action, their influences and their consequences. Ineveryday personal affairs few of our decisions use any directed analytical processes – and even fewerof these require any sort of rigorous approach – arguably due to relatively minor consequences of thedecisions we face.

The same is not true for large-scale human affairs – such as the business of nations and corporations –where the complexity of the environment and the relative consequences of decisions can have enormous

The work reported in this paper has been funded in part by the Co-operative Research Centre for Enterprise Distributed Systems Technology(DSTC) through the Australian Federal Government’s CRC Programme (Department of Education, Science, and Training).

2 ANALYSIS OF COMPETING HYPOTHESES USING SUBJECTIVE LOGIC (ACH-SL)

impact on the well-being and survival of a nation’s citizenry or a corporation’s ability to compete. Thisdistinction in and of itself is cause enough to consider whether human ‘everyday reasoning’ is robust andreliable enough for use in these larger contexts.

As it happens, humans systematically make substantive errors in reasoning due to problems of framing,resistance of mental models to change, risk aversion, limitations of short-term memory, and other cognitiveand perceptual biases [3], [4], [5], [6], [7]. This has severe implications for the process of intelligenceanalysis, and may lead to incorrect conclusions, especially in situations that appear familiar but whichactually result in different outcomes; in situations where the gradual assimilation of information intoestablished mental models results in the failure to detect ‘weak signals’ that should have triggered amajor re-evaluation; and in situations where the complexity of the mental models are untenable due tohuman limitations of short-term memory [8], [7], [9], [10].

When applied to the business of nation states, the consequences of intelligence failure can be disastrous,so much so that the recorded history of the world – both ancient and modern – is replete with a litany ofdevastating intelligence failures too numerous to list. Examples of these are easily found in any periodof history – such as the failure of the United States to perceive an impending attack on Pearl Harbor –and the failure of Japan to reason that Midway Island was a trap, with the consequent sinking of fourJapanese aircraft carriers and the loss of all crews, aircrews and aircraft.

It is therefore foolhardy to believe that good intelligence can be developed by relying solely onhuman cognition without resort to products, methodologies or frameworks that attempt to augment humancognition while also mitigating its defects. The management of intelligence analysis should encourage theapplication of products that allow clear delineation of assumptions and chains of inference; the specificationof the degree of uncertainty about the evidence and resultant conclusions; and the elaboration of alternativeperspectives and conclusions [7].

This paper proposes that the use of Subjective Logic [11] should be coupled with the Analysis ofCompeting Hypotheses (ACH) approach, [7], [12] as a basis for reasoning about alternative hypothesesas part of the process of intelligence analysis – an approach that is referred to as ACH-SL. The paperalso argues that the ACH-SL approach allows uncertainty of evidence, opinion about their influences, andthe resultant conclusions to be expressed in both quantitative and qualitative forms that are applicable forhuman and machine interpretation. Lastly, the paper shows that ‘diagnosticity of evidence’ can be derivedfrom this approach and need not be considered as a separate input to the process, except as a means oflimiting the initial set of evidence to be considered for inclusion into the model.

II. ALTERNATIVE ANALYSIS

Intelligence analysis generally requires that analysts choose from among several alternative hypothesesin order to present the most plausible of these as likely explanations or outcomes for the evidence beinganalyzed. Analysts that do not use some rigorous methodology will often work intuitively to identify whatthey believe to be the most likely explanation and then work backwards, using a satisficing approach wherethe ‘correct’ explanation is the first one that is consistent with the evidence [7]. The major downfall of thesatisficing approach is that there may be more than one explanation that is consistent with the evidence,and unless the analyst evaluates every reasonable alternative, they may arrive at an incorrect conclusion.Other common problems with using this strategy include the failure to generate appropriate alternativehypotheses; the propensity to filter and interpret the evidence to support the conclusions; and the failureto consider the diagnosticity of the evidence and how well it differentiates between hypotheses. Therecognition of these problems with their disastrous consequences has led to the development of AlternativeAnalysis techniques that are widely employed within the intelligence services1.

1Other strategies that are less commonly used in intelligence analysis and are also ineffective are discussed in detail elsewhere [13]

SIMON POPE AND AUDUN JØSANG, DSTC 3

Many alternative analysis techniques attempt to address the problems of fixed mind-sets and incompletegeneration of alternative hypotheses, while still others attempt to address the problems of reasoningabout the alternative hypotheses [10]. One way in which some of the problems of reasoning aboutalternative hypotheses can be overcome is to require the analyst to simultaneously evaluate all reasonablehypotheses and reach conclusions about their relative likelihood, based on the evidence provided. However,simultaneous evaluation of all non-trivial problems is a near-impossible feat for human cognition alone.Recent research suggests the number of individual variables we can mentally handle while trying tosolve a problem is relatively small – four variables are difficult, while five are nearly impossible [14].The Analysis of Competing Hypotheses (ACH) [7] was developed to provide a framework for assistedreasoning that would help overcome these limitations.

Alternative Analysis, and in particular ACH, is seen as so important that the CIA’s Sherman KentSchool for Intelligence Analysis runs a monthly Alternative Analysis Workshop and has introduced anAlternative Analysis unit into the Career Analyst Program (CAP) – the basic training program of the CIADirectorate of Intelligence (DI) which introduces all new employees to the basic thinking, writing andbriefing skills that are needed as an analyst [15].

III. HEUER’S ANALYSIS OF COMPETING HYPOTHESES (ACH)

The ACH methodology was developed in the mid- to late-1970’s by Richard Heuer, a former CIADirectorate of Intelligence methodology specialist, in response to his “never-ending quest for betteranalysis” [7]. His ACH methodology is still considered to be highly relevant today [12]. ACH consistsof the following eight steps:

Step-by-Step Outline of Analysis of Competing Hypotheses

1) Identify the possible hypotheses to be considered. Use a group of analysts with different perspectivesto brainstorm the possibilities.

2) Make a list of significant evidence and arguments for and against each hypothesis.

3) Prepare a matrix with hypotheses across the top and evidence down the side. Analyze the “diag-nosticity” of the evidence and arguments–that is, identify which items are most helpful in judging therelative likelihood of the hypotheses.

4) Refine the matrix. Reconsider the hypotheses and delete evidence and arguments that have nodiagnostic value.

5) Draw tentative conclusions about the relative likelihood of each hypothesis. Proceed by trying todisprove the hypotheses rather than prove them.

6) Analyze how sensitive your conclusion is to a few critical items of evidence. Consider the conse-quences for your analysis if that evidence were wrong, misleading, or subject to a different interpre-tation.

7) Report conclusions. Discuss the relative likelihood of all the hypotheses, not just the most likely one.

8) Identify milestones for future observation that may indicate events are taking a different course thanexpected.

– Heuer, Psychology of Intelligence Analysis [7]

These eight steps are intended to provide a basic framework for identification of assumptions, argumentsand hypotheses; consideration of all evidence and hypotheses – including its value relative to the hypothe-


ses; a method of disconfirmation for identifying the most likely hypotheses; an approach to reporting theresults of the analysis; and an approach to detecting future changes in the outcomes.

In simple terms, ACH requires the analyst to simultaneously evaluate all reasonable hypotheses andreach conclusions about their relative likelihood, based on the evidence provided. Heuer acknowledgesthat while this holistic approach will not always yield the right answer, it does provide some protectionagainst cognitive biases and limitations [7].

Of particular interest is Step 5, which requires the analyst to draw tentative conclusions about thelikelihood of each hypothesis. It has been argued that ACH recommends analysts consider the likelihoodof each hypothesis h given the assertion of each item of evidence, e, i.e. p(h|e) [12]. However, thiscan reasonably be interpreted to mean that the negation of each item of evidence, e should also beconsidered (p(h|e)). Consideration of counterfactuals has the advantage that the model can be constructedindependently of known facts and continually evaluated if the value of the evidence changes over time.The difference in interpretation lies in whether the evidence with respect to the hypotheses is considereda priori or a posteori.

Evidence can be constructed a posteori by the analyst from the ‘facts at hand’, where the evidence hasalready been measured and valued, rather than from a general examination of the possible signs for eachhypothesis. While examination of available data is usually relevant, ‘hidden facts’ – conditions whichare not observable, or conditions which have not yet taken place – are also likely to be relevant to theanalysis. If reasoning is conducted a priori, then the value of the evidence is uncertain, and the analyst ismore likely to consider the consequences of it being false as well as the consequences of it being true. Ifthe reasoning is a posteori, the analyst may know whether the evidence is true or false, and not considerits counterfactual to be relevant in determining the likelihood of the hypothesis. This is a mistake, sincethe analysis model will no longer be relevant if the value of the evidence changes, or there is uncertaintyabout its value.

Richard Heuer makes the excellent point that analysts should interpret ‘evidence’ in its broadest senseand not limit oneself just to current intelligence reporting. Indeed ACH is able to model the absenceof evidence as well as its presence, and when done diligently presents no conceptual problem. However,ACH does not require analysts to consider both the assertion and negation of evidence, and this deficiencymay lead them to frame the problem in terms of a single view of evidence – which often leads to incorrectconclusions, especially if deception or denial is being undertaken by an adversary [12].

IV. ANALYSIS OF COMPETING HYPOTHESES – COUNTER DECEPTION (ACH-CD)

ACH-CD was developed by Frank Stech and Christopher Elasser of the MITRE Corporation as amodified variant of ACH to account for cognitive factors that make people poor at detecting deception[12]. They correctly argue that the use of ACH can lead to greater susceptibility for deception, especiallywhen reasoning about a single view of evidence, i.e. the likelihood of each hypothesis given the assertionof the evidence p(h|e). Their argument is that this type of reasoning neglects the base rates both of theevidence br(e) and of the hypothesis br(h) which can result in reasoning errors that lead to incorrectconclusions [16]. More correctly it should be said that reasoning using only one of the logical conditionals(usually the positive conditional, p(h|e)) is more likely to produce reasoning flaws than when both areconsidered [17]. Stech and Elasser make the same point when they argue that analysts’ judgements aremore susceptible to deception if they also do not take the false positive rate of the evidence into account[12].

An excellent example of this provided by Stech and Elasser is how the reasoning about the detection ofKrypton gas in a middle-eastern country can lead to the erroneous conclusion that the country in questionlikely has a nuclear enrichment program. For clarity, their example has been reproduced below [12]:


Detect Kryptonp(enrichment | Krypton) = high→ p(enrichment program) = high→ p(nuclear program) = high

They argue that the main problem with this reasoning is that it does not consider that Krypton gasis also used to test pipelines for leaks, and that being a middle-eastern country with oil pipelines, theprobability of the gas being used outside of a nuclear program is also fairly high, i.e.

p(Krypton | not enrichment) = medium to high

This additional information should lead the analyst to the conclusion that there is a fair amount ofuncertainty of a nuclear program given the detection of Krypton. The assignment of the ‘high’ value top(enrichment | Krypton) neglects the fact that an oil-rich middle-eastern country is likely to use Krypton gas– regardless of whether they have a nuclear program.

However, it can be argued that Stech and Elasser have interpreted Step 5 of ACH more narrowly thanperhaps was intended. Heuer makes no claim about which of p(h|e) or p(e|h) – and their correspondingcounterfactual p(h|e), p(e|h) – should be used. Heuer describes the process in such general terms asto be consistent with either interpretation, although consideration of counterfactuals is essential if basicreasoning errors are to be avoided.

In any event, it can be shown2 that p(h|e) and p(h|e) can be derived from knowledge of p(e|h), p(e|h)and the base rate of the hypothesis br(h). Therefore the choice of which logical conditionals to use is lessimportant then the soundness of the belief values assigned to them. The choice of logical conditionalsbecomes more important when the analyst considers whether the evidence is causal in nature with respectto the hypotheses, or is merely derivative. Section V-B discusses the problem of framing with respect tothe causal or derivative nature of evidence and the implications for reasoning.

V. ANALYSIS OF COMPETING HYPOTHESES USING SUBJECTIVE LOGIC (ACH-SL)

Heuer’s ACH process describes a general process that is largely independent of technology and maybe used by analysts that have little more than ‘pen and paper’ at their disposal. By contrast, the ACH-SLprocess is highly dependent on technology to perform Subjective Logic calculations that would be highlycomplex and time-consuming if they were performed manually. ACH-SL is not merely a theoretical concept– it is also a functioning, implemented DSTC technology known as ShEBA3, and provides a frameworkfor the analysis of multiple hypotheses with multiple items of evidence. It was developed to addresssome of the key analytical issues within the defense, intelligence, and law enforcement communities. Thissection will outline the ACH-SL process and discuss some of its key features, including:

• compatibility with ‘fuzzy’ human representations of belief;• interoperability with Bayesian systems;• formalized abductive and deductive reasoning support; and,• a priori derivation of diagnosticity from analyst judgements.

ACH-SL is not meant as a replacement of ACH, but instead is an elaboration of the basic ACH processthat is consistent with Heuer’s intent. Within the original ACH process, Steps 3 through to 6 can besubstituted with modified ACH-SL, described below.

2 See Section V-F.2.3ShEBA – Structured Evidence-Based Analysis (of hypotheses)


Step-By-Step Outline of Analysis of Competing Hypotheses – Subjective Logic

1) Identify the possible hypotheses to be considered. (ACH Step 1)

2) Make a list of significant evidence and arguments for and against each hypothesis. (ACH Step 2)

3) Prepare a model consisting of:

a) A set of exhaustive and exclusive hypotheses – where one and only one must be true.

b) A set of items of evidence that are relevant to one or more hypotheses; are influences that have a causalinfluence on one or more hypotheses; or, would disconfirm one or more hypotheses.

4) Consider the evidence with respect to the hypotheses:

a) For each hypothesis and item of evidence, assess its base rate.

b) Should the evidence be treated as causal or derivative? Decide and record for each item of evidence orevidence/hypothesis pair.

c) Make judgements for causal evidence as to the likelihood of each hypothesis if the evidence were trueand if the evidence were false.

d) Make judgements for derivative evidence as to the likelihood that the evidence will be true if the hypothesiswere true, and if the hypothesis were false.

e) From the judgements provided, compute the diagnosticity for each item of evidence.

5) Measure the evidence itself and decide the likelihood that the evidence is true. Supply the measured evidenceas input into the constructed model, and use the Subjective Logic calculus to compute the overall likelihood ofeach hypothesis.

6) Analyze how sensitive the conclusion is to a few critical items of evidence. Changes in the value of evidence withhigh diagnosticity will alter the calculated likelihoods more than evidence with low diagnosticity. Consider theconsequences for your analysis if that evidence were wrong, misleading, or subject to a different interpretation.

7) Record and report conclusions. Discuss the relative likelihood of all the hypotheses, not just the most likely one.(ACH Step 7)

8) Identify milestones for future observation that may indicate events are taking a different course than expected.(ACH Step 8)

A. Determining base rates of evidence and hypotheses

One of the main problems of applying probability theory and belief calculi to real world problems isdetermining the base rates for evidence and hypotheses. A distinction can be made between events thatcan be repeated many times and events that can only happen once. Events that can be repeated manytimes are frequentist events and the base rates for these can be derived from first principles, or reasonablyapproximated through empirical observation. For example, if an observer knows the exact proportions ofthe different colored balls in an urn, then the base rates will be equal to the probabilities of drawing eachof the colors. For frequentist problems where base rates cannot be known with absolute certainty, thenapproximation through prior empirical observation is possible. For events that can only happen once, theobserver must arbitrarily decide what the base rates should be, and are often elided as a consequence.

The difference between the concepts of subjective and frequentist probabilities is that the former can


be defined as subjective betting odds – and the latter as the relative frequency of empirically observeddata, where the two collapse in the case where empirical data is available [18]. The concepts of subjectiveand empirical base rates can be defined in a similar manner where they also converge and merge into asingle base rate when empirical data is available.

As an example, consider how a public health department establishes the base rate of some diseasewithin a community. Typically, data is collected from hospitals, clinics and other sources where peoplediagnosed with the disease are treated. The amount of data that is required to calculate the base rate ofthe disease will be determined by some departmental guidelines, statistical analysis, and expert opinionabout the data that it is truly reflective of the actual number of infections – which is itself a subjectiveassessment. After the guidelines, analysis and opinion are all satisfied, the base rate will be determinedfrom the data, and can then be used in medical tests to provide a better indication of the likelihood ofspecific patients having contracted the disease.

1) Base rates of hypotheses: As a consequence of the ways in which base rates can be formed, thereis an inherent danger in assigning base rates to hypotheses when dealing with events that can onlyhappen once, or when a hypothesis does not have a strong relationship with a well-established modelfor the approximation of its base rate. Typically, events that have humans as causal influences are poorcandidates for empirical base rates since they are highly contextual and may be subject to slight variationwhich cause large perturbations in their appearance. In addition, much of strategic intelligence deals withhypotheses that can only happen once and for which empirical data that can be used to approximate baserates simply does not exist. In these cases, the base rates for non-repeatable hypotheses should be evenlyweighted since they form a exhaustive and exclusive state space – one and only one of the hypotheses istrue. So, for a set of k hypotheses Φ = h1, h2, . . . hk, the base rate of each hypothesis should be 1

k, i.e.

∀ hi ∈ Φ, br(hi) =1

k(V.1)

For three hypotheses, the base rate of each hypothesis should be 13; for four hypotheses, the base rate

should be 14, and so on. This follows from the Principle of Indifference which states that if we are ignorant

of the ways an event can occur, the event will occur equally likely in any way [19].

Setting the base rates to other than equal values for non-repeatable events is strongly discouraged. Doingso introduces an inherent bias in the model and may produce erroneous conclusions. Any direct reasoningabout base rates under these conditions should be discarded in favor of consideration of the evidence. Ifreasoning is applied to subjectively determine base rates, then that reasoning needs to be explicitly statedin the model instead of implicity included in the hypotheses’ base rates. The evidence for the reasoningshould be included as a standard part of the model and treated like all other evidence. Doing so reducesthe likelihood of erroneous conclusions and eliminates the possibility of ‘double counting’ evidence –where the evidence has already been taken into account in setting the base rate but is also used as partof the model to reason about the hypotheses.

2) Base rates of evidence: Base rates for evidence should be considered in the same way as forhypotheses. The set of hypotheses form an exhaustive and exclusive state space, such that one and onlyone hypothesis is true. Similarly for each item of evidence, consideration must be given to the otherelements of the the state space in which the evidence is situated. When the base rate for a item ofevidence can not be derived from first principles or approximated through empirical testing, then the baserate should be set according to proportion of the state space that the evidence consumes.

For example, the blood type being found at a crime scene to be AB might be considered evidence for oragainst certain competing hypotheses. The complete state space in which the evidence “Blood sample istype AB” allows for three other possibilities, namely that the blood type is A, B, or O. If the prevalenceof the four different blood types within the community was known as a result of analysis of statistical


data, then the empirically approximated base rates should be used. If they are not known or cannot bereliably derived, then the base rate for a result of blood type AB should be set at 1

4.

Typically, most evidence will have a base rate of 12

when dealing with simple true/false statements.However, consideration of the complete state space in which the evidence is situated is important so asnot to introduce base rate errors into the model.

B. Causal and Derivative Evidence

Abduction reasons about the likelihood of the hypothesis, given the likelihood of the assertion of theevidence under conditions when the hypothesis is true and false, i.e. using p(e|h), p(e|h). Physiciansprimarily use abduction for medical diagnosis when considering the likelihood of a patient having anparticular disease, given that the patient presents specific symptoms that are associated with the disease.

By contrast, deduction directly reasons about the likelihood of the hypothesis, given the likelihoods ofthe hypothesis under conditions of the assertion and negation of the evidence, i.e. using p(h|e), p(h|e).Deduction is most often applied when there can be a causal link from the evidence to one or morehypotheses, such as when reasoning about the likelihood of a patient having a particular disease, giventhe possibility of recent exposure to the same infectious disease.

The original ACH describes a process that uses deductive reasoning [7], while Stech and Elasserexplicitly use an abductive approach to reasoning with ACH-CD [12].

Both deductive and abductive reasoning have their uses and limitations. Deductive reasoning is bestsuited for reasoning about causal evidence, while abductive reasoning is best suited for reasoning aboutderivative evidence.

Causal evidence has a direct causal influence on a hypothesis – such as the presence of a persistentlow pressure system is causal evidence for rain, since a low pressure system has a direct influence onprecipitation. The ‘state of mind’ of an adversary is often regarded as causal evidence since it usually hasa direct influence on their decision making processes.

Derivative evidence [20] – also known as diagnostic evidence [21] – is indirect secondary evidence –not causal in nature – and is usually observed in conjunction, or is closely associated with the hypothesis.For example, a soggy lawn should be considered derivative evidence for rain – but soggy lawns are alsoassociated with the use of sprinklers, and recently-washed automobiles. In the nuclear enrichment examplefrom Stech and Elasser (Section IV), the detection of Krypton gas would be considered derivative evidence,since the presence of Krypton gas does not causally influence the likelihood of a nuclear enrichmentprogram.

In theory, both deductive and abductive reasoning can be used for analysis of competing hypotheses,providing the logical conditionals have suitable belief assignments. In practice though, there is an inherentdanger in applying deductive reasoning to derivative evidence, just as there is a danger in applyingabductive reasoning to causal evidence. The problem lies in how the questions about the evidence andthe hypothesis are framed, and the nature of causality that can be inferred [20], [4], [21].

1) Reasoning about causal evidence: Using abductive reasoning to reason about causal evidencerequires more cognitive effort than using deductive reasoning. It requires the analyst to suppose theassertion or negation of the consequent and reason about the likelihood of the antecedent, p(e|h) andp(e|h). At best, it is likely that the analyst is actually reasoning about the likelihood of the consequentgiven the assertion and negation of the antecedent, p(h|e) and p(h|e), and simply approximating p(e|h)and p(e|h) as a result. At worst, the analyst can draw completely different inferences that violate thecausal nature of the evidence and lead to incorrect reasoning about the hypotheses.


For example, during the Cuban missile crisis of 1962, President Kennedy publicly warned that theUnited States would view any Soviet strategic missile placements in Cuba as a grave threat and wouldtake appropriate action [20]. An abductive approach to this problem would require the analyst to ask ofthemselves:

p(e|h) If the Soviets are shipping strategic missiles to Cuba, what is the likelihood that the President made a publicstatement that the U.S. would perceive strategic missile placement as a threat?

p(e|h) If the Soviets are not shipping strategic missiles to Cuba, what is the likelihood that the President made a publicstatement that the U.S. would perceive strategic missile placement as a threat?

The way the question is framed suggests that the analyst should consider how likely there will be apublic statement given the existence of strategic missiles in Cuba. If the analyst was unaware of the likelyintention of the public statement, they might reasonably conclude that it is more likely that the a publicstatement would be made if the Soviets are in the process of shipping strategic missiles to Cuba, thusincreasing the likelihood of the ‘missiles’ hypothesis. However, this is almost certainly not as PresidentKennedy intended, instead likely reasoning that a public statement would serve to dissuade the Sovietleadership from shipping strategic missiles to Cuba – or at worst have no appreciable effect. In otherwords, he reasoned that the public statement would act in a causal manner to increase the likelihood ofthe ‘no missiles’ hypothesis [20].

If we apply a deductive approach to the same question, the analyst is required to ask of themselvesdifferent questions that preserve the causal nature of the evidence and appear less likely to facilitate thesame framing errors:

p(h|e) If the President made a public statement that the U.S. would perceive strategic missile placement as a threat, whatis the likelihood that the Soviets are shipping strategic missiles to Cuba?

p(h|e) If the President did not make a public statement that the U.S. would perceive strategic missile placement as a threat,what is the likelihood that the Soviets are shipping strategic missiles to Cuba?

Here the framing of the question suggests that the statement will influence the outcome, and one wouldprobably conclude that the statement will serve to lessen the likelihood of missiles being shipped to Cuba– all other things being equal. This effect is similar to the paradoxical probability assessments problemof logically-equivalent pairs of conditional propositions, discussed by Tversky and Kahneman [21].

2) Reasoning about derivative evidence: The same paradox holds true for deductive reasoning aboutderivative evidence. Deductive reasoning requires the analyst to ask questions in a way that implies that theevidence is causal in nature. Deductive reasoning about derivative evidence also requires more cognitiveeffort than abductive reasoning, and can also lead to incorrect conclusions.

For example, consider the questions that an analyst might ask themselves if deductive reasoning wereapplied to the nuclear enrichment hypothesis (Section IV).

p(h|e) If Krypton gas is detected in Iraq, what is the likelihood that the Iraqis have a nuclear enrichment program?

p(h|e) If Krypton gas is not detected in Iraq, what is the likelihood that the Iraqis have a nuclear enrichment program?

The framing of the question does not prompt the analyst to consider the likelihood of Krypton gaswhen there is no nuclear enrichment program, i.e. p(e|h). If the analyst is unaware that Krypton gas isalso used for detecting leaks in oil pipelines, they will likely erroneously conclude that the likelihood ofp(h|e) is high. The analyst might reasonably infer from the framing of these questions that p(e|h) maybe a good approximation for p(h|e) since Krypton gas is a by-product of nuclear enrichment – whichmay cause them to miss the fact that Krypton gas is merely diagnostic – and incorrectly conclude that itspresence implies a nuclear enrichment program. When the problem is framed using abductive reasoning,


the analyst is prompted to consider the likelihood of Krypton gas in cases where there would be no nuclearenrichment program and the framing problems disappear.

p(e|h) If the Iraqis have a nuclear enrichment program, what is the likelihood that Krypton gas will be detected?

p(e|h) If the Iraqis do not have a nuclear enrichment program, what is the likelihood that Krypton gas will be detected?

Even though humans routinely attribute causality where none exists [21], framing the question topreserve the derivative nature of the evidence makes it less likely that the analyst will misattribute causality.Similar conclusions may be reached as for the use of abduction for causal evidence (Section V-B.1). Whileit is possible to use either style of reasoning for both types of evidence, less cognitive effort is requiredwhen deductive reasoning is used for causal evidence and abductive reasoning is used for derivativeevidence, and therefore erroneous reasoning due to framing is less likely to occur.

C. Constructing the analytic model

Logical conditionals [22] are a pair of conditional beliefs that is used for reasoning about the likelihoodof a hypothesis, given some evidence. The values of these conditionals constitute the judgements that theanalyst supplies as part of the ACH-SL process to reason about each hypothesis in respect of each itemof evidence.

Assigning belief values to the conditionals requires that the analyst answer certain questions of them-selves, and probably others, including experts. The style of reasoning that is used for the evidence andhypothesis will largely determine the type of questions to be answered, which in turn will be stronglyinfluenced by the causal or derivative nature of the evidence with respect to the hypothesis (see SectionV-B). For deductive reasoning, the questions should use the following or an equivalent form:

p(h|e) If [the evidence is true], what is the likelihood that [the hypothesis is true]?p(h|e) If [the evidence is false], what is the likelihood that [the hypothesis is true]?

By contrast, abducive reasoning should use questions in the following or an equivalent form:

p(e|h) If [the hypothesis is true], what is the likelihood that [the evidence is true]?p(e|h) If [the hypothesis is false], what is the likelihood that [the evidence is true]?

For each question, the analyst must assign a value while considering only the evidence and the hypothesisto which it relates. They must assume when providing a judgement that no other information is knownabout other hypotheses or other items of evidence, other than the base assumptions on which the modelis predicated. This is done for two reasons. Firstly, as a human being, the analyst will be incapable ofweighing more than four factors simultaneously in supplying a judgement [14] – even though that theymay believe that they are considering dozens [7]. Secondly, the analyst runs the risk of introducing biasif each judgement is not considered in isolation – which will also bias the overall conclusions about thelikelihoods of the hypotheses and make it difficult for others, including policy makers, to understand thereasoning employed by the analyst.

1) Constraints on logical conditionals: The two types of logical conditionals used in inductive reasoninghave different mathematical requirements placed on their values as a result. The logical conditionals usedfor deduction are termed deductive logical conditionals, i.e. p(hi|e) and p(hi|e). These conditionals mustobey certain constraints in order to satisfy basic requirements of probability theory. For an exhaustive andexclusive set of hypotheses where one and only one hypothesis can be true, it logically follows that thesum of the probability expectations of the positive conditionals must be one; and similarly the probability


expectations of the negative conditionals must also sum to one, since

p(h1) + p(h2) + . . . p(hn) = 1p(h1) + p(h2) + . . . p(hn) = 1

(V.2)

The logical conditionals used for abduction are termed abductive logical conditionals, i.e. p(e|hi) andp(e|hi). These have different constraints to deductive logical conditionals and there is no requirement thatthe probability expectations of their positive conditionals sum to one since the items of evidence are inseparate state spaces. The probability expectation of the positive conditional p(e|hi) is unconstrained foreach hypothesis. However, the probability expectation of each negative conditional p(e|hi) must be equalto the average of the probability expectations of the other positive conditionals of the other hypotheses,i.e.

p(e|h1) =p(e|h2) + p(e|h3) + . . . p(e|hn)

n− 1(V.3)

Details on the constraints of logical conditionals are described in the Appendix.

2) Model coherence: Across multiple items of evidence, there are further constraints to ensure that themodel is coherent. The requirements for coherence concern the assignment of belief values for differentitems of evidence for the same hypothesis. The constraints apply only to deductive logical conditionalsp(hi|ej) and p(hi|ej) which are either explicitly provided by the analyst when deductive reasoning isused, or are implicitly derived from knowledge of the abductive logical conditionals and base rate of thehypothesis4.

The belief values assigned to the deductive logical conditionals of different items of evidence for thesame hypothesis must have overlapping or adjoining ranges for the model to be coherent. If the ranges ofbelief assigned to the logical conditionals do not overlap or join, then at least one of the logical conditionalsof an item of evidence must be incorrect. Put simply, model coherence requires that the minimum likelihoodof the hypothesis for one item of evidence can not be greater than the maximum likelihood for anotheritem of evidence, since both judgements were supplied independently. If the minimum for one item ofevidence were to be greater than the maximum for another item of evidence, then this indicates that atleast one of the judgements is incorrect since it is a logical and a physical impossibility for non-quantumevents. Model coherence is described further in the Appendix.

D. Subjective Logic Fundamentals

This section introduces Subjective Logic, which is extensively used within the ACH-SL approach tomodel the influence of evidence on hypotheses, and provide a calculus for the evaluation of the modelwhen measurement of the evidence is provided.

Belief theory is a framework related to probability theory, but where the sum of probabilities over allpossible outcomes not necessarily add up to 1, and the remaining probability is assigned to the union ofpossible outcomes. Belief calculus is suitable for approximate reasoning in situations where there is moreor less uncertainty about whether a given proposition is true or false, and is ideally suited for both humanand machine representations of belief.

Subjective logic[11] represents a specific belief calculus that uses a belief metric called opinion toexpress beliefs. An opinion denoted by ωA

x = (bAx , dA

x , uAx , aA

x ) expresses the relying party A’s belief in

4See Section V-F.2 for details on transforming abductive logical conditionals to deductive logical conditionals.


the truth of statement x. Here b, d, and u represent belief, disbelief and uncertainty, and relative atomicityrespectively where bA

x , dAx , uA

x , aAx ∈ [0, 1] and the following equation holds:

bAx + dA

x + uAx = 1 . (V.4)

The parameter aAx represents the base rate of x and reflects the size of the state space from which

the statement x is taken5. In most cases the state space is binary, in which case aAx = 0.5. The relative

atomicity is used for computing an opinion’s probability expectation value expressed by:

E(ωAx ) = bA

x + aAx uA

x , (V.5)

meaning that a determines how uncertainty shall contribute to E(ωAx ). When the statement x for example

says “Party B is honest and reliable” then the opinion can be interpreted as trust in B, which can alsobe denoted as ωA

B.

The opinion space can be mapped into the interior of an equal-sided triangle, where, for an opinionωx = (bx, dx, ux, ax), the three parameters bx, dx and ux determine the position of the point in the trianglerepresenting the opinion. Fig.1 illustrates an example where the opinion about a proposition x from abinary frame of discernment has the value ωx = (0.7, 0.1, 0.2, 0.5).

a

ω = (0.7, 0.1, 0.2, 0.5)x

x

xω

xE( )

0.5 00

1

0.5 0.5

Disbelief1 Belief100 1

Uncertainty

Probability axis

Example opinion:

Projector

Fig. 1. Opinion triangle with example opinion

The top vertex of the triangle represents uncertainty, the bottom left vertex represents disbelief, and thebottom right vertex represents belief. The parameter bx is the value of a linear function on the trianglewhich takes value 0 on the edge which joins the uncertainty and disbelief vertices and takes value 1 atthe belief vertex. In other words, bx is equal to the quotient when the perpendicular distance between theopinion point and the edge joining the uncertainty and disbelief vertices is divided by the perpendiculardistance between the belief vertex and the same edge. The parameters dx and ux are determined similarly.The edge joining the disbelief and belief vertices is called the probability axis. The relative atomicity isindicated by a point on the probability axis, and the projector starting from the opinion point is parallelto the line that joins the uncertainty vertex and the relative atomicity point on the probability axis. Thepoint at which the projector meets the probability axis determines the expectation value of the opinion,i.e. it coincides with the point corresponding to expectation value bx + axux.

Opinions can be ordered according to probability expectation value, but additional criteria are neededin case of equal probability expectation values. We will use the following rules to determine the order ofopinions[11]:

5See Section V-A for a discussion on the assignment of base rates.


Let ωx and ωy be two opinions. They can be ordered according to the following rules by priority:

1) The opinion with the greatest probability expectation is the greatest opinion.2) The opinion with the least uncertainty is the greatest opinion.

Opinions can be expressed as beta PDFs (probability density functions). The beta-family of distributionsis a continuous family of distribution functions indexed by the two parameters α and β. The beta PDFdenoted by beta(α, β) can be expressed using the gamma function Γ as:

beta(α, β) =Γ(α + β)

Γ(α)Γ(β)pα−1(1− p)β−1 (V.6)

where 0 ≤ p ≤ 1 and α, β > 0, with the restriction that the probability variable p 6= 0 if α < 1, and p 6= 1if β < 1. The probability expectation value of the beta distribution is given by:

E(p) = α/(α + β). (V.7)

The following mapping defines how opinions can be represented as beta PDFs.

(bx, dx, ux, ax) 7−→ beta(

2bx

ux+ 2ax,

2dx

ux+ 2(1− ax)

). (V.8)

This means for example that an opinion with ux = 1 and ax = 0.5 which maps to beta (1, 1) is equivalentto a uniform PDF. It also means that a dogmatic opinion with ux = 0 which maps to beta (bxη, dxη)where η →∞ is equivalent to a spike PDF with infinitesimal width and infinite height. Dogmatic opinionscan thus be interpreted as being based on an infinite amount of evidence.

When nothing is known, the a priori distribution is the uniform beta with α = 1 and β = 1 illustratedin Fig.2a. Then after r positive and s negative observations the a posteriori distribution is the beta PDFwith the parameters α = r + 1 and β = s + 1. For example the beta PDF after observing 7 positive and1 negative outcomes is illustrated in Fig.2b below. This corresponds to the opinion of Fig.1 through themapping of (V.8).

0

1

2

3

4

5

0 0.2 0.4 0.6 0.8 1pProbability

Prob

abili

ty d

ensi

ty B

eta(

| 1

,1 )

p

(a) Uniform beta PDF: beta(1,1)

0

1

2

3

4

5

0 0.2 0.4 0.6 0.8 1

Prob

abili

ty d

ensi

ty B

eta(

| 8

,2 )

p

pProbability

(b) Uniform beta PDF: beta(8,2)

Fig. 2. Uniform beta PDF examples

A PDF of this type expresses the uncertain probability that a process will produce positive outcomeduring future observations. The probability expectation value of Fig.2b. is E(p) = 0.8. This can beinterpreted as saying that the relative frequency of a positive outcome in the future is somewhat uncertain,and that the most likely value is 0.8.

The variable p in (V.6) is a probability variable, so that for a given p the probability density beta(α, β)represents second order probability. The first-order variable p represents the probability of an event,whereas the density beta(α, β) represents the probability that the first-order variable has a specific value.


By definition, the PDF and the corresponding opinion always have the same probability expectationvalue, and can be interpreted as equivalent. This makes it possible to fuse opinions using Bayesian updatingof beta PDFs.

E. Representations of Subjective Opinions

Opinions that are expressed in Subjective Logic can be transformed to and from other belief rep-resentations and can be visualized in a number of different ways. This section describes techniques forvisualization; mapping between verbal fuzzy adjectives and Subjective Logic opinions; and transformationopinions to and from Bayesian representations of belief.

Subjective Logic opinions, Subjective Opinions, can be transformed without loss to and from Bayesianbelief representations6. This makes the Subjective Logic calculus ideal for reasoning about machine-supplied data that correspond to, or can be interpreted as Bayesian representations of belief. The immediateimplications of this are that systems that use Subjective Logic – such DSTC’s ShEBA technology – areable to be interfaced with systems that use Bayesian representations of belief to provide data to otherBayesian systems, or take data from Bayesian systems to be used as input into an ACH-SL system.

In the earlier sections we have shown that bipolar beliefs in the form of opinions, as illustrated in Fig.1,can be mapped to and interpreted as beta PDFs. While these two graphical representations give a concisemathematical visualization of bipolar beliefs, people unfamiliar with the underlying mathematical conceptscan have difficulty interpreting them. For this reason, more intuitive graphical and verbal representationscan be used. This is shown in Fig.3, which is a screen capture of an online demonstration7.

The example visualizes bipolar beliefs about three different statements x, y and z. Each belief isvisualized in different ways, i.e. in the form of 1) points in an opinion triangle, 2) beta density functions,3) coloured/shaded bars, and 4) fuzzy verbal categories. The interpretation of the opinion triangle and thebeta PDF need no further explanation, as they have been described in the previous sections.

The horizontal shaded bars are actually colored in the online demonstration, which makes them easierto interpret. The first horizontal bar, representing the belief in x, consists of a dark shaded area representsbx, and a light shaded area represents axux – i.e. the amount of uncertainty that contributes to E(x) –so that the total length of the dark and light shaded areas together represent E(x). The second horizontalbar, representing the belief in y, consists of a green (leftmost) area representing by, an amber (middle)area representing uy, and a red (rightmost) area representing dy, as well as a black vertical line within theamber area indicating E(y). This uses a ‘traffic light’ metaphor, where green indicates “go”, red indicates“stop” and amber indicates “caution”. The third horizontal bar, representing the belief in z, simply has asingle dark shaded area representing E(z).

A fuzzy category is also indicated directly above each horizontal bar. The fuzzy verbal categories canbe defined according to the need of the application. The example of Fig.3 (p.15) uses categories from thetwo-dimensional matrix defined in Fig.4 (p.16).

Note that the certainty category “Certain” is implicit, and need not be mentioned together with theapplicable likelihood category. These fuzzy verbal categories can be mapped to areas in the opinion triangleas illustrated in Fig.5 (p.16). The mapping must be defined for combinations of ranges of expectationvalue and uncertainty. As a result, the mapping between a specific fuzzy category from Fig.4 (p.16)andspecific geometric area in the opinion triangle depends on the base rate. Without specifying the exactunderlying ranges, the visualization of Fig.3 and Fig.5 indicate the ranges approximately. The edge ranges

6See the Appendix.7http://security.dstc.edu.au/spectrum/beliefvisual/BVDemo.html


Fig. 3. Example visualizations of bipolar beliefs

are deliberately made narrow in order to have categories for near dogmatic and vacuous beliefs, as wellas beliefs that express expectation values near absolute 0 or 1. The number of likelihood categories, andcertainty categories, as well as the exact ranges for each, must be determined according to the need ofeach application, and the fuzzy categories defined here must be seen as an example. Real-world categorieswould likely be similar to those found in Sherman Kent’s Words of Estimated Probability [23]; based onthe Admiralty Scale as used within the UK National Intelligence Model8; or could be based on empiricalresults obtained from psychological experimentation.

Fig.5.a illustrates the case with base rate a = 13, which was also the case in the visualization in Fig.3.

Whenever a fuzzy category area overlaps, partly or completely, with the opinion triangle, that fuzzycategory is possible.

The possible mappings depend on the base rate. For example, it can be seen that the category 7D:“Unlikely and Very Uncertain” is possible in case a = 1

3, but not in case a = 2

3. This is because the

expectation of a state x is defined as E(x) = bx + axux, so that when ax, ux −→ 1, then E(x) −→ 1, sothat the likelihood category “Unlikely” would be impossible.

Mapping from fuzzy categories to Subjective Opinions is also straight-forward. Geometrically, theprocess involves mapping the fuzzy adjectives to the corresponding center of the portion of the grid cellcontained within the opinion triangle (see Fig.5). Naturally, some mappings will always be impossible fora given base rate, but these are logically inconsistent and should be excluded from selection.

It is interesting to notice that although a specific fuzzy category maps to different geometric areas inthe opinion triangle depending on the base rate, it will always correspond to the same range of beta PDFs.

8http://www.policereform.gov.uk/implementation/natintellmodel.html


! " # $ %

& & & & !& "& #& $& %&

! " # $ %

' ( ( ( ( !( "( #( $( %(

! " # $ %

Fig. 4. Fuzzy Categories

1D

5B

7E 1E2E3E4E5E6E9E 8E

4D 3D 2D5D7D9D 6D8D

1B9B 2B8B 3B4B6B7B

2A 1A3A4A5A6A7A9A 8A

4C 1C8C 2C9C 7C 6C 5C 3C

(a) Fuzzy categories with a = 13

9E 8E 7E 6E 5E 4E 3E 2E 1E

1D9D 8D 7D 6D 5D 4D 2D3D

8C 1C9C 7C 6C 5C 4C 3C 2C

9B 1B5B 2B7B 6B 4B 3B8B

9A 8A 7A 6A 5A 4A 3A 2A 1A

(b) Fuzzy categories with a = 23

Fig. 5. Mapping of fuzzy categories to ranges of belief as a function of the base rate

It is simple to visualize ranges of bipolar beliefs with the opinion triangle, but it would not be easy tovisualize ranges of beta PDFs. The mapping between bipolar beliefs and beta PDFs thereby provides avery powerful way of describing PDFs in terms of fuzzy categories, and vice versa.

F. Calculating opinions about the hypotheses

The likelihood of each hypothesis can be calculated from knowledge of the base rate for each hypothesisbr(hi); the base rate for each item of evidence br(ej); the logical conditionals for each hypothesis/evidencepair (hi, ej); and measurements/opinions about the evidence p(ei). Two distinct steps are required:

1) The likelihood of each hypothesis, hi, for each item of evidence, ei is inferred using either abductionor deduction, depending on whether abductive or deductive logical conditionals that are provided.This produces knowledge of p(hi) for each ej , i.e. p(hi‖ej).


2) The overall likelihood for each item of evidence, p(hi) for each hi, is computed by fusing theseparate p(hi‖ej) opinions using the consensus operator.

This section discusses these three basic Subjective Logic operators that are used within the ACH-SLsystem for inferring the likelihoods of the hypotheses from what is known about the evidence.

1) Deduction: Deduction reasons about the likelihood of a hypothesis, given knowledge of the likeli-hood of the hypothesis being true when some evidence is observed; and the likelihood of the hypothesisbeing true when the evidence is not observed (see Fig.6).

Fig. 6. Deductive reasoning from the evidence to the hypothesis.

Deductive reasoning is often used for reasoning when the appearance of the evidence temporallyprecedes the appearance of the hypothesis, where there is a causal link from the evidence to the hypothesis.As an example, suppose that a physician working in an area with a high rate of disease as the result ofsome natural disaster is trying to determine the likelihood of a patient having contracted an infectiousdisease. The patient has no visible signs or reported symptoms, but the physician knows that direct contactwith a carrier of the disease results in an infection 95 percent (0.95) of the time. However, of those whoare infected, about 10 percent (0.1) have contracted the disease without contact with disease carriers. If thelikelihood of contact with a disease carrier for a particular patient is 10 percent (0.1), then the likelihoodof infection is approximately 19 percent (0.19).

The details of the deduction operator, , are described in [17]. The operator is written as ωh‖e =ωe

(ωh|e, ωh|e

).

2) Abduction: Abduction reasons about the likelihood of a hypothesis, given knowledge of the likeli-hood of some evidence being observed when the hypothesis is true; the likelihood of the evidence beingobserved when the hypothesis is false; and the base rate of the hypothesis.

Abductive reasoning is often applied to interpret medical test results. As an example, suppose that aspecific disease has a base rate of 1

100(i.e. one in every hundred people on average has the disease).

A particular test for this disease has a false positive rate of 1100

and a false negative rate of 150

(i.e.br(h) = 0.01, p(e|h) = 0.99, p(e|h) = 0.02). The false negative rate means for two percent of thosewho have the disease and are tested, the test will erroneously report that they do not have the disease.Similarly, the false positive rate means that for one percent of those who do not have the disease and aretested, the test will erroneously report that they do have the disease. Under all other conditions, the testreports the results correctly. If the test is applied to a random person for whom it is not known if theyhave the disease, and the result is positive, then the likelihood that they actually have the disease is 1

3or

0.33 – and not 0.98 as might have been supposed if the base rate was ignored.

Deriving the likelihood of the hypothesis from the logical conditionals p(h|e) and p(h|e) and thelikelihood of the evidence p(e) is straight-forward. Likewise, deriving the likelihood of the evidence fromthe logical conditionals p(e|h) and p(e|h) and the likelihood of the hypothesis p(h) is also straight-forward. However, deriving the likelihood of the hypothesis with only the logical conditionals, p(e|h) and


p(e|h), is not possible without knowledge of the base rate of the hypothesis br(h) (see Fig.7).9

Fig. 7. Different conditionals are needed for reasoning in different directions.

Using the knowledge of these three pieces of information, the logical conditionals p(h|e) and p(h|e)can be derived, and the problem can solved using deduction (see Fig.8).

Fig. 8. Abductive reasoning from the evidence to the hypothesis.

Using Subjective Logic, the implicit logical conditionals that allow direct reasoning from the evidence tothe hypothesis are derived and used with the deduction operator (Section V-F.1) to obtain the correct result.The logical conditionals used for deducing the hypothesis from knowledge of the evidence, ωe|h and ωe|h,can be derived from knowledge of the supplied conditionals, ωh|e and ωh|e, and knowledge of the base rateof the hypothesis, br(h). While the abduction operator will be discussed in depth in a forthcoming paper,the general derivation of deductive logical conditionals from abductive logical conditionals is summarizedbelow.10

Definition 5.1 (Abduction): Given knowledge of the base rate of the hypothesis br(h) where the ωh isa vacuous subjective opinion about the base rate of the hypothesis, defined as

ωh = (bh, dh, uh, ah)

bh = 0dh = 0uh = 1ah = br(h)

(V.9)

and given the abductive logical conditionals about evidence e, expressed in Subjective Logic form ωh|e, ωh|e,then the deductive logical conditionals ωe|h, ωe|h are derived using the following formula

9Similarly, with knowledge of the base rate of evidence br(e) and p(h|e) and p(h|e), the conditionals, p(e|h) and p(e|h), could bederived.

10The authors wish to acknowledge the important contributions made by David McAnally ([email protected]) in developing theabduction operator.


ωh|e =ωh·ωe|h

ωh(ωe|h,ωe|h)

ωh|e =ωh·¬ ωe|h

ωh(¬ ωe|h,¬ ωe|h)

(V.10)

and ωh can be solved using the deduction operator (Section V-F.1) where

ωh‖e = ωe (ωh|e, ωh|e

)(V.11)

The abduction operator, , is written as ωh‖e = ωe(ωe|h, ωe|h, br(h)

). Details on the multiplication and

division operators can be found in [24].

3) Consensus: The consensus operator is used for belief fusion. It allows independent beliefs to becombined into a consensus opinion which reflects all opinions in a fair and equal way, i.e. when there aretwo or more beliefs about hypothesis h resulting from distinct items of evidence, the consensus operatorproduces a consensus belief that combines them into one.

For example, suppose that for the hypothesis, h, there two distinct items of evidence, e1 and e2,that are either causal or derivative with respect to the hypothesis. Assume that for each of item ofevidence some inference is drawn about the likelihood of h (using deduction or abduction), then the twoindependent opinions, ωe1

h and ωe2h , can be fused into a single consensus opinion ωh which provides an

overall assessment of the likelihood of the hypothesis.

The details of the consensus operator, ⊕, are described in [25] and discussed further in [26]. Theoperator is written as ωh = ωe1

h ⊕ ωe2h · · · ⊕ ωen

h .

G. Diagnosticity of Evidence

Not all evidence is created equal – some evidence is better for distinguishing between hypothesesthan others. Evidence is considered to be diagnostic when it is indicative of the relative likelihood ofthe hypotheses being considered. If a item of evidence seems consistent with all the hypotheses, it willgenerally have little diagnostic value.

Heuer’s Analysis of Competing Hypotheses [7] describes a process by which diagnosticity is indicatedby the analyst for each evidence-hypothesis pair. Under a fairly narrow interpretation, such as given byStech and Elasser [12], the ACH process appears to consider only the assertion of each item of evidenceh|e (i.e. the evidence is true), and does not consider its negation h|e (i.e. the evidence is false). Under abroader interpretation, consideration of both logical conditionals is implied.

Consider Heuer’s medical analogy as an illustration of this point. A high temperature, e, might havelittle diagnostic value in determining which illness a person is suffering from – yet the absence of a hightemperature, e, may be more significant for distinguishing between possible illnesses.

In the ACH process, diagnosticity is explicitly provided by the analyst as an input [7], and it is usedboth to eliminate evidence from the model that does not distinguish well between hypotheses, and toprovide a means of eliminating hypotheses based on the relative weight of disconfirming evidence. Theseinputs are ‘second-order’ judgements, since the analyst must first consider the relationship between theitem of evidence and the hypotheses in order to determine the diagnosticity of the item of evidence. Thisreasoning is usually hidden from the final analysis and may be subject to the cognitive limitations andbiases which significantly contribute to errors in reasoning.


In the modified ACH-SL system, diagnosticity is not explicitly provided by the analyst. Instead, it isderived from the ‘first-order’ values that the analyst assigns to the logical conditionals, independently ofthe actual value of the evidence. This allows analysts to concentrate on the judgements they make, ratherthan requiring them to consider diagnosticity as a separate, ‘second-order’ measure of the evidence.

Diagnosticity is represented as a real number between 0 and 1 – with a value of 0 indicating that theevidence does not distinguish between the hypotheses in any way; and with a value of 1 indicating thatthe evidence is capable of completely distinguishing between the hypotheses.

Diagnosticity can also be derived for any subset of hypotheses and provides the analyst with detailas to how well the evidence distinguishes between the members of the subset. For example, the overalldiagnosticity of a item of evidence may be poor in distinguishing between a set of six hypotheses, yet itmay be very good at distinguishing between just two of those six.

Diagnosticity is derived using the logical conditionals p(h|e) and p(h|e). If these conditionals are notknown, then they can be derived from knowledge of the p(e|h) and p(e|h), and from the base rate ofthe hypothesis br(h) (see Section V-F.2). Details on how diagnosticity is derived, including examples, aredescribed in the Appendix.

VI. CONCLUSION

The approach of Analysis of Competing Hypotheses using Subjective Logic (ACH-SL) has beendeveloped by the authors to address some of the key analytical issues within the defense, intelligence,and law enforcement communities. ACH-SL is not meant as a replacement of ACH, but instead is anelaboration of the basic ACH process that is consistent with the original intent. More than just a theoreticalconcept, ACH-SL is also a functioning, implemented technology known as ShEBA – developed by DSTC– which provides a framework for the analysis of multiple hypotheses with multiple items of evidence.

The Subjective Logic calculus used by the system provides a means of translating between both termsused by human agents, and bayesian data used by other systems. The ability to express calculated opinionsin everyday human terms allows the model and results to be more easily translated into appropriatelanguage for presentation to policy makers and other non-analysts.

ACH-SL uses a formal calculus, known as Subjective Logic to make recommendations about thelikelihoods of the hypotheses, given individual analyst judgements, uncertain knowledge about the valueof the evidence, and multiple items of evidence. In addition, ACH-SL derives measures of diagnosticityfor each item of evidence with respect to the hypotheses directly from the judgements that analysts make,rather than requiring them to consider diagnosticity as a separate, second-order measure of the evidence.

ACH-SL allows analysts to apply both deductive or abductive reasoning, and minimize the errors thatoften occur as a result of consideration of only one logical conditional. The ability of ACH-SL to allowboth approaches helps to ensure that the individual tasks undertaken by analysts require less possiblecognitive effort. This in turn allows analysts to focus on the judgements they make to produce higher-quality analysis as a result.


REFERENCES

[1] M. Clark, “Intelligence activities,” Commission on Organization of the Executive Branch of the Government [the Hoover Commission],Tech. Rep., June 1955, interim report to Congress prepared by a team under Gen. Mark Clark.

[2] M. Warner, “Wanted: A definition of “intelligence”,” Studies in Intelligence, vol. 46, no. 3, 2002. [Online]. Available:http://www.cia.gov/csi/studies/vol46no3/article02.html

[3] A. Tversky and D. Kahneman, “Judgement under uncertainty: Heuristics and biases,” Science, vol. 125, pp. 1124–1131, 1974.[4] ——, “The framing of decisions and the psychology of choice,” Science, vol. 211, no. 4481, pp. 453–458, 1981.[5] M. Thurling and H. Jungermann, “Constructing and running mental models for inferences about the future,” in New Directions in

Research on Decision Making, B. Brehmer, H. Jungermann, P. Lourens, and G. Sevon, Eds. Elsevier, 1986, pp. 163–174.[6] K. J. Gilhooly, Thinking: Directed, Undirected and Creative. Academic Press, 1996.[7] R. J. Heuer, Psychology of Intelligence Analysis. Washington, D.C.: Central Intelligence Agency Center for the Study of Intelligence,

1999. [Online]. Available: http://www.cia.gov/csi/books/19104[8] G. A. Miller, “The magical number seven, plus or minus two: Some limits on our capacity for processing information,” The

Psychological Review, vol. 63, pp. 81–97, 1956. [Online]. Available: http://www.well.com/user/smalin/miller.html[9] W. Fishbein and G. Treverton, “Making sense of transnational threats,” CIA Sherman Kent School for Intelligence Analysis, Tech.

Rep. 1, 2004. [Online]. Available: http://www.odci.gov/cia/publications/Kent Papers/pdf/OPV3No1.pdf[10] R. Z. George, “Fixing the problem of analytical mind-sets: Alternative analysis,” International Journal of Intelligence and Counter

Intelligence, vol. 17, no. 3, pp. 385–405, Fall 2004.[11] A. Jøsang, “A Logic for Uncertain Probabilities,” International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, vol. 9,

no. 3, pp. 279–311, June 2001.[12] F. J. Stech and C. Elasser, “Midway revisited: Deception by analysis of competing hypothesis,” MITRE Corporation, Tech. Rep.,

2004. [Online]. Available: http://www.mitre.org/work/tech papers/tech papers 04/stech deception[13] A. George, Presidential Decisionmaking in Foreign Policy: The Effective Use of Information and Advice. Boulder CO, USA: Westview

Press, 1980.[14] G. S. Halford, R. Baker, J. E. McCredden, and J. D. Bain, “How many variables can humans process?” Psychological Science, vol. 16,

no. 1, pp. 70–76, January 2005.[15] J. Davis, “Improving CIA analytic performance: Strategic warning,” CIA Sherman Kent School for Intelligence Analysis, Tech.

Rep. 1, 2002. [Online]. Available: http://www.odci.gov/cia/publications/Kent Papers/pdf/OPNo1.pdf[16] K. Burns, Mental Models and Normal Errors. Lawrence Erlbaum Associates, 2004, ch. How Professionals Make Decisions. [Online].

Available: http://mentalmodels.mitre.org/Contents/NDM5 Chapter.pdf[17] A. Jøsang and S. Pope, “Conditional deduction under uncertainty,” in Proceedings of the 8th European Conference on Symbolic and

Quantitative Approaches to Reasoning with Uncertainty (ECSQARU 2005), 2005.[18] B. de Finetti, “The true subjective probability problem,” in The concept of probability in psychological experiments, C.-A. Stael von

Holstein, Ed. Dordrecht, Holland: D.Reidel Publishing Company, 1974, pp. 15–23.[19] J. M. Keynes, A Treatise on Probability. Macmillan, 1921, ch. 4 “Fundamental Ideas”.[20] J. Zlotnick, “Bayes’ theorem for intelligence analysis,” Studies in Intelligence, vol. 16, no. 2, Spring 1972. [Online]. Available:

http://www.odci.gov/csi/kent csi/pdf/v16i2a03d.pdf[21] A. Tversky and D. Kahneman, Judgment under Uncertainty: Heuristics and Biases. Press Syndicate of the University of Cambridge,

1982, ch. Causal schemas in judgments under uncertainty, pp. 117–128.[22] M. Diaz, Topics in the Logic of Relevance. Munchen: Philosophia Verlag, 1981.[23] S. Kent, Sherman Kent and the Board of National Estimates: Collected Essays. CIA, Center for the Study of Intelligence, 1994, ch.

Words of Estimated Probability. [Online]. Available: http://www.cia.gov/csi/books/shermankent/6words.html[24] A. Jøsang and D. McAnally, “Multiplication and Comultiplication of Beliefs,” International Journal of Approximate Reasoning, vol. 38,

no. 1, pp. 19–51, 2004.[25] A. Jøsang, “The Consensus Operator for Combining Beliefs,” Artificial Intelligence Journal, vol. 142, no. 1–2, pp. 157–170, October

2002.[26] A. Jøsang, M. Daniel, and P. Vannoorenberghe, “Strategies for Combining Conflicting Dogmatic Beliefs,” in Proceedings of the 6th

International Conference on Information Fusion, X. Wang, Ed., 2003.


APPENDIXDEFINITIONS, THEOREMS, PROOFS AND EXAMPLES

A. Equivalence of Bayesian and Subjective Opinion

Definition 1.1: Any pure bayesian opinion ϕ = (r, s) with base rate br(ϕ) = aϕ can be transformed intoa subjective opinion ω = (b, d, u) with base rate br(ω) = aω using the transformation function F(ϕ) = ω[11].

F(ϕ) = ω = (b, d, u)

b =r

r+s+2

d =s

r+s+2

u =2

r+s+2

aω = aϕ

(A-1)

Corollary 1.2: Any subjective opinion ω = (b, d, u) can be transformed into a pure bayesian ϕ = (r, s)using the transformation function F ′(ω) = ϕ.

F ′(ω) = ϕ = (r, s)

r =2bu

s =2du

aϕ = aω

(A-2)

B. Logical Conditionals

Logical conditionals are pairs of conditional beliefs that are used for reasoning about the likelihood of ahypothesis, given some evidence. There are two types of logical conditionals used in inductive reasoning– abductive and deductive conditionals – and they have different mathematical requirements placed ontheir values as a result. This section will define the mathematical requirements and constraints for bothtypes of logical conditionals.

1) Deductive Logical Conditionals: The logical conditionals used for deduction, p(hi|e) and p(hi|e),must obey certain mathematical constraints in order to satisfy basic requirements of probability theory.For a complete set of hypotheses where one and only one hypothesis can be true, it logically followsthat the sum of the probability expectations of the positive conditionals must be one; and similarly theprobability expectations of the negative conditionals must also sum to one, since

p(h1) + p(h2) + . . . p(hn) = 1

p(h1) + p(h2) + . . . p(hn) = 1

Definition 1.3 (Deductive Logical Conditionals): Let Φ = h1, h2, . . . hk be a complete set of k hy-potheses where one and only one hi ∈ Φ is true. Then the logical conditionals used for deduction on a


single item of evidence, e (ωhi|e and ωhi|e), must obey the following mathematical constraintsn=k∑n=1

E(ωhn|e) = 1

n=k∑n=1

E(ωhn|e) = 1

(A-3)

2) Abductive Logical Conditionals: The logical conditionals used for abduction, p(e|hi) and p(e|hi),have different constraints to deductive logical conditionals and there is no requirement that the probabilityexpectations of their positive conditionals sum to one, so the probability expectation of the positiveconditional p(e|hi) may be unconstrained for each hypothesis. However, the probability expectation ofeach negative conditional p(e|hi) must be equal to the average of the probability expectations of the otherpositive conditionals of the other hypotheses, i.e.

p(e|h1) =p(e|h2) + p(e|h3) + . . . p(e|hn)

n− 1(A-4)

Definition 1.4 (Abductive Logical Conditionals): Let Φ = h1, h2, . . . hk be a complete set of k, (k >1) hypotheses where one and only one hi ∈ Φ is true. Then the logical conditionals used for abductionon a single item of evidence, e (ωe|hi

and ωe|hi), must obey the following mathematical constraints

∀ hi, hi ∈ Φ, E(ωe|hi) =

n=k∑n=1

E(ωe|hn)−E(ωe|hi)

k−1 (A-5)

and consequently, the following two conditions must also hold true

E(ωe|hi) =

n=k∑n=1

E(ωe|hn)− (k − 1) · E(ωe|hi

) (A-6)

n=k∑n=1

E(ωe|hn) =n=k∑n=1

E(ωe|hn) (A-7)

C. Model coherence

The belief values assigned to the deductive logical conditionals of different items of evidence for thesame hypothesis must share a common overlapping range for the model to be coherent. If there is nocommon overlap in the ranges of belief assigned to the logical conditionals, then at least one of the logicalconditionals of either item of evidence must be incorrect.

Definition 1.5 (Model Coherence): Let Φ = h1, h2, . . . hk be a state space for a set k hypotheseswhere one and only one hi ∈ Φ is true. Let ξ = e1, e2, . . . en be a set of n items of evidence. Then amodel is defined to be coherent for a hypothesis, hi, if and only if

lower(hi) ≤ upper(hi) (A-8)

where the upper bound, upper(hi) and the lower bound, lower(hi) are defined as

lower(hi) = max (∀ ej ∈ ξ, min( p(hi|ej), p(hi|ej) ))

upper(hi) = min (∀ ej ∈ ξ, max( p(hi|ej), p(hi|ej) ))(A-9)


D. Diagnosticity

Diagnosticity is a measure of how well evidence distinguishes between hypotheses, based on knowledgeof the logical conditionals p(hi|e) and p(hi|e).

Definition 1.6 (Diagnosticity of evidence): Let Φ = h1, h2, . . . hk be a state space for a set k hy-potheses where one and only one hi ∈ Φ is true. Let ΩΦ = Θ1, Θ2, . . . Θm be the corresponding setof m state spaces for a single item of evidence, e (where Θi = ei, ei, Θi ∈ ΩΦ) that represent theconditionals ωhi|e, ωhi|e for each hypothesis hi ∈ Φ. Then we define the diagnosticity of the evidence ewith respect to an arbitrary subset of hypotheses H ⊆ Φ and H * Φ with elements k > 0 to be

D(e,H) =

Etotal(e,H) = 0, 0;

Etotal(e,H) > 0,

n=k∑n=1|E(ωhn|e)−E(ωhn|e)−Dmean(e,H)|

Etotal(e,H)

(A-10)

where Dmean(e,H) is the mean of the sum of the differences, and Etotal(e,H) is the sum of theirexpectations, defined respectively as

Dmean(e,H) =

n=k∑n=1

[E(ωhn|e)−E(ωhn|e)]k (A-11)

Etotal(e,H) =n=k∑n=1

[E(ωhn|e) + E(ωhn|e)

](A-12)

Then the diagnosticity of the evidence e with respect to an arbitrary subset of hypotheses H can berewritten as (substituting A-11 and A-12 into A-10):

D(e,H) =

n=k∑n=1

∣∣∣∣E(ωhn|en)−E(ωhn|en)−n=k∑n=1

[E(ωhn|en )−E(ωy|en )

k

]∣∣∣∣n=k∑n=1

[E(ωhn|en)+E(ωhn|en)](A-13)

Remark 1.7: It can be seen that D(e,H) ∈ [0..1] where a value of zero indicates that the evidence lendsno weight to any of the hypotheses, while a value of 1 indicates that at extreme values for the evidence(i.e. E(ωe) = 0 ∨ E(ωe) = 1), one of the hypotheses, hi ∈ H , will be absolutely true and for the otherextreme, one or more will be absolutely false.

Lemma 1.8 (Diagnosticity of evidence for a complete set of hypotheses): The diagnosticity of the evi-dence e for a complete set of hypotheses Φ can be expressed in a simplified form as

D(e, Φ) =

n=m∑n=1

∣∣E(ωhn|e)− E(ωhn|e)∣∣

2(A-14)

Proof: For a complete set of hypotheses, where k = m,P = Φ, the sum of the expectations of theconditionals will be exactly two, (i.e. Etotal(x, Φ) = 2) since

n=m∑n=1

E(ωe) = 1,n=m∑n=1

E(ωe) = 1 (A-15)


and also Dmean(e, Φ) = 0, sincen=m∑n=1

[E(ωhn|e)− E(ωhn|e)

]= 0

So the diagnosticity of the evidence e for a complete set of hypotheses Φ can be simplified from A-13:

D(e, Φ) =

n=m∑n=1


2

1) Example 1: Consider an exhaustive set of hypotheses H = h1, h2, h3, and the respective expec-tations of their conditionals with respect to a item of evidence, e to be:

E(ωh1|e1) = 1.0 E(ωh1|e) = 0.0E(ωh2|e2) = 0.0 E(ωh2|e) = 0.35E(ωh3|e3) = 0.0 E(ωh3|e) = 0.65

Since both the sums of the positive conditionals and the sums of the negative conditionals of H add upto 1, i.e.:

n=3∑n=1

E(ωhn|e) = 1,n=3∑n=1

E(ωhn|e) = 1

and therefore the mean of the difference between the positive and negative conditionals is zero, and thesum of all conditionals is exactly 2, i.e.

Dmean(e,H) = 0, Etotal(e,H) = 2

it follows that the simplified form (A-14) can be used to obtain:

D(e,H) =|1.0− 0|+ |0− 0.35|+ |0− 0.65|

2= 1.0

The diagnosticity of the evidence e in respect of all hypotheses is D(e,H) = 1.0 – since when the evidenceis true, h1 must be true and all other hypotheses (h2, h3) must be false. Furthermore, the diagnosticitieswith respect to the subsets of H (using A-13) are:

D(e, h1, h2) = 1.0

D(e, h2, h3) = 0.3

D(e, h1, h3) = 1.0

so that it can be seen that the evidence is capable of distinguishing perfectly between h1 and h2, andbetween h1 and h3, but cannot distinguish well between h2 and h3, (assuming h1 is false).

2) Example 2: Consider another exhaustive set of hypotheses H = h1, h2, h3, h4, and the respectiveexpectations of their conditionals with respect to a item of evidence, x to be:

E(ωh1|e) = 0.5 E(ωh1|e) = 0.1E(ωh2|e) = 0.2 E(ωh2|e) = 0.0E(ωh3|e) = 0.15 E(ωh3|e) = 0.5E(ωh4|e) = 0.15 E(ωh4|e) = 0.4


The diagnosticity of the evidence e with respect to a subset of the hypotheses H ′ ⊂ H, H ′ = h1, h2, h3will be D(e,H ′) and since both the sums of the positive conditionals and the sums of the negativeconditionals of H ′ do not add up to 1, i.e.

n=3∑n=1

E(ωhn|e) = 0.85,n=3∑n=1

E(ωhn|e) = 0.6

and thereforeDmean(e,H

′) = 0.25, Etotal(e,H′) = 1.45

it follows that the more usual form (A-13) must be used to obtain D(e,H ′):

D(e,H ′) =|(0.5− 0.1− 0.083|+ |0.2− 0.0− 0.083|+ |0.15− 0.5− 0.083|

1.45= 0.6

with the diagnosticities with respect to the subsets of J ′ being:

D(e, h1, h2) = 0.25

D(e, h2, h3) = 0.65

D(e, h1, h3) = 0.3

E. Relevance

Relevance is a measure of how relevant the evidence is for determining the likelihood to a hypothesis,based on knowledge of the logical conditionals p(hi|e) and p(hi|e).

Theorem 1.9 (Relevance of evidence): The relevance of evidence e with respect to a single hypothesish is defined as the difference between its conditionals (e, e)

R(e, h) = |p(h|e)− p(h|e)| .

It can be seen that R(e, h) ∈ [0, 1], where R(e, h) = 0 expresses total irrelevance/independence, andR(e, h) = 1 expresses total relevance/dependence between e and h. For belief conditionals, the same typeof relevance can be defined being equivalent to the diagnosticity of evidence e with respect to a singlehypothesis hi and its complement hi.

R(e, hi) = D(e, hi, hi) =∣∣E(ωhi|e)− E(ωhi|e)

∣∣ (A-16)

Proof: Let Φ = h1, h2, . . . hm be a state space for a set m hypotheses. Let ΩΦ = Θ1, Θ2, . . . Θmbe the corresponding set of m state spaces for a single item of evidence, x (where Θi = x, x) thatrepresent the conditionals ωhi|e, ωhi|e for each hypothesis hi.

Then, the diagnosticity for a single hypothesis hi ∈ Φ and its complement hi = Φ\hi = h1, h2, . . . hm\hi (with k = m − 1 elements), can be defined by the diagnosticity of the set Q = hi, hi where theconditionals for hi are defined as

E(ωhi|e) =n=k∑n=1

E(ωhk|e), and E(ωhi|e) =n=k∑n=1

E(ωhk|e)


It can be shown from A-14 that Q = hi, hi form a complete set of hypotheses since both Dmean(e,Q) = 0(A-11) and Etotal(e,Q) = 2 Therefore, the simplified form for D(e,Q) can be used (A-14), i.e.

D(e,Q) = D(e, hi, hi) =

n=k∑n=1


2

=

∣∣E(ωhi|e)− E(ωhi|e)∣∣−

∣∣E(ωhi|e)− E(ωhi|e)∣∣

2

=2∣∣E(ωhi|e)− E(ωhi|e)

∣∣2

=∣∣E(ωhi|e)− E(ωhi|e)

∣∣

= R(e, hi)

F. Relative Diagnosticity

When considering a multi-dimensional problem with multiple items of evidence and multiple hypothe-ses, it is useful to consider the diagnosticity of evidence relative to the diagnosticity of all other evidence,rather than its actual diagnosticity (see 1.6).

Definition 1.10 (Relative Diagnosticity): Let the set of k items of evidence be Ψ = e1, e2, . . . ek, anda set of m hypotheses Φ = h1, h2, . . . hm and where the maximum of the evidence diagnosticities Dmax

is defined as:

Dmax(Ψ, Φ) =n=kmaxn=1

[ D(en, Φ) ] (A-17)

Then the relative diagnosticity Drel for each item of evidence ei ∈ Ψ be defined as the the diagnosticityD(ei, Φ) divided by the maximum of the evidence diagnosticities, Dmax i.e:

Drel(ei, Φ) =

Dmax(Ψ, Φ) = 0, 0;

Dmax(Ψ, Φ) > 0,D(ei,Φ)

Dmax(Ψ,Φ).(A-18)

analysis of competing hypotheses using … of competing hypotheses using subjective logic (ach-sl) 1...

Documents