evaluating trust in a public key certification authority

592

AbstractWith the growth of many different public key infras-tructures on the Internet, relying parties have the dif-ficult task of deciding whether the sender of digitallysigned message is really who the public key certificatesays they are.We have built an expert system that cal-culates the amount of trust, or trust quotient, that onecan place in the name to public key binding in a cer-tificate.The structure of the expert system is based onthe CPS framework of Chokhani and Ford (RFC2527), whilst the relative importance of the variousfactors that comprise the trust quotient, were deter-mined by interviewing PKI experts from around theglobe. This paper discusses the knowledge analysisstrategy employed to collect this expert informationand how we used it to develop the KBS.The analysisof the results of the interviews are also presented, andthey can be summarised succinctly as “there are somefactors concerning trust in a PKI which nearly allexperts agree upon, and there are other factors inwhich there is very little agreement at all”. Theimportance of identifying contextual factors whenbuilding a knowledge base is very important. Inmany cases, a disagreement between experts, as shownby a bimodal split in importance, was traced to differ-ences in context and we show how this can be asource of new knowledge.Keywords: Trust, Trust Quotient, Public Key Infrastructure,Certification Authority, Expert System, Certification PracticeStatement, Certificate Policy, X.509.

IntroductionOne relies on a public key infrastructure (PKI) toprovide a binding between the name of a subject andits public key. The subject is the holder of a public-private key pair, and can be a person or a computerapplication.The binding is provided by a PKI trustedentity called a certification authority (CA) that digi-tally signs the subject name/public key pair therebycreating a digital certificate. The internationallyrecognised standard for digital certificates is X.509[1].The relying party (RP) (the person or applicationthat receives a digitally signed message from the sub-ject) has to trust the PKI that this binding is still valid,otherwise the relying party can be duped into accept-ing a forged message from an attacker who has man-aged to either break the key pair or steal (possibly acopy of) the subject’s private key.

As the number of PKIs grows, and as e-commerceand inter-organisational data transfers increase, onecan increasingly expect the message sender and mes-sage receiver to be the subjects of different PKIs.Thusthere will be a need for a message receiver (relyingparty) to depend upon, or trust, a foreign PKI. Justhow is a RP supposed to base its trust decisions on aremote PKI that might be completely unknown to it?

Our research has tried to quantify the elements thatgo towards building the elusive quality of trust that isplaced in a remote PKI CA by the relying party.The

Evaluating Trust in a PublicKey Certification AuthorityDavid W Chadwick and Andrew Basden, IS Institute, University of Salford, Salford M5 4WT, England

Computers & Security Vol.20, No.7, pp.592-611, 2001

Copyright © 2001 Elsevier Science Limited

Printed in Great Britain. All rights reserved

0167-4048/01$20.00

593

work is based upon an Informational RFC byChokhani and Ford [2] that lists all the componentsthat a CA should take into account when writing itspolicy, practice statement and procedures.

Some people argue that trust cannot be objectivelycalculated and given a scalar value, and in many ways they are right. Trust does have a personal ele-ment to it, built from experience, personal knowl-edge, and bias. It also has a contextual element to it,based on the action that the trustee is about to take.However, whilst others might argue that objects can-not be given a scalar monetary value1, never-the-lessthis does not stop people from doing it and gainingsome benefit from the exercise. In the same vein, wehave attempted to give an objective scalar value totrust, which can then be used as is, or modified oreven ignored by the relying party. We believe thatthere is benefit to the relying party, not only in thetrust value calculated by our process, but also fromparticipating in the trust calculation process that wehave developed.

We have built an expert system that computes a “trustquotient” for a PKI CA, with output values that lie inthe range from 0 (meaning completely untrustwor-thy) to 1 (meaning absolutely trustworthy). Theexpert system asks the user a series of questions aboutvarious aspects of the CA, the answers to which canbe gleaned from the CA’s policy and practice state-ment. As an analogy, it is similar to calculating theintelligence quotient (IQ) of someone by asking thema series of questions and calculating a result based ontheir answers.

We know that our expert system is bounded correct-ly, in that it gives a score of 0 when every question isanswered with the least trustworthy answer possible,and a score of 0.97 when every question is answeredwith the most trustworthy answer possible.The reasonthat the expert system can never provide an answer of1 is due to the bayesian logic that is employed in some

places, to which unity is an asymptote that is neverfully reached. One can see the use of bayesian rea-soning as a tacit admission of a lack of knowledge: thatthere are yet other factors that might provide slightevidence that are not included in the knowledge basefor some reason. One reason for knowledge beingomitted is that not all the relevant factors are known,and this is particularly true for an issue like trust.

This paper describes the process that we wentthrough in order to determine the weightings to begiven to each node in the inference net of the knowl-edge base.The results of our research can be summedup in one sentence “there are some aspects of trustthat nearly all the experts agree upon, and otheraspects upon which there is little agreement at thecurrent time”. This result is at the same time bothquite worrying but not entirely unexpected. It is quiteworrying that if world leading experts in the field ofPKI cannot agree upon which aspects are the mostimportant in computing trust in a PKI, then whathope do the vast majority of less skilled users of PKIshave? However, it is not an entirely unexpected result,given that PKI products are still in their infancy andthat the practical experience and body of knowledgeabout what makes one PKI more trustworthy thananother is simply not available.There has been insuf-ficient time for this knowledge to grow, or for con-sensus to be reached, or for it to be assimilated by theprofession. In comparison, this is not the case in amore well established profession such as quantity sur-veying, where one could expect that many differentexperts will all arrive at approximately the same costfor a new construction project.

Knowledge Sources andKnowledge AcquisitionTo make progress in the real world assessment of trustwe have three alternative sources for the knowledgeto be used (and which knowledge we would thenseek to encapsulate in a knowledge base):

1. Use some agreed standard reference work. Theadvantage of this is that such a source encapsulatesmuch of the real previous experience, already

Computers & Security, Vol. 20, No. 7

1 For example, what price do you place on a cup of water destined for i)a person in a restaurant or ii) a person in the Sahara desert, or how do youuse the same scalar to put a price on a can of baked beans, an exquisitediamond and a symphony by Mozart.

Evaluating Trust/David W Chadwick & Andrew Basden

distilled into categories and rules. But it has sever-al disadvantages, chief of which are that it is un-likely to apply in every situation, and secondly, thatthe knowledge that it holds is often several yearsold, so in a fast changing sector it becomes suspect;

2. Seek to capture the experience and views ofexperts or practitioners in the field. Such a knowl-edge source can be more up to date, and it canhighlight factors that vary between situations. Butexperts often disagree - and this approach providesno means of resolving such disagreements - andknowledge of this kind often lacks generality;

3. Seek principles of the domain of knowledge.Principles are the answers to the questions “Why?”or “Why not?” and provide both generality andunderstanding. Because of the former the knowl-edge base can be made more robust across a vari-ety of situations, as long as the context specific fac-tors have been separated out. Because of the lat-ter, a knowledge base can provide higher qualityaid to its user, both in educating them and in giv-ing them a basis on which to depart from theresults that the knowledge base gives. Principlesinclude theory, but go beyond it to include thosegeneralities found in real life that are intuitive inform.

For a fuller discussion of the relationship betweenprinciples and experience, see Attarwala and Basden[3]. An approach based on seeking principles can notonly overcome disagreements between experts butcan in fact turn them to our advantage, in that theyare pointers to rich nuggets of knowledge yet to be uncovered. This can be done by asking the“Why?” question, which has the effect of separatingout the agreed principles from the expert’s own spe-cific context of working. This is possible because thetype of experience encapsulated in 2. above is in factan amalgam of principles and context expressed bythe equation:

E = U + CDPS

where E is Experience, U is Understanding andCDPS is Context-Dependent Problem Solving.This means that, if we have some E in our posses-sion, then if we are able to discern either of its com-

ponents, U or CDPS, then the other also becomesapparent.When this occurs a higher quality knowl-edge base can be built. In all, four types of questionare helpful in achieving this (Basden, Watson andBrandon [4]):

• “Why?” - to obtain understanding of the linksbetween concepts;

• “What is?” - to obtain understanding of the mean-ing and importance of concepts themselves;

• “When not?” - to obtain context-dependent factors in the links between concepts;

• “What else?” - to obtain context-dependent factors from concepts.

In this project we have used all three approaches inharmony. First, we based our knowledge base onChokani and Ford [2], a standard reference work onCPSs. Whilst this is the primary standard referencework to date, never the less it is approximately 2 yearsold, and due to the rapid developments in PKI tech-nology and experience, a revised version of this is cur-rently being worked upon. Second, working from theoverall structure, and some detailed knowledge, thatthis gave us, we sought the opinions of experts as towhat they believed to be the relative importance ofthe factors.As expected, we found much disagreementbetween them, and so, third, when we asked them toexplain their views, using especially the “Why?” ques-tion above, it transpired that quite often they wereassuming different context-dependent factors. As aresult, we were then able to make a decision about thecontent of the knowledge base.

The Process of KnowledgeRefinementThe process of building a knowledge base (KB)involves identifying the relevant pieces of knowl-edge, deciding how they relate to each other, assign-ing weightings to the interactions, and all of this toan appropriate degree of detail. Traditionally, theideal for this process has been monotonic in nature,such that pieces of knowledge are assembled into theKB and once there never need removing or chang-ing.With well-structured knowledge this ideal might

594

595

be approached if the knowledge acquisition processis carefully executed by a skilled knowledge engi-neer. Any need to remove or change knowledgealready in the KB is seen as a deficiency in theknowledge engineering process. Winograd [5] callsthis a ‘constructor’s-eye-view’ of KB development,and Basden and Hibberd [6] call it the assemblyapproach. The ideal sequence of steps to building aKB under this approach is a linear one of steps ofincreasing detail.

At first we attempted to follow this in the project,because it is suited to working from standard refer-ence works, as shown in Fig.1.

We built a KB largely based on Chokani and Ford,and this provided the overall structure. Weights had tobe assigned, as the next level of detail, so we drew upa questionnaire with which to approach experts, sothat we could obtain their weights, which we wouldthen insert into the KB. It was hoped that the overallstructure of the KB would not need any modification.

Fig 1. Initial Conception of Process of Building aKB

But most knowledge is not sufficiently well struc-tured to allow this ideal to be reached, even withhighly skilled knowledge acquisition, and our KBwas no exception. The ideal as well as the practiceof building a KB for less structured knowledge ininherently non-montonic, so that knowledge thathas been placed in the KB will often be changed,replaced or renewed by a process of continuous

reinterpretation. Winograd calls this a ‘designer’s-eye-view’ and Basden and Hibberd call it the cre-ative design approach.

In the creative design approach, knowledge is developed and refined during the process of build-ing the KB, and of using the KB, and as a direct resultof these. There are several reasons why this is so (Basden et al [7]). One is that even the ex-perts in the field do not possess all the necessaryknowledge (for the appropriate degree of detail) and,when required to provide it, find gaps therein whichmust be filled (Hines and Basden [8]). A related rea-son is that experts will disagree, largely because theyoperate in different contexts, and thus provide knowl-edge that is inconsistent with that from others.Another reason is that the knowledge required for theKB is not routinely exercised (for example knowledgeabout emergency or other rare situations) and thus isnot developed to a sufficient degree of detail. A fourth reason is that the knowledge is being appliedin a new context, and thus not only must it be reinterpreted for the new context, but new factorsbecome relevant that were previously not so. A fifth reason is that the field of knowledge is beingapplied in fundamentally new ways, such as the appli-cation of first principles where previously heuristicshad been sufficient (Basden and Hibberd [6]).

In our case most of these reasons applied to someextent (though, by using Chokani and Ford as ourmajor source it was hoped that we could avoid the fourth and fifth problems). This meant that the KB would require changing. The process bywhich a KB changes is often by repeated applicationof a simple cyclical development, such as in Fig. 2,in which a KB, at some degree of development,has its knowledge examined by those who havesome expertise in the field, sometimes by means of desk checking but often by actual usage of the knowledge in real situations. This identifies anumber of problems and the KB is then changedaccordingly. It often happens that later cycles changethe changed knowledge several times more beforethe final version is achieved. Methodologies havebeen developed for such processes e.g. KADS [9],CCM [4].


Chokani and Ford

KB 1

Questionnaire

Weights

Final KB


Fig. 2. Cycle of development

But in our project the process was more complex. Aquestionnaire, such as that which had been created ini-tially for the purpose of obtaining weightings, can beconsidered a knowledge base, in that it contains a rep-resentation of knowledge of the domain. When webrought this to the experts we found that its represen-tation of the knowledge was deficient, and it wasredesigned for future use. In all, the questionnaire wentthrough four versions (which are described below),and then the changes it had undergone were used tochange the KB from which the original version hadbeen created.This gave a double loop, as seen in Fig. 3.

Fig. 3. Double Loop of KB and QuestionnaireDevelopment

The Questionnaire

Experts were interviewed using a structured question-naire that was developed during the project by trialand error as explained above.The questionnaire basi-cally followed the internal structure of the KB direct-ed graph, and asked the respondents to weight thevarious aspects of trust that were the antecedents of anode in the graph.As the aspects were quite differentin many cases, it was a difficult task to perform, andwas perhaps analogous to asking gourmets to place avalue on the different fruits in a fruit salad. Four dif-ferent questionnaires were produced during theresearch: a pilot questionnaire, followed by versions 1,2 and 3. The pilot questionnaire attempted to askquestions about every node in the graph, starting withthe least significant ones (e.g. which of the followingways are the most trustworthy for authenticatingsomeone: a driver’s licence, a passport, a fingerprint, agun license etc.). Comprising of over 60 questions,this simply proved too large and cumbersome to use,and after interviews with two experts, each lastingtwo hours, the questionnaires were only half complet-ed. We quickly realised that we would never get 4hours of uninterrupted time with leading PKI worldexperts. Never the less, we used the results gainedfrom these two experts to populate the initial nodes inthe KBS graph.

Version 1 removed the least significant questions andconcentrated on the later nodes in the graph thatwould go towards calculating the final trust quotient.Its questions were derived from those nodes that feddirectly into the final goal ‘Can’t trust’, and thus con-cerned the six major broad concepts that directlyimpinge on trust.We found on using this questionnairethat the concepts were sometimes difficult to explainand difficult to elicit answers on, perhaps because someof the questions were worded too vaguely or ambigu-ously, or a context for thought had not been estab-lished prior to asking the question. It was a bit likepushing someone in at the deep end and hoping theycould swim. Consequently the questioner had tospend a lot of time talking to the respondents, explain-ing things to them, and helping them get through thequestionnaire. Nevertheless we had managed to reducethe time to completion to about 2 hours.

596

597

Version 2, the penultimate version that was complet-ed by the most experts and used to populate thenodes in our knowledge base, was a mixture of back-ground awareness raising questions, used to get therespondents thinking about the relevant issues andsetting a context (e.g. What measures should a CAadopt for effective protection of its private key), fol-lowed by questions asking them to weight the variousaspects of trust one against another.The time to com-plete was now under 1.5 hours. Another benefit thatarose from using the v2 questionnaire with experts,and the subsequent analysis of their responses, wasthat it highlighted one or two areas where we did nothave an optimal design for the KBS, as one or twoquestions were seen to be out of place.This indicateda misplaced arc in the KBS directed graph, which wassubsequently corrected. This led to version 3 beingproduced.

The Experts and the Interviews14 PKI world experts were interviewed over a peri-od of 18 months.The experts came from the USA,Canada, Germany, Austria, France, Belgium, Ireland,Italy and the UK. All experts answered version 2,and 5 answered the updated sections in version 3.The interviews were mainly carried out before,during or after conferences, project meetings andIETF meetings that the first author and the expertswere both attending. In this way the travel costs,time and inconvenience of both parties were min-imised.The interviews lasted just over an hour in allcases.The atmosphere was relaxed and informal, andwhilst the author knew of the experts and the workthey were doing in the field of PKI before the inter-views, in many cases they had not met face to faceprior to the interview.The experts are all involved inPKI either as product suppliers, researchers, imple-menters, designers or service providers, and all haveworked on aspects of Internet security for manyyears, some for well over 10 years. The expertsincluded a chair of the PKIX working group and anauthor of the RFC [2] that is the seminal work inthis area.The experts were primarily from a techni-cal background, although some are now in businessmanagement positions, and one is a lawyer specialis-ing in PKI.

The experts were asked to rate various aspects oftrust, one against another, and to provide their ownscale for comparative purposes.The reason for this wasthat the authors did not want to impose artificial lim-its on their expertise. In most cases the experts used ascale from 1 to 10, a few used 1 to 100, and one used1 to 7 (it is possible he may have had a mental topscore of 10 but thought that no single aspect warrant-ed full marks).When combining the marks together,the authors scaled them all so that the highest markfrom each expert in each set of questions was 10.Ratings in the range 8-10 were then given the label“high” importance, 4-7 “medium” importance and 0-3 “low” importance when calculating trust.

There were six different aspects of trust that werecompared.These were:

• Authentication and identification - this asked theexperts to compare, from a trust perspective, dif-ferent aspects of

• identifying and naming the certificate subject;• Obligations - this asked the experts to compare,

from a trust perspective, the different obligations ofthe various entities involved in a PKI;

• Procedures - this asked the experts to compare dif-ferent procedures that a CA might employ from atrust perspective;

• Compliance audit - this asked the experts to com-pare different aspects of the external audit from atrust perspective;

• Malpractice and controls - this asked for compar-isons, from a trust perspective, about the differentcontrols in place so that malpractice is difficult toperform;

• Legal redress - this asked the experts to comparevarious legal aspects from a trust perspective;

• Overall - the experts were asked to compare eachof the above against each other and against severalother aspects judged to be important to trust.

Experts sometimes gave reasons for the scores theyprovided and sometimes they did not. It was notapparent during the interviews, which scores wouldsubsequently prove to be at odds with those providedby other experts. This had to wait until the analysisphase was performed. During the analysis phase, the



first author contacted various participants who hadapparently anomalous rankings, asking them to giverationales for their scores. Their rationales, whereavailable, have been included in the results below.

The results of all the interviews, with all the differentversions of the questionnaire, were used, as appropri-ate, to make changes to the KB. Greater priority wasgiven to the later, more satisfactory, versions of thequestionnaire. As mentioned above, the pilot versionwas used to set a number of the input weightings at the antecedent side of the KB. The results fromversions 1, 2, and 3, which were derived from theconsequent side of the KB, were used not only to set weightings at the consequent side, near the can’ttrust goal, but also to make changes to the structureof the KB.

The ResultsAll the results are described in detail below, and asummary of approximately half of the results is givenin Table 1 in the Conclusions section.

Authentication and IdentificationOne would naturally expect that the primary role ofa certification authority is to identify and authenticatethe subjects, to allocate them a name, and then certi-fy (digitally sign) the name binding to the public keyof the subject. In our research we wanted to find outwhich were the most important factors to take intoaccount when calculating the trust that a relying partycan place in this name to public key binding.

Nine questions were asked about the authenticationof the subject and the identification of the subject viaits name in the certificate. The experts were askedhow important to trust is it that: the CA authenticatesthe subject and proves that it is in possession of thecorresponding private key, before issuing it with apublic key certificate.When creating the name of thesubject, how important to trust is it that the CA usesa globally recognised name form (so that softwarewill be able to handle it successfully); ensures that thename is locally unique to the CA (so that no twosubjects of the CA are allocated the same name);

ensures that the name is globally unique (so that notwo subjects anywhere are allocated the same name),and uses a name that is meaningful to the RP (so thatthe RP can identify the subject from its name).Furthermore, how important to trust is it that the CAmight use trademarks in the name. (This question wasasked because a trademark can typically be registeredin approximately 40 different contexts e.g. confec-tionery, pet food, electrical equipment etc. and thusseveral subjects from different sectors could poten-tially be given the same trademark name.) If thename of the subject’s organisation is included in thesubject’s name, then how important to trust is it thatthe CA authenticates the organisation and that thesubject has a relationship with it. Finally, if two sub-jects apply to be allocated the same name, howimportant to trust is it that the CA has a rival nameclaim resolution procedure.

The two most important factors were almost unani-mously agreed to be that the CA must issue locallyunique names, so that no two of its subjects wouldever get the same name in a certificate, and that thesubjects should be authenticated. 13 out of 14 expertsgave locally unique names a high mark, with over70% of the experts giving it maximum marks. Onegave it a medium score.Authentication of the subjectcame a close second, with 12 out of 14 experts giv-ing this a high mark and 78% the maximum mark.There was however one dissenting voice who gavethis a very low score of 1, and another who gave it amedium score. When the former was subsequentlyquestioned about the low score, this well respectedexpert replied:

If CAs were willing to assume residual liabili-ty for mis-identification (i.e. if the personturned out to be different from the “claimedperson”), then there would be a reasonableargument for requiring CAs to stronglyauthenticate the people to whom they issuecertificates. But this isn’t true today; todayCAs without exception in my knowledge dis-claim liability for mis-identification. All theydo is say “here’s what we do to authenticateusers; if this isn’t good enough for you, don’tuse our service”. I submit that CAs will

598

599

NEVER be able to assume this liability,because they don’t have a sufficient financialreserve to absorb losses due to fraud if they are deceived about identity. Therefore, theywill never be the source of authority for identity (because they can’t sustain the liabili-ty associated with being the source of author-ity). If they aren’t the source of authorityabout identity, then they need not stronglyauthenticate it.

However, most of the experts would not agree,because they see the issue from a different perspective.The CA has a duty of care to authenticate the subjectaccording to the procedures in its CPS. This is themeaning of authentication in this questionnaire. If theCA requires a person to be present face to face withtheir passport, then this is reasonably strong authenti-cation. However, if the person is a fraudster andknowingly presents a false passport, then a CA willusually disclaim liability for this. This seems reason-able. Similarly a passport office will not take the lia-bility for losses a company incurs due to a person pre-senting a false passport. In practice then, a CA willdisclaim liability from fraud.As an example, this is theliability excluded by Viacode [10] in its CertificatePolicy. It states:

…in particular the CA excludes:i) any warranty as to accuracy (sic) or reliabil-ity of any information contained in theCertificates which is not supplied by the CA,

But Viacode does still perform reasonably strongauthentication on subjects, in that it requires the faceto face visit of the subject along with one photo baseddocument such as a passport and two secondary doc-uments such as a utility bill, bank statement or driv-er’s licence.The authors therefore agree with the vastmajority of experts that subject authentication is aprimary factor in determining trust in a CA.

The above is an example of how using approach 2(seeking to capture views and experience of experts)led to a disagreement which was resolved to somedegree by then adopting approach 3 (seek principlesby asking “Why?”). The difference of view can be

understood as follows.The argument of the dissentingexpert is:

1. CAs do not have sufficient financial reserve.2. Therefore they cannot sustain liability in cases of

fraud.3. Therefore they cannot guarantee identity.4. Therefore they need not strongly authenticate it.5. Therefore strong authentication is not a good

indicator of trust.

However, that logic rests on an assumption that theuser of the knowledge base is only interested in theextreme cases. Most cases are less extreme, and in suchcases strong authentication is a good indicatorbecause, while it cannot guard against every fraud itwill prevent most cases of masquerading. Therefore,we can see that the difference between the two expertviews rests on contextual assumptions. The authorshave decided that, for present purposes, the knowl-edge base will assume the non-extreme case. Butwhether it is wise to do so is returned to below.

Authentication of the organisation (assuming theorganisation name was part of the subject’s name)came a little behind authenticating the person, butmost experts gave this factor the same score asauthenticating the person. If the organisation name isnot part of the subject name, then experts quite nat-urally said that organisation authentication was notrelevant to calculating trust in a certificate.

The remaining factors all gained medium scores, apartfrom the use of trade marks, which scored low.

Just over half the experts gave proof of possession ofthe private key high importance, the remainder weresplit between medium and low importance. If proof ofpossession (PoP) is not confirmed by the CA, then itwould be possible for an authenticated person to saythat someone else’s public key was their own, and getit certified as such. However, the types of attack thatcan launched with this false certificate are few and farbetween, the main one being an assertion that a signedmessage was created by the attacker instead of the gen-uine private key holder. Clearly if the latter still has theprivate key, it is trivial to rebut this assertion, but if a



long period has elapsed between the signing and thefalse assertion, and the private key no longer exists,then it may be more difficult to rebut. One expert saidthat PoP was not necessary to consider, since it shouldbe proved as a matter of course in the signed certifica-tion request message to the CA. In conclusion, theauthors do not judge this factor to be of high impor-tance when computing trust, and would agree withthose experts who gave this a low rating.

Half the experts said that using meaningful names wasof high importance, with a over a third giving it lowimportance, and only 14% medium importance. Howdo we account for this high/low split? It is almost cer-tainly due to the expert’s implied context of use of thecertificate. One expert said that a meaningful namecan never be sufficient to identify a subject in all casese.g. between a father and son with the same name.Another expert said that the application may have itsown way of linking the name in the certificate to thereal person. Perhaps it can be summed up by theexpert who judged it to be of maximum importance(10) if the certificate was to be used by humans, andminimal importance (1) if the certificate was to beused for automatic processing by an application.

Similarly, using a globally recognised name form andglobally unique names were judged by half theexperts to be of high importance, by over a third to beof low importance, and by only a minority to be ofmedium importance. Again this marked difference ofopinion can probably be accounted for by the expert’simplied context of use.To support this view, a coupleof experts qualified their answers and said that if theapplication was global in scope or being used in anopen environment, then globally recognised nameforms and globally unique names were of high impor-tance (9 or 10) but if the application was a locallydefined one then global factors were only of minimalimportance (1 or 2).

The use of trademarks scored a low rating by overthree quarters of the experts. Only two expertsthought this had any real significance towards calcu-lating trust. One was the operator of a CA service,who said that brand names can be very important insome contexts.

Obligations

The experts were asked to rank the obligations of thevarious parties involved in a PKI. These were: theobligations of a CA to its subjects (e.g. to issue certifi-cates and revocation lists in a timely manner), theobligations of a CA to its relying parties (e.g. to issuerevocation lists in a timely manner), the obligations ofthe subjects (e.g. to keep their private key from beingused by anyone else), the obligations of the relyingparties themselves (e.g. to make sure they have the lat-est revocation list if they need it), and the obligationsof the repositories (e.g. to publish certificates andCRLs to their clients).

A majority of the experts thought that the obliga-tions of the CA to the relying party was the mostimportant factor. It scored significantly higher thanthe next factor (subject obligations). Only one expertgave this a low rating, and four gave it a medium rat-ing. This is quite an important finding, especiallygiven that no formal contract will exist between arelying party and a remote CA.The CA will not evenknow that a particular RP exists, yet its obligations tothis unknown entity are judged to be the mostimportant of all its obligations from a trust perspec-tive. Interestingly, the lawyer placed this obligationbelow that of the CA to its subjects. This can beexplained by the fact that the security experts and thelawyer are viewing the obligations of the CA to theRP from different perspectives, namely from ethicaland juridical perspectives. The expert who gave thisfactor a low rating explained his answer by saying thatthe trustworthiness of a CA is initially determined byits subscribers (who pay it money) prior to RPsbecoming involved.

This difference of opinion raises an issue that is per-haps germane to the future shape of Internet activityand the smooth running of virtual organisations.Thedifference is whether the lack of a formal contractbetween CA and RP means that the CA’s obligationto the RP is important or unimportant.There are sev-eral layers to contract law: the contract itself, impliedterms, common law (or Roman Law in some coun-tries) and, finally, natural justice, all of which applywhether they have been codified or not.

600

601

The CA’s obligation to the RP lies in one of the lat-ter categories. Indeed, if we take the view that a majorreason for the existence of the whole edifice of CAs,CPSs and Certificates is that relying parties may rely,then the CA’s obligation to the RP is fundamentaland might come under the last category of naturaljustice.This view of law would provide a foundationin law for the beliefs of those experts who hold thatthe CA’s obligation to the RP is important eventhough not mentioned in a contract.

Its implication for the future shape of Internet activi-ty lies in the elevation of the importance of naturaljustice and the use of less formal mechanisms. Becausethe Internet is a new legal context, and also because itis international, we can expect that many long-heldassumptions of the legal fraternity must be questioned.New principles must be worked out, and these princi-ples will be based on those of natural justice. Jarvenpaaand Leidner [11] have discussed some of these issues,and perhaps go even further to advocate the impor-tance of informal mechanisms for regulating activityon the Internet, similar to those found in village life.Much depends on the attitudes and assumptions ofthose taking part (Kuosa and Basden [12]).

The second most important factor in trust was judgedto be the obligations of the subject (to the RP and tothe CA). Nearly half the experts gave this a high rat-ing, whilst the remainder were split between a medi-um rating and a low rating. Only three experts gavethis item a higher rating than the obligations of theCA to the RP, the majority gave it the same or alower rating. So we have a very good indication thatmost experts regard the obligations of the CA to theRP as being more important than the obligations ofthe subject to the RP.

The obligations of the CA to its subjects gave an inter-esting result. Nearly half the experts gave this a hightrust rating, and the other half a low or medium trustrating, with more giving it a low rating. Clearly thereis a split viewpoint over this item.The authors thinkthis is perhaps due to some of the experts misunder-standing the context. Clearly from a subject’s perspec-tive, the obligations of the CA are important. But froma relying party’s perspective, the obligations of the CA

to its subjects, over and above the CA’s obligations tothe RP, do not significantly affect the trust that the RPhas in a digitally signed message from a subject.As oneexpert commented, the issuing of a certificate to a sub-ject does not affect the trust of the RP, the certificateeither exists or it does not, and without it there is noPKI for that subject (and therefore nothing to be trust-ed). Consequently the authors agree with the expertswho gave this a low trust rating.

The obligations of the relying party gave anotherinteresting result. Nearly half the experts gave this alow score, with nearly a third giving it a score of zero,meaning that it has zero effect on the trust calcula-tion. One expert added: “because I am the relyingparty I obviously trust myself ” (to carry out my obli-gations), implying that trustworthiness does not effecthow much trust is placed in a remote CA. Otherexperts had completely the opposite viewpoint andthree gave it a maximum score of 10. One expert saidthat the RP has an obligation to distinguish anoma-lous behaviour and to continually re-evaluate the trustit places in each certificate subject. Another said thata RP would look particularly foolish if it had trusteda signed message after the certificate had already beenrevoked and the revocation list published. Clearlythrough its own actions (or lack of them) the RP hasonly served to hurt itself.Therefore its obligations areimportant, and how well it carries them out will affectthe trust decision it arrives at.

Such explanations force the designers of a knowledgebase to proscribe more precisely the meaning of thetrust quotient. In this case, it highlights the differencebetween the trust quotient as descriptive of the wholesituation, including the RP, and the trust quotient as atool to be used by the RP that includes only thosefactors external to the RP him/herself. In the formercase the RP’s own obligations are important to calcu-lating the trust quotient, in the latter case they are not.The authors have decided to opt for the latter mean-ing, and to leave it up to the RPs to recognise theirown limitations.

The obligations and use of repositories and their role in the calculation of trust proved to be a contro-versial issue in this research.The experts were almost



equally split between giving it a high, medium andlow score, with low slightly winning. How do weexplain this difference in viewpoint? At the oneextreme we had experts who gave this a maximumscore of 10 towards calculating trust, and said that therepository had to be there with high availability (24hours a day) making revocation information availableas soon as possible. Other experts however, agreed thatrepositories should have high availability with backupsand disaster recovery procedures, but that their role inthe calculation of trust was minimal. Repositorieswere there for efficiency reasons only, to provide revo-cation information as soon as possible, but their obli-gations do not effect the trust calculation. All therepositories do is reduce the time window for makingthe trust calculation, but they do not effect the trustcalculation in themselves. Yet other experts thoughtthat repositories had no role to play in the calculationof trust whatsoever, since one can never protectagainst denial of service attacks, and therefore onecannot rely on repositories to provide any revocationinformation for an RP’s trust decisions.

One expert perhaps summed it up by saying reposito-ries are like an on/off switch. If they are there andprovide current revocation information, then you canperform your trust calculation, if they are not thereand you don’t have current revocation information,then you cannot perform your trust calculation justyet (or if you do perform it then you have to add in arisk factor that the certificate has been revoked, or re-visit your decision once current revocation informa-tion is available). The authors would tend to agreetherefore that the obligations of the repositories havelittle to do with the calculation of the trust a RP canplace in a digital signature. If they are present and pro-vide revocation information, then an RP has the nec-essary information available to it to perform the cal-culation. If they are currently absent, then the RPshould wait until current revocation information isavailable.

ProceduresAt the heart of a CA’s operations are its procedures.Since it is well documented that humans are usuallythe weakest link in any security chain, clear effective

procedures, supported by strong controls, are two waysof strengthening this weakness. The experts wereasked how important it was that a CA has effectiveprivate key protection procedures, effective key man-agement procedures (for backup, recovery and renew-al of its private keys), effective personnel recruitmentand training procedures, effective subject key revoca-tion and suspension procedures, and that the require-ments placed on repositories are clear. Finally, if a CAchanges its policy or practices how important to trustis it that the mechanism for the notification of thechange is spelt out to the users in the CPS. Theexperts were also asked how important it was that theuse of the certificates by the users is made clear in theCPS. One expert noted that this question is really alegal one, and not a procedural one, and so it wasmoved to the Legal section in version 3.

There was almost unanimous agreement that the pro-cedures for the protection of the CA’s private key wasthe most important issue, with 13 out of 14 expertsgiving it a high score, and 12 giving it the maximumscore of 10.The sole dissenting voice that scored thisitem low was the operator of a public certificationservice, and his reason was that this procedure is aone-off occurrence, the root private key is created andthen split into fragments, so that protection is builtinto the system. He believed that the day to day oper-ations of the CA were the most important for keep-ing a high level of trust in the system.

CA key management procedures came second, with11 high scores (7 maximum ones), 1 medium, 1 lowand 1 no answer. The reason this procedure scoredlower than the protection of the CA’s private key, isthat two experts thought the CA’s private key shouldnever be backed up and managed, and therefore thisitem should count low towards trust in the CA (scoresof 0 and 4 were awarded by the two experts). In otherwords, a CA that does back up its private keys isinherently much less trustworthy than one that doesnot back up its private keys. However, this is not thecorrect meaning given to the weighting by these twoexperts. Rather, they should have said that the CA keymanagement procedures counted highly towards cal-culating trust in the CA, and no procedures or back-up meant the highest level of trust, good procedures a

602

603

medium level of trust, and poor procedures a very lowlevel of trust.

Several factors scored medium importance towardscalculating trust in a CA. Personnel recruitment andtraining gained the top medium score, as this wasmade up of approximately a third of the experts giv-ing it each of high, medium and low scores. No rea-sons were given by any of the experts for their scores,the only comment recorded was that it dependedupon the job function as to how important this itemwas. Perhaps we have an example here of where thequestionnaire is not specific enough, and if it had per-haps differentiated between job functions, we mighthave got more agreement between the experts.

Concerning certificate revocation procedures half theexperts gave it a score of 7 or above, whilst 4 expertsgave it a low score.This caused us to review the ques-tionnaire and we realised that the question was badlyworded in the v2 questionnaire.When it was reword-ed in the v3 questionnaire, all the experts gave this ahigh score. This perhaps shows the difficulty of for-mulating clear questions without ambiguity. Had therevised question been given to all experts, we feel surethat revocation procedures would have scored morehighly than personnel recruitment and training.

Certification suspension procedures scored slightlylower than certificate revocation procedures in bothversions of the questionnaire.Whilst the vast majorityof experts gave suspension and revocation the samescores, three gave suspension a lower score. One rea-sons given for this was that suspension is less impor-tant to trust, since revocation can always be usedinstead of suspension if a subject wants to make surethat no-one can use his certificate from this time on.

Notification about changes in a CPS also scored amedium rating for trust, with approximately one thirdof the experts scoring each of high, medium and low.Unfortunately very few reasons were captured fortheir opinions, except for one expert as discussedbelow.

Half the experts gave a low rating to two questions,namely that the use of the certificate is made clear to

the users, and the requirements on repositories areclear. However, 4 experts judged both of these issuesto be of high importance, with 2 giving it full marks.So whilst the majority agree that these factors are lowor medium low importance to calculating trust in thecertificate, a significant minority think otherwise.One expert commented that all the CA is doing isauthenticating the user, and not authorising the userto do anything, therefore the CA has little controlover the use of the keys by their subjects. However,one of the experts giving this factor full marks forimportance is the operator of a commercial CA. Soclearly he thinks that the use should be restricted, per-haps for liability reasons, and that the certificateshould not be trusted if it is being used for unautho-rised purposes.Whilst we agree with the latter senti-ment, never the less we do not think that certificateuse should count highly towards the trust calculation.Our reasoning is as follows. A certificate will score acertain trust quotient dependant upon the CPS butindependent of the actual use.The relying party canthen decide if the trust quotient is high enough for itsintended use. For example, say a certificate scores 0.75from its governing CPS; the RP may decide that thisis perfectly adequate to allow a book to be purchasedover the Internet, but not to allow a car to be pur-chased. Further, the RP may decide that whilst a cer-tificate is trustworthy enough for its intended use,never the less it was not issued for this purpose, so itwill be rejected. We thus conclude that whether theuse of the certificate is made clear or not counts littletowards the trust calculation.

As a result of the interviews and the subsequent anal-ysis it was discovered that one procedural questionwas missing, namely, how important to trust is it thatthe CA has effective disaster recovery procedures. Inorder to determine the ranking of this, 5 of the orig-inal experts were asked to answer the version 3 pro-cedural questions again. In some cases this was imme-diately after answering the v2 questionnaire, and inother cases it was up to a year later.Therefore we havesome objective measure of the consistency of anexpert’s answers.

Using a scale of 0 to 10, of the 33 questions for whichwe had both sets of answers, 12 were identical for



both questionnaires, 9 were just one mark different, 5were 2 marks different, 3 were 3 marks different and4 were >3 marks different. In the latter case oneexpert was responsible for 3 of these 4 variances, andthere was a time lag of a year between him answeringthe two questionnaires. When asked to justify hiswidely differing marks, it became obvious that differ-ent contexts had been assumed in both cases. In onequestion, concerning revocation procedures, he hadanswered for qualified certificates [13] in v3 and fornormal certificates in v2. (As the signatures with qual-ified certificates are equivalent in a court of law tohand written signatures, more trustworthy revocationprocedures should apply to them.) For another ques-tion, concerning notification of change of CPS, forversion 2 he had answered that a change of CPSshould not involve notification to RPs, but rather anew policy id should be inserted in all new certifi-cates, indicating a change of policy.Therefore notifi-cation of changes counts zero to trust calculations, aschanges to existing users should never occur.However for v3 he had said that if a CA can changeits CPS and issue new certificates under the new CPSwhilst still using the old policy id without notifyingrelying parties about the change, then this would seri-ously undermine his trust in the CA and so shouldcount 9 (high importance).

So one can see that the expert had not really changedhis opinion about the importance of a change of CPStowards the calculation of trust, but rather had givenopposite scores to the question by taking first a view-point that the issue should not arise and secondly athe viewpoint that if it did arise it was very important.We have seen this phenomenon occur a number oftimes during this research, where some experts havegiven an item a high score and others have given it alow score, as they have been looking at the issue fromdifferent perspectives.

The final large variance was in the answer to the ques-tion about repositories.The question read:“The CA’srequirements on the use of repositories are clear” towhich a score of zero was given to version 2. Howeverto version 3 the expert said that the question wasambiguous, and could be the requirements on theusers, in which case it counts zero to trust, or the

requirements on the repositories themselves, in whichcase it counts 8.The latter was the intended meaning,but it could be that the expert had assumed the wrongmeaning for version 2, in which case his answers wereidentical in both cases.This again shows the difficultyof correctly formulating questions to elicit knowl-edge, and without an expert interviewer being presentto give the intended meanings of questions and toclarify ambiguities, different experts with the sameviews could give apparently conflicting answers to thesame questions.

Disaster recovery scored 2 high and 3 medium scores,and whilst the number of experts is small compared tothose answering the v2 questionnaire, never the lesswe can infer that this factor is judged of mediumimportance to calculating trust in a CA, and is similarto the personnel recruitment procedures.

Compliance AuditThere were six questions in the section concerningthe compliance audit.The experts were asked to com-pare the importance of various audit items. Ideally, formaximum trust, a compliance audit should (i) be car-ried out by a suitably qualified auditor, where (ii)there is no a-priori relationship between the auditorand the CA organisation (e.g. it might diminish trustif an accountancy firm sets up a Trusted Third Partyservice and then used its own auditors for the com-pliance audit).The auditor should (iii) be able to callunannounced at any time to carry out an audit, (iv) beable to audit a complete range of topics e.g. docu-mentation, procedures, archives etc. held by the CA,(v) be able to provide the audit results to all relyingparties, and (vi) have extensive powers of sanctionshould the CA be found to be failing in its duty ofcare.

But what is the relative importance of these items? Amajority of the experts (65%) agreed that two itemswere significantly more important than the rest. Thetop two items were: the list of topics that are auditedand the qualifications of the auditor. Both of thesewere rated low (3/10) by only one expert, but it couldbe for related reasons. One expert stated that if theaudit procedures were clear then the qualifications of

604

605

the auditor were not that important. Another wentfurther than this, and stated that the audit resultshould be independent of the auditor, and thereforethe list was the important factor, not the auditor’squalifications. To counter this position, one expertadded that a well-qualified auditor would know tolook beyond any list of audit topics when appropriate,implying that the list itself is not that important to awell qualified auditor.Another expert only ranked thelist of audit topics as medium importance because hethought it would be obvious to an auditor what thelist should be. We might therefore conclude that thelist of audit topics should essentially be unbounded inorder to cover every issue. In support of this, oneexpert did state that some compliance lists he hadseen are deficient.Thus we can conclude that the top-ics that are audited should be all inclusive even if acompliance list itself is not. From a trust perspective,we can conclude that the list of audit topics, if small,might be a reason to place less trust in a CA, but witha well qualified auditor any written list of audit top-ics is not that important providing the auditor is will-ing to go beyond the list and inspect whatever isdeemed to be necessary. If the auditor is not that wellqualified or experienced, then the compliance listbecomes extremely important, and needs to be com-prehensive in scope.

The sanctions that can be applied against a CA failingan audit interestingly obtained the next largest num-ber of high scores (8), but also obtained a relativelyhigh number of low scores (4). Unfortunately no rea-sons were recorded as to why four experts thoughtthat sanctions were not that important to trust. Thissplit does however highlight the fact that experts havedivergent views about sanctions, and unanimousagreement about the importance of sanctions is a longway from being reached.

The relationship between the CA organisation andthe auditor was judged high importance by half theexperts, the rest were split almost equally between lowand medium. One expert only ranked it as mediumimportance providing that no relationship existedbetween the auditor and the CA, adding that an auditwill not work if a relationship does exist.This impliestherefore that the expert does believe that a proper

relationship is highly important to trust, even thoughhe only ranked it medium importance. At the otherextreme one expert said that the relationship was ofno consequence at all, and scored it zero. However,the majority of experts believe that the relationshipbetween the auditor and the CA should purely bebased on carrying out the audit and nothing else.Anyother relationship will decrease the trust that we canhave in a CA. One expert said that the type of organ-isation carrying out the audit was the most importantfactor, more important than any of the other ones inthe questionnaire.

The frequency of the audit was judged by the major-ity of experts not to very important, only two expertsgave it a high rating, the majority (9) gave it a medi-um rating. One expert said the frequency was not thatimportant providing it was reasonably frequent, whilstanother expert added that it was far more importantto have the ability for unsolicited audits.This is pre-sumably on the premise that humans naturally tend toset their houses in order immediately prior to animportant guest visiting them, so that calling unan-nounced is far more likely to result in seeing the CAin its natural operational state, rather than booking avisit in advance.This was confirmed by the CA oper-ator who was interviewed, who said that audits shouldbe at a high frequency, but initiated by the CA. If theauditor called unannounced it might be during a hec-tic period and he might spot a temporary operationalweakness that if publicised could lower the trust inthe CA, which is precisely the opposite effect that anaudit is supposed to have.We must conclude thereforethat a CA that allows unsolicited audits is more con-fident of its operations, and therefore more trustwor-thy, than one that only allows CA scheduled audits.Asa result of these interviews, we updated our expertsystem to ask if unsolicited audits were allowed by theCA.

The wide sharing of the audit results was judged to bethe least important factor in calculating trust.The vastmajority judged it medium or low importance.Indeed, one expert added that you might want to goas far as preventing the sharing of the audit results ifthey detailed a weakness in the operation of the CA.Even though six experts rated this trust factor as low,



never-the-less three experts still gave it a high rating,indicating that no well established consensus has yetemerged about the relative importance of this factorin calculating trust.

Malpractice and ControlsThis section addresses the trust issues surrounding theuse of appropriate controls to prevent malpracticefrom occurring.The interviews and subsequent anal-ysis of the version 2 results revealed that this sectionhad omissions in it, and so a more comprehensive ver-sion 3 was constructed. The v2 questionnaire askedabout the relative importance to trust of: having anappropriate audit trial, having appropriate archiveprocedures, having physical security controls to itsbuildings, and having appropriate procedural controls,in particular procedures for key recovery and revoca-tion.Version 3 in addition added: having appropriatenetwork security controls, computer security controls,and personnel security controls.

An unexpected advantage of the v2 omissions wasthat it gave us a consistency check.We now had theability to compare the experts’ answers to two sets ofsimilar questions, when asked between one hour andone year apart.The correlation between the two setsof results was as follows. Out of the 16 questionsanswered twice by the same experts, 7 had the samescores, 3 were within 1 mark, 2 were within 2 marksand an anomalous 4 had large differences.The incon-sistent experts were invited to explain the apparentanomalies in their replies, and two out of the threedid so (one was responsible for 2 of the inconsisten-cies). One of them had rated physical security con-trols of high importance in v3, and low importancein v2.This was explained due to the different inter-pretations he had applied to the two differentlyworded questions. V2 had asked how important totrust is it that professional standards are used for phys-ical security of a CA’s building, whilst v3 simplyasked how important it was that appropriate physicalsecurity controls are used.The expert said that he hadinterpreted professional standards as formal standards,rather than appropriate mechanisms, and that the useof formal standards in physical security is not thatimportant, but that the use of appropriate physical

security is important. Another had rated both audittrials and archives as high importance in version 2and a year later as low importance in version 3.Therationale for this was as follows: a competent CAneeds to keep proper audit trails and archives. If it isnot capable of doing this simple task properly, thenhow can it be trusted to do the more difficult tasksproperly, hence audit gets a high mark for impor-tance. However, on reflection a year later, the sameexpert thought that if a CA does everything else to ahigh standard except keeping proper audit trials andarchives, then these do not contribute so much to thetrust he would place in it, hence it gets a low scorefor importance.

Overall, network security controls were judged to bethe most important controls to be applied. Physicalsecurity controls came second.Whilst we only had alow sample of 4 experts who answered the v3 ques-tionnaire, never the less they were in unanimousagreement about the importance of network securitycontrols, with each expert giving it a score of 10.Thismakes sense, since physical security controls only pro-tect a CA from attack by people in the same geo-graphic locality as it, whilst network controls protecta CA from attacks by people in any geographic local-ity who are connected to the same network as theCA. The importance of this control is further sup-ported in the Overall section below, where networkand computer controls scored highly.

In the version 2 questionnaire, procedural controlswere seen to be the most important factor in ensur-ing trust in a CA.Whilst there are many different pro-cedures in use by a CA, the experts were asked specif-ically about the controls when subject key corruptionor compromise is reported, and the controls when keyrecovery is performed. A clear majority thought thatcontrols for key recovery were the most importantwith 10/13 experts giving it a high mark. Key recov-ery scored less, primarily because some experts dis-agree with the whole principle of key backup as men-tioned earlier. In the version 3 questionnaire, proce-dural controls were collapsed into one question,for ease of comparison with the enlarged set of otherfactors, and it is interesting to note that procedures (in general) now lost a little of their importance

606

607

when compared to a greater range of other controls.Physical and network security controls were seen tobe of higher importance than procedural controls.This result is not entirely unexpected, as it is a com-mon phenomenon that when you draw people’sattention to something specifically, it will oftenassume a higher importance than when you do not.Never the less, some caution is advised when inter-preting these results, since the number of expertsanswering the v3 questionnaire was only a third ofthose answering the v2 questionnaire.

Personnel controls and computer controls were alsojudged to be of high importance, though not quite ashigh as network and physical security controls.Theywere given the same importance to trust as procedu-ral controls.

Proper audit trials and archives were judged to be ofmedium importance overall. Approximately half theexperts gave it a high mark, and half a medium mark,with audit being judged on average to be a bit moreimportant than archives.The importance of these twotopics dropped in importance between versions 2 and3 due to the introduction of the more important trustrelated controls in the v3 questionnaire.

Legal RedressThe experts were asked to rank five legal aspectsaffecting trust. In order to maximise an RP’s trust, aCA must have a comprehensive set of liabilities, war-ranties and disclaimers so that the RP knows exactlywhat legal liability the CA is claiming to provide. Ifthe CA has indemnification clauses, then a RP mustknow what indemnification the CA is expectingfrom its users (RPs and subjects), and from its repos-itories. If the CA is placing obligations on its reposi-tories, then these should be published. Finally, thegoverning law has to be one that is trusted, respectedand accessible to the RP, otherwise legal redress willnot be feasible.

The experts clearly identified one issue as being farmore important than the others, and this was the setof liabilities, warranties and disclaimers being provid-ed by the CA. All the experts gave this a high score,

apart from one, with 12 giving it a maximum score of10. The lone dissenting voice who gave this a lowscore, qualified his answer by saying that in aEuropean context, the liabilities etc. of a public CAthat issues qualified certificates will be decided by EClegislation and therefore one does not need to placeany reliance in what the CA may say, as these will beoverridden by the EC legislation. The lawyer com-mented that as a general rule, the shorter this list, themore trustworthy a CA will be, as a long list will usu-ally be there to limit the liabilities of the CA.

Most experts gave the governing law a high score,apart from three who gave it a medium score, andone who gave it a score of zero. Most surprisingly,the latter was the PKI lawyer. His view was that thetrustworthiness of CA depends entirely upon itsprocedures, personal, controls etc. as described earli-er, and not upon the country in which it is estab-lished. Presumably, once a RP takes a CA to court,its trust in the CA will be approaching zero, and thus the governing law provides the RP not withany trust in the CA, but rather with trust in the legalsystem.

The remaining three issues scored on average a lowmedium score, of approximately 4.6 each. However,there were some interesting differences of opinion.The indemnification of the CA by its users (relyingparties and subjects) gave a split result. Over a third ofthe experts gave this a high score, meaning that itcontributes substantially to calculation of trust in theCA, whilst half gave it a low score, meaning that it isinsignificant in the calculation of trust. Only 2 gave ita medium score. The authors are unable to explainthis dichotomy, but suspect that some of the latterexperts may have given this a low score, thinking(wrongly) it meant that a CA with a user indemnifi-cation clause is less trustworthy, and should score alow trust quotient. Of course, in order to score a lowtrust quotient, the factor has to have a high weightingin the knowledge base.

Indemnification of the CA by its repositories had adifferent score profile. Nearly half the experts gavethis a medium score, with slightly less a low score.Only two experts gave it a high score. The authors



think that this issue will only have a small contribu-tion to the trust that an RP can have in a CA, becausethe RP can gain redress from either the CA or therepository if it had to. Trust is therefore not signifi-cantly affected by the indemnification of the CA bythe repository.

Finally we have the publication of the repository’sobligations. The experts were almost equally splitbetween high, medium and low importance to trust.There was close correlation between the scores givento this topic and the scores given to the obligations ofthe repository discussed above under obligations. Inversion 3 of the questionnaire this topic was removedfrom the Legal section, as it is seen to be covered bythe earlier section.

OverallFinally the experts were asked to determine the rele-vant weightings of the six topic headings describedabove. In addition, due to the importance alreadygiven to the CA private key protection and manage-ment procedures, these two items were added to thelist for comparative purposes. A topic that could beapplicable to trust, but that could not easily be classi-fied under any of the previous headings, is the cryp-tographic standing of the algorithms used by the CA.As a PKI depends upon the strength of its underlyingcryptographic algorithms, any weaknesses here wouldmean that an attacker who could break the algo-rithms could easily masquerade as any subject of theCA. Thus the experts were asked how important totrust is the cryptographic standing of the algorithmsbeing used. Finally, in the v2 questionnaire, theexperts were asked to rank the computer and net-work security controls (this was added to theControls section in v3).

The clear winner in all of the above was protection ofthe CA’s private key. This is seen to be the mostimportant aspect in allocating trust to a CA. 13 of theexperts gave this a high score, with 12 giving it themaximum score of 10. Only one gave this a mediumscore, for reasons already discussed above under pro-cedures. Therefore it is recognised that if the privatekey of the CA is compromised, then the entire PKI

becomes untrustworthy. It will have to be dismantledand an entirely new infrastructure built again fromscratch. For a commercial CA, this would probablymean that it would go out of business.Therefore it ishighly unlikely that a commercial CA will ever pub-licise the fact that its private key has been compro-mised, if such a disaster were to occur.

The next two most important items were judged tobe the cryptographic standing of the algorithms usedby the CA, and the network and computer securitycontrols. 11 experts judged the cryptographic stand-ing of the algorithms to be of high importance, withjust 2 giving it medium importance, and 1 low impor-tance. Network and computer security controls camevery close behind, with 9 experts giving it a highscore.This further supports the high ranking of thesefeatures in the Controls section.

Identification and authentication came close behindthe above, with 9 experts giving it a high ranking (ofwhich 8 were full marks), 3 a medium ranking and 2a low ranking.Audit scored the same mean ranking asauthentication, with 10 experts giving it a high rank-ing and 3 a medium ranking (although only 2 expertsgave it full marks).

The remaining items all scored a high medium aver-age score, so no items were judged to be unimportantin calculating trust in a CA.The rankings were:

• Malpractice (9 high scores, mean of 7.65);• Obligations (6 high scores, mean of 7.14);• CA private key management (9 high scores, mean

of 7.08) – this figure is pulled down because someexperts disagree with having private key backupsas described earlier;

• CA Procedures (8 high scores, mean of 6.96); andfinally

• Legal Redress (5 high scores, mean of 6.56).

As can be seen, legal redress had the lowest number ofhigh scores, the highest number of medium scores (9),and no low scores. So whilst we have broad agreementthat legal redress is the least important factor in deter-mining trust in a CA, it is still seen as being of medi-um-high importance.

608

609

Conclusions

There are some factors used in determining the trustin a PKI, that nearly all the experts who were inter-viewed can agree are extremely important, and otherfactors that they agree are of little or medium impor-tance.Table 1 gives some examples of what factors areimportant, unimportant, and disagreed about.Bimodal split indicates that most experts thought itvery important or low importance, but not of medi-um importance. No agreement indicates that all lev-els of importance were scored by some experts.

As can be seen, there were a large number of factorsupon which the experts did not agree.The questionis: what should we do about such disagreements whendeveloping a knowledge base or otherwise compilingknowledge to be used across diverse situations. Wehave presented three approaches to knowledge analy-sis and have demonstrated how they can work in

harmony in developing a knowledge base for calcu-lating a trust quotient. Because trust is a highly ill-structured issue, the third approach, of seeking princi-ples, proved vital to the whole analysis, enabling us tocapitalise on expert disagreement.

In many cases a disagreement between experts, asshown by a bimodal split in importance, was traced todifferences in context. Contextual factors can cover atleast three things: knowledge of the working context,knowledge of the characteristics of the person work-ing in that context, and knowledge of their preferredproblem solving methods (Attarwala and Basden [3]).The problem the knowledge analyst has is that manycontextual factors are implicitly assumed rather thanexplicitly stated; this is the problem of tacit knowl-edge (Collins [14]). But the four questions mentionedearlier (“Why?”,“What is?”,“When not?” and “Whatelse?”) are an aid to making contextual assumptionsexplicit.


Trust Factor Relative Importance

Authentication – use of locally unique names for each subject HighAuthentication of the subject HighAuthentication of the subject’s organisation (if in the subjects DN) HighAuthentication – use of trade marks in names LowAuthentication – rival name claim resolution procedure No agreementAuthentication – use of meaningful names Bimodal splitCA to RP obligations HighCA to subject obligations Bimodal splitObligations of repository No agreementCA private key protection procedures HighNotification about changes in CP No agreementUse of certificate is made clear to subjects and RPs LowCA personnel recruitment and training procedures No agreementList of compliance topics in audit HighQualifications of auditor HighSharing of the audit results No agreementFrequency of the audit MediumApplying sanctions for failed audit Bimodal splitControls over private key recovery HighCA having an archive and audit trail MediumIndemnification of CA by its users Bimodal split

Table 1.The relative importance of some factors when computing trust in a CA


Once they have been made explicit each contextualfactor implies some switch in the knowledge base,perhaps changing the weighting of some other factoror even switching whole structures of knowledge onor off. For each contextual factor there are two possi-ble courses of action. One is to explicitly include thatknowledge in the base.The other is to decide that aparticular factor will be always either on or off and setthe knowledge base accordingly. Which course ofaction is appropriate must be decided for each indi-vidual factor. For example, it was decided that authen-ticating the identity of the subject would always beconsidered important in determining trust. But thepossibility of there being a different context in whichit is less important might cause us to reconsider thedesign of future versions of the knowledge base.

This incorporation of explicated contextual assump-tions into the knowledge base is frequent practice inknowledge analysis and engineering, and is one reasonfor the need for knowledge refinement discussed ear-lier. Indeed, it was stated in the introduction thatprobabilities can never reach unity because of theacknowledgement of the possibility of missing knowl-edge.We can now see the reason for this: most of themissing factors are not those that are visible to theordinary analysis process but are the contextualassumptions that are tacit and deeply hidden, andthere can never be any guarantee that all have beendiscovered.

This has implications for both the knowledge analystand the knowledge base. The implications for theknowledge analyst can be summed up in two words:bold humility. The knowledge engineer must alwaysrecognise the limits of his/her own ability to find andencapsulate all relevant knowledge, and must alwayspay great respect to expert knowledge - and yet theknowledge engineer should always be bold enough toreserve judgement about each of the expert’s facts andopinions, and to encapsulate that judgement in theknowledge base.The implications for the knowledgebase are that it can never be considered fully completenor guaranteed to be fully accurate in its results -though we would of course expect a certain statisticallevel of accuracy. Therefore such a knowledge basemust always be used with wisdom, and it is the

responsibility of the user to decide whether the trustquotient given is applicable in their case. But, at leasta knowledge base built with approach 3 (principles)should be able to provide the explanations the userneeds in order to make such a decision.

A working prototype version of the knowledge baseis available on the Internet (http://huan.isi.salford.ac.uk:7007) in a form that provides not only a trustquotient but also the ability for the user to explorewhy that quotient has been arrived at.

AcknowledgementsThe authors would like to thank the EPSRC andDERA for funding this research under grant numberGR/L 54295, and for the 14 PKI experts who tooktime to participate in the survey:You know who youare, and we thank you very much.

References[1] ISO/ITU-T Rec. X.509 (1997) The Directory: Authentica-tion Framework

[2] Chokhani, S., Ford, W. “Internet X.509 Public KeyInfrastructure Certificate Policy and Certification PracticesFramework”. RFC 2527. March 1999.

[3] Attarwala, F.T., Basden, A., “A methodology for constructingExpert Systems”, R&D Management, v.15, n.2, pp.141-149. 1985

[4] Basden, A., Watson, I. D., Brandon, P. S. “Client Centred: anapproach to developing knowledge based systems”, Council forthe Central Laboratory of the Research Councils, U.K. 1995.

[5] Winograd, T. “From programming environments to environ-ments for designing”, Comm.ACM., v.38, n.6, pp.65-74. 1995

[6] Basden, A., Hibberd, P., R. “User interface issues raised byknowledge refinement”, Int. J. Human Computer Studies, v.45,pp.135-155. 1996

[7] Basden, A.; Evans, J. B.; Chadwick, D. W.;Young, A. “Copingwith Poorly Understood Domains: The Example of InternetTrust”, Research And Development In Expert Systems - 1998 ;Issue 15 ; Pages: 114-132, presented at Expert Systems 98 confer-ence, December 1998.

[8] Hines, J.G., Basden,A.“Experience with the use of computersto handle corrosion knowledge”, Br. Corros. J., v.21, n.3, pp.151-156. (1986)

610

611

[9] Hickman, F., Killin, J., Land, L., Mulhall,T., Porter, D.,Taylor,R. “Analysis for Knowledge-Based Systems, a practical guide tothe KADS methodology”, Ellis Horwood. (1989)

[10] The Post Office (a British Statutory Corporation). “ViacodeCertificate Policy for Medium Assurance Certificates”,Version 2,11th August 1999. Available from http://www.royalmail.com/atwork/viacode/serviceinfo/policy.pdf

[11] Jarvenpaa, S.,L., Leidner, D. “Do You Read Me? TheDevelopment and Maintenance of Trust in Global Virtual Teams”.Journal of Computer-Mediated Communication, 1998.

[12] Kuosa,T., Basden, A. “Predispositions as determinants of thefuture”. Futures, 32:833-852, 2000.

[13] Santesson,S., Polk, W., Barzin, P., Nystrom, M. “InternetX.509 Public Key Infrastructure Qualified Certificates Profile”,<draft-ietf-pkix-qc-06.txt>,August 2000

[14] Collins, H., M.“The TEA-Set:Tacit knowledge and scientif-ic networks”, Science Studies, 4, pp. 165-186. 1974


evaluating trust in a public key certification authority

Documents