HPP REVIEWS

Stanford University LibrariesDept. of Special CoHectioos

*w tj Series _^±gVgS^-Foi. Title

eNATIONAL SCIENCE FOUNDATION

WASHINGTON,

D.C. 20550

Mathematical and Computer Sciences

May 8, 1979

Professor Edward A. FeigenbaumComputer Science DepartmentStanford University320 Galvez StreetStanford, California 94305

Dear Professor Feigenbaum:

Enclosed are verbatim comments on yourentitled,"The Automation of Scientific

proposal no. MCS-7906669Inference: Heuristic

Computing Applied to Protein Crystallography".

Sincerely yours,

Eamon BarrettProgram DirectorIntelligent Systems Program

Enclosures

MICiiSWEE

JUN 1 1 1979E.A. Ftiuci.LJr.U

E.A. FEIGENBAUM

BE€EK¥EPMAY 14 1979

TITLETHE AUTOMATION OF SCIENTIFIC INFERENCE: HEURISTIC COMPUTINGAPPLIED TO PROTEIN CRYSTALLOGRAPHY .

COMMENTS (CONTINUE

ON

ADDITIONAL SHEET(S)

AS

NECESSARY)

This proposal is highly recommended for renewal based upon clear andeiJnifiSE? progress towards the goals of the original grant appncatxon.The proposed research addresses important problems in the conceptualdesign and realization of knowledge-based heuristic Problem-solvingsystems. From this work and related research (much of it *"»;«*lIS?*inthe Stanford Heuristic Programming Project), machine models of scientific

inference are emerging that provide guidelines for the design of heuristicprograms that are able to infer formal structure from experimental data,

lnS9by extension, from direct observation of the problem space modeled

by the system. The "blackboard" concept, developed here as a suitableorganization for inferential knowledge-based heuristic problem-solvingprograms will probably work for ion-inferential (analytic search;

knowledge-based problem-solving systems as well.

The proposed plan for continuation of the project is reasonable and

litely to be Successful, given the favorable research environment in

which it will be pursued. E.A. Feigenbaum, the principal investigator

listed on the cover sheet and in the budget summary, « ?£ 611-

-known for the excellence of his work in inferential problem-solvingsys^ms? and is exceptionally qualified to direct the proposed research.Th peroLl list, however, indicates that Feigenbaum will serve only

in an advisory role, with R. Engelmore assuming responsibility as project

d?recto?T AsLning'that Engelmore has played the major role in bringing

the proiect to its current state, there seems to be little risk in his

becoming principal investigator, especially with the support availableto him within the Heuristic Programming Project taken into consideration.Arrangements for cooperation with protein crystallographers seem quite

adequate — the Stanford group has a record of outstanding success in

this sort of collaboration between computer scientists and experts in

the physical andletiifrge sciences.

In summary, this research is good computer science with the added feature

that it also advances the useful art of protein crystallography. Presum-

ably, this work may t?e generalized to encompass the interpretation of

less constrained classes of x-ray diffraction electron density maps.

□ good □ fair Dpoor□ VERY GOODRATING: EXCELLENT

Verbatim but anonymous copies of reviews, ratings and associated correspondencewill be sent only to theprincipal investigator/project *«»'«>" "ctionSubject to this NSF policy and applicable laws, including the Freedom of Information Act, 5 USC 552, reviewers comments will be given maximum protection

from disclosure. ■ ■ 1 -w [̂w-^^^g^^^^^^^^^Bj^g^^^^^^^^ma^MtaM^ll^^jmi^^^^^Bt^lgMtg^M

NATriONAL SCIENCE

" PROPOSAL EVALUATION FORM Jh NSF Form 1B (10-78)Supersedes Forms 173 and 990FOUNDATION

PLEASE RETURN BYPROPOSAL

NO.

MCS-7906669

INSTITUTION

STANFORD UNIVPRINCIPAL

INVESTIGATOR

NSF PROGRAM

FEIGENBAUH EDWARD INTELLIGENT SYSTEMS PROG

SCIENCE

* ' " PROPOSAL EVALUATION FORM A NSF Form 1B (10-78)FOUNDATIONI*— Supersedes

Forms,

173 and 990I PLEASE RETURN BYpR«SRS^7JO6669

INSTITU

"OTANFORD UNIV

TITLI

APPLIED TO PROTEIN CRYSTALLOGRAPHY

COMMENTS

(CONTINUE

ON ADDITIONAL

SHEET(S)

AS

NECESSARY)

This proposal is for continued support of a project designed to result in a computationalmeans of handling the electron density maps derived from the X-ray diffraction patternsaffo&iad '*y proteins. The problems of dealing with such data are acute and the authorsare correct in their assertion that much of this work, which is presently done by skilledinvestigators can, and should be done by computer.

Technically, the proposal draws from work by, for example, Greer, and as a result, itprobably is close to the frontier of current thinking as far as this problem is concerned.That the Stanford group should be involved in this work is consistent with the generalaims of that group, which has established for itself a record of attacking relativelyintractable problems in artifical intelligence, with some success.

Professor Feigenbaum* s background is such that he is well qualified to be the P.I. inthis project. He has tackled many similar problems in the past and is well able tohandle the computational aspects of the problem. The proposal, as well as the collabora-tion with Johnson of ornl and Freer of UCSD, suggest that adequate awareness of otherwork in this area is at hand. The computational facilities available to the project willbe more than adequate.

Both Category B criteria are met by this proposal. There seams no doubt that the scienceof this area is being advanced by work such as that proposed here. Conceptual generaliza-tion are flowing from the artificial intelligence work of Feigenbaum 1 s group and thisproject is no exception.

Two troublesome points concern the details of the programming and the availability oftimely and experienced assistance with the x-ray diffraction aspects of the problem.

Collaborative arrangements at San Diego and Oak Ridge are fine, but it would seem muchmore valuable to have a protein crystallographer on hand and involved upon a daily basiswith this project. There seems to be no such person identified in the proposal. On theother point, the absence in the proposal of any discussion of language choice seemstelling in that little thought (as usual) is being given to ultimate transportability ofthe CHRYSALIS system. There is continuing criticism of the SUMEX project on these grounds,

RATING: □ EXCELLENT □good □fair □ poor□ VERY GOOD

Verbatim but anonymous copies of reviews, ratings and associated correspondence will be sent only to the principal investigator/project director on request.Subject to this NSF policy and applicable laws, including the Freedom of Information Act, 5 USC 552, reviewers' comments will be given maximum protectionfrom disclosure. „

"

PR,^K&HflfigH

GATOR

EMIARD A

becausesystems.problembecause—

it tends to lead to immovable, and hence less useful, finishedIt would be worthwhile for the P.I. to give some thought to this

now, rather than arrive at 1981 with a system that is unexportableit relies upon a relatively unknown and unsupported compiler.

NATIOafAL SCIENCE

COMMENTS

(CONTINUE

ON

ADDITIONAL SHEET(S)

AS

NECESSARY)

This project deals with applying artificial intelligence methods tothe interpretation of protein electron density maps. This effort isclearly worthwhile although it is important to consider how the existenceof several interactive graphics interpretative systems such as that atChapel Hill and the Diamond system at MRC Cambridge England and at the NIHmake this project less necessary, certainly in its purist forms.

The investigators clearly have considerable expertise in artificialintelligence methods. My greatest reservation is their expertise inprotein crystallography. Dr. Freer is a fine crystallographer but he issome 400 miles away from where the research is going on. A project ofthis type, with its goals, should have top-notch crystallographersintimately involved in the development at each step, rather tfahn bytelephone or computer link or 4 to 5 visits per year. Their insight andexperience are essential; as is their constant drive to use the program.For example, as the language of the proposal is so heavily couched inartificial intelligence dialect, it is hard to discern what newinsights and approaches beyond Greer's and Johnson's work these investi-gators have innovated.

I feel the project also suffers from the lack of interactive graphicson site that can be used to constantly monitor the results and suggest new"KS's". The investigators are aware of this problem.

The work in this proposal should be performed. In addition, theapplication of artificial intelligence methods to the problem is signifi-cantly useful, maybe even necessary. With proper on site, fulltimeinvolvement of a very good protein crystallographer, this project could becarried out more effectively and with greater likelyhood of the systembeing a useful scientific tool in other crystallographic laboratories.

RATING: □ EXCELLENT □ good □ FAIR □POOR[)2 VERY GOOD

Verbatim but anonymous copies of reviews, ratings and associated correspondence will be sent only to the principal investigator/project director on request.Subject to this NSF policy andapplicable laws, including the Freedom of Information Act, 5 USC 552, reviewers' comments will be given maximum protectionfrom disclosure. ________

===

injatjon PROPOSAL EVALUATION FORM NSF Form 1B (10-78)Supersedes Forms 1 73 and 99!

'ROPOSAL NO.

INSTITUTION PLEASE RETURN BY

MCS-7906669 STANFORD UNIV

'RINCIPAL INVESTIGATOR NSF

PROGRAMFEIGENBAUM EDWARD INTELLIGENT SYSTEMS PROGfITLE

THE AUTOMATION OF SCIENTIFIC INFERENCE: HEURISTIC COMPUTINGAPPLIED TO PROTEIN CRYSTALLOGRAPHY

\W tit/&/fa& I¥U: VSA-

>~af

NATIONAL SCIENCE FOUNDATIONWASHINGTON. DC. 20550

«Mathematical and Computer Sciences

May 18, 1979

Professor Bruce G. BuchananComputer Science DepartmentStanford UniversityStanford, California 94305

Dear Professor Buchanan:

Enclosed are verbatim comments on your proposal no. MCS-7903753entitled, "Knowledge-Based Consultation Systems".

Sincerely yours,

Intelligent Systems Program

Enclosures

JUN 5 1979E.A. FEIGENBaUivi

*** * 5 *&

Eamon BarrettProgram Director

KNOWLEDGE-BASED CONSULTATION SYSTEMS

[COMMENTS (CONTINUE ON ADDITIONAL SHEET(S)

AS

NECESSARY)

This is a proposal for continuation of research in the area of knowledge-based consultationsystems. The main focus is on the development of a framework which will help designers ofconsultation systems to acquire knowledge from experts and to incorporate it in a system

! which can use this knowledge effectively. This is an extremely significant area of Alj research at present.

The proposed research builds on previous work with MYCIN and with a generalized frameworkabstracted from MYCIN, called EMYCIN. It will be carried out in the environment of theHPP unit at Stanford, where there is substantial experience with the design of consultationsystems in science and medicine. In presenting the background for this research, many ofthe existing relevant systems are mentioned; however, the CASNET system at Rutgers is notdiscussed, even though 1t has several similarities with aspects of the proposed research,

j e.g. using a roje compiler to increase system effectiveness.The specific ideas about the next stage of research are extremely promising. The PI is afirst-rate researcher with outstanding qualifications for work in this area. He is workingin an excellent environment which includes an outstanding group of senior computerscientists and graduate students, and superb computing facilities. Note that, even thoughsuch a project requires considerable computing resources, no funds for computing arerequired from this grant; computing will be available from the SUMEX-AIM national resource

\ which is supported by NIH.

In summary, I rate this proposal excellent, and I believe that it should receive NSFsupport.

RATING: [El EXCELLENT D VERY GOOD CHgood CDfair LJ POOR

Verbatim but anonymous copies of reviews, ratings and associated correspondence will be sent only to the principal investigator/project director on request.Subject to this NSF policy and applicable laws, including the Freedom of Information Act, 5 USC 552, reviewers' comments will be given maximum protectionfrom disclosure. .

r

FOUNDATION TION FORM NSF Form IB (10-78)Supersedes Forms 173 and 990

PROPOSAL

NO.

INSTITUTION PLEASE RETURN BYHCS-7903753 STANFORD UN IV

PRINCIPAL INVESTIGATOR NSF

PROGRAMB U C H A N AN BRUCE INTELLIGENT SYSTEMS PROG

i

j

NATIONAL SCIENCEFOUNDATION

NSF Form 1B (10-78)( PROPOSAL EVALUATION FORM ( Supersedes Forms 173 and 990I

PLEASE

RETURN BY

PROPOSAL NO. INSTITUTION

STANFORD UN IV

PRINCIPAL INVESTIGATOR NSF PROGRAM

INTELLIGENT SYSTEMS PROGBRUCE GTITLE

KNOWLEDGE-BASED CONSULTATION SYSTEMS

COMMENTS

(CONTINUE

ON ADDITIONAL

SHEET(S)

AS

NECESSARY)

Judging from the past work of the Knowledge-Based Systems Group at Stanford,this proposal has excellent prospects and ought to be given a high priorityfor funding. Professor Buchanan has been a pioneer in the development ofexpert knowledge-bases in real-world problem domains, and in this currentproject he applies a considerable amount of his accumulated practical ex-perience in knowledge acquisition to the task of automating the interactionbetween the computer system and the domain expert. The strength of theproposal, it seems to me, lies in the fact that all of the proposed system-building tools are derived from concrete experience in a variety of problemdomains: they are therefore quite well-conceived and seem quite feasible.Nevertheless, there are some worries here, mostly concerning the underlyingknowledge representation. Buchanan acknowledges some of the limitationsof a production system based on associative triples with attached certaintyfactors (see p. 9): the uncertainty mechanism here seems inappropriate forthe more highly structured domains (PUFF, SACON and the horn repair system) ,but these more structured domains might then require a full-fledged frame-like representation (as investigated by Aikins [77]). The problem here isthat the tools for knowledge acquisition seem to be highly dependent on thecurrent production-rule formalism, and might not generalize easily. Forexample, consider the hypothetical dialogue on pp. 12-16 (curiously, thisconcerns the Structural Analysis domain) , and imagine the problems thatwould arise if this domain turned out to require a much more complex under-lying representation. On balance, however, Buchanan's group can be reliedupon to approach these problems pragmatically, as they have approachedsimilar problems in the past, and their experience with the proposedsystem-building tools for the current production-rule formalism shouldprovide invaluable information for the construction of the knowledge-basedconsultation systems of the future.

FAIR POOR

Verbatim but anonymous copies of reviews, ratings and associated correspondence will be sent only to the principal investigator/project director on request.

Subject to this NSF policy and applicable laws, including the Freedom of Information Act, 5 USC 552, reviewers' comments will be given maximum protectionfrnm disclosure.

,J

s CS-7 9 037 53

BUCHANAN

"

NATIONAL SCIENCEf 'PROPO^'.. 'EVALUATION FORM f~' NSF Form 1B (10-78)FOUNDATION

Supersedes Forms 173 and 990I PLEASE RETURN BY

INSTITUTION

STANFORD UNIV

RINCIPAL INVESTIGATOR

8 UCHANAN

NSF PROGRAM

BRUCE G INTELLIGENT SYSTEMS PROG

KNOWLEDGE-BASED CONSULTATION SYSTEMS

OMMENTS

{CONTINUE

ON

ADDITIONAL SHEET(S)

AS

NECESSARY)

I find a self contradiction as follows. On one hand the proposal stateson p. 2 that "High performance (of intelligent systems) has been achieved at thecost of generality and man years of work in knowledge base construction". Onthe other hand the tasks described in the proposed work (section 2) all seem tobe aimed at needed improvements in performance of systems. Truly, the proposalstates that these tasks will be performed in conjunction with the domainindependent EMYCIN system. Still the proposed tasks will have more impact onperformance than on the two problems of limited generality and man years of work.Recognizing these broader goals of the proposal, still this is an excellent planfor research, with highly qualified personnel.

<

ATING: MJ EXCELLENT VERY GOOD GOOD FAIR POOR

erbatim but anonymous copies of

reviews,

ratings and associated correspondence will be sent only to the principal investigator/project director on request.object to this NSF policy and applicable laws, including the Freedom of Information Act, 5 USC 552, reviewers' comments will be given maximum protectioncm disclosure.

_^__

ROPOSAL NO.

MCS-7903753

nTLE

3MMENTS

(CONTINUE

ON ADDITIONAL

SHEET(S)

AS

NECESSARY)

The stated objectives of the proposed research are "to understand and con-struct a system that scientists can use to build knowledge-based reasoningorograms in many scientific domains". My guess is that this research willfail to achieve the second goal, "to construct" , for reasons mentioned below.But the first objective, "to understand", is important enough in itself tojustify continued support for the project, for while I would have to characacterize a working EMYCIN system as specified in this proposal as a solutionin search of a problem, the insights that will be gained by attempting to isolatetthe domain-independent core of a fairly large class of knowledge-basedreasoning programs will certainly provide guidance for the design of effic-ient and useful heuristic problem-solving systems of this type.^ The part ofthis proposal in particular that addresses the problem of building, main-taining, and evaluating large judgemental knowledge bases is very well con-ceived, and is likely to produce results of wide-ranging usefulness.

My doubts about the system-construction phase of this research are engen-dered largely by the limited variety in the models upon which the generaliza-tion is based. Buchanan recognizes this in pointing out the difficulty inremoving the medical biases from the underlying formalism, and the problemis implicitly limned by the absence of proposed applications other than medi-cally*' based ones like HEADMED and PUFF, and the doubtful example of SACON.(IMeideatreservations aside, it is clear that Buchanan has made substantialcroqress towards understanding and solving the difficult problem ofacguir-ing large judgemental knowledge bases, making such knowledge accessible tomachine manipulation, and making the results of such manipulation accessibleto those who need to know them. Buchanan has played a major role in further-ing much important research in knowledge-based inference, consultation, andheuristic problem-solving systems at Stanford; I expect that he can make im-portant contributions to the search for generality in the design of such sg?g-

A

terns, and he should be supported in his effort to do so.A weakness of the proposal is that it fails to address an important issue

in considering the acquisition and evaluation of large judgemental knowledgebases. How will the system deal with disagreements among his expert know-ledge sources? Presumably, a powerful consultation system will draw upon thknowledge of a community of experts. Then one can predict (with CF = 1 )that there will be freouent and sometimes wide disagreements among them on,

(<

Clearly, any system designed to accept many sources of input knowledge,and where many experts differing in their expertize, experience, back-ground, and problem-solving philosophy will cooperate in evaluatingthe system's performance in order to refine the knowledge base willhave todeal with divergence of opinion without becoming unstable. Wefind this to be a potentially serious problem in our own research; theEMYCIN system design cannot ignore it.

One final remark. While the budget seems quite reasonable overall,$2800. appears to be somewhat excessive for one trip to IJCAI in Japanfrom the west coast.

erbatim but anonymous copies of reviews, ratings and associated correspondence will be sent only to the principal investigator/project director on request.jbiect to this NSF policy and applicable laws, including the Freedom of Information Act, 5 USC 552, reviewers' comments will be given maximum protectionom disclosure. 1

NATIONAL SCIENCE( PROPOSAL EVALUATION FORM ( nsf Form ib do-78)FOUNDATION

h.- ... .„ Supersedes Forms 173 and 990

"ROPOSAL

NO. INSTITUTION

PLEASE

RETURN BY

STANFORD UNIVORIbLCIPAL

INVESTIGATOR NSF PROGRAM

title

KNOWLEDGE-BASED CONSULTATION SYSTEMSCOMMENTS (CONTINUE

ON ADDITIONAL

SHEET(S)

AS

NECESSARY)

The proposal is very good and wall thought-out. The problem area ofmachine assistance in the acquisition of expert knowledge is very im-portant, and I believe this proposal will make important contributionsto this area.

My only qualms about the proposal are the simplicity of EMYCIN problemsolving structure; I wonder whether certain acquisition issues aren'ttremendously simplified byuusing such a structure.- But, from a researchstrategy viewpoint, this probably is the right direction to go.

I am also struck in a way by the lack of creativity of the proposal.It seems such a natural progression from previous work, a little changehere and there, as the ideas are made more generally applicable. Idon't see any new big ideas, just a lot of good engineering work. How-ever, these qualms don't at all reflect on my view that this researchshould be funded, but just a way of me expressing to the P.I. my long-term concerns about the research direction.

i.

fxfvERY GOODBATING EXCELLENT GOOD

~*

MCS-7903753

BUCHANAN BRUCE C | yw

Qfair D poor

4*

*(

Review of NSF Proposal MCS* J3753Knowlcdgc-Bnscd Consultation SystemsPrincipal Investigator: Bruce Buchanan

One of the key problems in the knowledge-based approach to computer system con-struction is the difficulty of transferring expertise from human experts to the computersystem. The proposed research is intended to explore ways of lessening this difficulty.While I appreciate the importance of the problem and believe the requested funds shouldbe granted, I think that certain comments arc necessary to put the proposed approachin perspective.

The first criticism concerns the use of production rules as a representation for exper-tise. While production rules arc an ideal way to express inference rules in an unstructureddomain like investment, counseling, they arc inappropriate in more structured domainslike mathematics. Contrast, the awkwardness of a production rule version of multi-digitmultiplication or division with the straightforward algorithmic description. Or considerthe awkwardness of a production rule version of the algorithm for tying one's shoes. Veryfew task environments are so unstructured that there is no expertise that would benefitfrom being expressed in another formalism. As a simple formalism for early experimentsin knowledge-based system construction, production rules were an excellent choice, par-ticularly for use by task environment experts unfamiliar with computers. However, theinteresting problem now is the invention of more flexible formalisms and mechanisms thatfacilitate their use. An effective general approach to the construction of expert problemsolving systems cannot be expected without such a formalism.

A more severe criticism of the proposed approach is the absence of any way todescribe the structure of knowledge in a task environment or across task environmentsand the absence of any predefined structures. For example, there is no way to describe thecommon features of the digestive system and the circulatory system. One can't describethe crucial abstraction of negative feedback from hormonal regulation and electronic cir-cuits. There is simply no vocabulary for describing abstractions and for relating differentfragments of knowledge. By comparison, when a person undertakes to master a new task:environment, he is not "empty", like EMYCIN, but rather possesses a fund of metaphorsto help in organizing the knowledge he is to acquire. The student actively assimilatesnew knowledge by fitting if into what he already knows, by adapting old metaphors andinventing new ones. In EMYCIN, there is no store of predefined or acquired structuresthat the system builder can take advantage of and no expertise for creating them.

The proposed program of research is not concerned with finding new and moreflexible representations for expertise nor is if concerned with encoding abstractions andmetaphors to facilitate the communication process. Apparently, the goal is simpljr toimprove and provide aids for the passive, "empty", production rule approach to system

1

-i

-> »

2

/-- L(

construction. As such, the projL, has more the flavor of a good engineering effort thana scientific investigation. Although a number of important unsolved problems are men-tioned (e.g. the representation of time-dependent knowledge and mcta-lcvel knowledge),the description of their proposed treatment is but sketchy and includes no innovations!

The purpose of these comments is to emphasize the nature of the proposed researchas an engineering project intended to facilitate one limited approach to knowledge-basedsystem construction, f think it is a job that must be done if we arc to understand theexact limitations of the approach, and the directions outlined in the proposal seem sound.If such engineering meets the funding objectives of the NSF, I strongly recommend thatthe project be supported.

The principal investigator, Bruce Buchanan, is more than qualified to supervise sucha project. I have read most of his published papers, and I believe his research sense andability arc very high. I am surcjic will make the project a success. I am not familiarwith the. support staff, but I think graduate students should easily be able to handlethe subprojeefs outlined in the proposal. Of course, Stanford has an excellent researchenvironment and adequate facilities.