FINAL REPORT TONATIONAL COUNCIL FOR SOVIET AND EAST EUROPEAN RESEARC H

TITLE:

DECISION ANALYSIS IN THE USSR :THEORY, METHODOLOGY, AND APPLICATIONS

AUTHOR: Vladimir M. Ozerno y

CONTRACTOR:

California State University, Hayward

PRINCIPAL INVESTIGATOR :

Vladimir M. Ozernoy

COUNCIL CONTRACT NUMBER :

629-5

DATE:

April, 198 6

The work leading to this report was supported by funds provide dby the National Council for Soviet and East European Research .

TABLE OF CONTENT

Page

EXECUTIVE SUMMARY 1

1.0

INTRODUCTION . : 4 -

2 .0 HISTORICAL OVERVIEW 7

3 .0 DECISION-MAKING METHODS AND TECHNIQUES DEVELOPE DIN THE USSR 1 3

4 .0 APPLICATIONS OF DECISION ANALYSIS 22

5.0 SOVIET USE OF AMERICAN EXPERIENCE IN DECISIONANALYSIS 28

6.0 LONG-TERM TRENDS IN RESEARCH AND APPLICATIONSOF DECISION ANALYSIS IN THE USSR 34

NOTES 3 7

REFERENCES 39

11

EXECUTIVE SUMMARY

Decision analysis has become a scientific discipline o fgrowing importance in the USSR . Today, quantitative methods an dcomputers are widely used to analyze complex decision- makin gproblems . The main areas in which decision-analytic method shave been applied have been in the evaluation of R & D project sand in engineering . Soviet specialists have done importan toriginal work in theoretical areas of decision analysis . TheSoviets remain far behind the U .S ., however, in such areas a sdecision support systems, expert systems, and decision analysi sapplications . They consequently devote considerable energy t ostudying American methods through translations of America nliterature, studies of American experience, join tSoviet-American projects, and scientific exchanges .

There is no prospect that Soviet decision analysis researc hwill help transform the Soviet state from an inefficient ,industrially based society into a modern information-base dsociety, because of intrinsic obstacles to the development an dapplication of information-based management systems . Thi sdevelopment is severely inhibited by the irreconcilabl econtraditions of Soviet reality . On the one hand, the Communis tParty and Soviet government encourage the use of quantitativemethods and computers to assist in decision-making in the area sthat are not directly related to ideological issues . On th eother hand, they are unwilling to use decision-analytic method sin the management of the economy, since these methods coul dchallenge the so-called "leading and managing" role of th eCommunist Party .

Nevertheless, the rapid growth of research in decisio nanalysis will go on . As long as the Soviet leadership continue sto attempt to bring Soviet industry and technology up to worl dstandards, research in decision analysis will be successfull ypromoted by the USSR Academy of Sciences and the USSR Committeefor Science and Technology . Such research is conducted today a tnumerous scientific research institutes of the USSR Academy o fSciences, universities, and classified research organization sworking for the military-industrial complex . Among the mos timportant institutes pursuing research in this area are th eInstitute for Systems Studies, the Institute of Contro lSciences, and the Computer Center of the USSR Academy o fSciences .

The motivations underlying Soviet interest in America nresearch, joint projects, and scientific exchanges do no tcorrespond to that of their American counterparts . As a result ,American scientists stand to gain little from joint research an d

1

scientific exchanges unless new ground rules for informatio nsharing are established and enforced . This has been clearl ydemonstrated in the most important joint Soviet-American projec t- the creation of the International Institute for Applie dSystems Analysis in Laxenburg, Austria . Whereas America ninterest in joint work grows out of a genuine desire to shar eand develop ideas and experiences, the Soviets' interest i sgeared toward a one-sided gathering of information about Wester nresearch; gaining access to Western research organizations ,industrial enterprises, computer systems, and data bases ; andeven conducting scientific and industrial espionage .

This report reviews and analyzes the current state of Sovie tcontributions in decision analysis theory, methodology, an dapplications . It provides a comprehensive survey of th edevelopment of decision analysis and its applications in th eUSSR from 1971 to the present . It considers five major issues :

1)

Causes for the development of decision analysis in th eUSSR during the last fifteen years ;2)

Recent progress made in decision analysis in the USSR ;3)

Practical applications of decision analysis methodolog yin the USSR ;4)

Adoption and use of American experience in Sovie tdecision analysis ;5)

Long-term trends in the research and application o fdecision analysis in the USSR .

Among the conclusions of this report are the followin grecommendations :

1.

Greater attention should be given to studying an dkeeping abreast of Soviet research in decision-making theory an dmethodology . In a number of areas Soviet scientists have openedoriginal paths of research and are making importan tcontributions to the development of decision theory an dmethodology . This work is largely unknown in the West .

2.

To implement the first recommendation, a center for th estudy of Soviet science and technology should be established a ta major American university . The work of such a center woul dencompass Soviet research in the decision sciences .Establishment of such a center would fill a significant void i nuniversity level Soviet studies sciences to the neglect of th epure and applied sciences . The center's investigation of Sovie tresearch in decision analysis would form a useful bridge betwee nexisting Soviet studies programs focusing on the social science sand the scientific and technological orientation of the ne wcenter .

3.

Much effort should not be expended in studying Sovie tapplications of decision making theory and methodology . Sovie tpublications on this subject are poorly documented, and th e

2

practice itself is severely limited . Many of the application sdiscussed in Soviet literature pertain only to Soviet realit yand hold no interst in the U .S . .

4.

To broaden the study of Soviet experience in th edecision sciences, more American researchers and graduat estudents should be encouraged and supported to participate i nexchange programs and conferences in the USSR . It is especiall yimportant that not only Sovietologists, but American scientist sas well, participate in such activities . Even though the g roun drules for such exchanges need to be revised to insure a mor ereciprocal sharing of information by the Soviets (see recom-mendation *6 below), no purpose is served by continuing to hav eso few American scientists aware of the work of their Sovie tcounterparts .

5.

More Soviet books and articles in decision analysi sshould be translated into English . This is essential i fAmerican specialists are to keep abreast of the lates tdevelopments in Soviet research .

6.

In establishing and conducting scientific exchange swith the Soviet Union, reciprocity in the exchange o finformation must be required . Currently such reciprocity doe snot exist . In particular, American specialists participating i nexchanges must be given access not only to universities an dthose institutes especially designated for dealing wit hforeigners, but also to the many scientific research institute sconducting important ongoing work in decision-making theory ,methodology, and practice .

3

2 . 0HISTORICAL OVERVIEW

As in all industrialized countries, the complexity of decisio nmaking problems has grown tremendously in the USSR. At the same timethe cost of making decisions has risen steadily, and the consequence sof unfortunate decisions have become more serious . These factors havespurred growing interest in tools to aid in decision making .

Recognition of the need for such tools emerged in the USSR durin gthe last fifteen years . Earlier, virtually no quantitative methodswere used in making business and industrial decisions . Decisionmaking was considered essentially an art based on experience andintuition. Moreover, the application of quantitative methods wa sperceived as alien and pernicious to socialism. It was understoo dthat the only necessary condition for arriving at the right decisionwas ideological devotion and Communist Party loyalty. But withtime, the cost of making decisions as well as the price of wron gdecisions rose dramatically, and the need for a quantitative approac hto decision making was realized . This laid the basis for thedevelopment of decision analysis methods in the Soviet Union . I nturn, advances made in these methods changed opinions about the veryprocess of decision making . Today, quantitative methods and, more an dmore often, computers are widely used in the USSR for analyzin gcomplex problems and reaching decisions . In the USSR, the analytica lapproach to decision making is becoming a scientific discipline whos eimportance is growing .

In the Soviet Union, as in the United States, there is n ouniversal agreement on the definition of decision analysis. However,the connotation of the term "decision analysis" is different in th etwo countries .

In the United States, decision analysis is usually identified wit hthe methodology for decision making under uncertainty (see, fo rexample, Keeney [1982]) . In the Soviet Union, decision analysis isunderstood as a collection of formal, analytical techniques offerin gpotential tools for improving the quality of decisions . Most of thes etechniques are designed either to help a decision maker identify asingle preferred course of action from among all feasible alternative sor to rank-order the alternatives .

Like many other scientific terms invented in the West, "decisionanalysis" does not have an exact equivalent in Russian. The closestterm is "analysis of decisions" . However, it is not used very often .

7

Synonyms used in the USSR for the term "decision analysis" arenumerous . A frequent synonym is "decision making" . Other terms are"decision science", "decision-making theory and methodology", an d"decision-making theory" . While a variety of names exist for the bod yof knowledge and methodology involving quantitative approaches t odecision making, two of the most widely known and accepted ar e"decision-making theory" and simply "decision making" .

Decision analysis is a relatively new research area in the USSR ,coming into existence some fifteen years ago . In the Soviet Union ,decision analysis emerged from and exists at the intersection ofoperations research, applied mathematics, cybernetics, managemen tscience, and military applications . The greatest influence ondecision analysis was produced by cybernetics, which in the USS Rincludes operations research, computer science, control theory ,statistical decision theory, information theory, production an doperations management, bionics, artificial intelligence, theory o ffinite automata, and many other disciplines (see, for example ,Korshunov [1972], Lerner [1967], and Glushkov [1964]) .

The early history of cybernetics in the USSR is similar to that o fother scientific disciplines born in the West . Such new discipline shave often been viewed by the Communist Party as incompatible wit hMarxist doctrine and ultimately as dangerous means for the penetratio nof so called "reactionary bourgeois ideology ." Major fields, fromgenetics to econometrics, were initially banned as anti-Marxist, andthe Soviet Union's research in these areas fell behind the West's .Cybernetics was one of these fields . Here, for example, is th edefinition of cybernetics given by the Concise Philosophica lDictionary (see Rosenthal and Yudin, [1954]) : "Cybernetics is areactionary pseudo-science which arose in the U .S .A. after World Wa rII and was widely spread in other countries . . . . Cybernetics clearl yreflects one of the major features of the bourgeois outlook, it sbrutality, its aim to transform the working masses into appendages ofmachines, instruments of production, instruments of war . On the otherhand, cybernetics features an imperialistic utopia : to replace athinking human being, struggling for his interests, by a machine, t omake this replacement both in the production process and in war .Instigators of a new World War use cybernetics for their own dirt ypurposes . . . . Thus, cybernetics is not only an ideologica linstrument of imperialistic reactionaries but a means of carrying ou ttheir aggressive military plans as well . "4

However, with years, practical considerations prompted th erehabilitation of cybernetics and related disciplines. In developingmilitary technology and in attempting to improve the effectiveness ofa poorly functioning centralized economy, many complex decisio nproblems emerged . The Soviet bureaucracy eventually realized tha tquantitative methods would not undermine its power and could be use dsuccessfully for many practical purposes, both in problem solving an ddecision making .

Ironically, after the rehabilitation of cybernetics, the Sovie tUnion was swept for many years with cybernetics and computer fever .

8

In numerous articles and books, the full utilization of cybernetic swas equated with the advent of communism .

When the rehabilitation of cybernetics began in the mid-1950's, aspecial unit within the USSR Academy of Sciences was created t odevelop and coordinate all research related to cybernetics . This wasthe Scientific Council on Cybernetics of the USSR Academy of Sciences .Its aim was to foster and promote cybernetics and related discipline sin the USSR. Research in cybernetics, operations research, an ddecision analysis became widespread in several institutes of the USSRAcademy of Sciences, at various universities, in military academies ,and at numerous classified research organizations working for th eSoviet military-industrial complex . Table 1 lists the mai ninstitutions conducting research in decision analysis and describe stheir basic research directions .

Besides practical considerations, several other factor scontributed to the development of decision-analytic research. One wasthe increased contact between Soviet and foreign scientists in the1960's and 1970's. Visits to the USSR by such scientists as N . Wiener,R. Ackoff, R. Hellman, H. Raiffa, and L. Zadeh, and subsequen tliterature exchanges promoted a good deal of interest in cybernetics,operations research and decision analysis among Soviet scholars .

Another factor contributing to the development of decision -analytic research was the desire of many officials to increase thei rpower and authority by the "scientific justification" of thei rdecisions . For example, A. Vendelin, former Deputy Chairman of th eCouncil of Ministers of the Estonian Soviet Socialistic Republic, use delements of decision analysis in the selection of the preferre dalternative of the water supply system for the city of Tallin and i nseveral other real-world problems (Vendelin [1973] and [1977]) . Hi sinterest in deoision-making problems spurred several researc horganizations to undertake decision-analytic studies .

Finally, many decision makers began to see decision-analyti cresearch as a convenient way for them to obtain advanced academi cdegrees . This also contributed to the broader use of decision -analytic methods . For managers and Communist Party officials seekin gadvanced academic status, decision analysis proved to be aparticularly convenient field since, on the one hand, decision -analytic methods and applications had a direct relation to managementand, on the other hang, they belonged to an officially recognize dscientific discipline .

Concurrent with the broader acceptance of decision analysi smethods, beginning in the 1970's, there appeared numerous Sovie tpublications related to decision analysis . During the past fiftee nyears, a great number of works has been published discussing variou stheoretical, methodological and practical aspects of decision making .See, for example, the comprehensive bibliographies in Podinovskii andNogin [1982] and Makarov et al . [1982] . The dramatic growth in thevolume of such publications reflects the increasing importanc eaccorded to decision problems by numerous Soviet research institutes .

9

Soviet Research InstituteDecision Analysis Relate d

Research Directions------------------------------------------------------------------ -

Institute of ControlSciences, Moscow

Computer Center of theUSSR Academy of Sciences ,Moscow

Institute for SystemsStudies, Moscow

International ResearchInstitute of Managemen tSciences, Moscow

Central Institute forEconomics and Mathematics ,Moscow

Institute of Cybernetic sof the Ukraine Academyof Sciences, Kie v

Institute of Cyberneticsof the Georgia Academyof Sciences, Tbilis i

Institute for MachineDesign, Moscow

Leningrad Stat eUniversity, Leningrad

Riga PolitechnicalInstitute, Riga

Multiple criteria decision making ,theory of choice, social choice, expertprocedures in decision making .

Fuzzy decision analysis, multiobjectivelinear programming, long-term planning .

Multiple criteria decision making ,social choice, expert systems .

Multiobjective linear programming ,interactive decision-making procedures ,resource allocation .

Decision making in long-term planning .

Multiobjective linear programming .

Decision making under uncertainty ,multiple criteria decision making ,fuzzy decision analysis .

Generating decision alternatives ,multiple criteria decision making ,interactive decision-making procedures .

Multiple criteria decision making .

Fuzzy decision analysis, decisionsupport systems .

--------------------------------------------------------------------- -

Table 1 : Main Soviet research institutes conducting studies i ndecision analysis

1 0

While many technical papers in decision analysis are publishedevery year in the USSR, there are no special journals comparable t oOperations Research, Management Science, or Decision Sciences .Similarly, there are no professional societies equivalent to theOperations Research Society of America (ORSA), the Institute o fManagement Sciences (TIMS), or the American Institute for DecisionSciences (AIDS) . In the USSR, decision analysis articles ar edispersed among numerous technical journals . Articles in decisionanalysis are also published in the proceedings of conferences, and th etransactions of research institutes. Some Soviet journals featurin garticles on decision analysis are translated into English . For a listof these journals, see Table 2 .

1 1

Soviet Journal

Translated as

Publisher--------------------------------------------------------------------- -

Avtomatika i Automation and Consultants Bureautelemekhanika Remote Control 233 Spring Street

New York, NY

1001 3

Tekhnicheskaya Engineering Scripta Publishing Co .kibernetika Cybernetics 7961 Eastern Avenue

Silver Spring, MD

20910

Zhurnal vychisli- USSR Computational Pergamon Presstelnoi matematiki Mathematics and Fairview Parki matematicheskoi Mathematical Physics Elmsford, NY

10523fizik i

Ekonomika i Selected articles are M.E. Sharpe, Inc .matematicheskie translated in : 80 Business Park Drivemetody Matekon (Translations Armonk, NY

1050 4of Russian and EastEuropean MathematicalEconomics )

Referativnyi Cybernetics Abstracts Scientific Informationzhurnal Consultants, Ltd ."Kibernetika " 661 Finchley Road

London NW2 2HNEngland

Automatika Soviet Automatic Scripta Publishing Co .Control 7961 Eastern Avenue

Silver Spring, MD

2091 0

Kibernetika Cybernetics Consultants Burea u233 Spring StreetNew York, NY

1001 3

Automatika Automatic Control Allerton Press Inc .i vychislitelnaya and Computer Sciences 150 Fifth Avenuetekhnika New York, NY

1001 1

Moskovsckii Moscow University Allerton Press Inc .universitet . Computational 150 Fifth Avenu eVestnik . Seriia 15. Mathematics and New York, NY

1001 1Vychislitelnaya Cyberneticsmatematika ikibernetika

Table 2: Some Soviet journals featuring papers on decision analysi sand their English translations .

1 2

3 . 0DECISION-MAKING METHODS AND TECHNIQUES DEVELOPED IN THE USS R

3 .1 Theoretical Research

The main areas of theoretical research in decision analysis i nthe USSR are (1) theory of choice, (2) group choice, (3) mathematica lmodeling of decision-making problems, (4) axiomatic substantiation ofdecision rules, (5) fuzzy decision analysis, and (6) multiobjectiv emathematical programming .

There exists an established school of scientists in Moscowworking in the area of choice theory . Aizerman et al. proved (see ,for example, Aizerman et al . [1977a] and [1977b], and Aizerman andMalishevsky [1981]) that numerous previous results in choice theory ,not related to each other, are components of a broader theory . Theydemonstrated that the conventional mechanisms for making the bes tchoice (such as scalar optimization, vector optimization, and pai rdominance) make explicit or implicit use of the principle of "pai rcomparability" for alternatives and of the "dominance" rule fo rchoice . Aizerman et al . also considered some "nonclassical" choic emechanisms--those that cannot be reduced to any of the conventiona lmechanisms of the best choice by pairwise comparisons . Unlike theclassical choice schemes, in which the results of pairwise comparisonshould be independent of the choice context, these "nonclassical "choice mechanisms are not necessarily restricted by this property o fthe independence of irrelevant alternatives . Aizerman et al . mad esubstantial progress in developing a general theory of choice whic hwould allow the description, classification, and study of bot hclassical and nonclassical choice mechanisms . They establishe dnecessary and sufficient conditions for the existence of choic efunctions generated by particular choice mechanisms . The survey byAizerman [1984] covers the research on the general theory of choic econducted at the Institute of Control Sciences . The survey by Naumov[1985] summarizes and discusses conditions for the existence o foptimal options in a choice on a fixed compact set of alternatives .The comprehensive survey by Emelyanov and Nappelbaum [1977] contain squite detailed description of Western work in the theory of choice .

There has also been much Soviet interest in group choic eresearch. The most important work in this area is the excellent boo kby Mirkin [1979] which has been translated into English . Othe rcontributions to group choice theory include the book by Kuzmin [1982 ]on collective choice with unfuzzy and fuzzy binary relations and th eanalysis by Aizerman and Aleskerov [1983] of the problem associatedwith the Arrow paradox in 'doting theory . Aizerman and Aleskerov[1984a] and [1984b] investigate voting operators which transfor m

1 3

collections of individual choice functions into a group choicefunction. They establish that these operators, which transform th epreference ordering of options used by the voters into a grou ppreference ordering, are functional analogs of the classical Arrowparadox. Aizerman and Aleskerov also investigate the reasons for th eobserved inconsistency of seemingly natural conditions imposed o nvoting operators and establish under what conditions analogs of theArrow paradox arise in the framework of the functional approach tovoting problems . Lezina [1985] discusses the relationship betwee ncharacteristic constraints imposed on group choice functions and th emanipulation of these functions .

There also is considerable Soviet literature on mathematica lmodels of decision-making problems. Many works have been publishe drelated to the conditions for effective (Pareto-optimal) solutions an dof the structure and properties of solutions (see, for example ,Germeier [1971b], Podinovskii and Nogin [1982], Makarov et al . [1982] ,and Beresovsky et al . [1981]) . Algorithms for determining the set ofnondominated solutions in discrete alternative problems have bee nproposed in Gaft and Ozernoy [1974], Vinogradskaya [1974] and[1977], and Makarov et al . [1982] .

Polterovich [1984] investigates the problem of determinatingPareto solutions which satisfy additional "equality type" condition sand proves a theorem for the existence of solutions . Lexicographicmodels and methods of optimization are developed in Podinovskii an dGavrilov [1975] . Set-theoretic approaches to the description andsolution of multiple criteria problems are used in Gaft and Ozerno y[1974], Makarov et al . [1982], and Beresovsky et al . [1981] .

Podinovskii [1975] introduces the concept of uniform equivalen tcriteria (i .e ., criteria or equal importance having the sam edimensionality) . Based on this concept, he investigates a preferenc erelation (which is a partial quasiorder) and examines the propertie sof the resulting nondominated strategies. When compensation of smal lchanges in the levels of certain criteria by large changes in th elevels of other criteria is either entirely forbidden or entirelypossible, Podinovsky suggests simple methods for reducing a discretemulticriteria problem to a scalar-valued optimization problem .

In Podinovskii [1976], [1978a], [1978b], and [1979], a genera lmethodology is developed for the evaluation of the importance o fcriteria in multiple criteria decision-making problems . Podinovski iintroduces definitions to be used in determining the importance ofcriteria and groups of criteria and investigates the consistency andmeaningfulness of information gathered on criteria . On the basis ofsuch qualitative information, he is able to specify preference andindifference relations as well as the transitive closures of thes erelations . In addition, Podinovskii discusses some properties of theserelations and demonstrates the possibility of using these propertie sfor the analysis and solution of multiple criteria decision-makingproblems .

There is a growing Soviet literature in the area of fuzz ydecision analysis . Orlovsky [1981] introduces various methods fo r

1 4

describing and analyzing decision-making problems using fuzzy sets .He analyzes mathematical programming problems with fuzzy areas o ffeasible solutions and fuzzy objective functions . In Orlovsky [1978 ]and [1981], fuzzy equivalence and strict preference relations ar edefined for a given fuzzy non-strict preference relation . Orlovskyintroduces a fuzzy subset of nondominated alternatives which i ssuggested as a solution to the fuzzy decision-making proble mconsidered . Two types of sufficient conditions are found for th eexistence of unfuzzy nondominated elements . The first is purel ytopological as it does not make use of the reflexivity an dtransitivity properties of the fuzzy relation . The second type i srelation-like insofar as it exploits the reflexivity and transitivit yof the fuzzy relation .

Borisov [1982] and Borisov and Merkuryeva [1983] discuss theconcept of linguistic preference relations . A set of axioms is pu tforth which imply the existence of the linguistic utility function.Some methods are suggested for deriving linguistic utility functions .

Borisov et al . [1983] introduce the concept of fuzzy time stream sof consequences and the notion of dominance relations on the set o ffuzzy time streams of consequences . It is assumed that with regard tothese time streams there exists a multiattribute value function, whic his either convex or concave and which may be represented by a Taylo rexpansion . The basic idea is to determine the values of partia lderivatives in the Taylor expansion by obtaining and then solving aset of independent equations for the desired values . Each of theindependent equations is generated on the basis of a nonfuzzy tradeof fprocedure for streams of consequences which the decision maker find sindifferent .

Multiobjective programming methods have been the subject ofresearch in the USSR for a long time. See, for example, Salukvadze[1972] and [1976] . Some Soviet contributions to this area ar ereviewed in Larichev and Polyakov [1980] . Recent contribution sinclude books by Lotov [1984] and Khomenyuk [1983] . In a recen tdevelopment, Korotkova [1983] proposes a method for solving multiplecriteria optimization problems when the dimension of the vector o fvariables is large (hundreds or thousands of variables) and there is alarge number of constraints . The case is investigated in which thedecision maker is minimally informed about the possible alternativ esolutions. Rastrigin and Eiduk [1985] provide a comprehensive surve yof man-machine procedures for multiple-criteria optimization an delucidate the structure of the man-machine interaction .

The consideration of uncertainty in decision-making problems ha sattracted little interest from Soviet researchers . Some theoretica laspects related to decision making under uncertainty are discussed i nTrukhaev [1972] and [1981] and Zhukovin [1981] and [1983] . However ,in methodological research and applications uncertainty is largel yignored . For example, Larichev [1979] claims that uncertaint yconditions are not typical in nonrecurring real-life decision-makin gproblems .

1 5

3 .2 Methodological Research

In the Soviet Union, the leading methodological approach employe din decision analysis is prescription : 1) identify the decisionsituation and formulate a decision-making problem, 2) structure th eproblem and develop a model, 3) develop a method for solving th edecision-making problem, 4) apply the method in order to evaluat edecision alternatives, 5) prescribe the "best" alternative .

Most Soviet studies start at the stage where the decisionsituation has already been identified and the decision-making proble mhas been formulated. Research is lacking in such areas as identifyin gand bounding the decision situation and diagnosing the decisio nproblem .

Structuring the Decision Problem

The second stage in the perscriptive methodology—structuring th edecision problem--includes the specification of criteria and criteri ascales, the generation of alternatives, and the quantification of th econsequences of each alternative in terms of multiple criteria. Thismethodological stage has also attracted only limited interest fromSoviet researchers .

Specifying Criteria . The problem of specifying a set o fcriteria is considered in Ozernoy [1974] as part of a study aimed a trevealing the relationship between decision alternatives and theirconsequences . A multistep procedure is proposed which allows th esystematic specification of criteria . This article also discusse sissues and makes recommendations for constructing a hierarchy o fcriteria and developing empirical indicators that reflect the degre eto which an associated objective is met . A multistep procedure i salso described for developing criteria scales and verifying th ecorrespondence between criteria and scales .

Procedures for specifying a set of criteria and criteria scale shave been used in several applications in the USSR. For example, ahierarchy of objectives was used in Eknelyanov et al . [1974] forquantitatively evaluating technological process control systems . Ahierarchy of criteria was used in Mechitov [1980] for simplifying theprocedure of evaluating R & D projects .

Generating Decision Alternatives . Soviet research has yet t orecognize fully the need for a systematic approach to the process o fidentifying decision alternatives . In most instances, decisionalternatives are assumed to be known in advance or to be identified b yexperts, even though in some applications, new decision alternative swere to be created rather than simply chosen from a list of availableoptions (Statnikov [1978]) .

In Sadovskaya et al . [1973], a hierarchical procedure i ssuggested for generating a set of alternatives--in this particula rcase, a set of alternative methods for improving the quality o fconsumer goods . First, alternatives are identified which meet th erequirements for those methods in the highest level of the hierarch y

1 6

of methods . However, such alternatives are not operationally usable .These alternatives are successively broken down to lower leve loptions, until at the lowest level of the hierarchy a set ofoperationally feasible alternative methods is identified .

Two normative methods for generating alternatives are wort hreviewing : morphological analysis and search in parameter space . Bothhave been applied to real-world problems .

Morphological analysis is an approach which allows the user togenerate all the potential options if all the parameters of a problemand their possible values have been specified . This approach wassuccessfully used by Ozernoy and Gaft [1977] for identifying a larg enumber of feasible alternative layouts of coal mines .

Another way to generate a large number of alternatives is toconduct a search in parameter space . This approach was used b yStatnikov [1978] and Sobol and Statnikov [1981] in their study o fparameter selection in machine design . The authors explored thefeasible region defined by giving constraints using sequences ofuniformly distributed non-random trial points .

An effective algorithm was developed and used for the selectio nof these trial points in the n-dimentional space of parameters .Performance criteria were given in implicit form as nonlinearfunctions of the design parameters with many local extrema . In orderto quantity machine design alternatives in terms of multiple criteria ,systems of differential equations were solved for each non-rando mpoint .

Another technique for quantifying a large number of options i nterms of multiple criteria was developed in Burchakov et al . [1972] toevaluate coal mine layout alternatives . The quantification of thelayout alternatives in terms of multiple criteria was greatl ycomplicated by the reappearance of some criteria at several levels ofthe morphological chart. The computations were significantly reduce dby evaluating all possible combinations of criteria values rather tha nall possible combinations of elements of each design alternative . Thenumber of possible combinations of the former is far less than of thelatter. Various combinations of criteria values of elements from twoor more layout levels were quantified by expert designers in adialogue with an analyst to evaluate all feasible layout alternativesby computer.

Decision-Making Methods

To facilitate the discussion of various decision method sdeveloped and/or used in the USSR, they are grouped into two mai ncategories: one-step methods and multistep methods. The distinctionbetween these two categories rests on the type of decision rule use din a method and the way it is employed .

One-step methods involve a single application of a decision rule .Multistep methods are based on the iterative application of a decisio nrule or on the successive application of several decision rules .

1 7

One-step Methods . One-step methods received the most attentionin the USSR at the beginning of the 1970's . Although these methodsseem very diverse, they all consist of the following components : 1) aset of feasible alternatives with specified attributes and attribut elevels, 2) numerical values expressing the attribute levels of eachfeasible alternative and numerical weights expressing the comparativ eimportance assigned to each attribute, 3) a well specified objectiv efunction (often referred to as a global criterion) which uses th eattribute weights to aggregate all the attribute levels of analternative into a single index, and 4) a rule for rank-orderin galternatives (or choosing the most preferred alternative) on the basi sof the highest value of this index .

Emelyanov et al . [1973] introduce the most exhaustive list o fglobal criteria, also referred to as "compromise schemes", and discus stheir relative advantages . Each global criterion is a function o fseveral parameters . These are a) an "ideal", "optimal", or "required "attribute level of each attribute, b) a normalized attribute level o feach attribute for each alternative, c) an attribute weight (o r"importance factor") for each attribute, d) a decision rule inde xenabling the generation of various decision rules .

Emelyanov et al . assume that by assigning particular weights andchoosing the decision rule index, the decision maker can generat evarious decision rules from which he can choose the one appropriate t othe particular decision-making situation at hand . However, whenattempting to use a global criterion, one faces the problem o fchoosing among various accepted principles for normalizating attribut elevels . Another problem involves determining attribute weights .Emelyanov et al . [1973] discuss some approaches for handling thes eproblems. Nevertheless, great methodological difficulties remain i nthe application of global criteria .

In reality, the choice and justification of attribute weights an ddecision rule indices are more difficult for a decision maker than th edirect choice of the most preferred alternative . Many works (e .g . ,Germeier [1971a] and Borisov [1972]) offer a set of decision rule sfrom which a decision maker could select one consistent with his o rher understanding of the decision-making problem . Such an approac hwas justly criticized by Venttsel [1972) who rightly pointed out tha tthis approach simply transfers the issue of making an arbitrar ydecision from the problem of selecting the best alternative to theproblem of selecting a mathematical model and an objective function .

Many one-step methods are complex procedures and includ etechniques for revealing and using the decision-maker's preferenc einformation . Buyanov and Ozernoy [1975] established sufficien tpreference conditions for ranking multicriteria alternatives using anadditive value function . In this work as well as in Buyanov an dOzernoy [1974] conditions are derived to determine whether tw oalternatives can be compared given the results of a comparison of som esimple alternatives. Buyanov [1978] suggested a procedure that guide sthe analyst in revealing preference information for determinin gwhether two alternatives can be compared. Larichev [1977] and [1978 ]

1 8

and Larichev et al . [1974], [1979] introduced some conditions fo rranking multicriteria alternatives implicitly using an additive valu efunction .

Glotov et al . [1978] developed an approach for the classificatio nof multicriteria alternatives using linguistic criteria and preferenc einformation in the form of classification tables . Shestakov [1978 ]introduced two techniques for revealing individual preferences : thegraphical plotting of preference patterns and the construction o fnumerical preference indicators by the least squares method . Bothtechniques use information about equivalent alternatives in order t oarrive at a general model of preference structure for the whole set o falternatives. Shapot [1978] suggested a method for deriving anadditive value function based on the comparison of some multicriteri aalternatives by the decision maker. This value function is consisten twith the revealed preference judgements and at the same time maximize sthe ability to distinguish values of alternatives whose criteri aestimates contain errors .

Multistep methods . Multistep methods are used in such decision -making problems that call for a partial ordering of decisionalternatives . An iterative procedure for formulating a sequence o fdecision rules was suggested by Ozernoy and Gaft [1974], [1977], an d[1978], Gaft [1979], and Emelyanov et al . [1976] . Each decisionrule permits at least partial elimination of the noncomparability o fdecision alternatives which existed at the previous stage of theprocedure. At every stage, additional information about the decisio nmaker's preferences is obtained. This information is used toformulate an intermediate decision rule . The sequence of decisionrules obtained must possess the following property : The initial rule(which is the weakest) is based on the simplest and most obviou sassumptions, and subsequent rules are formulated on the basis ofobtaining additional preference information and using additiona lassumptions which do not contradict those previously accepted. If thedecision rule formulated at some stage leads to the required orderingof the set of feasible alternatives, then it is adopted as the fina lrule and the procedure is terminated with it . Otherwise additiona lpreference information is obtained and a new decision rule i sformulated .

In Ozernoy and Gaft [1974] and [1977], general requirements ar eformulated for checking the meaningfulness and consistency o fadditional preference information . Every iterative procedure of thi stype must satisfy such requirements .

In proposing procedures for formulating decision rules, Ozerno yand Gaft substantiate the desirability of obtaining a sequence o fdifferent types of preference information from the decision maker . Aclassification of the various types of preference information i ssuggested in Ozernoy and Gaft [1974] and [1977] . The unifying"yardstick" in this classification scheme is that it is based o ngauging how a change in the level of one or more attribute will affec tthe decision maker's preferences .

1 9

Gaft and Podinovskii [1981] introduce a general method forspecifying preference relations when preference information i sincomplete and not uniform . Each fragment of the preferenc einformation is used to specify a fragmentary preference relation .Necessary and sufficient conditions are formulated for thenoncontradiction of fragmentary relations . For noncontradictoryfragmentary relations, it is shown that a global ordering can b especified as the transitive closure of the union of these relations .See Podinovskii [1978], and [1979], Ozernoy [1980], and Gaft et al .[1981] for more detail . In Gaft et al . [1981], methods are develope dfor (1) verification of the noncontradiction of the fragmentarypreference relations, (2) the development of the transitive closure o fthe union of the fragmentary relations, and (3) the selection of suc hnoncontradictory parts of contradictory fragmentary relations tha tensure the most comprehensive global ordering .

Soviet accomplishments in the area of multiple criteria decisionmaking are reviewed in Ozernoy [1975] and Emelyanov and Ozerno y[1975] . Additional references can be found in Podinovskii and Nogi n[1982] and Makarov et al . [1982] . Interactive decision-makingprocedures are discussed in Irikov and Kurilov [1978], Krumber g[1981], Kurilov and Larin [1982], Irikov [1982] and Shestakov [1983] .

3 .3 Descriptive Decision Research

In the USSR, there have been virtually no empirica linvestigations to determine whether people's behavior complies wit hthe assumptions of normative theories . Most Soviet descriptiveresearch simply surveys Western work in behavioral psychology (see ,for example, Moshkovich [1978], Larichev [1980], [1981], [1982], an d[1984)] . In addition, some attempts have been made to investigat ehuman ability in consistently comparing alternatives that differ wit hrespect of two or more criteria . The results of experiments conducte dby Glotov et al . [1978] suggest that people have difficulties i nconsistently comparing alternatives that differ in three criteria .

Similar results were obtained in laboratory experiments conducte dby Larichev et al . [1974] . In related research, Larichev et al .[1978] asked subjects to compare alternatives using binary criterio nscales to determine whether they could consistently assign th ealternatives to one of two classes . Larichev and Moshkovich [1984 ]studied subjects' inconsistency in assigning alternatives to one o fseveral sets of alternatives, the sets having been rank-ordered on th ebasis of multiple criteria .

Sheptalova et al . [1984] in laboratory experiments investigate dthe capability of subjects to consistently perform the followingoperations : (1) ranking criteria, (2) determining a least preferre dbut acceptable criterion level, and (3) changing acceptable levels fo rvarious criteria in order to arrive at an acceptable alternative .Several hypotheses related to these operations were formulated an dtested .

Recently, the Soviets began publishing papers of such prominen tWestern behavioral decision scientists as P . Slovic, B. Fischhoff and

2 0

P. Humphreis in Transactions of the Institute for Systems Studies .This is a very unusual departure from Soviet practice and indicate sboth the Soviets' keen interest in descriptive research and the goo dworking relationships that have been developed between Soviet an dWestern researchers in this area .

2 1

4 . 0APPLICATIONS OF DECISION ANALYSI S

4 .1 Genera l

Soviet work in decision analysis is not a health mix o ftheoretical/methodological research and applications of this research .Application-oriented publications represent only a small portion o fall decision analysis related publications .

Decision analysis has mainly been applied in R & D project sevaluation (see, for example, Larichev et al . [1974] and [1979], andZuev et al . [1979]), engineering (for example, see Sobol and Statniko v[1981], Ozernoy and Gaft [1977], Borisov et al . [1983] and Gaft et al .[1982] ), and the publishing business (see Larichev et al . [1981] ) .In addition, there are publications describing the use of decisio nanalysis methods in the design of control systems (see Makarov et al .[1971a] and [1971b], Emelyanov et al . [1974] and Vinogradskaya et al .[1984] ), construction (see Larichev et al . [1984)), and health car e(see Agievsky et al . [1983]) . Some attempts have also been made to usethe decision analysis approach in conjunction with simulatio ntechniques and expert judgments for studies in international relation s(see Moisseev [1979]) .

The main feature of Soviet publications on the application o fdecision analysis is their poor documentation. The publications neveridentify the decision maker . Neither the methodology applied nor th eresults of the study are discussed in detail. This is mainly due t othe especially strong censorship imposed on all application-relate dpublications .

Nevertheless, the analysis of Soviet applications holds interes tbecause it reveals some information about the methodologies that hav ebeen applied and by implication indicates the areas in which decisio nanalysis is not used (e .g ., on environmental problems) . Somerepresentative Soviet decision analysis applications are reviewe dbelow.

4 .2 Evaluation of R & D Project sat the USSR Academy of Science s

Larichev et al . (see, for example, Larichev et al . [1974] and[1979], and Zuev et al . [1979]) developed an approach for evaluating R& D projects proposed for study at the USSR Academy of Sciences .While R & D projects are not the main concern of the Academy, which i sprimarily involved in basic research, the Academy accepts a limite d

2 2

number of applied projects which it considers interesting an dbeneficial .

To systematize the evaluation and selection of R & D proposals, aprocedure was developed which allowed a carefully selected group o fscientists to make quantitative assessments of all proposals wit hrespect to numerous criteria and to use an explicitly stated decisio nrule to evaluate and compare the various R & D proposals . Thefollowing criteria were used for the evaluation of proposals (se eLarichev et al . [1974] and [1978] ) : (1) expected scientific value, (2 )importance for the national economy, (3) correspondence between th eproposed research project and the research directions of the institut eto be involved in the project, (4) status of ongoing research in th earea, and (5) resources required .

For each criterion, an evaluation scale was constructed . Thescales are discussed in detail in Larichev et al . [1978] and Zuev e tal . [1979] . As an example, here is a scale constructed for thecriterion "resources required" : (1) no additional resources arerequired for the project ; only some organizational arrangements ar eneeded, (2) more personnel and material resources are needed withi nthe framework of existing laboratories, (3) new laboratories should b eset up for the project .

In this study a decision rule was developed and used, based o ndecision maker's preferences. The application of the decision rul eresulted in the partial ordering of the R & D proposals . That is, theprocedure produced rank-ordered sets of proposals . Each of theseranked sets consisted of those R & D proposals which the procedureassessed as being either equivalent or simply non-comparable . Thi sapproach corrsponded to the scientific policy of the USSR Academy o fSciences which is geared to funding the most preferred proposals andto identifying least preferred proposals in order to improve them inthe future .

4.3 Evaluation of Most Important Research Proposals a tthe USSR Committee for Science and Technology

Every year, the USSR Committee for Science and Technology select sthe most important research projects to be funded (see Mechitov ,1980) . Research proposals submitted to the Committee are prepared b yseveral scientific-research institutions and then forwarded by theCommittee to various USSR Ministries for evaluation. These Ministriesreview the proposals and the requests for their funding . Th eCommittee makes funding decisions on the basis of the Ministries 'findings .

The major drawback of such a planning system is the lack o funiformity in the findings of the various review organizations . Thi sprevents the Committee from the developing and implementing a unifiedscientific policy . In order to remedy the situation, decisio nanalysts constructed a three-level hierarchy or objectives, whic hincluded a single top-level objective and 16 lower-level objectives .The top-level objective of the hierarchy was the "overall value of th eresearch proposal" . The lower-level criteria were divided into two

2 3

groups : criteria for evaluating the research proposal itself an dcriteria for evaluating the scientific-research institute that want sto carry out the research project .

In order to develop a decision rule, managers on theCommittee evaluated all possible combinations of scores at the lowe rlevel of the hierarchy using the criteria scales of the intermediat elevel of the hierarchy. Mechitov claims that such a decisio nprocedure corresponds to the features of the funding problem and i svery convenient for making funding decisions . The method was testedby evaluating four research proposals . Unfortunately, few detailsabout the procedure and its application are reported in Mechito v[1980] .

4 .4 Parameter Selection in Machine Desig n

The problem of selecting parameter values in machine design i svery involved since the machine performance criteria are nonlinea rfunctions of the machine parameters and are expressed in implici tform, with many local extremes . In addition, the feasible parameterarea may consist of several nonconnected subareas .

Sobol and Statnikov [1981] and Statnikov [1978] developed andimplemented a methodology for selecting the preferred machine desig nparameters . The methodology is based on a search in the machin eparameter space . The feasible parameter region defined by give nconstraints is explored using sequences of uniformly distributed non-random trial points. An effective algorithm is used to generate thesetrial points in the n-dimensional space of parameters . At each tria lpoint, the design alternative is evaluated in terms of all th emeasures of performance . The scores are arranged in the form o fspecial trial tables . Each row of the trial table corresponds to aparticular performance criterion and contains the numbers of thedesign alternatives arranged in declining order according to theparticular criterion .

In the process of analyzing the trial table, the analyst and th eclient determine criterion constraints, i .e ., acceptable deviationsfrom the best level for each of the performance criteria . Designssimultaneously satisfying all the constraints form the set of feasibl ealternatives. Finally, nondominated alternatives are selected fromamong these feasible designs .

This methodology has been widely used for selecting parametersfor table concrete vibrators, gears, and many other machines an dmechanisms .

4 .5 Large-Scale Project Selection : Coal Mine Layouts

Ozernoy and Gaft [1977] working with coal mine designer sdeveloped a methodology which was subsequently used for selecting coa lmine layouts . Layout selection was the most important consideratio nin the design of new coal mines and the reconstruction of workin gones. The best alternative was one which maximized the total expecte dprofit over the life of the coal mine .

2 4

A designer trying to select the best coal mine layout, was facedwith many difficulties in attempting to evaluate the effectiveness ofalternative designs of the mine . Usually there were a great number offeasible layouts for any geological condition, yet no reliabl etechniques existed for evaluating the relative benefits of the variou soptions . Accurate computations of the total expected profit wer eimpossible to program because of the uniqueness of each coal deposit .

The existing methods for selecting coal mine layouts were base don the intuitive experience of designers and proved not to be adequat efor several reasons (see Burchakov et al . [1972] and Ozernoy and Gaf t[1977] ) . Therefore a new methodology was proposed consisting of th efollowing steps : 1) generating a list of all the potential layouts ,2) identifying those layouts which were feasible, 3) specifyin gattributes and attribute scales according to which each layout woul dbe evaluated, 4) quantifying the consequences of each layout b yevaluating each alternative according to the specified attributes, 5 )determining the preference structure of the decision maker, and 6 )selecting a limited number of the preferred alternatives for furthe rdesign .

In some cases the number of potential layouts exceeded 200,000,and the number of feasible alternatives exceeded 7,000 .

The analysisof potential and feasible alternatives was carried out b ymorphological analysis . The use of morphological decompositio nallowed all the feasible layouts to be identified by a computer ,thereby greatly reducing the amount of information an expert or a nexpert team had to process .

In order to compare layout alternatives in a meaningfu lquantitative manner, technological, economic and technoeconomi cattributes were specified for each of the six levels of a layout :methods for opening, opening layout, sequence of preparatoryoperations, preparatory technology, sequence of steps in working themine, and the system of mining . A complete list of twenty-fiv eattributes and attribute levels is given in Burchakov et al . [1972] .All the attributes were qualitative measures ; all the attribute scaleswere ordinal and were constructed in a dialogue between decisio nanalysts and a group of coal mine design experts from the USS RMinistry for Coal Industry and the Moscow Mining Institute . Expertopinions were processed by computer in order to derive quantitativ eevaluations of all the feasible layouts in terms of all attributes .

The successive use of preference information in the thre edecision rules discussed in Burchakov et al . [1972] and Ozernoy an dGaft [1977] resulted in the identification of a number of mos tpreferred designs close to the required one .

The methodology outlined above was used to identify the most -preferred coal mine layouts for the Chicherbay area of Kuzbass i nSiberia . It produced a set of feasible layouts containing 7,70 4alternatives and a set of most-preferred layouts containing 2 1alternatives . The methodology was also used to identify the most -preferred layouts for the reconstruction of a V .I . Lenin "Tom' -

2 5

Usinskaya 1-2" mine in Kuzbass, one of the largest in the Sovie tUnion. In the latter case, the set of feasible layouts contained3,12b alternatives ; the set of the most-preferred layouts contained 4alternatives .

In both cases, the set of the most-preferred alternative scontained layout designs which had not previously been considered byexpert designers . In practice, these newly identified alternative sproved to be the best ones in terms of the total expected profit ove rthe life of the coal mine .

4 .6 Research and Development Planning : Electric Power Grids

Borisov et al . [1983] used fuzzy decision analysis to assis tdecision makers in the selection of a design for the three-stag edevelopment of an electric distribution network . In their study, thedesign department offered the project managers twenty nine alternativ edesigns for the electric power grid, each alternative to be develope din three stages . The design variants were created by adding ne wbranches to the network, by switching branches from one junction t oanother, by removing branches from the network or by changing branchparameters .

Electric power grid design problems of this type can beconsidered problems aimed at determining a sequence of actions, namel ythe most desirable sequence for introducing new and reconstructin gexisting transmission lines and electric substations . Each actionleads to a particular consequence in the future . The sequence o fconsequences form a time stream of consequences which is the mos timportant consideration in an electric power grid design problem .

The evaluation of time streams of consequences was the main focu sof Borisov's multiobjective decision analysis . For each of the thre estages of the network development and for each design alternative ,experts evaluated the alternative proposed designs, specifyin gattribute levels for the following attributes : capital investments ,losses of power, additional expenditures for voltage regulation, an dengineering reliability .

The preliminary analysis of twenty nine design alternatives wa sbased on a nonfuzzy dominance relation . The analysis allowed the setof alternatives to be reduced to ten Pareto-optimal (nondominated )alternatives .

One of the most interesting aspects of this study was that tw oattributes--capital investment and engineering reliability--weretreated as fuzzy variables . The decision maker's preferencestructure was quantified and a nonfuzzy value function of fuzz yattributes was specified. The value function allowed the nondominate dalternatives to be rank ordered and a single preferred design to b echosen for the development of the electric power grid . An interactiv ecomputer program was used in this study . However, no sensitivityanalysis was conduoted .

2 6

4 .7 Design Selection for Multifunctional Vans

Gaft et al . [1982] developed a method for selecting preferred ca rdesigns . The method is based on an approach suggested in Ozernoy[1974] for the solution of discrete multiple criteria decisio nproblems . With this approach, the process of designing a car i sbroken down into the following steps : 1) specifying criteria an dcriteria scales for the evaluation of design alternatives, 2 )identifying the set of feasible designs, 3) making a quantitativ eassessment of all the design alternatives according to each of th especified criteria, and 4) comparing feasible design alternatives andselecting the preferred designs . Gaft et al . used this approach fo rselecting the preferred designs of an advanced multifunctional van .The set of feasible designs contained some 3,000 variants . Eachdesign was evaluated with respect to eleven criteria which took intoconsideration technological and economic factors, safety, level o fcomfort, etc . Narrowing down the set of feasible alternatives to amanageable number of preferred designs was accomplished b ysuccessively obtaining preference information as discussed in Gaftand Podinovsky [1981] and Gaft et al . [1981] . By applying a sequenceof three decision rules eight preferred designs were identified . Al leight designs proved to be innovative and met the requirements fo rtechnologically advanced vans .

2 7

5. 0SOVIET USE OF AMERICAN EXPERIENC E

The Soviet government's scientific policy is aimed at adoptin gand using the scientific achievements of the Western World. Due t othis policy, a lot of attention is given to studying developments i ndecision-making theory, methodology, and applications in the U.S. Inaddition, the Soviets, feeling more and more the need for new ideas i nmanagement, make every effort to understand how the Western systemoperates. In the USSR, some officials and science bureaucrats stil lbelieve that certain methods and procedures developed and implemente din the United States can be adapted and used to improve th eeffectiveness of the Soviet economy. For example, Milner argues thatsome American decision-making models, methodologies, and procedure scan be used to improve the management of the Soviet economy (se eEvenko [1977], p . 4) . In order to justify the broad interest i ndecision-making theory and methodology as well as in applied decisionanalysis, some researchers point to the progress made by America ndecision analysts . See, for example, Koryakin [1983] .

The Soviet use of American experience in decision analysis i scarried out in the following four areas : 1) translation of America nliterature, 2) studies of the American experience in decision-makin gmethodologies and applications, 3) joint projects between Soviet andAmerican specialists, and 4) scientific exchanges .

These areas are discussed below in detail .

Translation of American Literature . Since the beginning o f1960's, many American books in the field of decision analysis hav ebeen translated and published in Russian . One of the first translate dbooks was the book by Luce and Raiffa [1961], first published in th eUnited States in 1957 . Today, the voluminous and fast-growin gAmerican literature on decision analysis being translated into Russia nis a sign of the interest and importance given to decision analysis i nthe USSR. Most interesting books of American authors in decisionanalysis are translated into Russian . Books by Raiffa [1977], Keene yand Raiffa [1981], Keeney [1983], Fishburn [1978], and Zadeh [1976 ]are some examples of such translations .

Nevertheless, the translation process in the USSR is a very slowone . For example, it took twenty-three years for von Neumann an dMorgenstern's classic work [1970] to be translated, five years forKeeney and Raiffa [1981], and three years for Keeney [1983] .

To give Soviet specialists access to recent Western publicationsin decision analysis, certain publishing houses in the USSR regularl yissue collections of translations of various articles on the subject .

2 8

See, for example, Chetyrkin [1972], Shakhnov [1979], and Shakhnov an dPospelov [1976] which feature many important papers by Dyer, Farguhar ,Fishburn, Geoffrion, Keeney, Zadeh and other foremost authorities i ndecision analysis .

Abstracts of Western papers in decision analysis are publishe dmonthly in the series of the Soviet journal of abstracts entitle d"Referativnyi zhurnal Kibernetika" (translated into English as"Cybernetics Abstracts", see Table 2) . In addition, translations o fmany interesting papers published in "Operations Research" ,"Management Science" and other Western journals are featured i n"Express Information. Technical Cybernetics" and in other subareas o fthe "Express Information" series . The circulation of these series i svery limited, and Soviet specialists need special permission to gai naccess to these publications . The "Express Information" issues can beread only in special roams at certain libraries . For example, such aroom exists at the technical library of the Institute of Contro l

Sciences in Moscow . It is forbidden to make photocopies of th etranslations published in the "Express Information" series . It i salso forbidden to give references to such translations . (Unti ljoining the copyright convention in the early 1970's the Soviet stranslated in full any article they wanted . After they joined theconvention, condensed versions of selected articles began to appear i nthe "Express Information" series . )

Studies in American Experience in Decision-Making Methodology andApplications. In the 1970's, Soviet researchers became increasingl yinterested in methodologies and applications of decision analysi sdeveloped in the U.S .A. For a number of years, a special departmentat the Institute of the U .S .A and Canada Studies of the USSR Academ yof Sciences was heavily involved in studying American decision-makin gmethods and procedures . The main purpose of these studies was t oexplore possibilities for using the American experience in th eapplication of quantitative methods, procedures, and computers t osolve real-life decision-making problems in the USSR . A number o fresearchers from the Institute of the U .S.A. and Canada Studies spen tseveral months in the United States studying progress in decision-making methodology and applications . One of the products of thei rwork was a book by Evenko [1977], which offers a detailed survey o fmethodological, informational, and organizational aspects of severa ldecision-making procedures employed in the U .S.A. both in the privat e

and public sectors .

A significant part of this book is devoted to sharp criticism o fthe American management practice, particularly its application o fthose decision-making methods which the authors consider mostappropriate for using to improve management in their own socialis teconomy . Here is a typical example of the criticism of the Americandecision-analytic approach (see Evenko [1977], p . 184) :

"This approach fully corresponds to the background premises o fthe bourgeois theory of organization that considers "the value system"of managers and hence the class goals of capitalists to be th eoverriding criterion for judging the quality of decisions . Naturally ,quantitative methodologies oriented toward the use of subjectiv e

2 9

criteria rather than objective factors are favored in capitalis tmanagement practice. The reason is that objective factors no tinfrequently contradict the purposes and policies of the top levelsof capitalist management ."

Despite such harsh criticism of the American decision-analyti capproach and other quantitative methods, Milner asserted in 1977 tha tthe studies of the American experience had been very instructive an dhad helped him and his team develop and implement the organizationa lstructure for the Kama automobile plant (KAMAZ) .8

Joint Projects . In the 1970's, representatives of America nconsulting companies such as SRI and McKinsey & Co . visited the USS Rto explore the possibility of obtaining contracts. In spite ofsignificant interest among several Soviet organizations and in spit eof the obvious likely benefits to the Soviets from such arrangements ,the consulting companies were never hired . One of the major reasonswhy such ventures proved to be impossible is the high level of secrec ypervading all areas of Soviet life . Most economic data, which woul dbe needed to carry out such projects, are highly classified, even whe nnot related to the defense sectors, e .g ., data on individua lconsumption, air pollution and accidental deaths in industry . Theeconomic data that is available is misleading and distorted .

The most important joint Soviet-American project in the area o fdecision analysis was the creation of the International Institute fo rApplied Systems Analysis in Laxenburg, Austria. The institute wa sfounded by the United States and the Soviet Union in the mid-1970s ,and many other countries became IIASA participants .

The basic idea in creating IIASA was to foster cooperation(between American and Soviet scholars and in particular to promot ejoint research projects and the exchange of ideas, methodologies, an dexperience . Moreover, it was assumed that such cooperation coul dcontribute to detente and to the improvement of relations between th etwo superpowers .

However, the Soviets pursued quite different goals .9 Their goalswere :

1) To gain access to Western scientific and technologica linformation, especially in such areas as applied systems analysis ,decision analysis, large scale systems, computer science, and regiona lplanning ;

2) To gain access to Western computer systems and networks an dhence to some very important data bases ;

3) To create the possibility of visiting Western industria lenterprises and research organizations closed to Soviet citizens bu topen to members of international organizations ;

4) To create a safe and convenient base for scientific andindustrial espionage .

3 0

All these goals were successfully achieved by the Soviets . Hereare some illustrative examples .

The Soviets obtained access to Western information retrieva lsystems and were given the opportunity to order through the IIASAlibrary technical reports and working papers from variou suniversities, research organizations, and consulting companies . Thus ,the findings of numerous research projects and the methodologie sdeveloped by American scientists were obtained by the Soviets at n ocost .

The Soviet computer specialists working at IIASA extensively use dWestern computer networks and in at least one instance were able t ogain access to the data base of a Silicon Valley company that was adefense contractor . This story was widely publicized in the Wester npress .

Through his IIASA affiliation, one of the leading Sovie tspecialists on automation in the iron and steel industries, was abl eto visit several modern Japanese steel plants and acquain

t himselfwith some of the latest developments in automatic control of rollin gmills . Before joining IIASA, he had been repeatedly denied access t othese same rolling mills over the course of twenty years .

Finally, one of the top ranking Soviet administrators at IIASAwas engaged in clearly illegal industrial espionage activity and wa scaught in Norway while trying to buy some confidential document srelated to equipment used for oil exploration. Having diplomati cimmunity, he just quietly returned to the Soviet Union . See, forexample, Sedych [1981] .

The USSR Committee for Science and Technology sends three type sof people to IIASA: 1) distinguished scholars, 2) "ideologicall yreliable" specialists with no real scientific credentials, and 3) KG Bofficers posing as administrative, scientific, or clerical personnel .All Soviet IIASA employees are considered to be fulfilling a Communistparty and Soviet government assignment : they are on missions a srepresentatives of the socialist state . They are required to find ou tand bring back to the USSR as much scientific and technologica linformation as possible and at the same time to reveal as littl einformation as possible about Soviet research. As a result, in man yprojects at IIASA the Western and Soviet scientists work separately .Several American scholars who have spent considerable time at IIAS Aconfirmed that they had no working relations at all with their Sovietcolleagues and had no idea about their research .

In summary, the Soviet Union benefits disproportionately from it sparticipation at IIASA, obtaining much scientific and technologica linformation while giving up little or nothing in return .

Some other joint projects in the area of decision analysis areworth mentioning . Many Soviet research organizations, such as th eInstitute for System Studies and the Institute of Control Sciences ,are very interested in cooperation with Western scholars . As aresult, several joint workshops have been organized in the USSR i n

3 1

recent years. The Institute for Systems Studies, Moscow, i ncooperation with IIASA, organized a workshop on Interactive DecisionAnalysis in Moscow, April 2-6, 1984. The workshop focused on a revie wof multiple criteria decision making work done in the USSR and als oincluded presentations by several Western researchers (see Zionts[1984]) . A Soviet-American workshop on normative and descriptiv edecision-making models was held in Tbilisi in 1979 . Papers presentedat the workshop were published in Lomov et al . [1981] . Anothe rworkshop was held in Moscow in May 1981 by the International Researc hInstitute for Management Sciences (IRIMS), Moscow, and the Finnis hOperations Research Society (FORS), The topic of the joint IRIMS-FOR Sseminar was Multiple-Criteria Decision Making . Papers presented a tthe workshop were published under the title "Theory and Practice o fMultiple Criteria Decision Making" (see Carlson and Kochetkov [1983]) .

Scientific Exchanges . The Soviets have given strong indication sof interest in scientific exchanges in decision analysis, however, o ntheir own terms . The Soviet decision science establishment gladl yinvites leading Western decision scientists to present lectures t oSoviet scholars at selected research institutes . There are severa lreasons for the Soviet interest in such visits . One is the desire o fSoviet bureaucrats and scientists to receive reciprocal invitation sfrom the American visitor and thus to create the possibility of bein gsent to the U .S . on a scientific exchange . Another reason for th eSoviet's interest in such exchanges is their desire to amass as muchscientific information as possible in the area of decision analysis t ouse in developing their own decision methodologies, writing surveys ,etc .

In recent years, many leading American decision scientists suchas W . Edwards, B. Fischhoff, R. Keeney, P . Slovic, A . Tversky, D. vonWinterfeldt, and S . Zionts have visited Soviet organizations involve din decision sciences research . The number of Soviet decisio nscientists who have visited the United States is relatively low ,possibly because the U.S. Government has reduced scientific contact sto protest Soviet intervention in Afganistan and Soviet involvement i nPoland . Sore of the recent Soviet visitors have been internationallyrecognized authorities, for example, M. Aizerman, while others havebeen science bureaucrats or "ideologically reliable" colorles sfigures .

Visits abroad are so rare in the Soviet scientific community tha tthey are considered a very high privilege, awarded by the Communis tParty and Soviet authorities for distinguished loyalty and faithfu lservices. In most cases the rewarded services have nothing to do wit hscientific contributions . That is why one sees so few real scholar svisiting the United States (see Taksar [1983]) . For genuine Sovietscholars such as M . Gaft, B. Mirkin, B. Statnikov and many others, wh ohave substantially contributed to the development of decision analysi sin the USSR, the chances of being permitted to attend an internationa lconference, even after receiving a formal invitation, are almos tnegligible. Their requests to participate in internationa lconferences on decision science are denied even before a thoroug hscreening has been completed .

3 2

Soviet scientists sent to international conferences must sign aspecial form (this form is reproduced in Taksar [1983]), where the ypledge to present and discuss only such information which has alread ybeen published in nonclassified and nonrestricted sources, authorize dfor dissemination abroad . The same restrictions apply for Sovie tscientists hosting American visitors .

It is not surprising that even after visiting Soviet institution sor hosting visits by Soviet researchers in the United States, America ndecision scientists often find that they have learned very littl eabout Soviet methodologies and accomplishments .

3 3

6 . 0LONG—TERM TRENDS IN RESEARCH AND APPLICATION S

OF DECISION ANALYSIS IN THE USS R

The development and use of decision analysis methods in the USSRare influenced by an irreconcilable contradiction inherent in th eSoviet system. On the one hand, the Communist party and the Sovie tgovernment encourage the use of quantitative methods and computers fo rmaking decisions in areas that are not directly related to th eCommunist ideology . On the other hand, they are reluctant to us edecision-analytic methods in management and economic planning, sinc ethese methods could challenge the so-called "leading and managing rol eof the Communist party" . This contradiction has an obvious influenceon decision analysis research and applications in the USSR .

During the last fifteen years, there has been a great deal o fwork in the USSR devoted to developing decision-making theory an dmethodology. Undoubtedly, the rapid growth of research in these area swill continue. As long as the Soviet leadership continues to call fo ra "profound restructuring" of the economy to eliminate waste andinefficiency and to bring the nation's industry and technology up t oWestern standards, research in decision analysis will be promoted b ythe USSR Academy of Sciences and the USSR Committee for Science an dTechnology .

While research in decision analysis is conducted in the USSR a tmany universities, research institutes, and military academies ,consulting companies, that could develop and use decision-analytictechniques for the solution of real-life problems, are nonexistent i nthe Soviet Union . In contrast to the United States, most research i ndecision analysis is carried out in the USSR not at universities bu tat scientific research institutes that employ most of the scientist sof the country . Most advanced studies in decision analysis areperformed at the scientific research institutes of the USSR Academy o fSciences, such as the Institute for Systems Studies, the Institute ofControl Sciences and the Computer Center of the USSR Academy o fSciences .

Unlike research organizations and consulting companies in th eUnited States, Soviet scientific research institutes do not compet efor projects in a market . Projects are assigned to them by uppe rlevel organizations in the administrative hierarchy, and are include dinto the State Plan. Nevertheless, leading scientists in institute sof the USSR Academy of Sciences have relative freedom in choosingtopics for their work . Usually the projects included in the Stat ePlan for these scientific research institutes are in keeping with theongoing activities of the institutes . As a rule, these activitie s

3 4

correspond to the research interests of leading scientists who usuall ytry to avoid direct involvement in the periodic, party sponsoredstate-wide campaigns, such as the recent campaign for the universa limplementation of management information systems or the perpetual an dhopeless campaign for increasing the cooperation of science an dindustry .

The system of research grants which is so important in the Unite dStates for the promotion of scientific research, does not exist in th eSoviet Union . In the scientific research institutes of the USS RAcademy of Sciences, decision scientists work without the pressure ofdeadlines and have no budget concerns . Thus, in no rush, they hav eexcellent opportunities for studying pure theoretical problems .

The application of methodologies developed in scientific researc hinstitutes of the USSR Academy of Sciences to real-life problems i sencouraged and awarded . Nevertheless, Soviet research in decisio nanalysis has focused on theoretical and methodological issues .Scientific research institutes do not become involved in consultin gwork on a regular basis since their main task is scientific research .Consulting work, if it occurs, does not involve any payment from th eclient to the consultant. Larichev [1979] acknowledges thatconsulting activities always emerge on the basis of the mutua linterests of the decision maker and the consultant . Specifically, thedecision maker is interested in increasing his influence and power an dthe consultant is interested in the application of decision-makin gmethodologies and in publications .

The trends in Soviet decision-analytic research is toward furthe rtheoretical and methodological studies . Topics of prior, on-going an dprospective research indicate that one can expect to see furthe rresearch in the following areas :

o theory of choic eo social choiceo mathematical modelling of decision problem so multiobjective mathematical programmin go interactive decision procedure so decision support systemso multistep decision procedure so fuzzy decision analysi so methods for the solution of ill-structured problem so expert systems .

Soviet scientists consider these as promising fields for researc h(see, for example, Dorofeyuk and Schneiderman [1985], Alexeeva andStefanyuk [1984], Tyurin [1981], Aizerman [1984], Larichev [1984]) ,and Gelovany et al . [1984]). One can therefore expect to see many ne wtheoretical and methodological works in all these areas .

The Soviets will continue to exhibit keen interest in th edecision-analytic methodologies developed in the U.S .A. Attempts toapply these as well as Soviet methodologies will go on. However, theexisting disbalance between theoretical/methodological work an dapplication-oriented work will only increase . Discussions of

3 5

applications with foreign scientists will be even more restricted, a sa result of a recently introduced law which provides severe punishmen tfor Soviets who discuss business matters with foreigners . Similarly ,papers on the application of decision-analytic techniques, which werepoorly documented in the past, can be expected to get even worse .

There have been some attempts in the USSR to design decisio nanalysis software tools that will make it possible to use interactiv eprocedures to solve practical decision-making problems . Previously ,decision analysis software generally required mainframe computers an dthe interaction of the decision analyst with the decision maker .Recent advances in the development of decision support systems in theUnited States make it practical to implement decision-analyti cmethodologies on personal computers . Since the Soviets are far behin dthe United States in both personal computers and decision suppor tsystems, they probably will make a strong effort to bridge this gap .Thus, they will both try to develop their own decision support system sand acquire American ones . But progress in this area is contingen tupon progress in hardware and software development . Even havingWestern computers or their copies, Soviets are many years behind i ntheir methods of communicating with computers . Hence, much emphasi sprobably will be placed on adopting American decision support systems .

In the years ahead, the Soviets will continue to try to modernizetheir economy, to move from being a not very efficient industriall ybased society to becoming an information-based society . Early hopesare already gone, however, that the Communist system can be improve dby cybernetics and computers . It is becoming increasingly clear tha twidespread successful use of computers for economic planning an dmanagement clashes with two fundamental principles of Soviet society :state secrecy about vital data and government control ove rinformation . Therefore, the prospects of wide use of computers fo rmaking decisions in industry, business, and management or for makin genvironmental decisions are bleak . Non-classified consultin gorganizations using decision analysis methodologies will not becreated in the Soviet Union because the authorities fear losin gfinancial control over consulting activities . In addition, even i fsuch consulting organizations were created, they could no tsuccessfully work without access to all kinds of information andcannot produce high quality projects without the stimulus o fcompetition. Both of these preconditions are absent in the USSR .Nevertheless, attempts to solve isolated real-life problems can b eexpected to go on . In many cases, the analyses will be performed witha view of supporting conclusions already reached by the clients. Theonly visible "bright spot" is likely to be in the evaluation of R & Dprojects and in engineering applications, where there is a real nee dfor decision-analytic methods and where ideological restrictions ca nreadily be disregarded .

3 6

NOTES

1 For original Soviet works which have teen translated into English ,references are provided to the English translations, where possible .For the remaining Soviet works, references are given to origina lpublications .

2See, for example, Tikhomirov [1972] and references therein .

3See, for example, Abchuk et al . [1979], p . 3 .

4This definition of cybernetics was published in 1954 . In 1984 ,Nikita Moisseev, Deputy Director of the Computer Center of the USS RAcademy of Sciences and a Corresponding Member of the USSR Academy o fSciences, claimed that cybernetics was invented by a Polishphilosopher B . Trentovsky in 1843 who predated and developed i tindependently from the cybernetics of Norbert Wiener . Moisseev alsowrote that he was surprised not by the fact that some Sovie tscientists initially rejected Wiener's cybernetics but rather by thefact that they eventually accepted it . See Moisseev [1984], pp. 14-24 .

5A manager or party official aspiring to an advanced degree woul darrange for a researcher to apply decision-analytic methods to aparticular real world problem falling under the official's authority .Any studies or publications resulting from this research would b ecoauthored both by the decision maker and the researcher, therebypromoting the decision maker's academic pursuits . Obtaining anadvanced degree would offer tangible long-term benefits to a decisio nmaker. For example, if dismissed or retired, an official with a nadvanced degree could usually obtain an executive position at aresearch institute or a department chair or professorship at a ninstitution of higher education . Often it would even be possible t ocombine several such positions .

6 In 1976, this department was transferred to the Institute fo rSystems Studies created shortly before, and the head of thedepartment, Boris Milner, became the Deputy Director of the Institute .

7 It is interesting to observe the course of Soviet reactions to th edecision-analytic methodology . Initially, the Soviets declared thatdecision analysis and systems analysis corresponded "to the clas sgoals of capitalists" (see Evenko [1977]), hence contradicted Sovie tideology and practice. Later they concluded that decision analysi sand systems analysis might be useful to a certain degree in someapplications (see, for example, Koryakin [1983]) . And finally they

3 7

concluded that these methodologies had been used by Soviets since th eBolshevik Revolution in 1917 (see Moisseev [1984]) .

8 Milner, B.Z. Personal communication, 1977 .

9The material in this subsection is partially based on (1) theauthor's participation in the preparation of documents justifying th ecreation of IIASA on behalf of the Science Department of the Centra lCommittee of the Communist Party of the USSR and the USSR Committe efor Science and Technology, (2) numerous discussions with both Sovie tand American scientists who have worked at IIASA, and (3) a visit t oIIASA by the author in 1973 as a Soviet representative and hi sthorough acquaintance with procedures regarding travel to IIASA b ySoviet scientists and requirements for subsequent reports by th escientists .

10Another case of the Soviet espionage is reported by Weiss [1983] .Weiss caught a Soviet spy who obtained a secret volume from an Arm yOperations Research Symposium and a very limited distribution set ofvolumes on modeling of the U.S .A. strategic defense system . Weis s[1983] indicates that the Soviets are placing great emphasis on th eoperations research/management science field . They are very muc hinterested in obtaining OR/MS materials and thoroughly cover th emeetings of ORSA/TIMS and MORS, Army Operations Research Symposia ,and similar events .

3 8

REFERENCES

Abchuk, V .A., F. A . Matveichuk, and L. A. Tomashevsky, Handbook ofOperations Research, Voyenizdat, Moscow, 1979 (in Russian) .

Agievsky, A.D., A.B . Petrovsky, and G .V . Raushenbakh, "Analysis ofExpert Preferences in Formulating Research Directions in Medica lSciences," in Methods for the Study of Complex Systems .Proceedings of the Conference of Young Scientists, Yu. S. Popkov ,(Ed.), Institute for Systems Studies, Moscow, 1983, 88-92 (i nRussian) .

Aizerman, M .A., Some New Issues in the General Theory of Choice ( ASurvey of One Line of Research ) " , Automation and Remote Control ,45, 9, Part 1 (1984), 1103-1135 .

Aizerman, M .A., N .V. Zavalishin, and E . S . Pyatnitskii, "Globa lFunctions of Sets in the Theory of Choice of Alternatives . PartI.", Automation and Remote Control, 38, 3, Part 2 (1977a), 393 -406 .

Aizerman, M .A., N .V . Zavalishin, and E . S . Pyatnitskii, "Globa lFunctions of Sets in the Theory of Choice of Alternatives. PartII.", Automation and Remote Control, 38, 5, Part 2 (1977b), 710 -721 .

Aizerman, M.A. and A.V . Malishevsky, "General Theory of Best Variant sChoice: Some Aspect s" , in IEEE Transactions on Automatic Control ,AC-26, 5 (1981), 1030-1040 .

Aizerman, M.A. and F.T. Aleskerov, 'The Arrow Paradox in Group ChoiceTheory (Analysis of the Problem)", Automation and Remote Control ,44, 9, Part 2 ( 1983), 1211-1232 .

Aizerman, M .A. and F .T . Aleskerov, " Functional Local Operators i nVoting Theory . I", Automation and Remote Control, 45, 5, Part 2(1984a), 613-620 .

Aizerman, M .A . and F.T . Aleskerov, "Functional Operators in Votin gTheory, II", Automation and Remote Control, 45, 6, Part 2 (1984b) ,775-682 .

Alexeeva, Ye. F. and V .L . Stefanyuk, "Expert Systems : State of theArt, and Prospects", Engineering Cybernetics, 5 (1984), 153-167 .

3 9

Beresovsky, B. A., V. I . Borzenko, and L. M. Kempner, Binary Relationsin Multicriteria Optimization, NauKa, Moscow, 1981 (in Russian) .

Borisov, A .N., (Ed.), Decision-Making Models Based on Linguisti cVariables, Riga, Zinatne, 1982 (in Russian) .

Borisov, A .N. and O.A. Krumberg, " Interactive Methods for the Analysi sof Alternatives in I11-Structured Decision-Making Problems , " i nMulticriteria Models and Methods for Formulating and ChoosingPreferred System Variants" . Transactions of the Institute forSystem Studies, 1, O.A. Kossov, (Ed .), Institute for Syste mStudies, Moscow, 1981, 104-114 (In Russian) .

Borisov, A. N ., Y. Y. Luns, and V . A. Popov, " Fuzzy Decision Analysi sin the Tasks of Electric Network Development " , in Fuzz yInformation, Knowledge Representation and Decision Analysis, E.Sanchez and M . M . Gupta, (Eds .), Pergamon Press, Oxford, 1983 ,287-292 .

Borisov, A. N. and G. V. Merkuryeva, "Methods of Utility Evaluation i nDecision-Making Problems under Fuzziness and Randomness", in FuzzyInformation, Knowledge Representation and Decision Analysis, E.Sanchez and M . M . Gupta, (Eds .), Pergamon Press, Oxford, 1983 ,305-310 .

Borisov, V.I., "Vector Optimization Problems", in Operations Research :Methodological Aspects", A.A. Lyapunov, (Ed.), Nauka, Moscow, 197 2(in Russian) .

Burchakov, A.S ., M .G. Gaft, L.A . Kaforin, O.I . Larichev, V .M . Ozernoy ,and V.A. Kharchenko, A Methodology for Selecting a Set o fPreferred Alternatives of Coal Mine Layouts . Lecture Notes for aCourse Entitled " Basics of Coal Mine Design " , Moscow MiningInstitute, Moscow, 1972 (in Russian) .

Buyanov, B .B ., "Developing Procedures for Eliciting Preferenc eInformation in Multiobjective Problems", in Multiobjectiv eDecision Problems, J .M . Gvishiani and S .V. Emelyanov, (Eds .) ,Mashinostoyeniye, Moscow, 1978, 96-105 (in Russian) .

Buyanov, B .B. and V .M . Ozernoy, "A Decision Method Using Vecto rCriterion", Engineering Cybernetics, 3 (1974), 49-54 .

Buyanov, B. B. and V.M. Ozernoy, "Sufficient Preference Conditions i nMultiple Criteria Decision-Making Problems", in Issues inCybernetics : Decision-Making Theory, 8, S. I . Samoylenko, (Ed .) ,Sovietskoye Radio, Moscow, 1975, 60-70 (in Russian) .

Carlsson, C. and J. Kochetkov, (Eds .), Theory and Practice of Multipl eCriteria Decision Making, North-Holland Publishing Company ,Amsterdam, 1983 .

Chetyrkin, Ye . M ., (Ed .), Statistical Methods for EstimatingQualitative Characteristics, Statistika, Moscow, 1972 (Collectio n

4 0

Darofeyuk, A.A. and M.V. Shneiderman, "Second All-Union Conference o nStatistical and Discrete Analysis of Nonnumerical Information ,Expert Estimates and Discrete Optimization " , Avtomatika iTelemekhanika, 3 (1985), 171-174 (in Russian) .

Emelyanov, S . V ., V. I . Borisov, A . A. Malevich, and A . M . Cherkashin ,"Models and Methods for Optimization under Vector-Value dCriterion", in Technical Cybernetics, Vol . 5, B. N. Petrov, (Ed .) ,VINITI, Moscow, 1973, 386-448 (in Russian) .

Emelyanov, S . V ., N. E . Kostyleva, B. P. Matich, V . M. Ozernoy, and V .A. Zimokha, Multicriteria Assessment of Control Systems forTechnological Processes, Institute of Control Sciences, Moscow ,1974 (in Russian) .

Emelyanov, S. V . and E . L. Nappelbaum, " Methods for Investigation o fComplex Systems . Part 1. Logic of Rational Choice," in Technica lCybernetics, Vol . 8, Parts 1, 2, B . N. Petrov and Z . B . Golembo ,(Eds.), VINITI, Moscow, 1977, 5-101 (in Russian) .

Emelyanov, S.V., and V.M . Ozernoy, "Decision Making in Multiobjectiv eProblems: A Survey", Problems of Control and Information Theory ,Jubilee Number, Publishing House of the Hungarian Academy o fSciences, Budapest, 1975, 51-64 .

Emelyanov, S . V., V . M. Ozernoy, and M . G. Gaft, "Formulation ofDecision Rules in Multicriterial Problems " , Soviet Physics ,Doklady, American Institute of Physics, 21, 5 (1976), 249-250 .

Evenko, L.I ., (Ed.), American Capitalism and Managerial Decisions .Decision-Making Theory and Methods . Nauka, Moscow, 1977 (i nRussian) .

Fishburn, P., Utility Theory for Decision Making, Nauka, Moscow, 197 8(Russian translation) .

Gaft, M. G., Multiple Criteria Decision Making, Znaniye, Moscow, 1979(in Russian) .

Gaft, M .G. and V.M . Ozernoy, "Isolation of a Set of Noninferio rSolutions and Their Estimates in Decision-Making Problems with aVector-Valued Criterion", Automation and Remote Control, 34, 11 ,Part 2 (1974), 1787-1795 .

Gaft, M. G. and V .V . Podinovskii, "Construction of Decision Rules i nDecision-Making Problems", Automation and Remote Control, 42, 6,Part 2 (1981), 806-815 .

Gaft, M . G, V. A. Osipova, V .V . Podinovsky, and N.P. Yashina, " On theUse of the Fragmentary Approach for the Development of PreferenceRelations in Decision-Making Problems", in Models and Methods forGenerating and Multicriteria Choice of Preferred System Variants .Transactions of the Institute for Systems Studies, 1, O.A. Kossov ,

4 1

(Ed .), Institute for Systems Studies, Moscow, 1981, 97-104 (i nRussian) .

Gaft, M .G ., N .N . Milovidov, V .I . Sergeev, E .D. Gusev, A .S . Levitsky ,and A .S . Chistyakov, "Decision Making Method for Selectin gRational Car Designs", in Problems and Methods in Decision Makingin Management Systems. Proceedings of the First All-UnionConterence on Decision-Making Problems in Management Systems, S.V .Emelyanov, (Ed .), Institute for Systems Studies, Moscow, 1982, 77 -83 (in Russian) .

Gelovani, V.A., V.N . Kim, O.V . Kovrigin, A. Yu . Kosulnikov, O .I .Larichev, V.P. Maligin, I .D. Smolyaninov, and Ye . M. Furems ,"Development of an Expert System for the Diagnosis of Ischemia" ,in Procedures for Multicriteria Evaluation of Alternatives .Transactions of the Institute for Systems Studies, 9, O.I .Larichev, (Ed .), Institute for Systems Studies, Moscow, 1984, 93 -112 (in Russian) .

Germeier, Yu. B., "The Convolution of Vector Efficiency Criteria int oa Single Criterion with Uncertainty in the Convolution Parameter sin Cybernetics for the Service of Communism, 6, A. I . Berg, (Ed.) ,Energiya, Moscow, 1971a, 175-184 (in Russian) .

Germeier, Yu. B., Introduction to Operations Research Theory, Moscow ,Nauka, 1971b (in Russian) .

Glotov, V .A., V .M . Grechko and V .V . Pavelyev, "Method of MakingDecisions with Linguistic Criteria", in Multiobjective DecisionProblems, J .M . Gvishiani and S .V. Emelyanov, (Eds .) ,Mashinostroyeniye, Moscow, 1978, 138-147 (in Russian) .

Glushkov, V.M ., Introduction to Cybernetics, Ukrainian SSR Academy ofSciences, Kiev, 1964 (in Russian) .

Irikov, V .A ., "Some Problems in Using Mathematical Methods i nPractical Decision-Making Procedures", in Problems and Methods inDecision Making in Management Systems . Proceedings of the Firs tAll-Union Conference on Decision-Making Problems in ManagementSystems, S .V . Emelyanov, (Ed .), Institute for Systems Studies ,Moscow, 1982, 12-24 (in Russian) .

Irikov, V.A. and A.E. Kurilov, Decision-Making Models in Man-MachineSystems, Moscow Institute for Physics and Technology, Moscow, 197 8(in Russian) .

Keeney, R. L. and H. Raiffa, Decisions with Multiple Objectives :Preferences and Value Tradeoffs, Radio i Svyaz, Moscow, 198 1(Russian translation) .

Keeney, R. L., "Decision Analysis :

An Overview," OperationsResearch, 30, 5 (1982), 803-838 .

Keeney, R. L ., Siting Energy Facilities : Making Decisions ,Energoatomizdat, Moscow, 1983 (Russian translation) .

4 2

Khomenyuk, V .V ., Elements of the Theory of Multiobjectiv eOptimization, Moscow, Nauka, 1983 (in Russian) .

Korotkova, T.I ., "Algorithm for the Solution of a Many-Dimensiona lMultiple-Criterion Optimization Problem", Automation and Remot eControl, 44, 3, Part 2 (1983), 350-358 .

Koryakin, Yu . I., "Editor's Preface", in Keeney [1983], 4-10 .

Kurilov, A .E . and B. Ya. Larin, " Interactive Combined Procedures fo rMaking Decisions", in Problems and Methods in Decision Making inManagement Systems . Proceedings of the First All-Union Conferenceon Decision-Making Problems in Management Systems, S.V. Emelyanov ,(Ed.), Institute for Systems Studies, Moscow, 1982, 47-52 (i nRussian) .

Kuzmin, V. B ., Collective Choice with Unfuzzy and Fuzzy BinaryRelations, Nauka, Moscow, 1982 (in Russian) .

Larichev, O .I ., "A Practical Methodology of Solving Multicriterio nProblems with Subjective Criteria", in Conflicting Objectives inDecisions, D.E. Bell, R.L . Keeney, and H . Raiffa, (Eds .), JohnWiley, London, 1977, 197-208 .

Larichev, O .I ., " Methods for Multicriteria Evaluation o fAlternatives", in Multicriteria Choice in Solving I11-StructuredProblems . Transactions of the Institute for Systems Studies, 5,S .V . Emelyanov, (Ed .), Institute for Systems Studies, Moscow ,1978, 5-3U (in Russian) .

Larichev, O.I ., The Art and Science of Decision Making, Nauka, Moscow,1979 (in Russian) .

Larichev, O.I ., Process Tracing in Evaluating, Comparing and Selectin gMulticriteria Alternatives in Decision-Making Problems " , i nDescriptive Approach for the Study of Decision-Making Processeswith Multiple Criteria. Transactions of the Institute for SystemsStudies, 9, S .V . Emelyanov and O.I . Larichev, (Eds.), Institutefor Systems Studies, Moscow, 1980, 26-36 (in Russian) .

Larichev, O.I ., "Analysis of Human Decision-Making Processes Wit hMultiattribute Alternatives (Review ) " , Automation and RemoteControl, 41, 8, Part 2 (1981), 1102-1109 .

Larichev, O.I., "On Human Capabilities in Individual Decision-Makin gProblems with Multiple Criteria", in Problems and Methods inDecision Making in Management Systems . Proceedings of the firs tAll-Union Conference on Decision-Making Problems in ManagementSystems, S .V . Emelyanov, (Ed .), Institute for Systems Studies ,Moscow, 1982, 5-12, (in Russian) .

Larichev, O .I ., " Problems In Man-Computer Interaction in Decisio nSupport Systems", in Procedures for Multicriteria Evaluation ofAlternatives . Transactions of the Institute for System Studies ,

4 3

9, O.I . Larichev, (Ed .), Institute for Systems Studies, Moscow ,1984, 20-28 (in Russian) .

Larichev, O .I ., Yu . A . Zuev, and L .S . Gnedenko, " Method fo rConstructing a Classification of R & D Project s" , in Long-RangePlanning of Research and Development, S .V . Emelyanov, (Ed .) ,Nauka, Moscow, 1974, 28-57 (in Russian) .

Larichev, O.I ., V .S . Boichenko, Ye . M. Moshkovich, and L .P .Sheptalova, "Revealing Decision Make r ' s Preferences in Binar yClassification Problems with Binary Attribute Levels", i nMulticriteria Choice in Solving I11-Structured Problems .Transactions of the Institute for System Studies, 5, Institute fo rSystems Studies, Moscow, 1978, 61-77 (in Russian) .

Larichev, O.I., Yu . A. Zuev, and L .S . Gnedenko, ZAPROS Method for theSolution of I11-Structured Multicriteria Choice Problems ,Institute for Systems Studies, Moscow, 1979 .

Larichev, O .I . and Ye . Moshkovich, "On the Possibilities of Obtainin gNoncontradictory Human Evaluations of Multidimentiona lAlternatives", in Descriptive Approach for the Study of Decision-Making Processes with Multiple Criteria . Transactions of theInstitute for Systems Studies, 9, S .V. Emelyanov and O .I .Larichev, (Eds .), Institute for Systems Studies, Moscow, 1980, 58-67 (in Russian) .

Larichev, O .I . and O.A. Polyakov, "Man-Machine Procedures for SolvingMulticriteria Mathematical Programming Problems", Ekonomika iMatematicheskiye Metody, 16, 1, (1980), 129-145 (in Russian) .

Larichev, O.M ., V .M . Grechko, and Ye . M . Furems, Planning Problems i nthe Publishing Business, Institute for Systems Studies, Moscow ,1981 (in Russian) .

Larichev, O .I ., V.S . Naginskaya, and A .I . Mechitov, "An InteractiveProcedure for the Selection of Industrial Building Designs", i nProcedures for Multicriteria Evaluation of Alternatives .Transactions of the Institute for System Studies, 9, O.I .Larichev, (Ed.), Institute for System Studies, Moscow, 1984, 84-9 3(in Russian) .

Lerner, A. Ya ., Principles of Cybernetics, Nauka, Moscow, 1967 (i nRussian) .

Lezina, Z .M ., "Manipulation of Option Choice (Agenda Theory)" ,Avtomatika i Telemekhanika, 4 (1985), 5-22 (in Russian) .

Lomov, B .F ., V . Yu . Krylov, N.V . Krylova, R. D. Luce, and W .K. Estes ,(Eds.), Normative and Descriptive Models for Decision Making ,Nauka, Moscow, 1981 (in Russian) .

Lotov, A.V., Introduction to Economic-Mathematical Modeling, Nauka ,Moscow, 1984 (in Russian) .

4 4

Luce, R.D . and H . Raiffa, Games and Decisions, Nauka, Moscow, 196 1(Russian translation) .

Makarov, I .M ., V .M . Ozernoy and A .P. Yastrebov, "Selection of Desig nPrinciple of Complex Control Systems on the Basis of ExpertOpinion", Automation and Remote Control, 32, 1, Part 2 (1971a) ,112-120 .

Makarov, I .M ., V.M . Ozernoy, and A.P. Yastrebov, " Decision Making i nthe Choice of the Variant of a Complex Automatic Control System" ,Automation and Remote Control, 32, 3, Part 1 (1971b), 421-426 .

Makarov, I . M ., T. M . Vinogradskaya, A . A. Rubchinsky, and V . B .Sokolov, Theory of Choice and Decision Making, Nauka, Moscow, 198 2(in Russian) .

Mechitov, A .I ., "Using Hierarchical Systems of Criteria in Decisio nMaking", in Descriptive Approach for the Study of Decision-MakingProcesses with Multiple Criteria. Transactions of the Institutefor Systems Studies, 9, S .V. Emelyanov and O .I . Larichev, (Eds .) ,Institute for Systems Studies, Moscow, 1980, 67-74 (in Russian) .

Mirkin, B .G ., Group Choice, Winston, New York, 1979 (Englis htranslation) .

Moisseev, N.N . Mathematics Conducts an Experiment, Nauka, Moscow, 197 9(in Russian) .

Moisseev, N .N ., People and Cybernetics, Molodaya Gvardiya, Moscow ,1984 (in Russian) .

Moshkovich, Ye . M ., " Linear and Nonlinear Descriptive Models : ASurvey", in Multicriteria Choice in Solving Ill-StructuredProblems . Transactions of the Institute for Systems Studies, Vol .5, S .V . Emelyanov, (Ed .), Institute for Systems Studies, Moscow ,1978, 52-b1 (in Russian) .

Naumov, G. Ye ., "Existence of an Optimal Choice on a Compact Set o fAlternatives", Avtomatika i Telemekhanika, 3 (1985), 5-27 (i nRussian) .

Orlovsky, S. A., "Decision-Making with a Fuzzy Preference Relation" ,Fuzzy Sets and Systems, 1, (1978), 155-167 .

Orlovsky, S. A. Decision-Making Problems with Fuzzy Information ,Nauka, Moscow, 1981 (in Russian) .

Ozernoy, V.M. "Principles for Constructing and Using Multiple Criteri aDecision Making Models " , in Decision-Making Problems .Transactions of the Institute of Control Sciences, 5, S. V.Emelyanov, (Ed .), Institute of Control Sciences, Moscow, 1974, 3 -15 (in Russian) .

Ozernoy, V . M ., " Problems and Methods of Decision Making under aVector-Valued Criterion", in Issues in Cybernetics, Decision -

4 5

Making Theory, 8, S. I . Samoylenko, (Ed .), Sovietskoye Radio ,Moscow, 1975, 53-60, (in Russian) .

Ozernoy, V. M., "Using Preference Information in Multistep Methods forSolving Multiple Criteria Decision Problems", in Multiple CriteriaDecision Making. Theory and Applications, G. Fandel and T . Gal ,(Eds.), Springer-Verlag, Berlin, 1980, 314-328 .

Ozernoy, V .M . and M .G. Gaft, " Developing Decision Rules in Multipl eCriteria Decision-Making Problems", in Decision Making Problems .Transactions of the Institute of Control Sciences, 5, S .V.Emelyanov, (Ed .), Institute of Control Sciences, Moscow, 1974, 30-44 (in Russian) .

Ozernoy, V. M . and M . G. Gaft, "Multicriterion Decision Problems", i nConflicting Objectives in Decisions, D. E. Bell, R. L. Keeney, andH . Raiffa, (Eds .), John Wiley, London, 1977, 17-39 .

Ozernoy, V . M . and M . G. Gaft, " Methodology for Solution of Discret eMultiobjective Decision Problems" , in Multiobjective DecisionProblems, J . M. Gvishiani and S . V. Emelyanov, (Eds .) ,Mashinostroyeniye, Moscow, 1978, 14-47 (in Russian) .

Podinovskii, V .V., " Multicriterion Problems With Uniform Equivalen tCriteria " , USSR Computational Mathematics and Mathematica lPhysics, 15, 2 (1975), 47-60 .

Podinovskii, V .V., " Multicriteria Problems With Importance-Ordere dCriteria", Automation and Remote Control, 37, 11, Part 2 (1976) ,1728-173b .

Podinovskii, V.V., " Importance Coefficients of Criteria in Decision -Making Problems . Serial, or Ordinal, Importance Coefficients" ,Automation and Remote Control, 39, 10, Part 2 (1978a), 1514-1524 .

Podinovskii, V .V., "Relative Importance of Criteria in Multiobjectiv eDecision Problems " , in Multiobjective Decision Problems, J. M.Gvishiani and S .V. Emelyanov, (Eds .), Mashinostroyeniye, Moscow ,1978b, 48-92 (in Russian) .

Podinovskii, V. V., "Axiomatic Solution for Evaluating Criteri aImportance in Multicriteria Problem s" , in N. N . Moisseev, (Ed .) ,Operations Research Theory : State of the Art, Nauka, Moscow, 197 9(in Russian) .

Podinovskii, V . V . and V . M . Gavrilov, Optimization by Sequentiall yApplying Criteria, Sovietskoye Radio, Moscow, 1975 (in Russian) .

Podinovskii, V . V . and V . D. Nogin, Pareto-Optimal Solutions ofMulticriteria Problems, Nauka, Moscow, 1982 (in Russian) .

Polterovich, V.M ., "Balanced States in Vector Optimization Problem" ,Automation and Remote Control, 45, 5, Part 2 (1984), 621-628 .

4 6

Raiffa, H ., Decision Analysis, Nauka, Moscow, 1977 (Russia ntranslation) .

Rastrigin, L.A. and Ya. Yu. Eiduk, "Adaptive Methods of Multicriteria lOptimization", Automation and Remote Control, 46, 1, Part 1(1985), 1-21 .

Rosenthal, M. and P. Yudin, (Eds.), "Cybernetics " , in ConcisePhilosophical Dictionary, Gospolitizdat, Moscow, 1954, 236-237 (inRussian) .

Sadovskaya, V. A., I . A. Berezansky, V . M . Ozernoy, O. I . Larichev, M .G. Gaft, V . M . Chernyagin, A . S . Burchakov, V . A. Kharchenko, an dL . A . Kaforin, Some Recommendations for Preparation an dComparative Worth Assessment of Alternatives in Production QualityControl Problems, National Research Institute for Standardization ,Moscow, 1973 (in Russian) .

Salukvadze, M. Ye ., "Linear Programming Problem With a Vector-Value dPerformance Criterion", Automation and Remote Control, 33, 5, Part1 (1972), 794-799 .

Salukvadze, M . Ye ., Methods of Vector Optimization, Metsniereba ,Tbilisi, 1976 (in Russian) .

Sedych, A., (Ed), "Caught Red-Handed", Novoye Russkoye Slovo, (Apri l17, 1981), 1 (in Russian) .

Shakhnov, I . F ., (Ed .), Statistical Models and Multiple CriteriaDecision-Making Problems, Statistika, Moscow, 1979 (Collection o fRussian translations) .

Shakhnov, I . F . and G. S . Pospelov, (Eds .), Some Issues in DecisionAnalysis and Decision-Making Procedures, Mir, Moscow, 197 6(Collection of Russian translations) .

Shapot, D.V., "Approximation of a Partial Quasiorder Using an Additiv eFunction" , in Multiobjective Decision Problems, J.M. Gvishiani an dS .V. Emelyanov, (Eds.), Mashinostroyeniye, Moscow, 1978, 106-126(in Russian) .

Sheptalova, L.P., A.D. Nikiforov, and C .B. Rebrik, "Experimenta lStudies of the Stability of a Decision Maker's Preferences i nDecision-Making Problems", in Procedures for Multicriteri aEvaluation of Alternatives . Transactions of the Institute forSystems Studies, 9, O.I. Larichev, (Ed .), Institute for SystemsStudies, Moscow, 1984, 69-80 (in Russian) .

Shestakov, O.A., "Methods for Revealing Continuous Individua lPreferences", in Multiobjective Decision Problems, J.M . Gvishian iand S.V. Emelyanov, (Eds .), Mashinostroyeniye, Moscow, 1978, 83-9 5(in Russian) .

Shestakov, O.A., "SIMS: An Interactive Multicriteria Search Syste m" ,in C . Carlson and Y . Kochetkov, (Eds .), Theory and Practice o f

4 7

Multiple Criteria Decision Making, North Holland Publishin gCompany, Amsterdam, 1983, 107-118 .

Sobol, I .M . and R.B Statnikov, Selecting Optimal Parameters inMulticriteria Problems, Nauka, 1981 (in Russian) .

Statnikov, R. B., "Solution of Multicriteria Machine Design Problemsby Parameter Space Investigation", in Multiobjective Decisio nProblems, J . M. Gvishiani and S . V . Emelyanov, (Eds .) ,Mashinostroyeniye, Moscow, 1978, 148-155 (in Russian) .

Taksar, M ., "Soviet Visitors to the U .S .", in Campus Report, StanfordUniversity, Stanford, March 16, 1983, 16-18 .

Tikhomirov, Yu . A., Managerial Decision, Nauka, Moscow, 1972 (inRussian) .

Trukhaev, R. I., Methods for Analyzing Decision-Making Processes underUncertainty, VMOLUA, Leningrad, 1972 (in Russian) .

Trukhaev, R.I., Models for Decision Making under Uncertainty, Moscow,Nauka, 1981 .

Tyurin, Yu . N., (Ed .), First All-Union Conference on Statistical andDiscrete Analysis of Nonnumerical Information, Expert Estimatesand Discrete Optimization . Abstracts, VINITI, Moscow-Alma-Ata ,1981 (in Russian) .

Vendelin, A. G., Process of Making Decisions, Valgus, Tallin, 1973 (i nRussian) .

Vendelin, A. G., Managerial Decision Making . Methodological Aspects ,Economika, Moscow, 1977 (in Russian) .

Venttsel, Ye . S ., "Presentation at the Symposium on "Operation sResearch and Analysis of Progress in Science", in OperationsResearch . Methodological Aspects" , A.A. Lyapunov, (Ed .), Nauka ,Moscow, 1972 (in Russian) .

Vinogradskaya, T.M ., "Using Properties of Partially Ordered Sets i nMultiple Criteria Decision Problems", in Decision-Making Problems .Transactions of the Institute of Control Sciences, 5, S .V .Emelyanov, (Ed .), Institute of Control Sciences, Moscow, 1974, 56 -60 (in Russian) .

Vinogradskaya, T .M ., "Two Algorithms for Selecting a Multidimentiona lAlternative", Automation and Remote Control, 38, 3 Part 1 (1977) ,376-380 .

Vinogradskaya, T .M., I.M . Makarov, A.A. Rubchinskii, V .B. Sokolov, andA.V . Shcherbakov, "Multicriterion Optimal Control", Automation andRemote Control, 45, 2, Part 1 (1984), 163-168 .

von Neumann, Y., and O. Morgenstern, Theory of Games and Economi cBehavior, Nauka, Moscow, 1970 (Russian translation) .

4 8

Weiss, A . B ., (Ed .), "OR/MS Today Editor Catches Soviet Sp y" , OR/MSToday, 10, 5, (1983), 2 .

Zadeh, L., The Concept of Linguistic Variable and Its Application toApproximate Reasoning, Mir, Moscow, 1976 (Russian translation) .

Zhukovin, V . Ye ., Models and Procedures for Decision Making ,Metsniereba, Tbilisi, 1981 (in Russian) .

Zhukovin, V. Ye ., Multicriteria Models for Decision Making underUncertainty, Metsniereba, Tbilisi, 1983 (in Russian) ,

Zionts, S ., Moscow Meeting on Interactive Decision Analysis, in Facet :The Newsletter of the Special Interest Group on Multiple CriteriaDecision Making, S. Zionts, (Ed .), State University of New York ,Buffalo, June 1984 .

Zuev, Yu .A., O.I . Larichev, V .H. Filippov, Yu. V. Chuev, " Problems i nthe Evaluation of Basic Research Proposals", in Vestnik Akademi iNauk SSSR, 8, 1979, 29-39 (in Russian) .

4 9