db: a lisp-type data base system

Volume 9, number 3 INFClRMATION PROCESSING LETTERS 5 October 1979

DB: A LISM’YPE DATA BASE SYSTEM

Marco FERRARI and Giovanni GLJIDA Milan Polytechnic Artificial Intelligence Project, Milan, Italy

Received April 1979

LISP, data base systems, file management systems, data-driver programming

1. Introduction

The need for systems which support maintenance of LISP-type data bases has been often stressed in the literature [lo- 121. ln fact, when using LISP in the solution of non-toy problems or in the development of large artificial intelligence programs, a LISP-type data base management system is often desirable [10,13].

The purpose of this paper is to supply an early progress report on an ongoing research in this field, in order to stimulate fruitful discussion. In particular, the paper is devoted to describe DB, an experimental system intended to supply a LISP-type data base management facility for the 1108 LISP [9,‘15! a The system developed at the Milan Polytechnic Artificial Intelligence Project and presently running on a ‘UNIVAC 1108 computer, has not to be considered as an industrial product but has been conceived as an experimental project centered around the following main goals: - definitic\:r of a valid set of requirements for a sys-

tem supporting the maintenance of a LISP-type data base,

- development of a running system for 1108 LISP satisfying the above defined specifications,

_ eva!uation on an actual problem of the performance of the system rru_r;lemented. Let us illustrate, now, the main design specifica-

tions assigned to our project. The DB system has been primarily oriented toward the management of LISP objects (atoms together with their values and property lists [ 1,8]) used as data. The implementation of a

126

virtual memor 4 system [6] has been purposely left outside the scope of this research. However, the extension to the files on mass storage of the data-driven programming possibility, proper of the LISP language [ 11,121, has been considered as one of the central aims of the project.

The main request imposed to the data base has been that one of preserving the great flexibility of the LISP data types. Therefore, the possibility of dynamically changing (at run time) the logical structure, the type and the dimensions of the stored data (records) has been requested. Moreover, the fully transparency for the user of creation, access, and integrity mechan- isms implemented has been considered as a desirable characteristic. The interaction between the user and the data base has therefore been limited to a very simple set of functions necessary, and hopefully suffl- cient, for an easy management of the data stored on mass storage (creation, open-close, inquiry, updtting, utilities).

At last, the system has been requested to reflect the same performance efficiency of the 1108 LISP system, in order to ensure the possibility of develop ing actual applications with it.

In the following sections we shall illustrate in detail all the relevant aspects of the project developed. More precisely, Section 2 is devoted to illustrate the user’s view of the DB system. The general arch& tecture of the system and the design criteria adopted are presented in Section 3. In Section 4 an experimental project developed with DB is reported and critically evaluated. Section 5 is devoted to a general discussion on the experience done with DB and to the illustration of some promising directions for future research.

Volume 9, number 3 INFORMATION PROCESSING LETTERS 5 October 1979

2. The user’s view of the DB system

The DB system has been conceived as a utility

tc.) which cm be loaded, ith the fund~ent~ LISP

main features of the system we will consider it as splitted in two logical levels: - the definition of a virtual machine, and . its implementation.

I vel is just the user interface of the

s)pst d one, which is fully transparent for the user, constitutes the actual software implementation chosen for the system.

This section is devoted to describe the first logical levelof DB, whereas the illustration of the second one will be postponed to Section 3.

The description of the user’s view of the DB system involves three distinct objects. Namely, the (logical) description of the data base, the user’s view of the core memory, and the definition of the available functions for the management of the data base.

The data base is a collection of files, assigned by means of ASG EXEC-O statements [ 141. A fire is an ordered sequence of records partitioned in b!ocks. The block is the physical entity which is us& in load-store operations between core memory and mass storage whenever an information item contained in a file is requested.

The initial (approximate) dimensions of the blocks of a file can be assigned by the user at creation time. The dimensions of each block in a file are allowed to dynamically change (at run time) in an automatic way, as a consequence of the updating activity on the content of the stored records. Each record has the logical structure of a property-list [ 1 ,g] , as illustrated in Fig. 1. The key atomic symbol is an atomic symbol,

local to the particular file in which it appears, used as master key in ordering the records in the file. The attributes are atomic symbols and the values associated to them can be general S-expressions (numbers, strings, lists, functions, atoms). If a value is an atom, then the whole structure of the atom itself (print-name, value, property-list [ 1,8]) is also part of the record. In this way complex-structure records are allowed, such as that one of Fig. 2, where VALUE 1, VALUE 11, ar!d VALUE 12 are atoms with non-empty property lists. Note that the use of complex-structure records is not the only way, and often also the most convenient one, for representing informationq having a very rich logical structure. For example, using lists as values allows to b

represent complex logical structures in a very simple and flexible way.

Each record may dynamically change (at run time) in content and dimensions (i.e., attribute-value pairs can be added or deleted, values of existing attributes can be changed, complex structures can be created or deleted).

The core memory is viewed by the user as a static area (assigned in standard dimensions at system loading or defined by the user by means of a GRAW LISP statement [ 151) in which a buffer can be dynamically allocated, in an automatic way, for storing the blocks which are loaded from the data base for inquiring and updating activities. The maximum number of blocks which can be stored in the buffer can be controlled by the user at fde opening.

Let us now illustrate the main functions available to the user for the management of the data base. We divide this set of functions into five classes: creation, open-close, inquiry, updating, utilities. Let us examine each class in detail.

2.1. Creation

The function for creating a file (previously assigned by means of an ASG EXEC-8 statement [ 14]),

Fig. 1. Property-list structure of a record.

127

5 October 1979 Volume 9, num’ber 3 INFORMATION PROCESSING LETTERS

Fig. 2. A complex structure record.

i.e. for organizing ii file of sequential information intO

the logical structure described in the previous para- graphs, is:

(*CREATE ‘Z N ‘*W)

KEY1 ATT11 VALll ATT12 VAL12 . . . * KEY2 ATT21 VAL21 ATT22 VAL22 . . . * . .

*‘END

where 2 is the name of the file, N the approximate dimension (expressed in words [ 141) of the blocks of the file Z and *W a control string, assigned to the file Z, for security and integrity reasons.

KEY 1 ATTRl1 ?AL! 1 ATTR12 VAL12 . . . * KEY2 ATTR21 VAL211 ATTR22 VAL22 . . . *

are the records (alphabetically ordered) which will constitute the content of the file. This information can be convenient\) prepared on a distinct sequential data-file which is then recalled by means of an ADD EXEC-8 statement [ 141.

This function is devoted to road the sequential information supplied, to divide ,t in blocks of appropriate she N, and to create the file on mass storage

128

(together with its index-block). Complex-structure records can be easily created by specifying, in the input sequential file. the property lists possibly bound to atoms used as values. This is obtained by means of a simple parenthetical description. For example, the complex record of Fig.2 is created by:

KEY ATTRl VALl (ATTRll VALll (ATTRlll VALlll

ATTRl12 VALl12) ATTR12 VAL12

(ATTRl21 VALl21 ATTRl22 VAL122)

ATTR13 VAL13)

ATTR2 VAL2 *

If the creation procedure is correctly performed the message

FILE 2 CREATED

K BLOCKS GENERATED

is produced. The content of the core memory is left unaltered after a creation function has been executed. The value of the *C E function is NIL.


2.2. Open-close

These functions are intended to initialize and ter- minate the activity of the system before and after each processing operation on the files.

Their syntax is:

- ;*OPEN ‘2 M ‘*W)

where Z and *W are as in the above function, and M denotes the maximum number of blocks of the file Z which can be resident at the same time in the core memory.

If the open operation is correctly performed the message

FILE Z OPENED

is produced.

- (*CLOSE ‘Z)

If the close operation is correctly performed the message

FILE Z CLOSED

is produced. The value of *OPEN and *CLOSE is NIL.

2.3. Inquiry

Two main functions are available for the inquiry activity:

- (*GET ‘X ‘Y ‘Z)

where X is a key atomic symbol, Y an attribute appearing in the record which has X as key atomic symbol and Z the name of the file in which the record having X has key atomic symbol is stored.

This function provides to load in the core memory the block which contains the record having X as key atomic symbol. Its value is the value as:>ociated to the attribute Y in the record of the file Z having X as key atomic symbol.

A slightly different function is

- (*GETLIST ‘X LY ‘Zj

where X and Z are as in the above function and LY is a list of attributes appearing in the record which has X as key atomic symbol.

is function returns as v e the list of the v

associated to the attributes LY in the record of the file Z having X as key atomic symbol.

2.4. Updating

The updating activity can be performed by means of the following basic functions:

- (*PUT ‘x ‘Y v ‘Z)

where X, Y and Z are as in the *GET function and V is a value.

This funct ran provides to load in the core memory the block which contains the record having X as key atomic symbol.

Then, it adds the attribute-value pair ‘Y-V to the record X or, if the attribute ‘Y already exists in X, it updates the value bound to ‘Y. It returns as value NIL.

An extension of the *PUT function, similar to *GETLIST for the function *GET, is

- (*PUTLIST ‘X LY LV ‘Z). - (*ADD ‘X ‘Z LY)

where X and Z are as above, LY is a list of the attribute-value pairs.

This function provides to add to the file Z the new record having X as key atomic symbol and LY as attribute-value pairs list. Its value is NIL.

- (*REMPROP ‘X ‘Y ‘Z)

where X, Y, and Z are as above. This function eliminates the attribute Y (together.

with its value) from the record of the file Z having X as key atomic symbol. It returns as value NIL.

An extension of the REMPROP function, similar to the *GETLIST for the *GET, is

- (*REMPROPLIST ‘X LY ‘Z) - (*REMOVE ‘X ‘Z)

where X and Z are as above. This function eliminates the record of the file Z

having X as key atomic symbol. Its value is NIL.

2.5. Utilities

Two utilities are available:

- (*SEQRE ‘x ‘Z)

where X and Z are as above.

129

This function is provided to initialize the sequential reading of the file Z stareing from the record having X as key atomic symbol (?X is BEGIN, the scanning starts from the first record in the file). The value of *SEQREAD is NIL.

The actual sequential reading is obtained by the following function:

- (*NEXT ‘LY ‘Z)

where LY is a list of artributes and Z is as above. This function provides to load in core memory the

block which contains the next record in the sequential scanning of the file 2, and returns as value a list the CAR of which is the key atomic symbol of the above record and the CDR of which is the list of values bound to the attributes of LY.

- (*COPY LY ‘Z FILE)

where LY and Z are as above and FILE denotes a data-file previously assigned by means of an ASG EXEC-8 statement [ 141. The list LY must contain the names of all the attributes present in the records of the file Z.

This function performs a sequential scanning of the file Z and copies its content on the data-file [14] FILE. The format of this data-file is such that it can be used as input sequential file for the function *CREATE.

For all the afore illustrated functions, a diagnostic is provided in the case of failure.

III_ creation

4 inquiry

5 October 1979 Volume 9, number 3 IWORMATION PROCESSING I&I-TERS

3. Architecturs and implementation

This section is devoted to illustrate the general architecture of the DB system and the main technical decision underlying its implementation.

In order to outline the main structural characteristics of the system, we are going to examine in detail

Fig. 3 shows a general view of the system. DB has been designed around a modular structure organized

each one of the following topics: background deci-

in three hierarchical levels. The first level represents

sions, structure of the data base, access, security, core

the user interface of the system, the second level is concerned with the management of the core memory,

memory management.

and the third one implements the access and security procedures. The second and third level are fully transparent for the user and can just be considered as the implementation of the virtual machine defined at the first level. This architecture has been chosen as the most adequate and simple one in connection with the basic tM.nical decisions about the development of the project.

The basic decision taken about the implementation of the system concerns the choice of the language. We have decided to adopt high-level tools such as LlSP [9,15 ] and EXEC-8 control language [ 141. This choice is primarily due to the possibility supplied by these languages of following the classical structured and modular programming methodologies [ 161.

Fig. 3. General architecture of the DB system.

130


Moreover, a skill use of LISP (PROG feature and compilation [ 1,5,9]) allows to maintain the loss of efficiency due to the high-level implementation within acceptable limits.

Let us now go further into the description of the implementation of the DB system. As we have seen in the preceding section, the dimensions and the type of the records (and, hence, of the blocks) in a file are allowed to dynamically change, at run time, as a consequence of the user’s updating activity.

This possibility is obtained by implementing each block as an EXEC-8 program-file element [ 141. The program-file consists of a collection of elements the content of which is often, but not necessarily, con- stituted of programs. The information necessary to reach each element within the program-file is maintained in a table, which is examined before any access operation. Each element in the program-file can be updated, or deleted and new elements can be added by means of the appropriate EXEC-8 functions.

From inside of LISP it is possible to create or up- date elements of a program-file by means of the function DUMP and to copy elements in core memory by the LOAD function [9,15].

The access procedures to the information contained in the files of the data base are implemented in a vrry simple way. At creation time, while the input information is read, an index-block is created in which, for each block, the key atomic symbol of the last record of the block itself is stored. Then, when an access to a record is requested, the key atomic symbol of this one is passed as an argument to the procedure *FIND which provides to determine in which block the record is stored by means of a linear search [4,5] in the index-block. The index-block is loaded in core memory by a call to the *OPEN procedure, that must be done before any operation on a file, and remains in core memory until the *CLOSE procedure is executed.

At the call of the *OPEN procedure the user must also specify the maximum number of blocks of the file which can be resident in core memory during the processing on the file. The *GTB function, which controls blocks loading and storing, provides to load the block requested by *FIND, if it is not already in core memory, and when the number of blocks present in core memory e:ozeeds that one fried by the user, it provides to store (if it has been updated) or

to erase (by means of the LISP function ERASE 19,151) the block less recently loaded. In other words, we have adopted to manage the core memory buffer dev,)ted to load-store operations a FIFO policy [4,5,6]. In order to implement this strategy a block-table (managed by *GTB) is maintained in core memory which contains the necessary information about the blocks of each file (present m core memory,, updated, etc.) and a queue is adopted to remind the loading order of the blocks [6].

4. Some experimental results

The topic we have chosen for a first evaluation of the performance of the DB system has been supplied by the School of Veterinary Medicine, Cattle Brecd- ing Institute, of the Milan State University. The problem faced is a classical one and consists of the compu- tation of the inbreeding coefficients of cattle of a given population [2,3,7,16]. More precisely, the source information about the population is contained in a sequential file (the CATTLE file) of about 2000 records, having the logical structure illustrated in Fig. 4, and devoted each one to describe the relevant characteristics of an individual of the population. The first field of the record contains the individual’s identification and is the record key, the second and third fields contain the dam’s and sire’s identifications respectively, the fourth field is devoted to store the inbreeding coefficient and is not empty only if this last has already been computed or is known as an original datum.

The inbreeding coefficient F, of an individual x can be computed by the following Wright-Malecot formula [7,17] :

F, = c ,,,i;+ir;+ ‘(1 + F~)

Y

where the summation is extended to all the ancestors y of x such that: _ they belong both to the maternal and paternal

generalogic subtrees of x; _ the two paths which connect y with x in the

maternal and paternal generalogic subtrees of x have no common nodes; and

_ ib is the length of the path which connects y with x in the maternal genealogic subtree;


individusl's dam's sire's il inbreedrng L

identification identification identification coefficient

Fig. 4. Logical structure of the CATTLE file.

$ is the length of the path which connects y with x in the paternal genealogic subtree. The problem consists in computing the exact in-

breeding coefficient of a given individual (assigned by means of its identification) applying the above recursive formula.

This topic has been considered as a suitable one for developing a test application, since the recursive nature of the solution, which implies the processing of trees of a priori unknown depths, suggest LISP (with DB) as a valid programming language.

Moreover, the logical structure of the available mformations about the population and the type of processing to be performed, which requires a large number of direct accesses to mass storage, show this problem as a good one for evaluating the efficiency of the DB system,

Let us outiine how the above illustrated problem has been solved by using the DB system. A one-fde data base has been first created from the original information available about the population (the CATTLE file). A LISP program (140 lines of about 20 characters each) has then been developed implementing the Wright-Malikot formula [7,17] and based on the classical preorder traversing algorithm of a binary tree [4,5].

Structured programming techniques [ 161 have been adopted both in the des&gn and in the coding of the program. Moreover, for efficiency reasons, the PROG feature [8,9] has been used, whenever possible, throughout the whole program and the LISP code has been compiled [9,1 S 1.

The main performance results obtained in this experimental project are sketched in two diagrams.

100 150 200 250 300

Fig. 5. Creation time versus number of blocks.

132


Fig. 5 shows the creation e versus the number of blocks in which the file has been partitioned. The creation has been performed from an ordered input

2000 records. The file generated on has a siztt of about 30 K words [ 141.

for computing one 170 direct accesses to

etail the results obtained. creation of the data base

(Fig. 5) increases with the number of the blocks because of the overhead caused by the increased number of store (DUMP 5)) operations and of

ater dimensions of index-block. The growth rate of the time-blocks function is, however, less than linear since the dimensions of the blocks are decreasing with their number. The time requested for izomputing the inbreeding coefficient of an individual (Fig. 6) is nearly constant (or also slightly decreasing) within a first short range and, then, fast increasing with the number of the blocks. In fact, it is quite intuitive that the efficiency of the system increases when a great amount of information is available in

t

eeconde

core memory and, hence, the number of load-store operations (and of the blocks) is small. Nevertheless, if the number of the blocks is decreased beyond a certain limit, the thereby increased dimensions of the blocks cause the load-store time necessary for each block to increase up to equalize, (or even to over- come) the gain due to the reduced number of input- output operations. Note that the number of blocks which can be resident in core memory at the same time is constant (three in our example).

5. Discussion and perspectives

The work done in the development of the DB system has allowed to achieve a great insight into tlhis topic and, moreover, to envisage several promising research directions.

The problem of supplying a LISP system with the possibility of interacting with large data bases on mass storage can be stated in a clear and definite way only if the type and the expected use of the items to be stored in the data base are precisely known. In

blocks

133

Volume 9, number 3 INFORMATION PROCESSING LETTERS 5 October I979

fact, an exact distinction between data and programs (formally undistinguishable in LISP [ 1 $1) becomes necessary when information on mass storagf: is largely used in a dynamic way at run time. We claim that such a distinction can ensure a good efficiency still preserving the highly desirable features of data-driven programming [ lo,1 1 ] and flexibility [ 1,8] proper of the LISP language.

In the development of the DB system this require- ment, which constitutes the result of an a posteriori reflection on ,the work done, has not been always considered with the due attention. In fact, some minor inadequacies in the present version of the system (concerning, e.g., data protection and consis- tency, and blocks dimensioning) have their origin, in our opinion, just in this remote cause.

The most promising research direction disclosed by the work reported in this paper can be possibly viewed in setting up a new, fully revised, version of DB. This could be organized around the following main goals: - definition of a simple software architecture devoted

to extend the 1108 LISP system in the two directions of virtual memory (for programs) and files management (for data);

- definition of an appropriate set of basic LISP functions (written in ASSEMBLER) devoted to allow a flexible and direct interaction between LISP and the EXEC-8 operating system (file system and the core memory management);

- development of a system (written in the extended version of LISP) supporting the above mentioned extensions and maintaining the flexibility and data- driven programming features of LISP. We believe that the particular experience done

with the present implementation of DB can be of great help in the development of such a project. In this way, the DB system will probably have served his main purpose.

Acknowledgment

We are grateful to the researchers of the School of Veterinary Medicine, Cattle Breeding Institute, Milan

State University, who have provided a good topic for a first evaluation of the performance of our system.

Moreover we are indebted to our colleague M. Ceccatelli for stimulating discussions.

References

[S] D. Knuth, The Art of Computer Programmiw - Sortin and Searching (Addison-Wesley, Reading, MA, 1973).

[ 61 S.E. Madnick and J.J. Donovan, Operating Systems u (McGraw-Hill, New York, 1974).

] G. Mal&ot, Les Mathematiques de l’Hete& (Masson, Paris, 1948).

17

18

19

] J. McCarthy et al., LISP 1.5 Ptogrammer’s Manual (MIT Press, Cambridge, MA, 1962).

] E. Norman, Reference manual for the UNIVAC 1108

[ 1 ] E.C. Berkeley and D.G. Bobrow, Editors, ming Language LISP: Its Operation and app (M.I.T. Press, Cambridge, MA, 1966).

[ 21 M.J. Falco, The calculation of inbreeding coefficient% Biometrics 12 (1975).

[ 31 R. Hanset, La genetique at;, f‘noulations d’effec reduit I: Le coefficient de consanguinitb, Ann. ‘Vbt&inaire 5 (1962).

[4] D. Knuth, The Art of Computer Programming - Funda- mental Algorithms (Addison-Wesley, Reading, MA, 1968).

LISP, The University of Wisconsin Computing Center (1968).

lo] E. Sandewall, Ideas about management of LISP data bases, Proc. 4th Internat. Joint Conf. Artificial Intelli- gence, Tbilisi, USSR (1975) 585 -592.

1 l] E. Sandewall, Some observations on conceptual programming, in: E.W. Flcock and D. Michie, Eda, Machine Intelligence 8 (Wiley, New York, 1977) 223-265.

121 E. Sandewaii, Programming in an interactive environ- ment: the LISP experience, Comput. Surveys 10 41978) 35-71.

1131 E. Sandewall, A survey of artificial intaI&ence, Ppgc. IFIP W.G. 5.2 Working Conf, on Artificial In c@ and Pattern Recognition in Computa+Aided Grenoble (1978).

[ 141 UNIVAC 1100 Operating System, Programmer Refer- ence UP-4144.

[ 1 S ] UNIVAC 1108 LISP, Reference ManuaI. [ 161 N. Wirth, Systematic Programming: An Introduction

(Prentice Hall, En&wood Cliffs, NJ, 1973). [ 171 S. Wright, The method of path coefficients, Ann. Math.

Statist. 5 (1934).

134

db: a lisp-type data base system

Documents