report no. 8106 november 1981 · mecurity classification of thit 'age (whe., dota esteed) _...

REPORT NO. 8106NOVEMBER 1981

CID Donald A. NormanO'ý Diane Rsher -

WHY ALPHABETIC KEYBOARDS

ARE NOT EASY TO USE:

KEYBOARD LAYOUT DOESN'T MUCH MATTER

Q ,% CENTER FOR HUMAN INFORMATION PROCESSING

UNIVERSITY OF CALIFORNIA, SAN DIEGO LA JOLLA, CALIFORNIA 92093

The research reported here was conducted under Contract N00014- 79-C-0323, NR 157-437 with the Personnel and Training ResearchPrograms of the Office of Naval Research, and was sponsored by the Office of Naval Research and the Air Force Office of ScientificResearch. The views and conclusions contained ih this document are those of the author and should not be interpreted as necessarilyrepresenting the official policies, either expressed or implied, of the sponsoring agencies. Approved for public release; distributionunlimited. Reproduction in whole or in part is permitted /or uny purpose of the United States Government.

81 12 26 011

r

UNCLASSIFIEDMECuRITY CLASSIFICATION OF THIt 'AGE (Whe., Dota Esteed) _

REPORT DOCUMENTATION PAGE READ INSTRUCTIONSREP T DBEFORE COMPLETING FORM

I. REPORT NUMBER 12. GOVT ACCESSION No. 3. RECIPIENTr CATALOG NUMBER8106 p• •oCOVERED

I.. TITLE (and Subi•',e) S. TYPE OF REP.ORT &ERIOD COv&Eo

Why Alphabetic Keyboards Are Not Easy to Technical Report

Usez Keyboard Layout Doesn't Much Matter *. PERORMING o REPORT UMUER

7. AUTHOR(@) '. CONTRACT OR GRANT NUMDER(s)

Donald A. Norman N00014-79-C-0323Diane Fisher

S. PERFORMING ORtGANIZATION NAM•E AND ADDRESS 10. PROGRAM ELEMENT. PROJECT. TASK

Center for Human Information Processing AREAA WORK UNIT NUMBERS

University of California, San Diego NR 157-437La Jolla, California 92093

II. CONTROLLING OFFICE NAME AND ADDRESS 12. RtPORT DATEPersonnel and Training Research Programs November 15, 1981Office of Naval Research (Code 458) 13. NUMUEROFp*i PAGESArlington, Virginia 22217 18

14. MONITORING AGENCY NAME & A0O6"REts(U different hro Cnt.l•.•n• Office..) IS. SECURITY CLASS. (of thu. "poett)

UNCLASSIFIED1Sa, OECASS| FIC ATION/OOWN GRAOING

,CHkIAULF i

16. DISTRIBUTION STATEMENT (of thli Report)

Approved for public release; distribution unlimited.r -- , o

I',.: .. . •. •••

IT. DISTRIBUTION STATEMENT (.5 the abstract ontered In Bleak 20. It different boo. RsePe) -:74

IS. SUPPLEMENTARY NOTES t

I1. KEY WORDS (Countinwe on revers, olde It necessay amd Identify by black n•mber)

Alphabetic keyboards QWERTYDvorak TypewriterHuman Factors

20. ABSTR ACT (Continue on~ reverse aide if nec•seay and Ide.ntify by block neeber)

(over)

DD 'j`13 1473 EOITION OF I NOV 65 IS O6SOLZTE UNCLASSIFIEDAN 7 S/N 0102-LF-014-6601

SECURITY CL..ASSIFICATION OF THIS PAGE (Wtihn Dle Snteered)

S ".eq

JUNCLASSIFIED

SECURITY' CLASSIFICATION OF THIS PAGE Mlih•n Date Entered)

Abstract

It is well known that the Sholes or Aqwerty" keyboard is deficient indesign, hard to learn, and awkward to use.- Many improvements have beensuggested. Alphabetically organized keyboards would appear to be supe-rior for novice typists, but this has not been demonstrated in previousstudies of typing., The Dvorak keyboard has been promoted as superiorfor experts, but many studies show only a small improvement, not enoughto warrant changing.

In "these experiments we confirm the lack of virtue for alphabeti-cally organized keyboards over both a randomly organized one and overthe standard Sholes keyboard. More important, we show why. To takeadvantage of the alphabetic keyboard requires considerable mental pro-cessing on the part of the user, and this processing is neither easy todo nor does it appear to offer much savings over visual search of thekeyboard. Simple visual search of the keyboard fails to take advantageof the alphabetical arrangement, but is relatively easy to perform. Thenovice is faced with a tradeoff between mental processing and visualsearch, and this tends to make different keyboard layouts equivalent.In addition, many people know at least something about the Sholes key-board, and even this little knowledge is useful, with the result thattheir performance will usually be better on Sholes than on alphabetic

keyboards (and certainly no worse).

Comparison of different keyboard layouts by a computer simulationof expert typing shows suprisingly little effect of keyboard arrangementfor a wide class of keyboards. Some alphabetical layouts are quiteslow, but others are within 2% of the speed of the Sholes keyboard. Thefastest layout is Dvorak, but the improvement is only around 5% overSholes. So for experts, keyboard layout doesn't seem to matter much.-Tlhe conclusion is that it is not worthwhile to use alphabetic keyboardsfor novice typists, nor to change to the Dvorak layout for experts.Keyboards can probably be improved, but only though radical redesign ofthe present physical key configuration.

-- U

'oU U..

UNCLASSIFIEDSECUIRTy CLASSIVICATION OF THIS PAGEII'Wln Deat Entered)

Why Alphabetic Keyboards Are Not Easy to Use:Keyboard Layout Doesn't Much Matter

Donald A. NormanDiane Fisher

Department of Psychologyand

Program in Cognitive ScienceUniversity of California, San Diego

La Jolla, California 92093

Copyright Q1961 Donald A. Norman and Diane Fisher

Approved for public release; distribution unlimited.

These experiments were performed by the second author as a senior honorsthesis under the supervision of the first author. The studies were ini-tiated as a result of our frequent discussions with Stu Card and TomMoran of the Xerox Palo Alto Research Center. We thank them for theirvaluable suggestions. The research reported here was conducted underContract N00014-79-C-0323, NR 157-437 with the Personnel and TrainingResearch Programs of the office of Naval Research, and was sponsored bythe Office of Naval Research and the Air Force Office of ScientificResearch.

Requests for reprints should be sent to Donald A. Norman, Programin Cognitive Science C-O15; University of California, San Diego; L~a Jol-la, California, 92093, USA.

Table of Contents

Possible Advantages of Alphabetic Keyboards ................. 2How Can an Alphabetic Keyboard Help?

The alphabetic search algorithm:The visual search algorithm:

Experimental Analysis of Alphabetic Keyboards ........... 6Method

Subjects.Design.

Experimental Results and DiscussionLearning rate.Subjective preferences.Conclusion.

The Effect of Keyboard Arrangement on Experts .....

Concusin ............... .... 1

Refeencs ..... ............ ............. I1

Norman & Fisher Typewriter keyboardsNovember 10, 1981 2

It seems obvious that the traditional typewriter keyboard -- the"qwerty" or Sholes" keyboard - presents many difficulties for non-expert typists. The arrangement of the letters on the keyboard seemsarbitrary and difficult to learn. There appears to be no system to thelayout, and beginners often ask why the keys cannot be laid out inalphabetical order. IIndeed, a number of typewriter-like devices doarrange the keys in alphabetical order, including children's electronictoys (e.g., Texas Instruments' "Speak and Spell"), hand-held languagetranslators, note-taking devices, and at least one model of P co,,aer-cial badge-making machine.

Although an alphabetical arrangement might be best for novices,different considerations are relevant for expert typists. Here, onewishes to lay out the keys so as to maximize typing rate. One majorredesign of the keyboard is known as the "Dvorak keyboard." This is akeyboard arrangement based upon human factors (time and motion study)principles that emphasizes an efficient layout of the keys to minimizehand and finger motion (Dvorak, 1942; Dvorak, Merrick, Dealey, & Ford,1936). Proponents of the Dvorak keyboard have frequently demonstratedadvantages in learning time and typing speed, but to little availagainst the established dominance of the Sholes arrangement. The ques-

tion ta be examined here is, how big a difference does keyboard layoutmake for novice and expert.

Possible Adatages of Alphabtic Keboards

Alphabetical ordering of the keys makes logical sense, and earlykeyboards were indeed alphabetical. The qwerty arrangement was inventedto minimize jamming of keys, but jamming is not a major concern withmodern type bars and typing balls, electric typewriters and computerterminals. An alphabetical arrangements would make sense to inexperi-enced typists, who today must spend considerable time learning the. arbi-trary arran-e'tent of the Sholes keyboard. Perhaps the strongest argu-ment against any proposed change is simply that the Sholes keyboard isso well known and used that it would not be practical to change. Theargument is persuasive enough to have defeated attempts to implement theIDvorak keyboard. The strongest argument for alphabetically arrangedkeyboards applies to the performance of the novice typist. Bath theSholes and Dvorak keyboards seem arbitrary, and for the novice, thereappears to be no reason for the location of the letters on the keyboard.Even the arrangement of keys is peculiar, with several rows of diago-nally structured keys that are not well designed for the structure of

1.. The keys were organized by the Sholes brothers in 1873 to minimizethe jamming of typebars in their early design of a typewriter. They

placed the keys that were typed successively as far apart on the key-board as possible so that the type bars would approach each other at arelatively sharp angle, thus minimizing the chance of jamming. Ease oflearning or typing were not considered; another triumph of technologyover human factors (see Beeching, 1974).


the hands, and shift and RETURN keys located in ways that make typingawkward. Many people have experimented with other schemes (Alden,Daniels, & Kanarick, 1972; Litterick, 1981), but major change seemsunlikely, except for specialized application. But if the physicalstructure of the keyboard is taken as given, still, why not arrange thekeys into a structured format so that the novice can find them?

Is the alphabetical arrangement of keys superior for novices? Itseems obvious that the answer must be yes: how could it be otherwise?Surely, knowledge of the alphabet will aid in finding the keys on thekeyboard, thus enhancing typing speed and reducing mental workload forthe novice. However, studies of alpnabetic keyboards fail to supportthis view; novice typists type as rapidly on the Sholes keyboard as onalphabetically organized kyboards (Hirsch, 1970; Michaels, 1971; alsosee the review by Alden, Oaniels & Kanarick, 1972). These studies seemto contradict common sense. Most people believe that the alphabeticarrangement of keys should be superior, at least for ncvices. Why then

have previous studies not shown the superiority? One p.,ssible answer isthat alphabetic keyboards have been compared with the Sholes keyboards.It is difficult to f~ind people who have not had some exposure to theqwerty keyboard arrangement, and so it is possible that the results werebiased by past experience. This opinion was buttressed when one of us(DAN) watched operators of a plastic card-making machine type plasticnamecards at the opening hours of a design engineering conference (the

machine was the Farrington "Cardwriter II"). The typists had cons-Lder-able difficulty with the alphabetically arranged keys. When asked howthey liked the machines, they responded "horrible" and "I can't under-

stand it." When asked about the keyboards, they said "I can't find thekeys," and "I suppose it would be OK if you had time to learn where theywere%" It would sem that they were not much helpei by the alphabeticalarrangement, and that for those who knew anything about typing (with theSholes keyboard), this knowledge was interfering. If knowledge of theSholes keyboard interferes with the use of alphabetic ones, then thiP

means that in a study of keyboards, the proper control would be to use arandomly organized keyboard, for to the novice, random and Sholes mustboth look equally plausible, yet they could not have had any experiencewith the random one. Acccrdingly, we set out to retest the differencesamong keyboards, using a randomly structured keyboard as our control.But beforL we started the experiments, we did some brief observations ofkeyboards. The first question to ask is, how ca.i the alphabeticalarrangement help?

How Can an Alphabetic Keyboard Help?

We were surprised to discover that when we made up alphabetic key-boards and watched ourselves and colleagues type on them, the alphabeticarrangement of the keys did not appear to help. Everyone agreed thatalphabetical arrangement seemed more rational, but they did not seemable to use the alphabet in their typing. You can discover this foryourself simply by attempting to type a line of text on each of the key-boards shown in Figure I (type by pointing at the appropriate locations

with the index finger).

-•- . - - .-.- - ----


A: The Sholes (qwerty) keyboard

q w e r t y u i o pi ,~ as d f g h J k 1;

z x c v b n mspace-bar

B: The "horizontal" alphabetic keyboard

a b c d e f g h i jk 1 m n o p q r a

t u v w x y zspace-bar

C: The "diagonal" alphabetic keyboard

a d g J m p a v yb e h k n q t w z

c f 1 1 o r u x

space-bar

A: The random keyboard

c y i f m & z d nJ

q o x h b t r w 1v a u p k e s

space-bar

Figure 1. The keyboard arrangements used in these studies. A showsthe standard Sholes or qwerty keyboard. B shows the "hcrizontal" alpha-betic keyboard, C the "diagonal" alphabetic keyboard, and D the randomkeyboard.

F I


The alphabetically ordered keyboard helps only inasmuch as there isa correlation between letter position in the alphabet and key placementon the typewriter. Part B of Figure I shows a reasonably standardalphabetic layout Notice that the point where the leLters shift fromone row to another is dictated by the physical arrangement of the key-board and not by any logical structure of the alphabet. Suppose thatyou are typing the word flower. The f is located on the third row, nearthe middle of the keyboard; where should you look for the 1? Knowledgeof the alp'.abetic indicates that the 1 is after the f, and so perhapsyou should look to the right to find it. But whether or not this istrue depends upon where the linear string of the alphabetic is broken tofit it on the keyboard. In fact, on this keyboard you must look to theleft to find the 1. Knowledge of the alphabet does not correspond toknowledge of key location. Thus, a simple analysis of the alphabeticlayout indicates that maybe the alphabetical arrangement is not so use-ful after all.

Now consider the arrangement showu ii Figure 1 C. Thic "diagonal"arrangement of the letters allows position in the alphabetic to guideplacement of the keys. In the word flower, 1 occurs both later than fin the alphabet and to the right of it on the keyboard. The correlationbetween the position of the letter in the alphabet and the location ofthe key is considerably higher for the diagonally organized alphabetickeyboard than for the horizontally organized keyboard. This keyboardarrangement should therefore be better than the horizontal arrangement.However, our informal observations didn't support this outcomc- eicher.

How could the alphabet be of use, even for the diagonal keyboard?To be useful, a considerable amount of mental computation is required.Here is a basic algorit.o for alphebetic search.

The alphabetic search algorithm:

Al. Call the letter just typed the "current letter" and its key-board location the "current location";

A2. Determine the next letter to be typed from the text;

A3. Mentally determine the position in the alphabet of the letterto be typed relative to that of the current letter;

Ai,. Mentally translate the relative alphabetic position to a key-board location; call the new keyboard location the "targetlocation" (TL);

A5. Do a saccadic eye movement from the current location to TL;

A6. If the letter to be typed is at TL or in vtew, move the fingerto thE location and type the key, otherwise call the letter atTL the "current letter" and its keyboard location the "currentlocation" and repeat from step A3.

- .--. ,. - --- . . .- -.--- ~ ---. - I


Obviously, there is more to the use of an alphabetically organizedkeyboard than is obvious at first analysis. Compare now a person mighttype on a random keyboard. The algorithm is based on visual search, andi much simpler.

The visuial search algorithm:

RI. Call the letter just typed the "current letter" and its key-board location the "current location";

R2. Determine the next letter to be typed from the text;

R3. Do a saccadic eye movement from the current location to a newtarget location. (Many possible search strategies could beused to determine the new target location, and the resultingmovement could be rarndom, linear-horizontal, etc.);

R4. If the letter to be typed is at the target location (TL) or inview, move the finger to the location and type the key, other-wise call the letter at TL the "current letter" and its key-board location the "current location" and repeat from step R3.

A major difference between the two algorithms is the amount of men-tal computation; Steps A3 and A4 for the alphabetical keyboard have nocounterpart with the random keyboard. This means that there is a con-siderable amount of mental effort expended in the case of the alphabeticalgorithm, almost none in the visual search algorithm, In addtion, thealphabetical algorithm requires that typists be able to use informationabout the relative location of letters in the alphabet, but several stu-dies have showm that peope do not know the relative locaticns of allpairs of the alphabet, but must compute them by a search of their longterm memory knowledge of the alphabet structure (see Hamilton & Sanford,1978 and Lovelace & Snodgrass, 1971). So, it is now possible to under-stand why there might not be much aid given by alphabetically arrangedkeys; there is too much mental computation required, and very littlegain. Still, we weren't sure we believed either Lhis analysis or that ofthe previous experiment3, so we did a simple experiment.

Experimental Analysis of Alphabetic Kelboards

Method

Four keyboards wore studied (as shown in Figure 1). The Sholeskeyboard was a standard computer keyboard that was connected to one ofthe laboratory computers. The other three keyboards were made by takligthree of the older keyboards in the laboratory and moving the key capsto make the desired arrangement. These keyboards were connected to aset of lights (LEDs) that displayed the code of the key being pressed,but they were not otherwise operational. (These keyboards were leftoverfrom a previous generation of discrete transistor circuits and were not


easily interfaced with the voltage levels now in use in the laboratory.)All keyboards were mounted in a sloping panel of relatively standarddimensions, height, and angle, and the three non-qwerty keyboards wereidentical except for the arrangement of their keycaps.

The interkeystrolce intervals for the Sholes keyboard were recordedby the computer. Because the other keyboards could not be connected toour system, we videotaped the hand and eye positions, using a counterthat electronically put a number on each video frame. The video tapewas analyzed with a Sony stop-action recorder and video disc, allowingfor reasonably rough determination of keystroke intervals (to within1/60 second). (The times could be determined by coding the frame atwhich the LEDs went on that specified the depression of the key; the LEDcode gave an unambiguous specification of which key was pressed.)

Subjects. Twelve subjects were studied, all undergraduate studentsat UCSD who either received course credit for taking part in the experi-ment or who were paid. The subjects were screened for typing abilityand the advertisement requested that only non-typists apply. All sub-jects were given a pretest on the Sholes keyboard and any who typed over25 wpm (2.1 letters/second) were dismissed. The average typing speed of2the 12 subjects on the Sholes keyboard was 16 wpm (1.3 letters/second).

Design. The subjects were first given a pretest on the Sholeskeyboard. They were then were tested for 10 minutes on each of the otherthree keyboards in counterbalanced order, with a 5 minute rest betweentests. (With three keyboards and 12' subjects, this yielded two replica-tions of each order.) The same text was used throughout; we opted forconsistency of the letter strings in the experimental material at thecost of some learning of the materials, counting on the counterbalancingto even out the factors. The text was adapted fromn a Reader's Digestarticle on diets. 3 It was approximately 12,000 character-s l1ong and waspresented as double-spaced, typewritten copy, printed in all upper casecharacters. The design of the keyboard was explained to each subjectprior to each trial. The instructions stressed speed with accuracy.Subjects typed as much of the text as they could within the 10 minutesallowed them. After all the trials had been completed, we asked thesubjects for their subjective impressions and preferences.

Experimental Results and Discussion

2. The speed criterion (25 wpm) is is the same as used by Hirsch(1970). The mean typing rate of his "non-typists" was 0.9letters/second, compared to our value of 1.3.

3. The text war~ prepared by Donald Gentner for use in other typing stu-dies in our laboratory (Gentner, 1981).

--------- ~At

Nomn&Fisher ad pae, re Typewriter keyboardsj

The number of letters typed by each subject during each ten minute

Sesoexcluding punctuation adscerepresented in Table 1.

The keypresses were determined by viewing the videotapes and noting theLED indicators activated by the keyboards (the keyboards produced nei-

did not type very much in each 10 minute session; most subjects onlymanaged to complete 60 to 100 words with the alphabetic and random key-boards. Most errors were substitutions of words similar in sounds ormeanings to the target words. Errors also included skipped words.Thus, most typing errors were perceptual or memorial in nature, not key-board typing errors. (Typing rates are usually computed by dividing thenumber of letters typed per minute by five, the standard conversionbetween letters and words. Spaces are typed with relative ease, and donot differ across keyboards. Thus, by not counting the typing ofspaces, we amplify any possible differences among keyboards. Whenqwerty typing speeds are computed including the spaces, the mean speedincreases from the 13 wpm shown in Table I to 16 wpm.)

An analysis of variance performed on the non-qwerty keyboardsshowed a significant effect of keyboard type (F - 6.235, p < .01). A95% Sheffe confidence interval on the data (with order used as a covari-ate) showed any difference of 29 key presses to be reliable. Thus,there is a small (10%) but significant superiority of the two alphabeti-cal keyboards over the random one, but no difference between the twoalphabetic types. The times for the qwerty keyboard are included forcomparison purposes; they are clearly superior. (The qwerty keyboardswere not tested within the same experimental design as the others and socannot be included within the ANOVA. However, for 11 of the 12 sub-jects, the qwerty speeds exceed that of all the other keyboards.)

Lering rate.

It is possible that the learning of the alphabetical keyboard couldproceed more quickly than that of the random or qwerty keyboard, for thestructure of the keyboard would help in the learning of key placement.ITwo factors indicate that this is not likely to b~e the case. First, thething to be remembered is which finger types which keys, and this is nothelped much by horizontally organized alphabetic keyboards (e.g., themiddle finger of the left hand types c, m,1 and v; the alphabetic natureof the keyboard arrangement is not apparent).- Second, the srudies ofHirsch (1970) and Michaels (1971) showed no superiority in learningrates. To asses differences in learning, we compared the number ofkeypresses typed by each subject in the first half of each session withthe number typed in the second half. In general, subjects did improvein the second half. However, the improvements were small; the differ-ences in keystrokes between the second half and the first half was 18,19, and 25 for the horizontal, diagonal, and the random keyboards,respectively. Moreover, the standard deviations were relatively large:15, 21, and 18, respectively. Thus, there appears to be no differencesbetween the learning rates of the two alphabetical arrangements, and thelearning of the random keyboard is greater than that of the alphabetic,contrary to expectations. However, it is also the case that typing rate

Norman & Fisher typewriter keyboardsNovember 10, 1981 9

Table 1

Letters typed in 10 minutes(not including spaces and punctuation)

KEYBOARDSOrder of

Subject keyboardQwer ty &riz. Diag. Rand. presentation

(alphabetic)

1 1 505 292 302 362 q h d r2 1 815 395 367 346 q r h d3 295 384 340 313 q d r h4 605 411 453 323 q h r d5 405 331 354 292 q d h r6 1 740 412 416 364 q r d h7 1 705 330 330 364 q h d r8 I 630 426 432 329 q r h d9 1 870 400 349 374 q d r h10 1 740 458 524 457 1 q h r d11 1 700 351 336 306 q d h r12 770 462 474 415 q r d h

Mean: 648 388 373 357

Letters/sec. 1.08 0.65 0.62 0.60

Words/min. 13.0 7.8 7.5 7.1

A ---.- ---

INorman & Fisher Typewriter keyboardsNovember 10, 1981 10

on the random keyboard was slower, so perhaps the greater learning ratereflects the fact that the subjects have more to learn.

Subjective preferences. Subjective preferences for keyboardsjvaried. Three subjects said that they preferred the random board. Sixsaid that they preferred the alphabetic boards, and three said they didnot find much difference among the three. When asked to compare the twoforms of alphabetic keyboards, two subjects said they had no prefer-ences, five preferred the horizontal, and. two preferred the diagonal.Two subjects said that the alphabetic keyboards were frustrating, whiletwo subjects said that the random board took more concentration. One.subject commented that while he performed better on the alphabeticalkeyboards, he would have preferred the random if he had a chance to getused to it. In general, the subjective preferences do not appear to beuseful. We did not expect this result; we believed that the subjectswould prefer the alphabetically organized keyboards, even if their per-formance did not warrant it. Unfortunately, we got the prefereace rat-ing last, after subjects had used all keyboards. Perhaps their naturalpreferences were changed after they had 10 minutes of frustration isattempting to use the keyboards..

To test the influence of experience on subjective impressions, we

selected new subjects in the same manner as we had selected the first '12. This group was asked to give their subjective impressions of threekeyboards; qwerty and the two alphabetical arrangements. In addtion,their typing speed on the qwerty keyboard was measured. 4 The subjectswere shown a picture of each keyboard, without being allowed any experi-ence with them. Unfortunately, there was still no difference in prefer-ence. Even when w2 looked at the slowest (poorest) typists in thegroup, no differences emerged. Thus, the subjective ratings of the 11slowest subjects whose qwerty typing rate was measured by us to be lessthan 20 wpm on a seven point rating scale (with 7 meaning "like verymuch"), is 4.2, 4.6, and 3.2 for the qwerty, horizontal, and diagonalkeyboards, respectively. (The standard deviations are 1.4, 1.8, amd1.3, and the mean qwerty typing speed is 15.5 wpm.) The small differ-ence in prpference between the qwerty and the alphabetic keyboards isclearly not significant, and the diagonal keyboard is the least pre-ferred.

Conclusion. The appropriate assesuient of alphabetic keyboards ismade by comparing their performance with the random keyboard, thusavoiding contamination with the qwerty arrangement. Any negative influ-ence of prior knowledge of the qwerty keyboard should have equal influ-ence on both the alphabetic and the random keyboards. There is a slightadvantage of the alphabetically organized keyboards over a randomlyorganized one. This advantage is around 10%, a difference of slightlyless than one word per minute. The advantage may be statistically sig-nificant, but it is of no practical significance.

4. Phil Mercurio performed this experiment and collected the ratings.j

Norman & Fisher Typewriter keyboardsNovember 10, 198J 11

It is t.nfcrtunate that we got subjects with knowledge of the stan-dard keyboard, even though wc attempted to get people with no typing'-xperience. It is difficult to find people who have not had experiencewith typewriters, ard this is especially true in a college environment.Note that W :sch (1970) founi essentially the same amount of knowledgewith his "non-typists." Nonetheless, the lack of meaningful differencebetween Lhe alphabetic and tne random keyboards is striking. Note thatour subjects were clearly superior on the qwerty keyboards. Even though

they could only type about one letter per second, the qwerty keyboardyielded a 67% improvement over the speed reached with the alphabetickeyboards and an 83% improvement over the random keyboard.

The Effect of Keyboard Arrangement on Experts

What would happen with expert typists? How would they be affectedby different keyboard arrangements? We despaired of doing the actualexperiments because to do so would. require months of training on avariety of keyboards to get people to expert status, and then a consid-

erable amount of retraining of the subjects so that they were back tonormal on regular typewriters. The experiment has actually been donewith the Dvorak keyboard. Estimates vary, ranging from a 5 to 10%improvement in typing speed to no significant difference (Alden, et al.,1972). I

It is possible to compute the effects of keyboard arrangement ontyping speed. Kinkead (1975) devised a computational method based on thekeystroke intervals for typing keystroke patterns, the relative fre-quency of English digraphs, and the physical arrangement of the keys.He estimated only a 2.6% improvement in typing speed with the Dvoraklayout, although he thought it possible to devise other keyboardarrangements that could give a 7.6% improvement. That size improvement,he concluded, "would probably be lost in the flurry of re-training andre-building 20 million typists/machines."

Card, Moran, and Newell (1981) have extended Kinkead's analyses tocover other arrangements of keys (they have also generalized and simpli-fied the analytical methods). They find that the Sholes keyboard f'sin the middle of a range of possible speeds, with the Dvorak arrangementyielding a maximiun improvement of 11% and certain inelegant alphabeticarrangements yielding a decrement of about 10%. (CL.rd & Moran, personalcommunication).

Rumelhart and Norman (1982) have developed a computer simulation ofthe hand and finger movements of a skilled typist. With this program,it is possible to estimate the effects of any keyboard layout, includingones in which the keys are physically quite different from that of thestandard typewriter (the Kinkead and Card, Moran, & Newell techniquesare tied to the standard typewriter by the use of empirical time esti-mates based on the qwerty layout). This simulation includes the cogni-tive and motor control mechanisms, and the output is a computer graphicsdisplay of the hands and fingers moving over the keyboard, plus the

reaction time distributions of interkeypress !ntervals.

____-- L


Using this simulation model, we have tested a variety of keyboardlayouts. The typing program was run with different keyboard layouts,using the same text as we used in our experiment. The typing modelyields reaction time distributions for the interkeystroke intervals inarbitrary model units, and these were transformed into words per minuteby ascuming thac each wodel unit was 40 milliseconds. This yields areasonable typing rate for an average, secretarial level typist (about55 wpm). In computing the typing speeds, carriage returns were notcounted, but spaces and simple punctuation were used (periods and com-mas). The text was in all upper casc, and so no shirt key operationswere required. The data from the simulations are shown in Table 2.Despite the very different assumptions and mechanisms of the typingmodel from the other methods, our estimates agree with the work of Cardand Moran witkin the accuracy of the estimations; about a 5% improvementfor Dvorak, and a decrement for the horizontal alphabetic keyboards thatranges from 2% to 9%, dependent upon the particular layout that is usLd.

Different layouts of keyboards produce considerably different load-ings of the two hands, as well as different percentages of keypressesrequired off the home row. Detailed studies of the timing characteris-tics of typing have shown that between-hand letter digraphs are typedfaster than within-hand, and between-finger digraphs faster than Iwithin-finger (Gentner, 1981; Kinkead, 1975; Rumelhart & Norman, 1982).These and other factors indicate that for optimum typing speed, key-boards should be designed so that:

A: The loads on the right and left hands should be equalized;

B: The load on the home (middle) row should be maximized;

C: The frequency of alternating hand sequences should be maxim-ized and the frequency of same finger typing should be minim-ized.

The Dvorak keyboard does a good job on these variables, especially A andB; 67% of the typing is done on the home row and the left-right handbalance is 47-53%. Although the Sholes (qwerty) keyboard fails at con-ditions A and B (most typing is done on the top row and the balancebetween the two hands is 57-43%), the policy to put successively typedkeys as far apart as possible favors factor C, thus leading to rela-tively rapid typing. Thus, the Sholes keyboard actually seems to be asensible design, superior to all of the alphabetical arrangements thatwe have studied, and only 5 to 10% slower than the D)vorak keyboard, theone that was based upon time-and-motion studies.

For the expert typist, the layout of keys makes suprisingly littledifference. There seems no reason to chose Sholes, Dvorak, or alphabet-ically organized keyboards over one another on the basis of typingspeed. It is possible to make a bad keyboard layout, however, and twoof the arrangements that we studied can be ruled out. Note that one of

the slower keyboards is "alphabetic I," or "horizontal alphabetical" in

• - - i . . .. ... • ; .. .. .. . ...... . . .• .. . ....... • .... , ÷ • •" •-.i ••...... • , • • .... Ii •q ! .... -, ,• • , . . .., V


Table 2

Typing Speeds Calculated from the Rumelhart & Norman Typing Model*(Words per minute assume each model time unit is 40 msec.)

keyboard % use % left % right moiel % deviation(rows) hand hand wpm from qwerty

QWERTY:qwert yuiop 42% 57% 43% 56 -

asdfg hjkl; 27%zxcvb nm,. 15%

-space- 16%DVORAK:?,.py fgcrl 21% 47% 53% 58 +5.4%aoeui dhtns 56%"qjkx bmwv 7%

-space- 16%ALPHABETICAL-i:abcde fghij 37% 64% 36% 52 -7.1%klmno pqrs; 30%tuvwx yz,• 17%

-space- 16%

ALPHABETICAL-2:;.,ab cdefg 29% 54% 46% 55 -1.8%hijkl mnopq 29%rstuv wxyz 24%

-space- 16%

ALPHABETICAL-3:abcde pqrst 43% 66% 34% 51 -8.9%fghij uvwx; 18%klmno yz,. 23%

-space- 16%ALPHABETICAL-4:abcde fghij 37% 57% 43% 55 -1. 8%klmno pqrst 38%

,•;uv wxyz 9%-space- 16%

DIAGONAL:adgjm psvy; 23% 55% 45% 55 -1.8%behkn qtwz, 29%cfilo rux. 32%

-space- 16%

• In these computations, the space key is assumed to be typed with

the right thumb.

S. . .. .. + -.y ++ , - _.. , • i i

Norman & 7:sher Typewriter keyboard&November 1b, 1981 14

our experiment. This configuration of keys matches well what peoplethink of when considering alphabetical arrangements. It is not verygood at all.

Conclusion

In these studies we have analyzed the novice and expert use of dif-*ferent keyboards. First consider the implications of keyboard layout

for novices. We conclude that in order for novices to make use of thealphabetical organization of the keys on the keyboard, considerableworkload is involved. Our experimental studies of typing rate show thatalphabetically organized keyboards are slightly superior to a randomlyorganized one, for beginners, but that difference is too slight to be ofJany practical significance. Moreover, because even a slight knowledgeof the qwerty arrangement is apt to help considerably when using theSholes keyboard (and to interfere with the alphabetical one). Webelieve that, on the whole, our subjects did not use the alphabeticalgorithm, but instead resorted primarily to visual search of the key-board, thus treating the key arrangement as if it were unstructured:(random). As the comparison of the two different algorithms wepresented earlier indicates, this reduces the mental workload consider-ably, with little effect on performance. There is no sense in introduc-ing the alphabetical keyboard; essentially no one will be helped, yetmany will be hindered.

The Sholes keyboard may be easier to learn and to use than a randomkeyboard. The home row is in quasi-alphabetical order (see Figure I A:"...d f g h j k l...') . Hirsch describes its virtues this way:

'..although not a perfect arrangement, the key arrangementof the standard typewriter is also not a random one. Whateverits limitations, it was 'human engineered' even as early as1873, and many of the most frequently used letters are, gen-erally speaking, clustered in the center of the keyboard.Hence, hunting for a letter can usually b~e confined to a rela-tively small visual area" (Hi~rsch, 1981, p. 139).

stuie ofsidiferentheyboardlaratingmns for expert typists. depeiultindNtdisow conidferen theybimplicratingmns for expert typists Oursiultind

upon the accuracy of the simulation program. However, in our detailedexamination of the simulation model for the Sholes keyboard, it seemedto provide an accurate simulation of many of the properties of the reac-tion time distributions of skilled typists (Rumelhart & Norman, 1982).In addition, the results presented here for the comparison of differentkeyboards agree reasonably well with other methods of computation andwith experiments comparing Dvorak and Sholes keyboards. These results,therefore, seem reasonable, if only because they are consistent withother methods.

Norman 6Fisher Typewriter keyboardsNovember 10, 1981 15

Finally, our analysis of the operations a novice typist would haveto perform in order to take advantage of the alphabetical arrangement ofkeys provides a possible explanation for the lack of improvement withthe introduction of chese keyboards; the mental load required to makeuse of the keyboard is too great. (And it is not clear that most peopleknow the alphabet well enough to take full advantage of the ordering.)In his recent review of studies of alphabetical keyboards Hirsch alsosuggests that this may be the case:

" ".the alphabetical keyboard probably requires, first, a

memory search to locate the letter in its approximate or rela-tiv alhbtclpstoadte Iiulsac ofnthe key on the board (where it is situated without regard toits frequency of its use). Accordingly, the combination ofthe memory plus visual searches may be less efficient than apurely visual search where the probability is high that thevisual area first scanned will contain the sought-for lettertf

(Hirsch, 1981, p. 139).

The conclusion is that both experimental results and various formsof computational models agree that it is a waste of time to re-az rangethe typewriter layout; there are pitifully few savings to be had. HL-w-ever, this does not mean that typewriter layout cannot be improved."Kinkead (1975) pointed out that considerable savings could be made byeither moving the RETURN key or by eliminating its use altogether (withcomputer assisted text editing). Elimination of the RETURN key gives aminimum of 7% improvement in speed, and "up to 30% when the originalcopy is not properly formatted." Thus, simple changes to the way thattypists use the keyboards can have a large effect, without changing thekey layout.

There is considerable room for dramatic revision of typewriterstructure (see Litterick, 1981 for a quick review). Thus, there is lit-tle justification for the present size of the keyboard, for the stag-gered arrangement of the keys, and for other aspects of the physicalarrangements. The left and right hands should have a mirror image con-figuration of keys, rather than the continual upper-left to bottom-rightdiagonal slope of the keys now used. The left and right hand portions ofthe keyboard could be separated (allowing the manuscript that is beingcopied to be placed between the hands where it can be seen withouttwisting the head). The space bar seems wasted opportunity (mostskilled typists only use the right thiumb :.o type Jpaces, and then onlyuse a small section of the bar). Alternative. y, thle keyboards could besensitive to both upwards and downward motion, or keyboards could allow(require) typing of several overlapping keys, or simultaneous depressingof several keys (these are called "chord" keyboards), or several keys-trokes could be required per letter, thereby allowing one to reduce the

number of keys. Gopher and Eilam (1979) have developed a successful

Ir

Norman & Fisher Typewriter keyboardeNovember 10, 1981 16

one-hand keyboard for Hebrew using this technique. our lesson is simpl.ythis; do not waste time re-arranging the letter arrangement of theexisting standardized keyboard. And when investigating novel typingIschemes, take advantage of computational methods in assessing their

validity

5. Many of these ideas have been tested out in various formal and infor-mal experiments. We have not seen a proper assessment of them, however,comparing typing speeds, error rates, and difficulty of learning. Thestandard "chord keyboard" has been studied, and although it is indeedvery fast (which is why it is used by court recorders), it is not easyfor a novice. A review~ of many of these methods is found in Alden,Daniels, and iKanarick, 1972.


References

Alden, D.G., Wainels, R.W., & Kanarick, A.F. Keyboard design andoperation: A review of the major issues. Human Factors, 1972, 14,275-293.

Beeching, W. A. Century of the typewriter. New York: St. Martin'sPress, 1974.

Card, S., Moran, T., & Newell, A. Applied Information-ProcessingPsychology: The Human-Computer Interface. Hillsdale, N.J.: ErlbaumAssociates, in press (tentative date: 1981).

Dvorak, A. There is a better typewriter keyboard. National BusinessEducation Quarterly, 1943, 11, 58-66

Dvorak, A., Merrick, N.I., Dealey, W.L., & Ford, G.C. Typewritingbehavior. New York: American Book, 1936.

Hamilton, J. M. E., & Sanford, A. J. The symbolic distance effect foralphabetic order judgements: A subjective report and reaction timeanalysis. Quarterly Journal of Experimental Psychology, 1978, 30, I33-43.

Gentner, D. R. Skilled Finger Movements in Typing. Center for HumanInformation Processing Technical Report. La Jolla, CA, 1981.

Gopher, D., & Eilam, Z. Development of the letter-shape keyboard: Anew approach to the design of data entry devices. Proceedings ofthe Human Factors Society - 23rd Annual Meeting, 1979, 40-49.

Hirsch, R. S. Effects of standard versus alphabetic keyboard formatson typing performance. Journal of hrlied Psychology, 1970, 54,484-490.

IBM Systems Journal, 1981, 20, 123-171.

Kinkead, R. Typing speed, keying rates, and optimal keyboard layouts.Proceedings of the Annual Meeting of the Human Factors Society,1975, 159 - 161.

Litterick, I., QWERTYUIOP - dinosaur in a computer age. New Scien-tist, 1981 (January 8), 66 - 68.

Lovelace, E. A., & Snodgrass, R. D. Decision times for alphabetic orderof letter pairs. Journa. of Experimental Psychology, 1971, 2,258-264.

Michaels, S. E., Qwerty versus alphabetic keyboards as a function oftyping skill. Human Factors, 1971, 13, 419-426.

Rumelhart, D. E., & Norman, D. A. Simulating a skilled typist: A studyof skilled cognitive-motor performance. Cognitive Science, 1982,5, in press.

S'9 •- .-. 9.3*:•:.'<WAW WLL-.AAIAA-O" • '• i :• .-PlA :tt ASA-IA..... .. PA.' I 9t. A•,;•y. A A•• •.:.••L•• ! CS• -LI., -• :•:LI . .. .

U.... 0i1 A A It,

A SI A t I.l NA -kt ' S ....L. WA' I9 .t .... A. ....... .t'n'A IIVAN-'I A , A

....... AWAL -, INCC~.

Na. .. .. -t- .... t.tLLLAA0 Wil • llpill 1. .9•* AbA A •*l 9 fl 'Ai.kIl.9 WA*.W ViAo Il* ll AAiA CiAASPA fi IL illliliI.t

C3Ail* III1 , .l.t l At3A.IL AKAWA[LH l Poi.Al ILASL , A'L IWCAA *•l~t•*, 1 I

A, ALA Sn i.s .. IA 1l. 1L,16A1-tA L AL AAPA9 AIA .IWA.5. C1 .

A' C WAA.11 UP . 94A..~~~~~~~~~~...... PAW PII.99 . AA.ACWAAS S ~ A . .WSS

NC bAA a? LA PALAA I At- ;%A: ......A... ..IWL mi..W

Ite AA9Ael, WA I~l I'il AIAA A A.AlL~lii 9 . CAA t3ChIlli 1 59W AilltIA35.•AAlA SAL.AL AAO

99. JAA PA9Ae I t. Al4StAAS,A 'APA .l WAlLt PAIWA 3.9 9L Ctt SAAAL A LAL.1 A~lt3Oi

IA. WIAWAYC AIWAAW 3 At.~~A I.e~ ..A.. I. ..AA .AAA *91 - 9A AI A A I U IA~ . A

A..A A.SA .A.AA....AA.A At9.A

Iit AlllIliliL 194W A.W L P iAAAA6 *1iWLA.9 IIACll IlllW AACOCAP9LL I.IA AI~ allA .13W:ll l

IAACAAAAAA, PAWS.IAAIII . IWA.3 .IlSAAAO tAAAI AAll . AAiJAAAAS AA9A..SA1A

CAVA WiA .AI l AA A...A

9CSAWPAAWAAAlli ets .e9AtA.A l . 9.. ..A.9" AA1 il A AC.LA .I.. I. II.-

PI-A1AC A S AI I I. .A WII II A i A A iA L l S e P IV !I A Wl A L A A A A W A I A A

t

AAA ALA* 9¢1 III.I ASCAi .6 .AP i C9AILNI AAl IA ISA LP CA.CILA* AOAIL SLLAI AA. lSI l6.CLLCA

WO W.WC.cCA A IA Se. LAASp,

Ct. IA. t 9.11.9 WAIL C ..A ..L 1i IAALAA . At. I..'W 01IA A

W~~o.ý SAW .....A....WAA~WS. A.CA.AAAASCCC N mA 9 A.SAAA I A.. IAPALILA 3 PAA .9A WAA:AC. 9093 A . IA aC1AA I*AAAA Wi SA

1st mIIcA., WA11913 Ai OSAA cLLL A.. lL~l AllK eAW ASAW 093 eA. Ill9l PAPASALAWA

II ~~~~~~~~ ~Z1. 111:11 .,111,I€ l ll 10Io IIlll

LAW l I illil il*I A I-AAWAAA WAA~ . CAleAAAAWA0 C AASAl 9W. A 1111IWAPASLAAIAA

Alt-LAW? ~ I I.L......C...t.L.ACIAAAI A 4AWWS ACA49 W L~

A.AWIAA.A - K AWAWIlS.A..IAAAI .iOAAIiClSILA9 A. AAAl WIAAA .~lI'l•' CAdell Ci~l I A9A t. PL-

.5 AA ..I.. W:m, d ...... P.I AlL.. LANLW CSILI I .t.A WL

SAA.A AA . AAAAAAAl9AACAAAW LdbIlL P9A1A P1leIAlL.. CA IA LO

*CAiSAWhl.LW WW 0.6. Al~~llA PAAA - AL Pp SAlillIWA l AWAClSA "LALl*ICA A[ AWSAIA I4

SA9CAi,,~I~llllli L AL CSASAIli I.4 il iePCl L ACACAAC.CPA9Ct 69AA0 lllI 9" AAAAO IAAA WA ACAAA

Ab" C . 9i AlleslIASAA A9A C 51WW A~l l pll PAPA SAWAl AIPS CblIIAII AAAAAll IAA11 SALeeAW CA CAWACl

* 9i.5 ALA 9AAASA C epWA API)CAACI.Ale,.PAAWlW LASLW . AA ALCO

WIl IIAAAA , AS 1661W$ il•iilllIilll I OP.9 0. • AWWA 64LAAAA W1'1. SIWA. I AlAIkWA

CAIWA ACI- I PA. AA l I, WAA • (WAW1AAA IinPALAAPAA SIet, tiA

WA.i AIIIA A 666A9i AILCWA LiAA WCWiAAPCil PISA AlIn IL P991.1 PAAAWAIAAL

A, WASAS L Sp. a... A A ...... tII.A I.It.

IAA Ilk*i. S |tWA~d LiS l..AA le IAACAIIACAI* e A A I* i.A A. .... A0 P..i... A.I f pitt

ASSiA9 CtAAAS ICWAAAll P~let9Ael CAAA5A A AAA ctb. AC L IIkolI L9AAAAA. WSWWC1AA

ACCA9 lt W. WWWW.AAI 56 SACCI i LLA LLAiA AIAhAPAAiIW. AW *,CAAA

PeAAAPA * il •l 56CC. Cl 61663 AFOCC PSCWWAAAA9I.i 3*4llll 3 91 •Ill~lllilli•l*l

A AL &i, IAPS, VA 10113 0.1.. i9li-l. C p" #111 tA¢dt, IAAAIILA1 I P. PALAAt LAAeoLAl

AtC~ eNte W

1 A. e .e AA , 5 . . ..AI ..L*L ....... .9 A t.5

Ai, WIe.Al ,AAAIAAI&~ll ;IilllIIWAAAA.4 LAW-tI€., 0.-.A 1 *1114.95-' C'ALAiAPL 9L I Ct-

C'LAC .06i AWA111t St.i.A. lli All, LA OL II 1 i It , l* II1iACAA UIPt AtIAL .AeAAA• 3*~i 31- I~l-tASALAWA~ ~l~ li~l

WA, AA AC .sASI1

A.. S iee*. 04 CiilA ll ti iW A SI A 'tAA. PAAA9A. P Al C1IAltL, AIIA A AAlliepIA lL,,*A. llll

IiPA - 9AASAI7eA 343 l.A. .i AW SICO S.d .. IS•WAAA

AAA9LAAPIA A9 A2pLtt1e ... ...A

A...eSAA , CA 96111 9., l i , IASe ¢AAASit9A IN' AL.iAL i'eL.W9WLCL.5i1.9. II #11113 PIliSIW PA bll lT IAAAAWII WA1 3.A1 .1, il l iit

WAAWIeSAA A'L, CAL3 WLACCpC. IAi Celtll l 0A3AAItA I Ai WAA9A In, IA1AAAA ll~t

AAAA PCCSSCA 546fl WASIL LAAAIiIIAAA ll ililil i C.. IZIAWIA.lCALIAW. CAWAO ,A.il WA CIh

ilSALA SASACAL. A* W..*nt.. let - 3 99. StAi AA!ll.IA-t ALAf•ll .'I*. i • .

ISA .A .... .. CiLIA A ..... k~ AAW *'t.9.3( , 6.43Ae

AA9 SW~liA. IA AWiA lle llkl i I ~l~l i~ ~it . lAWi SO,k ll Il

0SI ASett6 CACIACI IIIWA- 069W I 0.-S..lCliAAA .•.....iiA. C.

I05PIIAASLCA.PA ClIA iA 11 1W LW..

90AL ,DIV..LAAlLLALlCl CA 9i3 .- -M

AClil llll(4 I SA. WAli AALAWI P. C• e.9i A I ,WA - Al 19. AlleA

tWA A C SLA 121, A A

m

S..t,.,CW Cl~lIC LW iAASIAI.WA.Slk eA.LlA IiiiLi•I9, e.AWA 9.l*O CAtAI PA.CALAP iA. ,,,

1.i .2.%4 1 0, A.

PtAAAALASlIAI AA.AnL•I .9

l19A.h A L 1

Ci 3 I•i II L'.)l~IAAAA ¢WAiI liil• ilIIIA •I LISAA'. LW PA.~li CW LAi.AAAAi iWS llt

CeS i*AiA~ LL A II$ WAII lli *l lLLS iAAL A. Atior~ l ALLA. .9 AASp sAlil PAIWAIAWI=i 31119iI9I9SA•S "INAL

I L., AAW IAWC AIIAAA Also.•l~l.l, I,. CtAW* e.ACAA , ~il ,•,i*•lIii , Il

Il ICt ,130tl t Ille~ SAALA iILA WIAn 3 PA JASAS i•.l i iPA. IA CWACAA,, ii e 1, II,1911

A'ntA.,WAiAAIAI 9AAIIAACLl ItAIALAAAAIIA 3 It - AIIAAIW AA4AAIOl

alI I¢,IIIIlll iI iii .•I• II • I SOi I•I .I I

MII A . 1.IAI I3 A .A

4 A AIt .. AA0AAAA1,

II ......... l i~~l~l

t PC llS. ....ISI 3.031. C. 965l.A iAC9 LiLleA PSAAWA.oSl

AIiAL - LA 1113' 39. It .*~i A-Ai..lt.,. ,t C. ...IA.. ...I..AA l .ASAAAA3PPII PAASA SASLA ¢A A...,

i41 il1li. l• I-StI WAAA I WOCLLAAS3A I WAllili AASIA II ALl ,5 1} llIl .lll

.... AL ..... .. ..... IS -i

A-AAI•.SIAW 13A CA.A CAtA AA CO 11 5,¢*i~4i -fil ,i Wtt*.,AI I l~u

ILLIetttW.~ IW1 IL I i,, A ii ll~t IIAA- J.lWA Lli lt I 9533 A IIItt.11,IAIWIIASC Al ,ll li I-I ,I.1 , i ll

PSl. IL5II,*tilIr I lePAPAIe l~~. P000 SSAW9li.o . PS LOCIA NAlSAAAC k .9k PSA WAIWA As~~~yi

.9Wl S e,,WI •1¢I l~ilI ll'll• l~ I, 11911AWPW S@Yle LAW •t ll~r. llll t11

- I It b. ACII, .. l i ALA, St A. l 9AeAAA.,tA IAIAAl i ll. I l A. ........

•LI. .Lt •III.CWdA AA IPAI•A A L3ALC ISLt' P.0.9. i*,LW* * l I Pt PAA9A9AIAA.!, I *,i~~i

LAIIeCI A .lil A II ltllt I PtIAC IA. tet 9 I 9 lt. ,- Al uICA ~lll• 1Ill

LA.i I lilI AAAPAll hi *~ W1t•, t ll W ACSAW W NCCSl., ISA.l

tAitlCl A. S,.I lSW.O AIIAA ileAI* .9Wl 03319il Sept tAll PLAI9AAI#

C lpllh . W.ipi .S W9l ,AtS.AAAA LW I A.. C eei .PALWC Seltl .il ¢ IAlt S~lCS l I.I iil v-i

PAAAAI'IA •l~ lil. LllCSAl I.A AIAA. AASAAIl WA SIIItItAS CAWltt

WAIL,•il1il~l11. .. tWiiA Il I,[ ~ 'i, SA~l - 1 At W PAAAA

35.LI IS P. t 12•L tI, PAAAAI.P9CSSWA5AA. lll ii KItliti ll~ AJAIC AAI*.AAIA Atli l l

I,,.CLAA'. C CAILI CACSAL66e,

... .C...A

ll~~~~l. .. ..... Ai.l I/I l l~ l l I i l

... rl .. l .. ......- All*,.;;, Iol~l .. I ...... ...r....

CA..l... IA. W'--i-CC-rn---. A Art- III.., -- ' ,:.. - 1

report no. 8106 november 1981 · mecurity classification of thit 'age (whe., dota esteed) _...

Documents