poorly formed input and miscommunication in natural-language keyboard dialogue: an exploratory study

Computers in Human Behavior, Vol. 4, pp. 275-283, 1988 0747-5632/88 $3.00 + .00 Printed in the U.S.A. All rights reserved. Copyright © 1988 Pergamon Press plc

Poorly Formed Input and Miscommunication in Natural-Language Keyboard Dialogue:

An Exploratory Study

Ronan Reilly

Educational Research Centre St. Patrick's College

Eoghan Mac AogMn

Linguistics Institute of ire/and

Abstract -- Subjects (n = 21) completed a series of tasks involving data-base query using a simulated natural-language interface yielding a corpus of 476 user inputs. Ill-jCormedness and miscommunication were studied using a variety of taxonomies proposed in previous research. Misspelling ran at 14 % (oJ user inpuO, extra-grammaticality, largely consisting of ellipsis, at 40%, and misunderstanding, much of which remained undetected, at 20%. The study verified a number of expectations of the "naturalistic" approach to interface design, in particular the correlation of low-level ill-formedness with miscommunication at planning level, and the existence of strong temporal effects in the data according as the dialogue proceeded from beginning to end. The implications of the findings for interface design are discussed.

1. INTRODUCTION

1.1 Naturalistic Interface Design

A central content ion of the "natural is t ic" app roach to interface design (Reil ly, 1987a) is that the user will tend to adopt p e r s o n - p e r s o n modes of c o m m u n i c a t i o n in p e r s o n - m a c h i n e communica t i on wherever the env i ronmen t is conducive to this end. Th i s is not to suggest that the user will t reat the sys tem as a person. T h e r e is already considerable evidence (e.g., Richards & Underwood , 1984a, 1984b) that users change their no rma l pe r son -pe r son modes of communica t ion in the pe r son - machine situation. W h a t the naturalistic approach predicts, rather, is that the adap-

The research reported here was partly supported by the ESPRIT programme of the Commission of the European Communities (ref. AIP P527).

Requests for reprints should be sent to Dr. Ronan Reilly, Educational Research Centre, St. Patrick's College, Dublin 9, Republic of Ireland.

275

276 Reilly and Mac Aogdin

tation of person-person modes of communication to the person-machine situation will follow the lines of similar adaptations in the person-person situation, for example, when natural language dialogue is adapted to facilitate communication between partners of unequal fluency. The resulting language, be it "foreigner talk" (native speaker to foreigner), "motherese" (mother to child), or "computerese" (person to computer), reveals certain general features of "interlanguage" (Corder, 1981; Faerch & Kasper, 1983). Of particular interest in the present context is the use ot simplified syntactic and discourse forms designed to allow communication to con- tinue in spite of large amounts of ill-formed input.

The naturalistic approach suggests that interface design can benefit from the study of the naturally occurring "interlanguage" of naive computer users, and that the correlation of this interlanguage with ill-formedness and communication failure are essential preliminaries to the design of a robust natural-language interface (Egan, 1987).

1.2 III-formedness and Miscommunication at the Interface

Ill-formedness of input and communication failure in person-machine dialogue have been described by a number of authors, and several important distinctions have been clarified as a result (see Reilly, 1987b for a review). Ringle & Bruce (1982) distinguish between input failures, where no internal representation of the input is achieved, possibly because of ill-formedness, and model failures, where the representation generated is incorrect. Riley (1980) uses the traditional linguistic distinction between lexical, syntactic, semantic, and pragmatic features of input to distinguish levels of communication failure. The fact that communication can succeed on one level and fail on another underlines the distinctness of the notions of ill-formedness and miscommunication. At the same time it shows the close dependence of the concepts on each other. It is true that in person-person communication low-level ill-formedness (misspelling, mispronunciation) does not nor- mally interfere with communication. Yet it is miscommunication, as becomes apparent when, for one reason or another, corrective action from higher levels is not possible.

In the context of data-base query Webber and Mays (1983) contrast Is and Can- Be misconceptions of the user about objects referred to in the data-base. Similar forms of misconception are dealt with by McCoy (1987) and Carberry (1987) under the headings of"proper ty misconceptions" and "pragmatic overshoot." Reference failure is dealt with most explicitly by Grosz (1981) and Goodman (1987).

Carbonell and Hayes (1983) propose four categories of extra-grammaticality in user input.

1. Missing words (Copy new files my directory). 2. Spurious words (Copy if you would be so kind the new files to my directory

please; Delete I mean copy the new files to my directory). 3. Out of order words (New files to my directory copy). 4. Constraints violation (Copy the two new file to my directory; Copy my direc-

tory to the new file).

The Missing words category captures many features of "computerese," such as the omission of function words, articles, and so no. The Spurious words category refers

Poorly forrned input in keyboard dialogue 277

not so much to misspellings, neologisms, etc., but to discontinuities at discourse level, incomplete and interrupted speech acts, abrupt transitions between discourse levels, and so on. The Out of order words category refers to syntactic anomalies, and Constraints violation includes the various kinds of "misconceptions" referred to earlier.

The most complete formal taxonomy of communication failure in person-machine dialogue proposed to date is that of Reilly (1987b). An advantage of the taxonomy is that it subsumes ill-formedness and miscommunication under the single notion of"mismapping" from input to the internal representation which a natural- language interface builds from it. The taxonomy is based on a formal model of communication incorporating decoding modules at lexical, syntactic/semantic, discourse, and goal levels. The lexical and syntactic/semantic levels of the taxonomy are on traditional lines. The discourse level assigns utterances to categories of speech-acts which are then parsed by a discourse grammar. The lexicon and grammar are based on the work of Sinclair and Coutthard (1975) and Burton (1981), and were expressed as an ATN in Egan, Harper, Harris, and Reilly (1986). The planning level is formalized along the lines suggested in the plan-based approach to dialogue (Allen & Perrault, 1980). The taxonomy handles all the miscommunication types mentioned earlier as special cases.

1.3 Objectives of the Study

Below we report the findings of an exploratory study of person-machine communication in which the user's task was to extract information from a data-base using a simulated natural-language interface.

The primary objective of the study was to add to the small, but growing corpus of person-machine dialogue, with special reference to naturally occurring patterns of ill-formedness and miscommunication in user input. We expected to find, as others have (Eastman & McLean, 1981; Richards & Underwood, 1984a, 1984b; Thompson, 1980) that lexical failure would run at about 3-4% (of user inputs), that syntactic failures and ellipsis would be in the order of 10-20%, and that the corpus would exhibit the usual features of"computerese," most notably the omission of brief dialogue-management signals and other function words from the user input.

With the advantage of a comprehensive taxonomy of miscommunications in person-machine dialogue (Reilly, 1987b), we also hoped to be able to confirm some implications of the naturalistic approach. In particular, we predicted that ill- formedness at plan-level would be related to deformities of input at lower levels. In the study of interlanguage, it has been shown that miscommunication at plan level is signalled by irregularities at lexical, syntactic/semantic, and discourse levels (Egan, 1986; Faerch & Kasper, 1983). In practice, high-level miscommunication in interpersonal dialogue could be difficult to detect in many instances if it did not leave a trace at lower levels. We predicted that high-level miscommunication in person-machine dialogue would also leave traces of ill-formedness at lower levels.

Finally, we expected that the corpus would reveal a definite temporal structure according as the dialogue unfolded in time. While no explicit hypotheses were for- mulated, we expected to find phenomena analogous to the "entrainment" (Leiser, Carr, & Rogers, 1988) and "negotiation of meaning" (Sixt, 1985) phenomena noted in person-person communication. These refer to the gradual convergence on a mutually agreed subcode in the course of dialogue. We also expected that cogni-


tive strain, arising from task demands, would reveal itself in fluctuating levels ot accuracy and efficiency of input over time.

2. METHOD

2.1 Subjects, Task, and Experimental Setting

The subjects were 21 clerical and junior management staff in public service offices. All but two were female. All were efficient typists, and none had familiarity with computers or with the task which they were asked to perform.

Subjects were asked to fill in the required figures on a form requesting the num- bers of students and staff in various administrative and academic categories in a national teacher training college. Three subtables had to be completed. The task took about 25 minutes.

Subjects sat at one of two interconnected microcomputers. They typed their requests to the "system," the other microcomputer, and the answers appeared on their monitor. Subjects were told that the "system" was in fact being operated by a person (the junior author of this article), but they could not see the other microcomputer since there was an intervening partition.

The two interconnected microcomputers were Olivetti M24 SPs, with 20 mb disc drives and were connected via their RS232 ports. The monitor on each microcomputer was divided into three main windows. The topmost window was used by the system to display tabular output from the database, the middle window displayed the typed output of the system, and the bottom window displayed the typed utterances of the user. Tabular displays were called up by preset function keys, a facility that was not available to the user. Various prompts to the system regarding tables to be displayed were shown in a prompt line displayed at the bottom of the screen and were only visible to the system. In effect the "database" was a set of prestored tables which anticipated all the possible requests that the user might make. The interface, which was written in GW-Basic, kept a complete record of the dialogue, including response times.

The system responded in ordinary language and was as cooperative as possible, requesting clarification, and removing mistaken assumptions as necessary. How- ever, in order to increase the ecological validity of the experiment, knowledge of the user's task (i.e., knowledge of the particular form to be filled out), was never used by the system. Mismatches between the categories of information-seeking forms and those of the database which contains the information are typical of bureaucratic form-filling exercises, and result in a high incidence of"model failure" (Ringle & Bruce, 1982). In effect, the user misinterprets information successfully retrieved from the system.

2.2 Measurement

Basic statistics relating to quantity of words and utterances, and time taken to complete them, were compiled directly from the logs.

Extragrammaticalily was recorded using the modified Carbonell & Hayes (1983) taxonomy, and miscommunication was recorded using the Reilly (1987b) taxonomy. In our application of the former we distinguish two levels of ellipsis associated with

Poorly formed input in keyboard dialogue 279

Missing Words category, one (intra-sentential) in which the fully restored utterance is semantically self-sufficient, and another (extra-sentential) in which the fully restored utterance still depends semantically on a previous utterance, thus being, as it were, doubly elliptical.

Goal achievement was computed for each subject from the number of subgoals completed. Rate of goal achievement was derived by introducing data on quantity of utterances per goal and amount of time taken.

Deviation from optimal planning was computed from the sequences of subgoals attempted by each subject. The observed sequence was assigned a deviation score from a theoretically optimal sequence. The latter was the sequence determined by the order of the tasks on the form to be completed and the maximally efficient path through the subtasks to complete it. A high deviation score indicates poor planning, resulting in erratic movement from tasks to subtasks and back again, and general lack of orientation (see Egan et al., 1986, for further details).

Time scores were assigned to utterances depending on their Location in the dialogue (beginning, middle, or end) and Elapsed time, namely the amount of time they took to complete. The beginning section of a dialogue consisted of the successive thirds of the dialogue, in terms of number of exchanges. Residual time was derived from Elapsed time by partialling out individual differences in typing speeds. Thus, it records at each point in the dialogue whether subjects are typing faster or slower than is usual for them.

2.3 Analyses

For the statistical tests reported below, utterance-level data (n = 952) were ag- gregated to subject level (n = 21), thus yielding a somewhat conservative test of the null hypothesis. When the data are divided further into beginning, middle, and end of dialogue segments, a repeated measures design was used (Cohen & Cohen, 1975, Ch. 11). In some analyses the sample was also divided post hoc into two groups (n 1 = 11, n2 = 10), namely high and low scorers on some variable of interest.

3. RESULTS

3.1 General Characteristics of the Corpus

Some ten hours of dialogue were recorded, yielding a usable corpus of 952 utterances obtained from 21 subjects. There were, therefore, a total of 476 utterance pairs, and the same number of user inputs. The 952 utterances consisted of 1,070 sentences, which gives an average of i. 12 sentences per utterance and 8.71 words per sentence. The size of the corpus gathered is somewhat smaller than other similar studies. For example, Thompson (1980) in an extensive study of a database querying system collected 1615 inputs, compared to our 476. More comparable, was the study of Eastman and McLean (1981), which gathered 693 inputs.

The histograms of words-per-utterance, one each for the users and the system, are presented in Figure 1. Median utterance length was 10 words for the user, and five words for the system. Zero-length utterances are virtually all unintentional strikings of the return key in the case of user. In the case of the system they cor- respond mostly to tabular displays, unaccompanied by other input from the key-


Words per

Utterance User System

3,4 ****** ****

5 , 6 * * * * * * * * * * * * * * * * * * 7,8 ************ **********

9,10 **************** *****

11,12 ************* ****

13,14 ********** ****

15,16 ******* **

17,18 **** ****

19,20 **** ****

21,22 *** **

23,24 ** **

25,26 ** *

27,28 * *

29,30 * *

31,32 ** **

33,34 * *

35,36 *

37,38 *

Figure 1. Histograms of words per utterance for user and system.

board. When zero-length utterances are excluded the medians are 10 and 6.5, respectively for user and system. The difference is due principally to the fact that the cluster of one- and two-word utterances which is typical of person-person dialogue has dropped out of the user's corpus, but remains in the system's. This finding has been reported previously (Richards & Underwood, 1984a, 1984b).

3.2 Types of Miscommunication

The ill-formed utterances in the dialogues were classified under three main headings, (a) misspells, (b) extra-grammaticalities, and (c) misunderstandings. Extra- grammaticalities were placed in subclasses using a version of the Carbonell and Hayes (1983) taxonomy, and misunderstandings were subclassified using the taxonomy described in Reilly (1987b).

In all, 186 misspellings in 134 utterances were generated by both user and system. This accounts for about 2% of all the words used and 14% of all utterances. The rate of misspelling differed markedly from that found by Thompson (1980) who reported that 3.8% of the 1,615 utterances he gathered contained at least one misspell.

In our corpus, 219 utterances were classified as extra-grammatical. This rep- resents 40 % of the user's input. Intra-sentential ellipsis accounts for 44 % of the extra-grammaticalities, and extra-sentential ellipsis for 18%. Therefore, ellipses as a whole are responsible for 62 % of all extra-grammaticalities in the corpus.

Our findings, with regard to ellipses, are in rough agreement with the findings of similar studies. In the Thompson (1980) corpus "fragmentary inputs" accounted for 13% of the utterances. In the Eastman and McLean (1981) corpus, various forms of ellipsis accounted for 14 % of the utterances. This compares to the 17 %


found in our corpus. The remaining extra-grammaticalities in our study were accounted for by 6% of the utterances, as compared to 4% in the Thompson (1980) corpus and 12.3 % in the Eastman and McLean (1981) corpus. Despite the varying definitions of extra-grammaticality employed by the different studies, our percentages are therefore in rough agreement with those already reported.

Using the taxonomy described in Reilly (1987b), 21% of the user's input was classified as misunderstanding. Most (64%) resulted from the user's failure to grasp the task, and most (74%) of these remained undetected by the user.

3.3 Task Efficiency and Miscommunication

The task proved moderately difficult. Only five subjects successfully completed all subtasks. All except one completed at least half the subtasks correctly.

Deviant planning is related to task completion (r = - .39, p < .05), and to number of exchanges required (r = .39, p < .05). Subjects using nonoptimal strategies were less successful in completing the task, and required a greater number of exchanges.

Efficiency of input, measured in terms of the goals achieved per section, divided by the number of attempts, increases towards the end of the task (F(1,18) = 7.1; p < .01). The low ellipsis group shows greater efficiency throughout (F(1,19) = 5.0; p < .05).

Deviation from optimal planning rises in mid-dialogue (F(2,18) = 10.1; p < .001). The high ellipsis group is also significantly higher on the deviation index than the low ellipsis group (F(1,19) = 8.9; p < .01).

3.4 Temporal Effects

There was a significant increase in the user's rate of misspelling in the middle section of the dialogue (F(2,19) = 4 .1 ;p < .05). Residual time to complete input was lower for the low ellipsis group (F(1,19) = 10.1; p < .01). In other words lack ot ellipsis was associated with faster input. This finding should probably be taken in conjunction with the previous findings that efficient input (in terms of planning and goal achievement) tended to lack ellipses.

4. DISCUSSION

The rate of ill-formedness and miscommunication observed in our study is higher than previously estimated. Misspells (14 %), misunderstanding (21%), and ellipsis (40%) were the most notable deviations in user input. Admittedly, the simulated interface was tolerant, which probably increased ill-formedness. On the other hand the users were expert typists and had adequate facilities for correcting input before entry. The least that may be concluded is that naive users would ideally expect natural-language interfaces to cope with large amounts of ill-formed input.

The high rate of ellipsis poses a special problem since conventional parsers can- not deal easily with it (Weischedel & Sondheimer, 1983). Users expect the system to be able to cope with telegraphic input. At the same time, the more competent users tended to use relatively fewer ellipses. This suggests that ellipsis may be partly a side-effect of cognitive strain. Alternatively, over explicitness may be a deliberate strategy used by the better problem solvers.

It should be noted that by comparison with person-person communication, the typical input in the present study was highly redundant. Users in general were


reluctant to use anaphoric reference based on the previous input, as one would naturally do if the same task were being done in person-person dialogue. Thus, their input, in general, was "inefficient" in the Situation Semantics sense (Barwise & Perry, 1983). Users tried to ensure that each input was referentially self-sufficient. If it can be confirmed that this is a feature of person-computer interlanguage, then perhaps entrainment towards nonelliptical forms in the early stages of the dialogue would be appropriate as a strategy for coping with ellipsis as an alternative or perhaps a complement to robust parsing.

About one input in five was based on a misunderstanding, and only one in four of these misunderstandings was detected and corrected. A high degree of"model failure" (Ringle & Bruce, 1982) must therefore be expected in natural-language input, even when tasks are relatively simple. It is true that the users in the present study had little or no experience with computers. On the other hand, one of the objectives of natural-language interface design is to cater to exactly such users. This is a particularly reasonable objective in the area of database interrogation, since the expertise needed to access a database directly, in a standard query language, is not great in any case. Thus, there is little point in moving towards a natural-language interface for such tasks unless one can go all the way, or a good part of it.

Looked at from the plan-based perspective on dialogue, the data show the expected relationship between planning capacity, efficiency of input, and task completion. As predicted, successful planning and execution of dialogue, in terms of goal achievement, were linked to dialogue structure. Poor planning led to longer dialogues and less efficient input. It was also associated with a higher rate of ellipsis and a slower rate of input. All of these effects are potential indicators of incipient communication failure at goal level, and are therefore significant for interface design.

Temporal effects were also in evidence. Planning deteriorated in mid-task. Mis- spells also increased at this point. Efficiency of input improved towards the end of the task. The existence of temporal effects in the data also has implications for interface design. Interfaces are typically constant over time and do not make any attempt to "shape up" the user's behavior. There is evidence, however, that users would welcome some degree of coaching on preferred forms of input in the early stages of person-machine dialogue (Leiser, Carr, & Rogers, 1988). Indeed, "negotiation of meaning" (Sixt, 1985) and other "reduction" strategies (Faerch & Kasper, 1983) are typical in interlanguage, as participants try to establish a working "subcode" early in the exchange. According to the naturalistic hypothesis one would therefore predict that users would expect, and benefit from, coaching on preferred input forms in the early part of the dialogue. One should also question whether interfaces which are maximally tolerant of ill-formedness right from the beginning of the dialogue might not give the user unrealistic expectations and lead to increased ill-formedness in later stages, as was observed in our data.

It was also noted, consistent with the findings of previous studies (Richards & Underwood 1984a, 1984b), that users tend to omit short dialogue-management signals from their input, such as go-ahead signals, or signals of perplexity. Since these play a crucial role in detecting communication failure in person-person dialogue, it follows from the naturalistic approach adopted in this paper that some means of replacing them should be found, perhaps in a nonverbal input mode, since comparable signals are often transmitted in this manner in person-person dialogue.


REFERENCES

Allen, J . , & Perrault, C.R. (1980). Analyzing intention in utterances. Artificial Intelligence, 15, 143-178.

Barwise, J . , & Perry, J. (1983). Situations and attitudes. Cambridge, MA: MIT Press. Burton, D. (1981). Analysing spoken discourse. In M. Coulthard & M. Montgomery (Eds.), Studies

in discourse analysis (pp. 61-81). London: Routledge & Kegan Paul. Carberry, M.S. (1987). The use of inferred knowledge in understanding pragmatically ill-formed

queries. In R. Reilly (Ed.), Communication failure in dialogue and discourse. Amsterdam: North- Holland.

Carbonell, J. , & Hayes, P.J. (1983). Recovery strategies for parsing extragrammatical language. American Journal of Computational Linguistics, 9, 123-146.

Cohen, J. , & Cohen, P. (1975). Applied multiple regression~correlation analysis for the behavioral sciences. Hillsdale, N J: Erlbaum.

Corder, S.P. (1981). Error analysis" and interlanguage. London: Oxford University Press. Eastman, C.M., & McLean, D.S. (1981). On the need for parsing ill-formed input. American Jour-

nal of Computational Linguistics, 7, 257. Egan, O. (1987). Induction and decision-making in communication failure. In R. Reilly (Ed.), Com-

munication failure in dialogue and discourse. Amsterdam: North-Holland. Egan, O., Harper, G., Harris, J . , & Reilly, R. (1986). CFID Project Deliverable 4: Dialogue

Studies-Main Phase (Contract No. ESPRIT P527). Burssels: Commission of the European Communities.

Faerch, C., & Kasper, G. (1983). Strategies" in interlanguage communication. London: Longman. Goodman, B. (1987). Repairing reference identification failures by relaxation. In R. Reilly (Ed.),

Communication failure in dialogue and discourse. Amsterdam: North-Holland. Grosz, B. (1981). Focussing and discription in natural language dialogues. In A. Joshi & B. Web-

ber (Eds.), Elements of discourse understanding. Cambridge, England: Cambridge University Press. Grosz, B., & Sidner, C.L. (1985). The structures of discourse structure. (Report No. CSLI-85-39).

Stanford, CA, Centre for the Study of Language and Information. Leiser, B., Carr, D., & Rogers, Y. (1988). Improving natural language understanding by implicit

entrainment. In O. Egan, (Ed.), CFID Pro3~ct: Empirical studies of the interface. (Contract No. ESPRIT P527). Brussels: Commission of the European Communities.

Litman, D.J. (1985). Plan recognition and discourse analysis: an integrated approach for understanding dialogues. TR170, Department of Computer Science, University of Rochester.

McCoy, K.F. (1987). Generating responses to property misconceptions using perspective. In R. Reilly (Ed.), Communication failure in dialogue and discourse. Amsterdam: North-Holland.

Reilly, R. (1987a). Communication failure in dialogue and discourse. Amsterdam: North-Holland. Reilly, R. (1987b). Types of communication failure in dialogue. In R. Reilly (Ed.), Communication

failure in dialogue and discourse. Amsterdam: North-Holland. Richards, M.A., & Underwood, K. (1984a). Talking to machines: How are people naturally inclined

to speak? In Procedings of the Ergonomic Society Annual Conference. Richards, M.A., & Underwood, K.M. (1984b). How should people and computers speak to one

another. Interact, 2, 33-36. Riley, P. (1980). When communication breaks down: Levels of coherence in discourse. Applied Lin-

guistics, 1, 201-216. Ringle, M.H., & Bruce, B.C. (1982). Conversation failure. In W.G. Lehnert & M. H. Ringle (Eds.)

Strategies for natural language processing. Hillsdale, N J: Erlbaum. Sinclair, J . , & Coulthard, M. (1975). Towards an analysis of discourse. London: Oxford University

Press. Sixt, D. (Ed.). (1985). Comprehension as negotiation of meaning. Amsterdam: Goethe Instut. Thompson, B.H. (1980). Linguistics analysis of natural language communication with computers.

In Proceedings of the Eight International Conference on Computational Linguistics, Tokyo, Japan.

Webber, B., & Mays, E. (1983). Varieties of user misconceptions: Detection and correction. In Pro- ceedings of IJCAl '83, Karlsruhe, Germany.

Weischedel, R., & Sondheimer, N.K. (1983). Meta-rules as a basis for processing ill-tbrmed input. American Journal of Computational Linguistics, 9, 161 - 177.

poorly formed input and miscommunication in natural-language keyboard dialogue: an exploratory study

Documents