conceptual, communicative and pragmatic aspects of interaction forms - rich interaction model for...

8/3/2019 Conceptual, Communicative and Pragmatic Aspects of Interaction Forms - Rich Interaction Model for Collaborative Virtual Environments

1/6

Conceptual, Communicative and Pragmatic Aspects of Interaction Forms - Rich

Interaction Model for Collaborative Virtual Environments

Tony ManninenDepartment of Information Processing Science, University of Oulu, FINLAND

[email protected]

Abstract

This paper provides a form-oriented description and

applicable model of interaction in the context of

Collaborative Virtual Environments (CVE). The

construction and the main categories of the conceptual

interaction form model are described through examples

and previous work. The evaluation and validation of the

model is illustrated by delineating the research process

conducted by the author. Furthermore, the benefits and

limitations of the model are discussed in the light of CVE

analysis and design.

1. Introduction

The lack of intuitive and non-intrusive non-verbal cues

is one of the distinctive features that separates computer-

mediated communication settings from face-to-face

encounters. One solution to the interaction problems is the

CVE design approach that draws from the theories and

conceptual models of available interaction forms. Theinteraction form model, which has been constructed

during this research, directs the design to start from the

natural areas of interaction. By taking a holistic view of

the concept of interaction forms, the model enables

designers to take into account all the necessary

manifestations and representations of interaction.

The aim of this research is to conceptualise and

delineate the mutually perceivable interaction forms

available for avatar-based CVEs. Interaction forms are

actions that can be perceived as manifestations of the

user-user and user-environment interaction. These forms

are used to convey the actions of the user to oneself, and

to others. The forms enable awareness of actions byoffering mutually perceivable visualisations and

auralisations within the virtual environment.

The scope of this work covers the manifestations of

interaction. The emphasis is on every action and

interaction that can be perceived in CVEs. The mutually

perceivable actions and behaviours have been described

without tackling the social or cultural aspects of

communication, co-ordination and collaboration. The

point of interest, thus, is to find more understanding about

the possibilities and effects of rich interaction in the CVE

context.

2. Theoretical background and related work

The forms of interaction have been described and

discussed in the existing literature in the field of

communication, for example, under topics such as non-

verbal communication channels and communication

codes [2, 8, 10]. The focus and starting point for these

models is on the face-to-face interaction between humans

who share the same physical place.

Within the context of computer-supported co-operative

work (CSCW), the focus of research is not limited to

specific communication channels. One area of interaction

research relates to the embodied actions that cover the

movements and actions of the participants who interact

with each other and with their environment [16].

However, the interaction forms described in this paper

only exist in virtual environments, and thus, the actions

occurring in the physical world are out of the scope of this

paper.

The concept of rich interaction is not only a

quantitative measure describing the amount of available

interaction forms [8]. However, a basic set of interaction

form categories helps the CVE designers to consider all

the necessary areas of action representations. Rich

interaction can be enabled by a set of interaction forms

which is large, flexible and focused on the content. The

contextual and communicative support for interaction is

essential in providing users with meaningful ways to

express themselves and their actions. The richness itself

is, at the end of the day, achieved by users who are able to

exploit the available interaction forms in an intuitive and

non-deterministic style.

The related research of interaction forms covers a wide

area, ranging from CVE design to communication and

collaboration support. This section outlines some of the

previous research that is closely related to the line of work

described in this paper.

Proceedings of the 16th International Conference on Computer Animation and Social Agents (CASA03)087-4844/03 $17.00 2003 IEEE


2/6

Benford et al. [4] have created a spatial model of

interaction which provides a basic set of abstractions for

managing interactions in a wide range of spatial systems.

The authors argue that the spatial model of interaction

provides a novel and powerful set of abstractions for

managing interactions in a variety of large-scale virtual

spaces. However, the model does not imply any specific

interaction forms that could be used, for example, to

represent and execute participants aura, focus and

nimbus.

Robertson has constructed a taxonomy of embodied

actions [16], which consists of individual actions (in

relation to physical objects, other bodies, and to the

physical workspace) and group activities constituted by

individual embodied actions. While Robertson's

taxonomy does not contradict the model presented in this

paper, the level of abstraction is different. Robertson

delineates actions that describe the incentive of the

participant. Furthermore, there are actions that can be

presented with various interaction forms.

Additional descriptions of modes and types of

interaction have been presented, for example, in the areas

of autonomous agents [9]. The non-verbal communication

aspects of CVEs have been studied, for example, in the

context of user embodiment [3], communicative

behaviour [18], conversational interface agents [6], and

realistically expressing avatars [17]. These approaches,

however, tend to concentrate on a highly specific and

limited interaction support.

The approach selected by this author follows, to a

certain degree, the lines of the related research. The

interaction form model has been constructed in order to

obtain a clear overall picture of the concepts related to

interaction. The basic theories have been selected from

the areas analysing physical world communication in

order to enable the continuum in the level of

communication. In addition, the relatively mechanistic

perspective to interaction (i.e., the focus is on forms

instead of functions) is believed to enable user-driven

communication, control and collaboration.

3. Construction of the model

The interaction form model has been constructed by

collecting theoretical knowledge (e.g., communication

literature) and empirical material (video recordings,interviews, walkthroughs, observations, and heuristic

evaluations) from networked games, game events and

from self-organised gaming sessions [12].

The main aim of the model is to provide as exhaustive

a set of interaction forms as possible. However, due to the

high number of available interaction forms, the model is

structured into several main categories, which, in turn,

contain a number of sub-concepts.

The starting point for the conceptual modelling was the

interaction form support offered by the existing CVEs. In

order to acquire a basic set of currently available

interaction forms, a number of contemporary multi-player

games have been studied and the material has been

expanded with heuristic evaluations [18, 19]. The games

that were observed include, for example, a 3D multi-

player action game called Counter-Strike. Additional

games that have been studied include action games (e.g.,

Action Quake, Team Fortress) and role-playing games

(e.g.,EverQuest, Ultima Online). Some material was also

obtained from text-based games and flight simulators.

The next step in constructing the concept model was

the categorisation of different interaction forms. The basic

criteria for categorisation were the closeness and relations

of different sub-concepts. For example, all the interaction

forms that relate to the appearance of the avatar were

grouped together. The construction was based on the

conceptual analysis, and thus, offered a somewhat

coherent illustration of the corresponding interaction

forms. The first versions of the model were constantly

refined and revised according to the additional

evaluations of games and other CVEs.

The theoretical framework that would support the

aforementioned preliminary model was applied on a later

stage. Before the integration, the preliminary model was

used as a basic design guideline in constructing an

empirical experiment called Tuppi3D. The experiment

was then evaluated by using the synthesised model of

non-verbal communication channels (hereinafter referred

to as NVC) described in the communication literature (cf.

[2, 8, 10]). The perceivable interaction forms encountered

in the Tuppi3D experiment were analysed based on the

NVC model. Manninen and Kujanp [14] have

elaborated on the experiment and the outcome of the

analysis.

After the analysis of perceivable non-verbal

communication forms in the experiment, the concept

model was modified in a way that the structure of non-

verbal communication forms created the backbone of the

model. The categories from the preliminary concept

model were then added to the base model. The combined

model, thus, covers a wider range of interaction forms.

Furthermore, it takes into account the aspects of virtual

environments by also describing the forms that are not

necessary applicable in physical world interactions.

4. Description of the concepts in the model

Figure 1 represents the first layers of the

decomposition that forms the proposed concept model of

interaction forms. The map illustrates the main interaction

types that can be found within current multimedia games.

The forms have been categorised into 12 classes, each



3/6

consisting of a number of sub-concepts. The basis for this

taxonomy is the categorisation of various interaction

forms in terms of communication channel, context, and

acting entities (e.g., body parts, environment, fellow team

members, etc.).

INTERACTION

FORMS IN CVEs

Avatar

Appearance

Kinesics

Autonomous

AI

Artefacts

Language-based

Communication

Physical

Contact

Setting

Physique Equipment

Clothes

Postures

Environmental

DetailsFollow-me

Auto-actions Reflection

Use of

Object

Object

Exchange

Destruction

Construction

ObjectMoving

Tactile

DefensiveSignals

Emotional

Agressive

BodyMovement

ChatSpeech

Phrases

SignLanguage

Modification

Non-verbal

Audio

Sound

effects

Silence

MutualAffects

Face & SkinHair Adornment Paralanguage

OlfacticsChronemics

Spatial

Behaviour

Orientation

Positions &Locations

Proximity &Distance

HeadMovement

Gestures

Facial

expressions

Music

OcculesicsEye

Movement

Visual

Orientation

Eye

Contact

Figure 1. Concept model of interaction forms.

It has been attempted to keep the naming of the sub-

concepts within the level of interaction forms without

describing the functions that can be executed by using the

forms. However, there are some sub-concepts that can be

considered as styles of interaction. For example,

emotional physical contact does not imply any specific

interaction forms. It merely directs the thinking towards

the instances of emotional forms, such as hugging, petting,

holding hands, etc. In relation to this, the consistency of

the model is compromised. Still, the naming convention

aims at providing as clear a tool for analysis and design as

possible.

The following section illustrates the forms, or

manifestations, of interaction. A brief description of each

of the main categories is provided. The model does not try

to replicate the physical world interaction forms. The

relation to the physical world is taken into account when

applicable, but the potential of virtuality is also harnessed.

Autonomous AIcategory includes a set of pre-

programmable actions and reactive behaviour that

resembles subconscious and intuitive actions in the

physical world. Some of these actions can be regulated by

the system while others are modifiable by the participants.

This group of interaction forms overlaps with most of the

other categories. However, the autonomous actions are

considered as a separate group because of the specific

nature of most of the actions and their importance in terms

of design.

Avatar appearance defines the attributes of image and

presentation of self [8, 10]. Appearance contains the

visual aspects of one's presentation. Argyle [2] divides

this into two: those aspects under voluntary control -

clothes, bodily paint and adornment - and those less

controllable - hair, skin, height, weight, etc. The aspects of

appearance can, thus, be thought of as static or dynamic

communicational messages, depending on the attribute.

The CVE context enables novel ways of utilising

appearance as a form of interaction because the physical

constraints are not necessarily replicated from the real

world. For instance, avatar size and shape can be

dynamically altered in order to convey particular

messages. The Tuppi3D experiment included only static

representations of avatars that conveyed the identities of

the players. However, the participants felt that they would

have liked more freedom in customising their visual

images. A dynamic way to change ones appearance was

considered as effective channel for communicating.

Chronemics involves the use and perception of time

[5]. Masterson [15] describes the example of being

punctual versus being late as one illustration of this group.

However, there are several other possibilities in using time

as a communication tool. For example, pauses can be used

to increase anticipation and to make others pay closer

attention to ones actions. The interaction analysis of

Tuppi3D [14] suggested that the chronemical forms

emerge when there is flexible set of various interaction

forms available for the participants.

Facial expressions may be broken down into the sub-

codes of eyebrow position, eye and mouth shape and

nostril size. These, in various combinations, determine the

expression of the face, and it is possible to write a

'grammar' of their combinations and meanings [7].

Furthermore, Argyle [2] classifies blushing and

perspiration as facial expression.



4/6

Although face is the most significant channel of non-

verbal communication in physical world, the virtual

environments tend to diminish the role of face due to

graphics resolution limitations. However, in close

encounters and discussion-oriented situations the facial

expressions are fully perceivable even in games and

CVEs.

Environmental details define the appearance of

surroundings providing contextual cues [15]. These

include artifacts that can be used and manipulated within

the environment [5]. Argyle [2] states that moving objects

and furniture, leaving markers, and architectural design

can be used to communicate through space and place. The

mutual affect involves the effect environment has on the

user and vice versa. For example, physical boundaries

(e.g., walls), lighting (e.g., shadows), and the matter

filling the virtual space (e.g., water) can change the

performance of the participants. The dynamics of the

setting vary according to the implementation. However,

usually at least a restricted destruction and modification of

the environment is allowed.

Kinesics includes all bodily movement commonly

referred to as body language [5]. Head movements are

involved mainly in interaction management, particularly

in turn taking in speech, and can consist of one or several

sequential (rapid) nods at various speeds [7]. Posture

defines the way of sitting, standing and lying [2]. Gestures

involve the hand and arm as the main transmitters, but

gestures of the feet and head are also important. They are

usually closely co-ordinated with speech and supplement

verbal communication.

Computer games support kinesics relatively well. There

are numerous examples of crawling, walking, running,

jumping and waving animations that portray the action of

the player.

Language-based Communication is the major channel

for interpersonal information sharing in most of the

current CVEs. The use of language, or symbols, can be

modelled and conveyed textually, aurally, and in the form

of images. Text chat and voice-over-IP speech support are

examples of the channels that support this group of

interaction forms. Although language itself is not within

the focus of this work, the level of language abstraction

and automation is highly relevant. Speech audio with

spatial auralisations represents the level of highest

'manual' control. Instead, pre-programmed phrases and

textual parser support represent a somewhat abstracted

and automated utilisation of language.

Non-verbal audio includes the use of voice in

communication, which is often referred to as

paralanguage [5]. The non-verbal aspects of speech

contain prosodic and paradigmatic codes [7]. The former

is linked to speech (e.g., timing, pitch, and loudness) and

the latter are independent of the speech (e.g., personal

voice quality and accent, emotion, disturbances). Non-

verbal vocalisations [2] are an essential part of

communication as they can significantly change the

meaning of the message. In addition to these, CVE

systems can contain various sound effects and background

music that can be used as manifestations of interaction.

Furthermore, successful, i.e., purposeful, silence is a

strong form of interaction that often causes problems in

networked settings because of lag (e.g., communication

partner does not mean to be silent but the network lag

makes this happen).

Occulesicsare movements of the eyes, e.g., gaze [15].

Eye movement and eye contactdepict the focus, direction

and duration of the gaze in relation to other participants

[7]. Allbeck and Badler [1] use the term visual orientation

to differentiate occulesics from spatial behaviour. Argyle

[2] describes two groups of variables associated with the

gaze: amount of gaze (e.g., how long people have eye-

contact) and quality of gaze (e.g., pupil dilation, blink

rate, opening of eyes, etc.). In CVEs, the effective use of

this category requires modelling of eye movements and

detailed enough visuals.

Olfactics refer to the non-verbal communicative effect

of one's scents and odours [15]. Perhaps the most common

example of this category is the use of perfumes. This

group is not widely supported in CVEs, because of the

user-interface limitations. However, there are examples,

especially in games, of the successful application of

simulated olfactics as an interaction form.

Physical Contactreflects the use of touch in

communication situations [5]. This category consists of

actions such as handshakes and patting [1]. Furthermore,

bodily contact stimulates several receptors that are

responsive to touch, pressure, warmth or cold, and pain

[2]. In the context of CVEs, this group consists of virtual

interaction between the avatars of the participants.

Spatial behaviour consists of proximity, orientation,

territorial behaviour and movement in a physical setting

[2]. Burgoon and Ruffner [5] use the term proxemics to

include actions relating to the use of personal space.

Proximity consists of the various actions corresponding to

the use of personal space, i.e., how closely we approach

someone can give a particular message about our

relationship. Different distances usually convey different

meanings. Orientation defines the direction to which a

person has turned to. This code conveys information about

our point of interest, or, focus [7].

5. Evaluation of the model

The usefulness of the interaction form model has been

evaluated in two ways. First, the model has been used as a

framework for analysing the existing CVEs. Second, the

model has been used as a design guideline in constructing



5/6

new CVE experiments. The evaluation has been

conducted in an iterative manner. The results of previous

evaluation have been used to refine the model, which, in

turn, have then been used as a framework for further

analyses.

The preliminary version of the model was used as a

framework in the analysis interaction forms perceivable in

multi-player game sessions [11]. The results of the study

indicate the successful application of the model as a tool

for structuring the data into coherent and descriptive

categories. Furthermore, the model was helpful in pointing

out several areas of interaction forms that were not

adequately supported by the systems.

The model was analysed and compared against the

social theoretical framework [10]. The different

approaches to interaction (i.e., the interaction form

approach and the higher-level social action view)

adequately supported each other. The model of interaction

forms was successfully mapped as a set of executing

instances for the higher-level social activities.

The next phase in the evaluation was the design and

development of the Tuppi3D experiment. The interaction

form model was used as a design guideline in constructing

a CVE that would support rich interaction. A qualitative

video analysis was then conducted by using the NVC

model as the framework. The results indicated that the

purely communicational NVC model does not represent a

complete set of interaction forms [14]. Based on this, the

interaction form model and NVC model were combined in

order to obtain a holistic view towards the concept of

interaction.

The earlier interaction form analysis of multi-player

games was re-evaluated using the final version of the

model. The results indicated that the findings support the

new model relatively well. There were no interaction

forms that did not fit into the framework. On the contrary,

the model illustrated several concepts that were not

evident in the current data [13].

The final evaluation of the model was conducted from

a process point-of-view in a research project which

focused on a value-added service production for mobile

platforms. The importance of non-verbal communication

in multi-player game environments was successfully

brought forward and demonstrated in theory (i.e., the

concept model) and in practice (i.e., the experiment).

Furthermore, the rich interaction model and the

corresponding design philosophy led to a solution that

was, in many ways, superior to the solution designed and

developed purely with the technological focus.

The holistic approach to a complex phenomenon, such

as interaction forms, makes the modelling difficult

because the models may grow to be too large. There are

hundreds of specific interaction forms that can be applied

to the CVE context. Successful illustration of all these

concepts may prove to be a relatively difficult task. Still,

categorisation of the concepts enables the coherent

breakdown of the model into a set of sub-models that can

be considered individually.

The nature of multi-player games maintains the level of

abstraction relatively high, so, for example, the detailed

direct manipulation and complex artefact sharing that are

common in numerous groupware applications may not be

fully supported. Nevertheless, this type of interaction can

be illustrated with a set of corresponding interaction

forms.

Although the model is useful in analysing and

understanding the concept of interaction forms, it is

difficult to design and implement CVEs based on the

model alone. The implementation generally requires

abstraction, which, in turn, forces the designers to

consider higher levels of interaction. This may lead to a

traditional top-down design approach, in which the

available interaction forms are suffocated by the higher-

level activities [12].

The usefulness of the model is in providing a

conceptual framework that can be applied as a basis for

the analysis, evaluation and design of CVE applications.

The model can aid the designers to support meaningful

and useful interaction by providing a holistic view to the

applicable interaction forms. Furthermore, the conceptual

understanding of the phenomenon helps the researchers

and practitioners to tackle the issues of interaction design

collaboratively because it provides a common language

and terms for the various interaction form categories.

Due to the somewhat mechanistic approach, the model

serves well as a basis for lower level interaction support.

The focus being on forms, and not on functions, makes it

possible to design a number of interaction forms that

support numerous functions. The aim is not to dictate how

the participants should exploit the available forms of

interaction. Instead, the goals and incentives of the users

can be well supported by the lower level manifestations of

interaction described by the model.

The model is beneficial for CVE designers, as it

illustrates the available interaction forms. Thus, it is

possible to reduce the limitations and restrictions of

computer mediation by supporting more flexible and

natural interaction. Although the naturalness and

intuitiveness of face-to-face communication is hard to

achieve, the rich interaction model can direct designers to

additional and novel ways that could enhance the weak

areas of interaction.

6. Concluding remarks

The concept model of rich interaction forms illustrates

the various forms of communication, co-ordination and

collaboration in CVEs. A creative combination of



6/6

interaction forms makes it possible to enhance the overall

interaction and further increases the communicative,

collaborative and constructive uses of the virtual

environments.

Although the mechanistic approach to interaction

modelling may be criticised, the interaction form oriented

analysis and design could be a solution that would support

the construction of more communicative and collaborative

systems. The top-down approaches to system design have,

to date, been unable to solve problems of computer-

mediated group activities. However, the proposed bottom-

up approach could be used as a design guideline that

would help the implementation of flexible-enough

interactions.

The interaction form model is significant for CVE

designers, as it illustrates the possible representations of,

for example, non-verbal communication in networked

settings. Thus, it is possible to reduce the limitations and

restrictions of computer mediation by enabling more

flexible and natural interaction. When designers know the

possible means of interaction, they can direct their effort

to the ones that best suit the application domain.

Furthermore, the basic set of interaction forms, being

given by the interaction model, enables the designers to

apply other design approaches, such as artistic selectivity,

to enrich the interaction.

7. References

[1] Allbeck, J. M., and Badler, N. I. (2001). Consistent

Communication with Control. Workshop on Non-Verbal and

Verbal Communicative Acts to Achieve Contextual Embodied

Agents in Autonomous Agents 2001

[2] Argyle, M. (1975). Bodily Communication, International

Universities Press Inc., New York.

[3] Benford, S., Bowers, J., Fahlen, L. E., Greenhalgh, C., and

Snowdown, D. (1997). Embodiments, Avatars, Clones and

Agents for Multi-User, Multi-sensory Virtual Worlds.

Multimedia Systems, 5(2), 93-104.

[4] Benford, S., and Mariani, J. (1993). Requirements and

Metaphors of Interaction - COMIC Deliverable 4.1, Lancaster

University, Lancaster.

[5] Burgoon, M., and Ruffner, M. (1978). Human

Communication, Holt, Rinehart and Winston.

[6] Cassell, J. (2000). Embodied conversational interface

agents. Communications of the ACM, 43(4), 70 - 78.

[7] Fiske, J. (1982). Introduction to Communication Studies,Routledge, London.

[8] Laurel, B. (1993). Computers as Theatre, Addison-Wesley

Publishing Company, Inc.

[9] Maes, P., Darrell, T., Blumberg, B., and Pentland, A.

(1997). The ALIVE System: Wireless, Full-body Interaction

with Autonomous Agents. Multimedia Systems, 5(2), 105-112.

[10] Manninen, T. (2000). Interaction in Networked Virtual

Environments as Communicative Action - Social Theory and

Multi-player Games. In proceedings of CRIWG conference,

Madeira, Portugal, IEEE Press, 154-157.

[11] Manninen, T. (2001a). Virtual Team Interactions in

Networked Multimedia Games - Case: "Counter-Strike" - Multi-

player 3D Action Game. Workshop on Presence, Philadelphia,

USA.

[12] Manninen, T. (2001b) Rich Interaction in the Context of

Networked Virtual Environments - Experiences Gained from theMulti-player Games Domain. In proceedings of IHM-HCI, Lille,

France, Springer-Verlag, 383-398.

[13] Manninen T. (2002) Interaction Forms in Multiplayer

Desktop Virtual Reality Games. In Proceedings of Virtual

Reality International Conference, June 19-21, Laval, France. In

press

[14] Manninen, T., and Kujanp, T. (2002). Non-Verbal

Communication Forms in Multi-player Game Session. In

proceedings of HCI, London, UK. In press.

[15] Masterson, J. (1996). Nonverbal Communication in Text

Based Virtual Realities, Master of Arts Thesis, University of

Montana.

[16] Robertson, T. (1997). Cooperative Work and Lived

Cognition: A Taxonomy of Embodied Actions. European

Conference on Computer Supported Cooperative Work, 205-

220.

[17] Thalmann, D. (2001). The Role of Virtual Humans in

Virtual Environment Technology and Interfaces. Frontiers of

Human-Centred Computing, Online Communities and Virtual

Environments, R. Earnshaw, R. Guejd, A. van Dam, and J.

Vince, eds., Springer-Verlag, London, 27-38.

[18] Vilhjlmsson, H. H., and Cassell, J. (1998). BodyChat:

Autonomous Communicative Behaviors in Avatars. Second

International Conference on Autonomous Agents, Minneapolis,

MN, US, 269-276.


conceptual, communicative and pragmatic aspects of interaction forms - rich interaction model for...

Documents