rendering and affect - the university of edinburgh€¦ · tual environments. perceptual studies...

EUROGRAPHICS 2003 / P. Brunet and D. Fellner(Guest Editors)

Volume 22 (2003), Number 3

Rendering and Affect

D.J. Duke1 P.J. Barnard2 N. Halper3 M.Mellin3

1Department of Computer Science, University of Bath, Bath, UK.2MRC Cognition and Brain Sciences Unit, Cambridge, UK.

3Department of Simulation and Graphics, Otto-von-Guericke University, Magdeburg, Germany.

AbstractPrevious studies at the intersection between rendering and psychology have concentrated on issues such as realismand acuity. Although such results have been useful in informing development of realistic rendering techniques,studies have shown that the interpretation of images is influenced by factors that have little to do with realism. Inthis paper, we summarize a series of experiments, the most recent of which are reported in a separate paper, thatinvestigate affective (emotive) qualities of images. These demonstrate significant effects that can be utilized withininteractive graphics, particularly via non-photorealistic rendering (NPR). We explain how the interpretation ofthese results requires a high-level model of cognitive information processing, and use such a model to account forrecent empirical results on rendering and judgement.

Categories and Subject Descriptors (according to ACM CCS): I.3.m [Computer Graphics]: Miscellaneous

1. Introduction

Psychophysical experiments into colour perception and vi-sual acuity are ongoing in computer graphics. They pro-vide insight into human perception that informs develop-ment of new methods for realistic rendering, and an empiri-cal methodology for assessing progress in areas such as vir-tual environments. Perceptual studies have also underpinnedsome developments in non-realistic rendering (NPR). Thisincludes the use of NPR to convey shape and detail in tech-nical and artistic illustration10, 21, and work in visualization,for example techniques to differentiate regions14. NPR re-search also draws on wider sources of inspiration. Analysesof the perceptual merits of artistic techniques have under-pinned new rendering styles for multi-dimensional data16,and are currently an area of active research9.

Communication, however, is about more than just percep-tion. The use of NPR raises issues about interactions withinhigher-levels of cognitive process. For example, sketch ren-dered images of architectural designs have been shown toresult in qualitatively different dialogue between architectsand clients, compared with that elicited through more real-istic images19. This effect is not just a matter of perception,but involves also levels of cognition connected with judge-ment and articulation. In psychological terms, the image hasan affective content, separate from its structural detail. Sec-

tion 2 of this paper summarizes a series of empirical resultsthat suggest this is not an isolated effect.

Individual results do not in themselves provide insightinto why such effects occur, or how they might be utilizedwithin graphics applications. Our first contribution, in Sec-tion 3, is to provide an explanation of these results, using theconcept of invariants to establish a link between propertiesof representations and the cognitive resources involved intheir interpretation. The significance of this link for com-puter graphics is highlighted in Section 4, in which we sum-marize key results from a series of recent experiments as-sessing the effect of NPR style on judgements. These results,details of which have been reported in a separate paper12,provide empirical evidence that affective content can be usedto guide choice and decision-making in interactive graphics.Using the model developed in Section 3, we are able in Sec-tion 5 to explain the nature and variability of the invariantsaccessible through NPR. Through this, we gain insight intohow to scale these results up to graphics applications.

Running through the paper is an argument that to under-stand how rendered images are used requires models thatcan accommodate not only perception, but also the dynam-ics of how meaning is processed in a task environment. Theconclusion sets out a case for how such knowledge can con-tribute to open challenges in computer graphics.

c© The Eurographics Association and Blackwell Publishers 2003. Published by BlackwellPublishers, 108 Cowley Road, Oxford OX4 1JF, UK and 350 Main Street, Malden, MA02148, USA.

Duke, Barnard, Halper and Mellin / Rendering and Affect

2. Connotation in Graphics

Vision is one of five sensory modalities that we have evolvedto (firstly) survive in and (secondly) make sense of our envi-ronment. Computer graphics provides the capability of syn-thesizing virtual environments, and in describing user inter-action with and within such environments it is usual to thinkin rational, reflective terms: how well can features of the en-vironment be discerned, how realistic do they appear, andhow effectively does the environment support specific tasks.

Our understanding of the world is not based just on sur-face phenomena perceived directly. It also involves deeperlevels of representation that capture and reflect relationshipsand regularities at higher levels of abstraction. One accessi-ble example of these is in art forms, the use of onomatopoeiain poetry, or the ability of cartoonists to convey meaning andemotion through figures that bear little resemblance to theirinspiration. An understanding of such interactions involvesthe concept of invariant, developed by the work of J.J. Gib-son (see5 for an overview). An invariant is a property com-mon to (shared by) a range of entities or behaviours. Threeaspects of invariants are significant here:

1. they can reflect stable properties of entities, and proper-ties of behaviours over time;

2. they can apply across a range of abstraction levels;3. they can be restricted in some way by the context (culture,

task, or environment);

Even within a single modality such as vision, human re-sponse is affected by qualitatively different kinds of invari-ants extracted from the incoming data streams. Only someof these invariants correlate with what might be thoughtof as reasoned responses. For example, Figure1 contains asketch of a radar display, originally due to Provins et al18.When asked to glance at the display and immediately iden-tify which group (circles or triangles) of aircraft are the‘hostile’ forces, nearly all subjects identify the triangularsymbols18, 12. This is not a reasoned response, there is noovert information in the scene that ‘triangles’ are hostile. Itis, rather, an example of an affective (emotional) response.

Figure 1: Radar Display (after Provins et. al., 1957)

It has been argued2 that such responses utilize latentknowledge, in this case an invariant of sharp shapes invokesa connotation of threat. In evolutionary terms, rapid reactionto sharp shapes (e.g. in the form of fangs) may have con-veyed a survival advantage, and become encoded into the

neural architecture. The link between cognition and invari-ants forms a substantial part of this paper.

Whereas Gibson viewed invariants as relationships ‘in theworld’, our notion is more general, encompassing relation-ships and regularities within and across a range of men-tal representations. The pattern of properties captured bysuch invariants can be multimodal, drawing together percep-tual,cognitive and affective attributes. An example of this isdemonstrated by the following experiment, due originally toDavis7. When asked to associate the shapes in Figure 2 withthe sound of two words, ‘Ulloomo’ and ‘Takete’, most peo-ple associate ‘Takete’ with the shape on the left of the figure.

Figure 2: Two connotative shapes (after Davis, 1961)

Davis’ result can be explained by positing that certain vi-sual characteristics (i.e. invariants) of the shape match in-variants of the sound. When asked to explain the correspon-dence, subjects find it difficult to articulate why these qual-ities match; this is a characteristic of latent knowledge, itis automatic, and not necessarily subject to conscious intro-spection. Further, response to latent invariants is extremelyrapid, as shown by Guthrie and Weiner. In their experiment11

subjects were asked to make character judgements based onan outline image of a seated figure. The subjects were shownan extremely brief, subliminal image of a complete outline,followed, for a longer duration, by an outline with missingsegments. The study showed that the figure was judged asmore hostile and aggressive when the subliminal image con-tained angular features, as opposed to curved features.

How invariants come into play depends on the resourcesavailable to the human information processor. An examplecan be seen in studies of behavioural attribution. If a subjectunfamiliar with brownian motion is shown an instance ofthis phenomenon, they will often interpret it as, for example,a ‘lot of bugs’; the random motion of the particle ensembleis an invariant that maps to a familiar behaviour, in this casea cloud of insects. However, for someone with experienceof brownian motion, the invariant properties of the motioncan also be mapped to that internal knowledge. Similarly,the Heider-Simmel experiment13, in which viewers providean account of interaction between abstract shapes, elicits be-havioural invariants (e.g. ‘one thing chasing/controlling an-other’) that are mediated by cultural expectations.

The experiments described above show invariant proper-ties in the display being mapped to judgements, but involveonly small-scale displays and low-level tasks. More recentresearch has shown that invariants linked to rendering style

c© The Eurographics Association and Blackwell Publishers 2003.

360


also result in qualitatively different responses in larger-scaletasks, involving higher levels of judgement. A study on therole of sketch rendering in architectural CAD19 was men-tioned in the introduction. It has also been noted that sketchcharacteristics can be used to draw attention to/from partic-ular parts of a scene20. Taken together, these results suggestthat there are qualities of images, independent of overt struc-ture, that influence interpretation. These qualities involve in-variants that are not bound to a particular modality, but rathercome into play at a high level of cognitive process, at whichaffect and ideation interact. In the next section we describehow such invariants impact cognition; later in the paper wevisit results that show how such invariants arise from, andcan be controlled through, NPR styles.

3. Rendering Style and Affect

Early psychological experiments, described in Section 2,show that graphic displays carry more than just structuralinformation (i.e. ‘that object is there’). Images also evoke alatent, affective response. Recent experiments, summarizedin Section 4, have shown that such effects (a) are not limitedto simple images and tasks, and (b) can in principle be uti-lized to guide high-level judgements. By themselves, theseresults are isolated data. What we need is a basis for ‘in-terpolating’ from these experiments, understanding what iscausing these effects, how they might be deployed in graph-ics applications, and what freedom is available to designersin manipulating their use. In other words, experiments areonly useful in so far as they lead to a generalizable under-standing of the underlying phenomena.

Use of latent knowledge, including affect, has previouslybeen explored in the context of interactive systems2. Thissection returns to the theoretical framework used in that pa-per, and demonstrates how it accounts for the newer resultssummarized in the previous section. The concept of invari-ant, introduced in Section 2 is useful in this context, in thatserves as a ‘common currency’, bridging between comput-ing and psychological models. Invariants can be understoodboth as properties of graphics, and as phenomena utilizedin cognitive processing. By understanding the operation andprinciples of this processing we can better appreciate howsuch invariants can be utilized in graphics applications.

3.1. Interacting Cognitive Subsystems

Cognitive science attempts to understand the capabilities andoperation of the mind in terms of its ability to process in-formation. One form that this understanding takes are mod-els, at various levels of detail, that posit mechanisms bywhich observed phenomena can be explained. These obser-vations include narrowly focused experiments, clinical re-sults, and more recently detailed neurophysiological studiesof the mind operating in vivo. Because of the large number ofobservations available, differences in methodology, and dif-ferent approaches to modelling, there is no single ‘model’ of

cognition. Rather, there are a collection of models, account-ing for a different range of phenomena. Within this diversity,however, a number of trends have emerged. One is that cog-nition is a distributed process; damage to (say) one regionof the brain does not completely destroy one kind of faculty,and may leave other faculties untouched. Similarly, differentareas of the brain process particular kinds of information.

Interacting Cognitive Subsystems2, 3 (ICS) is one of anumber of models that provide a comprehensive account ofhuman information processing (EPIC15 is another). It is oneof the more developed models that can account for and ex-plain a broad range of phenomena, including task perfor-mance and affective properties4, 1. This breadth of applica-tion, and in particular its treatment of affect, motivates its usehere. In ICS, cognition is viewed as an activity distributedbetween nine subsystems, shown in Figure 3.

art¬

lim¬

implic¬

bs

som

visc

®implic

®mplac

®prop

®artmpl

®type

®speechart

vis ®obj

®implic ®arm

®handlim

®mpl

®prop

®limobj

®som

®prop

®viscimplic

®obj

®implic

®mplprop

Figure 3: ICS Architecture

Briefly, these nine systems are:

Sensory subsystemsVIS visual: hue, contour etc. from the eyesAC acoustic: pitch, rhythm etc. from the earsBS body-state: proprioceptive feedback

Structural subsystemsOBJ object: mental imagery, shapes, etc.MPL morphonolexical: words, lexical forms


361


Meaning subsystemsPROP propositional: semantic relationshipsIMPLIC implicational: holistic meaning

Effector subsystemsART articulatory: subvocal rehearsal and speechLIM limb: motion of limbs, eyes, etc

A feature of the model is its distinction between two lev-els of meaning, propositional and implicational, a featuredemonstrated recently through experiments using affectiveproperties derived from auditory perception22. The proposi-tional level is concerned with referential knowledge, that is‘facts about the world’, including properties of and relation-ships between entities. In the case of a scene from a com-puter game, this is facts such as ‘another player is standingnext to a door’, or that ‘the door is opening’. In contrast,the implicational level is concerned with schematic models,capturing among other things ideational and affective con-tent. In the game scenario, representations at this level wouldinclude plans that associate passing through the door withachieving some goal, and also awareness of any threat posedby the second player in the scene. Implicational level there-fore involves a higher set of invariances. This is the level ofknowledge representation that we need to be concerned with,as it is here that invariants related to affect and the analysisof consequences come together.

All subsystems have the structure shown in Figure 4. In-coming data streams arrive at an input array, on which anumber of processes operate in parallel. A single ‘copy’ pro-cess in each subsystem takes the incoming data and laysdown a record of this in the image record, an episodic mem-ory. Other processes are working on the input array, tryingto transform the available representation into other codes.Here and in the remainder of the paper, the notation ‘A-B’is used to denote the process in subsystem ‘A’ that trans-forms its input into representations for subsystem ‘B’; thus‘obj-prop’ is the process in the object subsystem that pro-duces propositional-level representations. For transforma-tions such as obj-prop that produce a higher-level represen-tation, this involves the abstraction of particular invariantsfrom the input. In the case of ‘obj-prop’, the processes is at-tempting to output a semantic interpretation of spatial (struc-tural) invariants over its input. Transformations producinga more ‘refined’ code, for example obj-lim, involve imple-menting abstract invariants in terms of lower-level capabil-ities, for example instantiating the muscular control neededto position a limb at some point in space.

In general a subsystem is receiving data streams from avariety of sources. Only some combinations of source arecoherent, that is, the invariants within the incoming repre-sentations are compatible. Transformations can only oper-ate on coherent products, and if none are available the out-put may become unstable and effectively unusable by otherprocesses8. If the incoming data is insufficient to form a sta-ble response, a transformation can disengage from the in-

®code B

®code C

®code Aincomingrepresentations

imagerecord

copyprocess

transformationsinput array

Figure 4: Subsystem

put array and instead take input from the image record. Inthis mode of processing, the transformation can operate ei-ther on the proximal end of the record, the data most re-cently copied from the input array, or on data revived fromdeeper within the experiential record. Either way, enteringthis ‘buffered’ mode has a significant effect on the trans-formation. Buffering brings in additional attributes that of-fer greater affordances for subsequent processing. Whena process is buffered, an experimental subject will be fo-cally aware of the extended attribute space associated withthat subsystem, and the effect of any invariants in the inputstreams may be influenced by patterns of invariants formedover long time scales or involving different combinations ofattribute. In contrast, in non-buffered (direct) mode, the ef-fect of invariants is exerted in the background, but will dom-inate processing. Significantly, a constraint on the architec-ture is that, within a configuration, only one process at a timecan be operating in buffered mode.

3.2. Affective and Structural Graphics

Although all ICS processes are attempting to operate on in-coming data, the products of only some of these processesare relevant to the current task, and/or are linked to focalawareness. The collection of processes that are deployed ona given task is called a configuration. Figure 5 shows a con-figuration corresponding to a task involving interpretation ofa graphical display, using a thick black line to show the flowof representations between the processes involved:

In this configuration there are processes extracting ob-jects from the retinal image (vis-obj), identifying structuralrelationships (obj-prop), and utilizing those relationships ininterpreting the situation relative to goals and plans (prop-implic). Essentially all of these processes are using invari-ants derived from the structure of the graphic. If the subject’sattention is focused on the meaning of the task, buffering willbe located at the propositional level.

In parallel with this structure activity, the vis-implic pro-cess is also operating, building a high-level schematic modeldirectly from the raw visual input. This gives rise to a secondcomponent of the configuration, shown in Figure 6 by a greyline, connecting vis to implic.


362


art¬

lim¬

implic¬

bs

som

visc

vis ®obj

®implic ®arm

®handlim

®mpl

®prop

®lim

obj

®som

®prop

®viscimplic

®obj

®implic

®mplprop

Figure 5: Visual Processing Configuration

In early organisms with a simpler cognitive mechanism(essentially just sensory and implicational systems), sucha model might indicate a basic ‘fright’ response on thedetection of hostile features, or an aggressive response inthe presence of food! In our evolved form, this stream isonly one voice in a fugue. What it does do, however, isto provide an affect-related invariant to the implicationalsystem. In the case of the radar display (see Figure 1), ratherthan responding just on the basis of the structural content(‘circles’ and ‘triangles’), processes in the implicationalsystem, operating in direct mode, are accessing an inputarray in which the invariants of a ‘threat’ schema (from vis)are correlated with spatial and task-relevant properties of thedisplay (from prop). When judgement is elicited, the blend-ing of the triangle/threat invariants leads to propositionaloutput, (implic-prop, carried on the second grey flow inFigure 6), that identifies the triangles with the threat. Manyprocessing tasks require reciprocal exchange of informationbetween the propositional and implicational levels, and theexistence of such data flows, particularly when coupled withbuffering moving between the prop and implic levels, isreferred to as the ‘central engine’ of cognition.

The results of Davis’ connotative shape experiment, alsodescribed in Section 2, are derived by a similar mechanism,this time though involving blending between productsderived from both visual and acoustic sources. The flowscarrying the affect-related invariants are shown by blacklines in Figure 7. The possibility of affective invariantsderived from body-state information (bs-implic) is alsoshown. For example, if a subject was asked to touch anobject and select a word (or visual shape) that corresponds

art¬

lim¬

implic¬

bs

som

visc

vis ®obj

®implic ®arm

®handlim

®mpl

®prop

®lim

obj

®som

®prop

®viscimplic

®obj

®implic

®mplprop

Figure 6: Affect in the Configuration

to the object, it would be expected that a sharp object such asa thorn would map to triangles/‘takete’, while a soft object,e.g. a tennis ball, would map to rounded shapes/‘ulloomo’.Although we cannot be exact, we note in this examplethat abrupt changes in form or constituents are amongthe features shared by one class of representation, but arelacking in the second. In interpreting the meaning of images,the central engine also has available structural informationcarried on the data streams vis-obj and obj-prop, shown bygrey lines in Figure 7. These too generate invariants at theimplicational level, via prop-implic. Thus invariants formedat implic involve correlation over general patterns within thesensory and proprioceptive environment, coupled with thosederived from referential meaning. Consequently the impli-cational level is critical for utilizing and matching perceptualinvariants, and that multimodal patterns of invariants form anessential part of the process of interpreting rendered images.

4. Evidence of the Nature and Variability of Invariantsin NPR

The results presented in Section 2 form the baseline for arecent series of experiments designed to explore how ren-dering style affects users’ decisions and judgements withininteractive graphics. The experiments also demonstrate thatthe effects of the type described above are not accidental,and, in particular, that they can be accessed and controlledthrough NPR techniques, for example stylized silhouettes.

Empirical results from nine experiments are reported andinterpreted in a separate paper12. In each experiment, thesubject has, under time pressure, to make a judgement based


363


art¬

lim¬

implic¬

bs

som

visc

®implic

®mplac

®prop

®artmpl

®type

®speechart

vis ®obj

®implic ®arm

®handlim

®mpl

®prop

®lim

obj

®som

®prop

®visc

implic

®obj

®implic

®mplprop

Figure 7: Blending of affect

on a displayed image. Subjects’ response and response timesare recorded, along with demographic data (including gen-der and first language, as the subject population coveredmultiple nationalities). These confirm that rendering stylecan have a significant effect on interpretation, and that thiseffect can be controlled. Some of the theory needed to ex-plain these effects has been set out in the previous section,but there are a number of results that point to more compli-cated interactions between levels of cognitive process, withimportant consequences for the use of rendering style. Theexperiments are summarized below; unless otherwise noted,all results are statistically significant.

4.1. Assessment of Danger and Safety

The example of aircraft representation18 was the startingpoint for three hypotheses exploring the jaggedness of con-tour as an invariant to denote danger.

• Experiment (a): A radar mock-up, similar to that shownin Figure 1, was displayed, and subjects were asked todetermine on which side the hostile forces were located.(This experiment was intended to re-establish the resultsof18, and provide a test for our methodology.)Result: 84% of subjects selected the side marked by tri-angles as being hostile.

• Experiment (b): Generalizing (a), it was hypothesized

that sharp contours carried an invariant connoting a gen-eral sense of threat. Subjects were presented with threerepresentations of a door, one in a simple outline, oneoutlined with jagged edges, and one with a wavy outline.They were told that there was danger behind one of thedoors, and were asked to identify which door this was.Result: 67% of subjects selected the jagged-contour door.

• Experiment (c): The hypothesis here was that the sharpcontour invariant could be used to connote danger, evenfor objects for which there might normally be a latent cul-tural invariant mapping to safety. Subjects were shownone of three images. Each image consists of a house onthe left, and a clump of trees on the right. They were askedto identify the safest location. The rendering of the houseand tree were varied: in the first image the house and treesare both rendered with no effects; in the remaining two,one of the house and trees are drawn with jagged edges.Result: Given a simple house and tree, 87% of subjectsselected the house. This corresponds to a cultural invari-ant of houses as ‘safe’. When the tree was rendered withjagged edges, 88% selected the house for safety; however,when the house was rendered with jagged edges, 81% se-lected the trees for safety. This demonstrated that sharp-ness can be used to connote threat, and this connotationcan override a cultural safety invariant.

4.2. Assessment of Strength and Weakness

Tests were conducted to investigate whether silhouette thick-ness is an invariant that can be mapped to an attributionof ‘strength’ or ‘weakness’ of graphical objects. Althoughthese invariants are not in themselves forms of affect, the in-terpretation required for a positive outcome requires interac-tion with affect-related invariants at the implicational level.

• Experiment (a): It was hypothesized that a dotted silhou-ette would carry a connotation of weakness compared toa thick, solid silhouette. Subjects were shown a radar dis-play with two groups of friendly and hostile forces, withone group of each rendered with a solid silhouette, andone group with a dotted silhouette. They were asked todetermine which of the friendly groups was the weaker.Results: 88% of subjects selected the dotted circles asrepresenting the weakest friendly force.

• Experiment (b): Subjects were shown three images ofmen, identical apart from silhouette strength, and wereasked to judge which was the strongest. The silhouettesused were dotted, normal, and thick. The hypothesis wasthat the image drawn with the thick silhouette would beidentified as the strongest.Results: Subjects selected the normal and strong silhou-ette figures as stronger, at levels above what could beattributed to chance. Reasons for the split between thethick and normal contours are suggested in12. This ex-periment shows how judgements can be constrained byfactors other than display invariants; further cognitively-grounded issues are raised in the next section.


364


• Experiment (c): The images from test (b) were re-used,with subjects now asked to identify the weakest man. Thehypothesis was that the man rendered with a dotted sil-houette would elicit the most responses.Results: 67% of subjects selected the dotted silhouette asbeing the weakest figure.

4.3. Goal-directed Interaction

The third set of tests investigated goal-related judgementsconcerning selection between alternative paths and objects.In this context, ‘approaching’ an entity represents a positiveprimitive of gain, avoiding an entity or path reflects a senseof indifference. As inquisitiveness is a key skill for younganimals, there is a rationale for thinking that invariants ofdetail or content will operate latently.

• Experiment (a): Subjects were shown an image of ascene with two divergent paths. One path was renderedin a high level of detail (LOD), the second at a low level.Subjects were asked to choose which path to explore first;it was hypothesized that the high LOD path would be cho-sen. To control for culturally-based preferences6, subjectswere divided into two groups, with one group shown themirror image of the scene.Results: The population of first-language speakersshowed a strong inclination (74%) to choose the highLOD path. Second-language speakers show a muchhigher variation and their results do not exhibit this bias.This result is important, in that differences between thecognitive processes deployed by these groups can provideguidance on how and where NPR effects can best be uti-lized. Further details of the empirical test results can befound in the interpretation section of12.

• Experiment (b): Subjects were shown a scene with twopaths in the foreground, and a path in the distance leadingto a house. They were asked to judge which of the twopaths, would lead to the house. Again, the paths differedin the level of detail at which they were rendered, and twogroups of subjects were used, with the image being mir-rored between groups. The hypothesis was that the highLOD path would be preferred.Results: As in the previous case, first language speakersshowed a demonstrable preference for the high LOD path.

• Experiment (c): The third test was intended to explorethe relationship between rendering style and attention.Subjects were shown a scene of a shelf with twenty ob-jects. Ten objects were rendered in an NPR style, ten ina realistic style. Subjects were asked to choose three ob-jects, with the hypothesis that subjects would tend to se-lect the NPR-styled objects. As a control, for one groupthe background scene was rendered in an NPR style, forthe second the background was drawn realistically.Results: A significant majority of subjects selected twoor more NPR objects.

The rationale for hypothesizing that subjects would pre-fer the high LOD paths requires further elaboration. At onelevel, it might appear obvious that more detail is ‘more in-teresting’, but this just pushes back the question to one ofwhy detail should be interesting, that is, what invariants arepresent within the detailed pathways. Our explanation isconcerned with rate-of-change. Data streams derived from‘regions’ of high variation (detail) will exhibit a high rate ofchange, as attention passes through the structure. High ratesof change can be linked to excitement or arousal; dually, lowrates of change with indifference. Inquisitiveness, mentionedearlier, is triggered by change.

These results create a challenge for explanation, one that astraightforward analysis of low-level perceptual mechanismscannot provide. They represent a significant bridge from thepsychological phenomena described in Section 2 to open is-sues of how and why to utilize non-realistic rendering incomputer graphics.

5. Invariants, Meaning, and Cognitive Mode

Variations in the experimental results described above, inparticular the effect of whether subjects were first or sec-ond language speakers, raise the question of how specificinvariants are used to meet subjects’ psychological require-ments, and how different types of invariant are deployed inperforming a given task. Our analysis in Section 3 of earlyresults explained how latent, implicational knowledge is re-quired for successful completion. However, within the cog-nitive system, content-related data streams, derived from thestructural systems (obj and mpl) will be co-present with theimplicational streams. The presence of these streams pro-vides an explanation for how and why subjects responsesdiffer from the expected. In particular, we can account forthe effect of second-language speakers on the recent path-selection experiments, described in Section 4.2.

5.1. Mode Oscillation

Recall from Section 3.1 that a process will try to enter abuffered mode of processing when it is unable to produceusable output directly from the available input streams. Wehave previously supposed that in the context of the test,buffering will be located at the propositional level, with di-rect processing of invariants at the implicational level (Fig-ure 8, left). However, in two situations this configurationcannot be sustained:

• If subjects have difficulty interpreting the task require-ments, central engine activity will be required to correlateperceptual and task invariants. This requires schematicjudgements, involving implicational processing.

• The vis-obj and obj-prop processes are making a semanticrepresentation of the scene available at the propositionallevel. However, this stream carries no overt informationsuitable for completion of the task. Consequently, if the


365


art¬

lim¬

implic¬

som

visc

®som

®prop

®viscimplic

®obj

®implic

®mplprop

from vis-implic from obj-prop

art¬

lim¬

implic¬

som

visc

®som

®prop

®viscimplic

®obj

®implic

®mplprop

from vis-implic from obj-prop

buffering at implic

buffering at prop

Figure 8: Oscillation in the Central Engine

prop-implic process engages with this stream, the impli-cational process will be unable to reconcile this (unstable)stream with the ongoing, high-level task description.

In either case, users will be trying to compare (inadequate)propositional models against implicational judgements. Asa result, the implic-prop process will also seek to operate inbuffered mode, with the buffer moving to the implicationallevel (Figure 8, right). However, implic-prop is then operat-ing on invariants in a much richer model than the incomingstreams accessed in direct mode, and further, automatic pro-cessing, including key latent responses, is suspended. And asimplic-prop is then unable itself to generate a task-relevantoutput, unstable input arrives at the propositional level. Con-sequently, as shown in Figure 8 the system configuration os-cillates over time, with the ‘central engine’ trying to resolvea task through overt reasoning, but failing to access the rel-evant latent knowledge. Under pressure of time to completethe task, subjects may then fall back on an arbitrary choice.

Experimental support for this scenario can be found in theempirical results12. In the study of radar targets, subjects se-lecting the triangles as the hostile forces tended to have sig-nificantly shorter response time than subjects who selectedthe circles. In the case of a latent response, subjects will beutilizing the vis-implic stream through direct mode. Just aswith Guthrie and Wiener’s experiment11, this should enablea rapid response. In contrast, subjects consciously attendingto the structural stream will override the latent response andenter the oscillatory configuration described above. With no‘rational’ reason for selecting one or other of the shapes ashostile, any response will tend to be delayed by the longercycle of processes involved.

5.2. Belief and Focal Awareness

ICS provides insight into an experiment carried out in con-junction with those described in Section 4, but not reportedin12 due to lack of statistically significant results.

The image in Figure 9 is from an experiment intendedto explore whether NPR images convey a high-level invari-ant connected a sense of disbelief. This was motivated byobservations of cartoon art. In cartoons, characters can andare made to perform acts which completely violate normalexpectations derived from real-world experience. Yet view-ers watch and engage with such features without a sense ofdisbelief that might be engaged if realistic characters andscenery were present. In the experiment, subjects are dividedinto two groups, with one group given the upper, realistic,image in Figure 9, and the other the lower, NPR, image. Eachsubject is then asked whether the character portrayed wouldbe likely to succeed in jumping between the buildings.

In ICS terms, the difficulties with this test stem from at-tention being drawn to the structural detail of the scene, and(consequently) focusing awareness on matching the seman-tics of the scene with the task description. For exactly thereasons given above, this increases the likelihood that buffer-ing at implic will bring propositionally-derived invariantsinto play. The difference with the situation described aboveis that in the earlier example the task was sufficiently clearto first language speakers that they could perform withoutimplicational buffering, whereas the jump analysis is impos-sible to resolve satisfactorily for any sample population. Abetter approach would be to set up an assessment that drewattention to some secondary task, leaving assessment of thephysical circumstances to latent implicational knowledge.For example, asking whether the character would be ableto retrieve some object (which happened to be located onthe other side of the gap) would encourage buffering at thepropositional level, allowing direct mode processing of anyimplicational invariants.

5.3. The Application of Affect

As we have discussed, for latent knowledge to be utilizedeffectively, the user should not be consciously attending to


366


Figure 9: ‘Suspension of Belief’ Images

affective content of streams. That is, it is desirable that in-variants within the external representation that activate latentknowledge should not be modified by invariants extracted bycultural experience or propositional reflection.

There are reasons to believe that these conditions aremet in a number of computer graphics applications, specifi-cally games and virtual environments. These combine task-centered activity with a rich visual environment that providesa sense of immersion and presence. In the case where at-tention is drawn into the visual environment, buffering willmove out of the central engine (implic and prop), and into thesystems involved in visual interpretation (vis and obj). Withthe central engine processes ‘on automatic’, allowing impli-cationally derived invariants to contribute maximally. Andindeed, our experiments12, summarized in Section 4, pointalready to two aspects of games and virtual reality whereimplicational invariants could be invaluable:

• Games, particularly the new generation of character-basedmulti-player environments, require users to engage inmonitoring their environment for threats and opportuni-ties, and to explore. For novice players in particular, agame is a large, complicated environment in which un-

familiarity may easily lead to disappointment. By usingaffect-related invariants in rendering a game, it should bepossible to provide cues about levels of threat posed byentities within the game, as well as guiding players, whenappropriate, towards useful or safe areas.

• Virtual environments are often a fa cade, presenting theappearance of a detailed environment, but in which theapparent capabilities of realistically rendered objects arein practice matched by few actually capabilities. By us-ing NPR rendering, it should be possible to unobtrusivelyguide attention away from imposter objects, towards task-relevant or usable features of the environment.

By using rendering style to convey meaning through theimplicational level, we support a mode of processing inwhich the more implic contributes, the more likely it is thata wide range of propositional material will be generated andtaken into account within central engine processing. In aconfiguration where central processes are able to operate di-rectly, the buffer (and focal awareness) can operate at the vi-sual and object levels, supporting the sensation of presencethat is a major design objective in these application areas.

6. Conclusions

This paper makes contributions to computer graphics at twolevels. The first level is the direct contribution. We have setout examples of how rendering style can convey meaningand influence judgement, we have demonstrated (throughreference to recent experiments) that these effects are con-trollable, and have provided a generalized account that ex-plains why these effects occur, how they are subject to vari-ation, and how they might be applied in practice.

At the second level, the paper calls for a re-appraisal ofthe relationship between graphics and psychology. Tradi-tionally, this has been focused at the level of psychophysics.The most important point of our analysis is that the effectswe describe are outside the realm of low-level perception.They involve semantics, affect, and other high-level invari-ants such as rate-of-change. Worrying solely about the per-ceptual adequacy or realism of an image is to miss the factthat meaning is ultimately interpreted at higher levels of cog-nitive process, and involves the integration of more than juststructural invariants. Understanding the meaning carried byrendered images requires broader approaches to human in-formation processing. Significantly though, thinking abouthigher levels of processing does not require thinking in theabstract: effects that exist at this level are measurable, andare accessible through empirical experiment.

There are other links between graphics and psychologythat have not been touched on here, most importantly per-haps the phenomenon of ‘change blindness’ 17, which may inpart be related to the issue of indifference to streams exhibit-ing low rates of change. There are also links to be exploredwith the use of sound in virtual environments; as the ul-


367


loomo/takete example shows, sound also has affective prop-erties, something which cinema uses to good effect in creat-ing and sustaining emotional states. As ICS includes acous-tic processing, it provides a natural route for exploring thetheoretical basis for multimodal blending.

It is clear that a model is needed to underpin the develop-ment and deployment of NPR in a way similar to how psy-chophysics and models of human vision have supported de-velopment of realistic rendering. Such a model may remaina long term goal, but the contribution of this paper representa first step towards achieving it.

Acknowledgements

The first author gratefully acknowledges the support of anAdvanced Research Fellowship (AF/001275) from the UKEngineering and Physical Sciences Research Council.

References

1. P.J. Barnard. Asynchrony, implicational meaning andthe experience of self in schizophrenia. In A. David andT. Kircher, editors, The Self in Neuroscience and Psy-chiatry. Cambridge University Press, 2003. In press.

2. P.J. Barnard and J. May. Interactions with advancedgraphical interfaces and the deployment of latent hu-man knowledge. In Eurographics Workshop on De-sign, Specification and Verification of Interactive Sys-tems, pages 15–49. Springer, 1995.

3. P.J. Barnard and J. May. Representing cognitive ac-tivity in complex tasks. Human Computer Interaction,14:93–158, 1999.

4. P.J. Barnard and J.D. Teasdale. Interacting cogni-tive subsystems: A systematic approach to cognitive-affective interaction and change. Cognition and Emo-tion, 5(1):1–39, 1991.

5. V. Bruce, P.R. Green, and M.A. Georgeson. Visual Per-ception: Physiology, Psychology, and Ecology. Psy-chology Press, third edition, 1996.

6. S. Chokron and M. De Agostini. Reading habits influ-ence aesthetic preference. Cognitive Brain Research,10(1–2):45–49, 2000.

7. R. Davis. The fitness of names to drawings. BritishJournal of Psychology, 52:259–268, 1961.

8. D.J. Duke, P.J. Barnard, D.A. Duce, and J. May.Syndetic modelling. Human Computer Interaction,13(4):337–393, 1998.

9. F. Durand, M. Agrawala, B. Gooch, V. Inter-rante, V. Ostromoukhov, and D. Zorin. Per-ceptual and artistic principles for effective com-puter depiction. SIGGRAPH 2002 Course. Seehttp://graphics.lcs.mit.edu/~fredo/SIG02_ArtScience/.

10. B. Gooch and A. Gooch. Non-Photorealistic Render-ing. AK Peters Ltd., 2001.

11. G. Guthrie and M. Weiner. Sublimal perception or per-ception of partial cue with pictorial stimuli. Journalof Personality and Social Psychology, 3(6):619–628,1966.

12. N. Halper, M. Mellin, D.J. Duke, and T. Strothotte.Implicational rendering: Drawing on latent humanknowledge. Submitted to ACM Transactions on Ap-plied Perception, 2003. Test details available atwww.opennpar.org/userstudies.

13. F. Heider and M. Simmel. An experimental study ofapparent behaviour. American Journal of Psychology,57:243–259, 1944.

14. V. Interrante. Harnessing natural textures for multi-variate visualization. Computer Graphics and Appli-cations, 20(6):6–11, 2000.

15. D.E. Kieras, D.E. Meyer, S. Mueller, and T. Sey-mour. Insights into working memory from the per-spective of the EPIC architecture for modelling skilledperceptuo-motor and cognitive human performance. InA. Miyake and P. Shah, editors, Models of WorkingMemory: Mechanisms of Active Maintenance and Ex-ecutive Control. Cambridge University Press, 1999.

16. D.H. Laidlaw, R.M. Kirby, and H. Marmanis. Visu-alizing multivalued data from 2d incompressible flowsusing concepts from painting. In D. Ebert, M. Gross,and B. Hamann, editors, IEEE Visualization’99, pages333–340. IEEE Computer Society Press, 1999.

17. J. May. Perceptual principles and computer graphics.Computer Graphics Forum, 19(4):271–279, 2000.

18. K. Provins, H. Stockbridge, D. Forrest, and D. Ander-son. The representation of aircraft by pictorial signs.Occupational Psychology, 31:21–32, 1957.

19. J. Schumann, T. Strothotte, A. Raab, and S. Laser. As-sessing the effect of nonphotorealistic rendered imagesin CAD. In R. Bilger, S. Guest, and M.J. Tauber,editors, Proc. Computer Human Interaction (CHI’96),pages 35–42. ACM Press, 1996.

20. T. Strothotte, B. Preim, A. Raab, J. Schumann, andD.R. Forsey. How to render frames and influence peo-ple. Computer Graphics Forum, 13(3):C–455–C–466,1994. Conference issue: Proceedings of Eurograph-ics’94.

21. T. Strothotte and S. Schlechtweg. Non-PhotorealisticComputer Graphics: Modelling, Rendering, and Ani-mation. Morgan Kaufmann, 2002.

22. P.G. Walz and R.M. Rapee. Disentangling schematicand conceptual processing: A test of the interactingcognitive subsystems framework. Cognition and Emo-tion, 17(1):65–81, 2003.


368

rendering and affect - the university of edinburgh€¦ · tual environments. perceptual studies...

Documents