choosing tools for virtual environments

Leonardo

Choosing Tools for Virtual EnvironmentsAuthor(s): Dan O'SullivanSource: Leonardo, Vol. 27, No. 4 (1994), pp. 297-302Published by: The MIT PressStable URL: http://www.jstor.org/stable/1575997 .

Accessed: 15/06/2014 00:46

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .http://www.jstor.org/page/info/about/policies/terms.jsp

.JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range ofcontent in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new formsof scholarship. For more information about JSTOR, please contact [email protected].

.

The MIT Press and Leonardo are collaborating with JSTOR to digitize, preserve and extend access toLeonardo.

http://www.jstor.org

This content downloaded from 185.2.32.21 on Sun, 15 Jun 2014 00:46:43 AMAll use subject to JSTOR Terms and Conditions

http://www.jstor.org/action/showPublisher?publisherCode=mitpress

http://www.jstor.org/stable/1575997?origin=JSTOR-pdf

http://www.jstor.org/page/info/about/policies/terms.jsp


SPECIAL SECTION: VIRTUAL REALITY

Choosing Tools for Virtual

Environments

Dan O'Sullivan

he world we know is imaginary. Our imagination is constantly constructing a world from the blur of sen-

sory information that comes at us. Artists have always taken advantage of our ability to construct reality in order to coax us to imagine-at least temporarily-that something beyond our experience is real. They make use of various tools of illusion to place us in a scene, but must ask us to engage our

imagination to bring the scene to life. The artist places her audience's imagination in a given set-

ting by learning the nervous system's methods of collecting information. Equipped with the knowledge of the sensory av- enues to the imagination, the artist can intercept input from the natural world and substitute alternative worlds. Motion

pictures and perspective in oil paintings are two artistic tools for achieving this. However, placing a scene in front of a viewer is no guarantee that he or she will bring the scene into consciousness.

The artist also engages the viewer by seducing the imagination. Most people approach art with a willingness-if not a

strong desire-to participate, if only at the level of suspend- ing belief in their sensory world in order to be transported into another world. An unadorned textual narrative leaves most of our senses untouched, yet it engages the imagination because the reader wants it to. Often the best thing an artist can do is get out of the way of a healthy imagination.

In my work with new technologies, I try to find the best mix of placement and engagement of the imagination to bring people to places they have never visited. I was initially excited

by the possibility of complete sensory illusion, of virtual reali- ties. In making do with limited resources, I found that nonimmersive interactive media invite the audience to co-

produce the art, thus involving viewer's imaginations without

fully assaulting their nervous systems. Although I am still excited about virtual reality technology, it is often unnecessary and can be an obstacle to setting the imagination in flight. Now I am much more interested in facilitating the

imagination's creations of virtual worlds from sparse sensory props than in fooling the imagination by blanketing the nervous system.

My excitement with virtual reality technologies is evident in

my early work, such as Dan's Apartment and navigable scenes, which made use of shortcuts to achieve the illusion of pres- ence without the use of virtual reality technologies. In Being There with the Melons, I added a linear narrative to these nonimmersive illusions to further engage the audience. Yorb-

Dan O'Sullivan (educator), 446 East 20th Street, 1 IE, New York, NY 10009, U.S.A.

Received 20 October 1993.

Manuscript solicited by Kiersta Fricke.

An Electronic Neighborhood, an on-

going project, uses more strictly defined virtual reality technology such as real-time rendering to create the environment. However, I ask the audience to add to this environment by sending in media over the telephone line. Finally, my recent project for Apple Com-

puter, called Mirror Play, concen- trates on facilitating projections of the imagination rather than placing the imagination in immersive environments.

ABSTRACT

The author describes how virtual reality tools can allow greater access to the imagination and explains how he has chosen his tools for creating virtual environments such as navigable movies and interactive cable television shows. He discusses how his work is moving toward minimizing special effects in order to enlist the cooperation of the user's imagination. He is now making use of less-immersive interactive multimedia to create virtual spaces, rather than relying on more realistic and expensive tools for sensory illusion.

VIRTUAL REALITY I see virtual reality technologies as a natural extension of exist-

ing tools of illusion. Motion pictures exploit the fact that our

perceptual system can only see the world at a certain speed. The eyes take readings of the world as fast as they can, and the brain blurs or interpolates between readings to make a coher- ent whole. Thus, when a viewer sees a slightly changing se-

quence of still images proceeding faster than the eyes can read them, the brain interpolates movement between the images. It is difficult to defeat the illusion of movement in a movie the-

Fig. 1. Dan's Apartment, a cable television show produced with an

Amiga 2000 computer, Amigavision software, videodisc, and cus- tom voice recognition hardware, 1991. Viewers could direct a virtual camera view through my apax tment by issuing voice com- mands through touchtone telephones.

LEONARDO, Vol. 27, No. 4, pp. 297-302, 1994 297 ? 1994 ISAST



ater or on television by trying to see the actual still pictures. However, one can

easily prove that the movement is not

part of one's real world by testing the environment-for instance, by turning away from the screen to see the rest of the room. Of course, people tend to cooper- ate with the illusion instead of testing it.

Virtual reality technology makes use of the illusion of motion pictures but also reacts to any moves the body might make to debunk what it is seeing. In

theory, if images of the world are not

only correct and convincing (such as those on film), but can also react to the movements of the viewer's body, then the nervous system is obliged to treat what it sees as reality. Virtual reality does not allow the viewer to shake the illusion

by turning away from it. Instead, the illusion continues and deepens as the viewer tests it by attempting to look

away: The illusion is no longer there, it is here and one is present in it. There are no media between the viewer and the illusory world; illusion is mainlined into the nervous system. This has under-

standably led to concern about forced hallucinations, but, for the foreseeable future, the realism and context of these virtual worlds will still require some co-

operation from the viewer. If it is the artist's or the author's en-

deavor to explore and convey our experience of life, virtual reality promises a

representation of experience that is as

good as any our senses can receive. Al-

though our experience of life goes be-

yond what our senses offer, we can only communicate about our interior experi- ences through our senses. Virtual reality promises tools for communication with a bandwidth as wide as the range of our

sensory perception. Virtual reality is defined by a set of

technologies that emphasize placement in a scene. Computer clothing is neces-

sary to track all the joints of the body as the user goes about testing the illusory world. The computer must be pro- grammed to react to the body's tests by creating the correct sensory output (for instance, visual perspective) for a given body position. This output has to be created faster than our perceptual systems can take it in (around 20 frames

per second for the visual, 22 kHz for the auditory). Finally, devices such as head-mounted display units are neces-

sary to input this sensory information into the nervous system. Although any-

speaking, it is this immersion equipment that defines virtual reality.

In some ways, a sparse representation through limited bandwidth is more

likely to require the viewer to engage his or her imagination to fill in the gaps. A book can quite often be more enjoyable than a movie because it requires engagement of the imagination. Ultimately, the skill of the communicator is more im-

portant than the bandwidth of the tools. Artful communication consists of involv-

ing the senses as deeply as possible within a scene without losing the appeal to the imagination to take flight.

Immersive virtual reality equipment is

presently not conducive to creating worlds that engage the imagination. The obtrusiveness of the equipment that is now necessary to deliver an immersive world, such as the head-mounted display unit, distracts more than it lends to the

experience. This equipment is currently being refined to be less uncomfortable. However, it is possible that it will always be a better trade-off to use more abstracted inputs and less immersive out-

puts. For instance, people might prefer to turn a knob instead of turning their neck to pan their view; a high-definition screen might be preferable to a television strapped over each eye. In fact, the

use of abstracted movements allows the viewer to overcome the limitations of the natural world, such as our inability to fly or to turn our heads continuously in a 360? circle.

The technical theory of virtual reality has also limited the flight of imagination by emphasizing the placement of a

person in the world above the creation of a world by that person. If virtual real-

ity is to be used as communication, we must be able to create worlds as easily as we can be placed in them. The current

emphasis on placement may be a con-

ceptual vestige of one-way broadcast television technology, which does not make it possible for most viewers to be

very creative. Interactive media natu-

rally invite participation from the audience and offer perfect opportunities to

engage the imagination. How much this

participation calls upon the imagination depends on whether the audience is being asked to pick a pay-per-view movie, branch in a narrative or deliver a line of dialogue. As of now, virtual real-

ity and interactive media in general are

designed for a low level of creative participation. Virtual reality could facilitate the highest level of expression: the user's creation of an entire world that is

complete and treated as reality. This

Fig. 2. Still from Pavlovsk Palace, a navigable scene created on a Macintosh with

HyperCard and QuickTime, 1992. Users can control the viewpoint from which they see this room by moving the mouse around in the QuickTime window. For instance, holding the mouse down in the lower right would cause the view to pan to the right and tilt down. Several rooms in the Pavlovsk Palace outside St. Petersburg were captured with QuickTime video.

thing from a film to a CD-ROM (com- pact disc-read only memory) could be said to create virtual reality, strictly

298 OSullivan, Choosing Tools for Virtual Environments



world-creation-in addition to world-

consumption-is the dream of people likeJaron Lanier [1]. As costs lower and these technologies fall into more hands, I am sure the world-creating possibili- ties will be explored in more depth.

DAN'S APARTMENT Dan's Apartment (1991) was a television show on Manhattan Cable that allowed one random viewer after another to control the viewpoint of a simulated walk

through my apartment by speaking com- mands into his or her telephone (Fig. 1). As the viewpoint moves through the un-

distinguished apartment, nobody is home and there is no sound. No explana- tion is offered. It is as if the viewer woke

up alone in a stranger's apartment and is now taking the opportunity to snoop around. Who is Dan, why is his apartment on TV, why is there a tooth on the desk? There are many questions to en-

gage the viewer's imagination. This show was made by systematically

videotaping many paths through my apartment, pressing them to videodisc and programming the relationship between the video segments on an Amiga computer. Viewers had a monoscopic view and two axes of freedom (left and

right, forward and reverse) in following about 20 different paths: a far cry from virtual reality, which has a stereoscopic view and freedom along six axes (x, y, z, pan, tilt and roll) and an infinite number of paths. However, because the im-

agery was recognizable and the mystery of the piece was engaging, the audience mentally supplied the missing degrees of freedom.

Dan's Apartment showed me that the

playback of a relationship between rec-

ognizable images can create a convincing environment. In other words, if I move left and an object in my view moves right, then I move forward and see the object grow bigger, I will believe the object is in front of me. The audience has to be allowed to control and test the environment to see if the rela-

tionships seem right. If the relationships hold up, one has successfully created a virtual space; however, the linear media used to create Dan's Apartment only allow a viewer to choose a premapped path within this space. If the user is given freedom to look in any direction, she is more likely to trust the environment. The greater the quantity of relationships between images that can be pro- grammed and played back, the more convincing the environment.

Fig. 3. Still from Being There with the Melons, television show produced on a Macintosh with HyperCard and videodisc, 1992. Viewers can move the camera around the room as a drama about two television viewers turned into honeydew melons unfolds around them.

NAVIGABLE SCENES In virtual reality, the relationship between objects is set by creating a 3D (three-dimensional) model. This gives the creator the advantage of being able to describe a complete relationship between objects. However, 3D modeling and real-time rendering have the disad-

vantage of creating unconvincing ob-

jects. Objects seen on video or film are

usually instantly recognizable, but cam- eras do not capture relationships between objects, and linear media like film and tape allow only linear relationships to be played back.

While I was an intern for Michael Mills at Apple Computer in 1991, QuickTime digital video was being de-

veloped. Most uses of digital video are for random access of linear clips of media. However, digital video can be used as an intrinsically nonlinear medium to create environments by portraying a nonlinear relationship between images.

I built a camera rig that would take

pictures in a very systematic way and thus capture not only the images but the

relationships between them. I used a

computer to control the pan and tilt of the tripod head so that the computer would know what the relationships were between the captured images. The result is what Apple calls "navigable scenes," which allow one to capture and

play back an entire panorama-a range extending 360? horizontally and 180?

vertically. The same technique can be

used to capture an object instead of a scene by mounting a camera on an arm

pointed inward at the object. In addition to pan, tilt, zoom and time lapse, any other number of relationships can be systematically captured and played back (Fig. 2).

Navigable movies composed of such scenes use the natural world instead of

trying to reproduce it. This technique prerenders a limited subset of information instead of rendering all possible information as it is needed. As a result, navigable movies are more accessible than virtual reality. The ability to take

photographs is all that is really needed to create these scenes. Virtual reality worlds suffer from a lack of people who are willing and able to use a computer to model in three dimensions. Navigable scenes can be played back on a $1,000 personal computer instead of the $200,000 graphics workstation required for virtual reality. Accessibility means less time hustling for money and more time creating environments that invite the imagination.

AUGMENTED REALITY The idea of augmented reality makes the

greatest use of the natural world and re-

quires the least amount of computing power. Instead of rendering images of the world in real time or even pre- rendering them, augmented reality allows the world to stand for itself. Only

O'Sullivan, Choosing Tools for Virtual Environments 299



Fig. 4. Still from Yorb-An Electrnic Neigh- borhood, computer BBS and cable television show (playing on Monday, Wednesday and Friday from 11-12 PM on Channel 34 in New York) using SGI rendering, HyperCard Media Server and NovaLink BBS, 1992. Viewers can navigate around a real-time rendered environment. One person at a time controls the viewpoint via a touchtone telephone. Other callers can be heard and have virtual lips to move along with their voices.

what does not already exist in the world needs to be produced by the computer, and what does exist in the world is presented with lifelike fidelity.

To test this idea, I overlaid computer graphics on a live video feed of a room. I used an office chair to track the user's motions because it limited the problem to two axes of freedom: swivel and tilt. I attached a screen for displaying the overlaid video to the chair in such a way that the screen stayed in front of the user as she moved in the chair. The camera providing the live-video feed was

just behind the screen and pointed away from the user. In this configuration, the screen acted as a piece of glass that was always in place in front of the user. The user could draw on the screen with a mouse and thus doodle on the walls. The computer monitored the movement of the chair and scrolled the

doodlings to correspond to the orienta- tion of the chair.

NAVIGABLE SCENES WITH LINEAR NARRATIVE In 1992 I went to Russia with Apple Com-

puter to make navigable scenes of the Czar's palace (see Fig. 2). The rooms in these palaces are ornamented from ceil-

ing to floor, and the navigable scenes ac-

curately captured what it is like to stand and look around in these museums. However, like most museums, these scenes are lifeless. The introduction of characters into the scenes woolid add life visually, but such additions would be

technically very difficult. However, it oc- curred to me that even a simple linear audio drama might help the viewer

project herself into the daily life of that time period.

Back at New York University, I ex-

plored this combination of a navigable environment with a linear narrative. Be-

ing There with the Melons (1992) was a television show that attempted to place the audience on stage with the drama and thus eliminate the "fourth wall." Viewers were invited to call in and adjust the camera's view within a full pan- oramic range-360? around and 180?

up and down-by using their touchtone

telephones while the drama unfolded

(Fig. 3). In this case, the drama revolved around two people who had turned into

honeydew melons. Because these characters were unmoving, all the action took place on the audio level. In addition to looking around, the viewer could find out more about objects on the set.

Although this idea deserves some more

investigation, it was not very successful. The navigation of the environment seemed to distract from the linear narrative without adding enough room for the imagination to roam.

YORB-AN ELECTRONIC NEIGHBORHOOD Most of these interactive techniques still concentrate on situating the audience's

imagination within a context, a prejudice that seems to linger from the tradition of

one-way television. The true value of interactive media is its ability to invite members of the audience to use their

imaginations to help create as well as to control what they see. I believe that these media should ask the audience not only to make choices, but to take initiatives.

In Yorb-An Electronic Neighborhood we are asking the audience not only to view a virtual world, but also to participate in

building it. Yorb-An Electronic Neighbor- hood is a television program that began in 1992 and airs on Manhattan Cable three times per week. Conceptually, the

program is a multimedia bulletin board rather than a traditional television show. The action takes place on what looks like a green cartoon planet (Fig. 4). When our show is on the air, one caller at a time, chosen at random, can use a touchtone telephone to fly around this

imaginary world. When the caller

bumps into something on the planet- for instance, the Zen Temple-she can browse through the media contained

Fig. 5. Still from Yorb-An Electronic Neighborhood, a computer BBS and cable television show using SGI rendering, HyperCard Media Server and NovaLink BBS, 1992. When the callers encounter an object in the world of Yorb, the most recent message sent to that ob-

ject is displayed. Using the * and # buttons on a touchtone telephone, the caller can page through all the previous messages. Messages can take the form of graphics, sound, QuickTime, text, HyperCard or MacroMind Director files. They can be left in the world and retrieved using a modem, fax machine or LAN.

300 O'Sullivan, Choosing Tools for Virtual Environments



within it, which other viewers have previ- ously transmitted to this location (Fig. 5). The planet and the Zen Temple are created using Swivel 3D software and are rendered in real time using VPL (Visual Programming Language) Research's virtual reality software. The media contained in the Zen Temple can be com-

puter graphics, faxes, QuickTime movies and sounds, HyperCard stacks, text files and MacroMind Director files. This media can be placed in (and retrieved from) the Zen Temple by calling our

computer BBS (bulletin board system) by modem, fax, or our LAN (local area network).

The Electronic Neighborhood is an ex-

periment both in using 3D space as an interface and in asking the audience to be creative and expressive. I find that the use of a 3D space as an interface is rather successful for a wide audience. However, contributions to the world that the show presents are generally made by a small group of insiders who have un- derstood the unusual concept of the

program. In addition to the novelty of the concept, several technical con- straints impair the quality and quantity of the participation in the creation of this virtual world. The skills and tools for creating multimedia are not widely available to our audience. The processes of putting the media into the world-for instance, by using fax and modem-are

separate from the show and too difficult for most of our viewers. Because we

opted to try real-time rendering, which

requires an expensive machine, we can afford to present only one viewpoint on one cable television channel for 3 hours

per week. The limited availability of our

program lessens the payoff for placing a

message in the Electronic Neighborhood by decreasing the likelihood of a reply.

PLAY In trying to break down the barriers to creativity I am constantly drawn to the

image of a child playing with a doll. A child projects her imagination onto the world: A wooden floor can become an ocean and a throw rug a raft as the child constructs a drama about a person being lost at sea. Computers can only get in the child's way. Adults need extra help projecting their imagination and often create things in a more purposeful way. The computer can help this process by providing some premade scenery as well as recording, layering, disguising or

transmitting imaginings. The real chal- lenge is to make the act of creation as

Fig. 6. Mirror Play, Macintosh applica- .I tion produced with HyperCard and QuickTime, 1993. . Users can compos- ite still and moving images into a scene simply by placing the image or object in front of the monitor. In this case, the users made a quick sketch about two ice :: skaters. ...

:.;:. :

easy as holding up a doll and moving one's lips.

I was at Apple Computer again in July of 1993 and developed a simple com-

puter application called Mirror Play. Mir- ror Play begins as an empty screen (Fig. 6). Users can construct a virtual world by simply holding objects in front of the monitor as if it were a mirror. The object is digitized and added to the scenery. In the same way, users can add moving pictures and sounds to a scene to give it a

plot. The following scenario illustrates a

possible use of Mirror Play: A user makes a line drawing of two stick figures lying on a beach and holds it in front of the monitor to be digitized. She takes a plant and holds it in front of the monitor in such a way that it appears to be the top of a palm tree. Placing her head in front of the monitor so that it appears to be on the shoulders of one of the stick figures in the drawing, she records a QuickTime movie of herself saying how glad she is not to be in the office. Next door, she asks an office mate to put his or her head on the other stick figure and deliver the next line. People using Mirror Play find it easy to construct imaginary scenes because it makes use of their knowledge of familiar things in the natural world, such as pens and paper, office plants, their own faces and mirrors.

The images that have been created

using MirrorPlay are disjointed and full of seams, resembling a cross between David Hockney's photo montages and

pictures made using Kid Pix children's

drawing software. However, because it was so easy to exercise their imagina-

tions, users projected what they wanted to see throughout the process. They needed only a rough symbol to hold the

place for what their imaginations had constructed. Roughlyjuxtaposed images were readily blended by the creator, who was anxious to believe in the scene. The- ater is a parallel example of a medium whose material tools of illusion are often rudimentary: theater sets are frequently stylized and unabashedly made of ply- wood. The audience "plays along" because of the unfettered creativity of live

performance. The process of creation is the end

product of Mirror Play: its results are linear narratives that might only be enjoyable to the people who created them. The dramas are disposable, as is the ex-

perience of movies in a theater where there is no rewind. The personal investment in creation can bring a person deeper into an imaginary world than a linear film made with the most seamless

special effects. Although the history of their creation-or the end product-resembles film in its linearity, during the

process of creation the user is presented with an infinite array of alternatives at

any given point. I believe this horizon of alternatives resembles our experience of life more closely than film does.

Because users fill in the gaps in Mirror

Play with their own imaginations, other

people lacking the same projections of

imagination might not understand the

play. A user can, of course, decide to communicate and use universal conventions, or just enjoy the process of creation without worrying about the end

O'Sullivan, Choosing Tools for Virtual Environments 301



product, as a child does. The increasing sophistication of visual literacy among such groups as MTV (Music Television) viewers suggests that the results of Mirror

Play, rough though they may be, could often be shared with other people.

People are willing to learn to read conventions or various forms of short- hand if these codes allow their ima-

gination to skip to the fun stuff. In film, the conventions are usually centered around time. One is glad to live with the nonrealism of 10 years passing in a frac- tion of a second on film if it allows rapid passage over tedious bits in order to get to the interesting parts. MirrorPlay takes

advantage of spatial as well as temporal conventions. A user will live with the fact that heads do not seamlessly fit on stick

figures' shoulders if it allows for the creation of a character without putting on a costume and going on location. Most of Mirror Play is a still graphic, with motion

appearing only over areas that really need to show movement. For instance, if one stick figure is waving to another, all that is necessary to signify this is a video of a hand waving superimposed over the arm of one stick figure. Although the use of motion graphics is limited in this

way for technical reasons, it produces an

interesting effect that has the feeling of

Cubism, with the close-up of the hand and the wide shot of the scene presented at the same time. In film these shots would be temporally separated.

There are many ways to tap our natural abilities to construct imaginary worlds. Virtual reality could allow communication using representations that are as complete and accurate as the data

they represent, in terms of what our nervous system will process. Although immersive virtual reality systems can thus

place the nervous system in another world, they do not allow enough room for the user to create worlds. This may simply be due to the cost and clumsiness of the systems that exist today.

Beyond producing perceptual illusions, interactive media can tap the

imagination by making a partner of it. If the audience has an investment in the creation of an imaginary world, less

technology is required to maintain the illusion. Less work maintaining the illusion means a greater ease of creativity. Creative work feeds back again to better

capture the imagination. Initially excited by new tools for plac-

ing the imagination in created situa- tions, I have discovered in the process of

my own work what a willing and able col- laborator the imagination is. I have also

discovered that even modest interactive

technology is able to reach the imagination once it has been co-opted. While I am still interested in virtual reality, I re- alize that it is best to use a mix of the tools in an artist's arsenal of illusion. For the moment, I am focusing on tools that facilitate a projection of imagination rather than tools that project onto it.

Acknowledgments Most of this work was done at New York University's Interactive Telecommunications Program and would not be possible without the support of the program's chair, Red Burns. I am also grateful to Michael Mills andJoy Mountford at Apple Computer.

Reference

1. Jaron Lanier, from a lecture given at New York University's Interactive Telecommunications Pro- gram in April 1993.

Bibliography Gombrich, E.H., Art and Illusion (Princeton, NJ: Princeton Univ. Press, 1960).

Illich, Ivan, Toolsfor Conviviality (Berkeley, CA:

Harper & Row, 1973).

Postman, Neil, Amusing Ourselves to Death (New York: Penguin, 1985).

302 O'Sullivan, Choosing Tools for Virtual Environments



choosing tools for virtual environments

Documents