interacting with dynamic real objects in a virtual environment

Interacting With Dynamic Real Objects

in a Virtual Environment

Benjamin Lok

February 14th, 2003

Outline• Motivation• Incorporation of

Dynamic Real Objects• Managing Collisions

Between Virtual and Dynamic Real Objects

• User Study• NASA Case Study• Conclusion

Why we need dynamic real objects in VEs

How we get dynamic real objects in VEs

What good are dynamic real objects?Applying the system to a driving real world problem

Assembly Verification

• Given a model, we would like to explore:– Can it be readily assembled?– Can repairers service it?

• Example:– Changing an oil filter– Attaching a cable

to a payload

Current ImmersiveVE Approaches

• Most objects are purely virtual– User– Tools– Parts

• Most virtual objects are not registered with a corresponding real object.

• System has limited shape and motion information of real objects.

Ideally

• Would like:– Accurate virtual representations, or avatars, of

real objects– Virtual objects responding to real objects– Haptic feedback– Correct affordances– Constrained motion

• Example: Unscrewing a virtual oil filter from a car engine model

Dynamic Real Objects

• Tracking and modeling dynamic objects would:– Improve interactivity– Enable visually faithful

virtual representations

• Dynamic objects can:– Change shape– Change appearance

Thesis Statement

Naturally interacting with real objects in immersive virtual environments improves task performance and presence in spatial cognitive manual tasks.

Previous Work: Incorporating Real Objects into VEs

• Non-Real Time– Virtualized Reality (Kanade, et al.)

• Real Time– Image Based Visual Hulls [Matusik00, 01]– 3D Tele-Immersion [Daniilidis00]

• Augment specific objects for interaction– Doll’s head [Hinkley94]– Plate [Hoffman98]

• How important is to get real objects into a virtual environment?

Previous Work: Avatars• Self - Avatars in VEs

– What makes avatars believable? [Thalmann98]

– What avatars components are necessary? [Slater93, 94, Garau01]

• VEs currently have:– Choices from a library– Generic avatars– No avatars

• Generic avatars > no avatars [Slater93]

• Are visually faithful avatars better than generic avatars?

Visual Incorporation of Dynamic Real Objects in a VE

Motivation

• Handle dynamic objects (generate a virtual representation)

• Interactive rates

• Bypass an explicit 3D modeling stage

• Inputs: outside-looking-in camera images

• Generate an approximation of the real objects (visual hull)

Reconstruction Algorithm

…1. Start with live camera images

2. Image Subtraction

3. Use images to calculate volume intersection

4. Composite with the VE

…

Visual Hull Computation

• Visual hull - tightest volume given a set of object silhouettes

• Intersection of the projection of object pixels

Volume Querying

• A point inside the visual hull projects onto an object pixel from each camera

Implementation

• 1 HMD-mounted and 3 wall-mounted cameras

• SGI Reality Monster – handles up to 7 video feeds

• Computation– Image subtraction is the most work– ~16000 triangles/sec, 1.2 gigapixels

• 15-18 fps• Estimated error: 1 cm• Performance will increase as graphics

hardware continues to improve

Results

Managing Collisions Between Virtual and Dynamic Real

Objects

Approach• We want virtual objects

respond to real object avatars

• This requires detecting when real and virtual objects intersect

• If intersections exist, determine plausible responses

Assumptions• Only virtual objects can move or deform at

collision.

• Both real and virtual objects are assumed stationary at collision.

• We catch collisions soon after a virtual object enters the visual hull, and not as it exits the other side.

Detecting Collisions

Resolving Collisions Approach

1. Estimate point of deepest virtual object penetration

2. Define plausible recovery vector

3. Estimate point of collision on visual hull

Results

Collision Detection / Response Performance

• Volume-query about 5000 triangles per second

• Error of collision points is ~0.75 cm.– Depends on average size of virtual object

triangles– Tradeoff between accuracy and time– Plenty of room for optimizations

Spatial Cognitive Task Study

Study Motivation

• Effects of– Interacting with real objects– Visual fidelity of self-avatars

• On– Task Performance– Presence

• For spatial cognitive manual tasks

Spatial Cognitive Manual Tasks

• Spatial Ability– Visualizing a manipulation in 3-space

• Cognition– Psychological processes involved in the acquisition,

organization, and use of knowledge

Hypotheses• Task Performance: Participants will

complete a spatial cognitive manual task faster when manipulating real objects, as opposed to virtual objects only.

• Sense of Presence: Participants will report a higher sense of presence when their self-avatars are visually faithful, as opposed to generic.

Task

• Manipulated identical painted blocks to match target patterns

• Each block had six distinct patterns.

• Target patterns:– 2x2 blocks (small)– 3x3 blocks (large)

Measures• Task performance

– Time to complete the patterns correctly

• Sense of presence– (After experience) Steed-Usoh-Slater Sense of

Presence Questionnaire (SUS)

• Other factors– (Before experience) spatial ability– (Before and after experience) simulator sickness

Conditions

• All participants did the task in a real space environment.

• Each participant did the task in one of three VEs.

Real Space

Purely Virtual

Hybrid

Vis. Faithful Hybrid

Conditions

Avatar Fidelity

GenericVisually

faithful

Interact with

Real objects

HE VFHE

Virtual objects

PVETask performance

Sense of presence

Real Space Environment

• Task was conducted within a draped enclosure

• Participant watched monitor while performing task

• RSE performance was a baseline to compare against VE performance

Purely Virtual Environment

• Participant manipulated virtual objects• Participant was presented with a generic avatar

Hybrid Environment

• Participant manipulated real objects• Participant was presented with a generic avatar

Visually-Faithful Hybrid Env.

• Participant manipulated real objects• Participant was presented with a visually faithful avatar

Task Performance Results

Small Pattern Time (seconds) Large Pattern Time (seconds)

Mean S.D. Mean S.D.

Real Space (n=41) 16.8 6.3 37.2 9.0

Purely Virtual (n=13) 47.2 10.4 117.0 32.3

Hybrid (n=13) 31.7 5.7 86.8 26.8

Visually Faithful Hybrid (n=14) 28.9 7.6 72.3 16.4

Small Pattern Time Large Pattern Time

T-test p T-test p

Purely Virtual vs. Vis. Faithful 3.32 0.0026** 4.39 0.00016***

Purely Virtual vs. Hybrid 2.81 0.0094** 2.45 0.021*

Hybrid vs. Vis. Faithful Hybrid 1.02 0.32 2.01 0.055

* - significant at the =0.05 level ** - =0.01 level *** - =0.001 level

Task Performance Results

Sense of Presence Results

SUS Sense of Presence Score (0..6)

Mean S.D.

Purely Virtual Environment 3.21 2.19

Hybrid Environment 1.86 2.17

Visually Faithful Hybrid Environment 2.36 1.94

Sense of Presence Results

Sense of Presence

T-test p

Purely Virtual vs. Visually Faithful Hybrid 1.10 0.28

Purely Virtual vs. Hybrid 1.64 0.11

Hybrid vs. Visually Faithful Hybrid 0.64 0.53

Debriefing Responses• They felt almost completely immersed while performing the task.

• They felt the virtual objects in the virtual room (such as the painting, plant, and lamp) improved their sense of presence, even though they had no direct interaction with these objects.

• They felt that seeing an avatar added to their sense of presence.

• PVE and HE participants commented on the fidelity of motion, whereas VFHE participants commented on the fidelity of appearance.

• VFHE and HE participants felt tactile feedback of working with real objects improved their sense of presence.

• VFHE participants reported getting used to manipulating and interacting in the VE significantly faster than PVE participants.

Study Conclusions• Interacting with real objects provided a quite substantial

performance improvement over interacting with virtual objects for cognitive manual tasks

• Debriefing quotes show that the visually faithful avatar was preferred, though reported sense of presence was not significantly different.

• Kinematic fidelity of the avatar is more important than visual fidelity for sense of presence.

Handling real objects makes task performance and interaction in the VE more like the actual task.

Case Study: NASA Langley Research Center

(LaRC)Payload Assembly Task

NASA Driving Problems• Given payload models, designers and engineers

want to evaluate:– Assembly feasibility– Assembly training– Repairability

• Current Approaches– Measurements– Design drawings– Step-by-step assembly instruction list– Low fidelity mock-ups

Task

• Wanted a plausible task given common assembly jobs.

• Abstracted a payload layout task– Screw in tube– Attach power cable

Task Goal

• Determine how much space should be allocated between the TOP of the PMT and the BOTTOM of Payload A

Videos of Task

Results

Participant

#1 #2 #3 #4

(Pre-experience) How much space is necessary?

14 cm 14.2 cm 15 – 16 cm

15 cm

(Pre-experience) How much space would you actually allocate?

21 cm 16 cm 20 cm 15 cm

Actual space required in VE 15 cm 22.5 cm 22.3 cm 23 cm

(Post-experience) How much space would you actually allocate?

18 cm 16 cm

(modify tool)

25 cm 23 cm

The tube was 14 cm long,

4cm in diameter.

Results

• Late discovery of similar problems is not uncommon.

Participant

#1 #2 #3 #4

Time cost of the spacing error

days to months 30 days days to months months

Financial cost of the spacing error

$100,000s -$1,000,000+

largest cost is huge hit in schedule

$100,000s -$1,000,000+

$100,000s

Case Study Conclusions

• Object reconstruction VEs benefits:– Specialized tools and parts require no modeling– Short development time to try multiple designs– Allows early testing of subassembly integration

from multiple suppliers

• Can get early identification of assembly, design, or integration issues that results in considerable savings in time and money.

Conclusions

Overall Innovations• Presented algorithms for

– Incorporation of real objects into VEs

– Handling interactions between real and virtual objects

• Conducted formal studies to evaluate

– Interaction with real vs. virtual object (significant effect)

– Visually faithful vs. generic avatars (no significant effect)

• Applied to real-world task

Future Work• Improved model fidelity

• Improved collision detection and response

• Further studies to illuminate the relationship between avatar kinematic fidelity and visual fidelity

• Apply system to upcoming NASA payload projects.

Current Projects at UNC-Charlottewith Dr. Larry Hodges

• Digitizing Humanity– If a virtual human gave

you a compliment, would it brighten your day?

– Do people interact with virtual characters the same way they do with real people? (carry over from reality -> virtual)


• Digitizing Humanity– Basic research into virtual characters

• What is important?• How does personality affect interaction?

– Applications:• Social situations• Human Virtual-Human Interaction

• Virtual Reality– Basic Research:

• Incorporating Avatars• Locomotion Effect on Cognitive Performance

– Applications:• Balance Disorders (w/ Univ. of Pittsburg)


• Combining Computer Graphics with:– Computer Vision

• Dr. Min Shin

– Human Computer Interaction• Dr. Larry Hodges, Dr. Jee-In Kim

– Virtual Reality• Dr. Larry Hodges

– Graduate and Undergraduate Research• Future Computing Lab has 4 PhD, 3 MS, and 6

undergraduates

Collaboration on Research• Digital Media

– Applying VR/Computer Graphics to:– Digital Archaeology (Digital records of historic data)– Digital Media Program (Getting non-CS people involved in VR)– Mixed Reality

• Computer Vision/Image Processing– Using VR technology to aid in object tracking– Using computer vision to augment VR interaction

• Computer Graphics Lab– Photorealistic Rendering

• Novel Visualization– Evolutionary Computing Lab– Central Florida Remote Sensing Lab

Future Directions• Long Term Goals

– Enhance other CS projects with Graphics, Visualization, VR.

– Computer Scientists are Toolsmiths– Help build the department into a leader in using

graphics for visualization, simulation, and training.– Effective Virtual Environments (Graphics, Virtual

Reality, and Psychology) – Digital Characters (Graphics & HCI)

• Additional benefit of having nearby companies (Disney) and military

– Assistive Technology (Graphics, VR, and Computer Vision)

Thanks

ThanksCollaborators

Dr. Frederick P. Brooks Jr. (PhD Advisor)

Dr. Larry F. Hodges (Post-doc advisor)

Prof. Mary Whitton

Samir Naik

Danette Allen (NASA LaRC)

UNC-CH Effective Virtual Environments

UNC-C Virtual Environments Group

For more information:http://www.cs.uncc.edu/~bclok

(VR2003, I3D2003)

Funding Agencies The LINK Foundation

NIH (Grant P41 RR02170)

National Science Foundation

Office of Navel Research

http://www.cs.uncc.edu/~bclok

Object Pixels

• Identify new objects

• Perform image subtraction

• Separate the object pixels from background pixels

current image - background image = object pixels

Volume Querying

• Next we do volume querying on a plane

Volume Querying

• For an arbitrary view, we sweep a series of planes.

Detecting Collisions Approach

Are there real-virtual collisions?

For virtual object i

Done withobject i

Volume queryeach triangle

Calculate plausiblecollision response

Determine pointson virtual object

in collisionN Y

Research Interests• Computer Graphics – computer scientists are toolsmiths

– Applying graphics hardware to:• 3D reconstruction • simulation

– Visualization– Interactive Graphics

• Virtual Reality– What makes a virtual environment effective?– Applying to assembly verification & clinical psychology

• Human Computer Interaction– 3D Interaction– Virtual Humans

• Assistive Technology– Computer Vision and Mobile Technology to help disabled

interacting with dynamic real objects in a virtual environment

Documents