Edward J. WegmanJuergen Symanzik Patrick Vandersluis

Xiaodong Fu Ying Zhu Rida Moustafa Qiang Luo

Nkem-Amin Khumbah Fernando CamelliAntoinette Dzubay Robert Wall

Workshop on Data Visualization in Statistics

July 8, 1998

Outline of Talk

4The CAVETM VR Environment

4Motivation for the MiniCAVETM

4The MiniCAVETM Environment– Windows NT/Pentium II Implementation

– Voice Interface

4Technical Challenges

Trademark Notes

4 CAVE is a trademark of the Electronic VisualizationLaboratory at the University of Illinois, Chicago

4 MiniCAVE is a trademark of the Center for ComputationalStatistics at George Mason University

CAVE Concept

4A Projection-based Immersive VR System– Silicon Graphics-based with 8 to 12 processors

– RE2 or RE Infinity graphics engines

– CRT-based projection system

– Stereographics Crystal Eyes shutter glasses

– Head tracking

– Usually 3 to 5 wall cube

4Developed originally at U. Illinois, Chicago– Carolina Cruz-Neira

CAVE Strengths

4Effective immersive environment– Lightweight non-intrusive glasses

– Can see own hands and other participants

4Effective for group VR– Good tool for group collaboration

CAVE Weaknesses

4CRT Projectors– Projectors not very bright

– Shock, vibration & heat, hard to keep focus

– Geometric distortion at wall interfaces

4Tracking– One user tracked, badly distorted stereo for

users not at viewpoint

4User Interface– Usually 3-D extension of desktop metaphor

CAVE Weaknesses

4Expensive– $1,000,000 fully outfitted

– $600,000+ SGI computers

– $30,000 per projector

Motivation

4 Installed MATLAB 5 on SGI Onyx and Pentium– Benchmarks on 200 megahertz Pentium Pro ($3000)

and 200 megahertz SGI Onyx ($120,000) similar

4 Liquid Crystal Projectors sharp, bright, and stableunder shock, vibration and temperature variation

4 Stereographics Crystal Eyes technology availablefor Windows NT

MiniCAVE Concept

4Windows NT/Intel Pentium II 400 mhz

4LCD/DLV-based projection systems

412 ft cubes scaled to 6 ft cubes

4Tracking optional, reduced latency

4Voice command metaphor

4$100,000 entry level

VR from Workstation to PC

4Project Purpose– Can the NT workstation really match the SGI

workstation in 3D graphics area?

– Possibility of VR implementation in PCenvironment, especially MiniCAVE.

– Explore the hardware and software capacities ofPC for VR application.

4Initial Approach1. Porting one SGI GL application, SkyFly, to

OpenGL, which is a platform-independent 3DAPI.

2. Porting this application to NT environment.

3. Performance comparison between SGI and NT.

4. Stereo display on PC using CrystalEyes.

5. Controlled by voice command.

Scene DB

Generate TerrainSetup textureCreate Scene DB

Update Viewer PositionUpdate Paper PlanesProcess Mouse Input ProcessKeyboard Input

Culling Drawing

Timer Event

Mouse/KeyEvent

Structure of Skyfly Flight Simulation Program

4Porting Skyfly to NT– Using OpenGL interface (GLUT) instead of

SGI’s proprietary mouse/key GL interface.

– OpenGL source code compiled on NT and SGIwithout modification.

GL/OpenGL features GL Version OpenGL Version CommentsTriangle Strips ü ü TerrainTexture Mapping ü ü Clouds & terrainLighting ü û* Terrain and paper planes

4Performance Comparison

Performance of Skyfly

0

10

20

30

40

50

60

70

320x240 640x480 800x600 1024x768

Resolution

FP

S

PPro200/NT/TrueColorw/o Hardware 3D

PPro200/NT/16bit Colorwith Hardware 3D

O2/IRIX6.3/CRM

ONYX/IRIX6.2/RE2

ONYX/IRIX6.4/IR

4Stereo using CrystalEyes– Above-below stereo.

– Image resolution 1024x384 each eye.

– Vertical refresh rate 120-150 (60-75 each eye).– SGI monitor can handle both 120 and 150.

– CRT projector can only handle 120 refresh rate.

– PII 333 + Permedia 2 can achieve 30 fps

4 Principles of above-below CrystalEyes stereo

Graphics Card

Computer

Video signal

CrystalEyeEmitter

Double Sync video signal

Monitor/Projector

Emitted pulse

Left Right

L R L

Speech Motivation

4User Interfaces (Van Dam)

4Shortcuts in Xgobi

4User Controls in the C2 Stats Application

Speech Recognition Products

Dragon Dictate

IBM ViaVoice

Speech Recognition TechnologyEvaluation

1. Can you train the software to understand additional words?

2. Is the software speaker dependent or speaker independent?

3. Can you store different pronunciations of one word into the samedatabase so a spoken word is compared with different pronunciations?

4. How good is performance (i.e., percentage of correctly identified wordsbefore and after training)?

5. Does accuracy depend on speaker / accent / training?

6. What improves accuracy relative to the various sources of imprecision?

7. How long does it take to reach various levels of performance?

8. What if we aren’t able to train? What should we expect?

Measures of PerformanceSummary

Attribute4 Setup Time

– Installation Time

– Basic Training Time

4 Accuracy (WYSIWYG)– Untrained and Trained

– Under Various Speakers

4 Speaker Independence– out-of-box performance

4 Robustness– Scope of Vocabulary

4 Response Latency– Command & Control

Metric4 Duration

4 Error rate (%)

4 No-Training Accuracy

4 Number of New Words Needed

4 Command-to-Text Cycle Time

– What’s the good format for voice-controllednavigation in this application? What are therecommendations on the maximum number ofwords for the application? What do theysuggest as maximum number of words in onecommand?We currently only have a simple commandformat: single word, with a around 20 wordsvocabulary. The generalization appearsachievable.

4Voice Control– command set

left, right, up, down, fast, slow, forward,reverse, stop, start.

– Link directly to Dragon Dictate (locally), orlink to custom-DLL(network possible).

Overview of Voice Interface

CAVEController

CAVEController

“Command”

> Command

C A V E

Microphone

CommandRecognizer

CommandRecognizer

Vocabularytext

Scenario

HardwareSoftware

TEST

CommandVocabulary

Command Recognizer

CommandRecognizer

CommandRecognizer

SpeechRecognition

Software

Visual BasicCommandFront End

text

API

CustomDynamic Linkable

Library (.dll)

Requirement:Capture Voice Command & Output Text

Speech Recognition Details

4 First Cut Assumption: Speech recognition and Skyflysoftware reside on same machine

4 Visual Basic (VB) Command Front End (FE)– Trap & redirect spoken word using

• DragonDictate DDSpeech Control

• VB Controls (e.g., text boxes)

• C-Interface defined in a custom DLL

– Flow

• MiniCAVE-immersed user issues a command

• DDSpeech control returns word and a recognition indicator to FE

– If (Word Found) Then Call interface to pass word

– Else VB displays message

Speech Recognition Details (continued)

4 Connecting VB to C Interface– Create DLL containing C function

– Link C function into VB FE using “Declare” statement

4 Custom Dynamic Link Library Components– Librarian

– Windows Exit Procedure

– .def File

– Make File

– C Interface (variable length string, integer length, integer return code)

• return code: 0 = No Error; 1 = Error (“command not recognized”)

. . . Use Visual C++ to create the DLL

CommandVocabulary

CAVE Controller

CAVEController

CAVEControllertext

API

Other appsSkyFlyC-Interface

Requirement:Capture & Execute Text Command

Scenario

Domain Components

Vocabulary

4 Command & Control Vocabulary4 Dictation Vocabulary

4 3-dimensional Graphical Renderings4 Visual Images, Maps, etc.

4 Textual Data

4 Application Functions, Formula, etc.

4 Numerical Data

Requirement:”Specialize” the Application

Approach

4Integrate successfully on one machine then

attempt a multi-machine solution

4Recognition of spoken word causes delay in

SkyFly program

Speech Recognition ProjectN-node Implementation Alternatives: Option 1: Single Machine

4 SkyFly program:– Receives the speech recognition event

– Interprets the word and produces the motion

4 No communication method such as RPC is required

4 Easy to implement, but...– Recognition of the spoken word causes a delay in the SkyFly program

– Motion is “jerky” as you move from left-to-right or vice versa

– Running both applications on the same NT machine may not be aviable alternative

Speech Recognition ProjectN-node Implementation Alternatives: Option 2: Single- or Multi-Machine

4 Uses the message passing interface PVM to communicate betweenthe speech recognition application and the SkyFly program

– Non-synchronized sending and receiving or messages

– MASTER-SLAVE architecture

4 Visual C++ or C++ program:

– Receives the speech recognition event & bundles information

– Acts as the MASTER process -- initializes PVM and spawns aSLAVE process

4 Fly-over program:

– Acts as the SLAVE process

– Receives and unbundles the information

Speech Recognition ProjectN-node Implementation Alternatives: Option 3: Single- or Multi-Machine

4 Visual Basic (VB) Application– Receives the speech recognition event

– Passes the recognized word via a custom built DLL

4 Fly-over program uses the same DLL to receive the word

4 Can be extended to a multi-machine solution using RPC or PVM

4 Easy to integrate speech controls into the VB application

4 Didn’t implement the back-end call to the DLL

4 We’re not sure it works; back-end call to DLL not implemented

Technical Challenges - Successes

4Port of SkyFly Stereoscopic Demo to NTsuccessful with adequate frame rates on 333megahertz machine.

4CrystalEyes interface on NT successful

4Voice recognition using Dragon Dictatesuccessful– but requires training of speech recognizer.

Future Directions

4Speech Enhanced ExplorN

4MRI Controlled through Speech

4Stat/GIS application such as VirGIS

4Military Applications to Command andControl

Remaining Challenges

4Easy Ones– “Stereo Lock” Synchronization

– “Image Lock” Synchronization

4Hard Ones– Edge Blending with Digital Projectors

– Digital Projectors Themselves• Frame Rates

• Decay

Current Status

4Patent Disclosure Filed

4Demonstrated at 1998 World Congress onInformation Technology

4CRADA signed with U.S. Army (WhiteSands Missile Range) - awaiting funding

4Planned EDA/Data Mining Applicationwith Voice Interface.

4Machine Stolen…Gads!

Contact

4Edward Wegman– ewegman@gmu.edu

4Juergen Symanzik– symanzik@galaxy.gmu.edu

4Website– www.galaxy.gmu.edu/papers/MiniCAVE.ppt