Collaborative Mixed RealityCollaborative Mixed Reality Visualization of an Archaeological Visualization of an Archaeological

ExcavationExcavation

Hrvoje BenkoHrvoje BenkoDepartment of Computer ScienceDepartment of Computer Science

Columbia UniversityColumbia University

MIT talk – January 12, 2005MIT talk – January 12, 2005

2

Multidisciplinary TeamMultidisciplinary Team

3

MotivationMotivation

►Excavation is destructive and Excavation is destructive and physically “unreconstructable” processphysically “unreconstructable” process

►Need to preserve as much data as Need to preserve as much data as possible for analysispossible for analysis

►Data interpretation happens off-siteData interpretation happens off-site►Current tools focus on 2D data and do Current tools focus on 2D data and do

not incorporate 3D informationnot incorporate 3D information►Many experts—collaboration is a must!Many experts—collaboration is a must!

4

Archaeological Excavation at Archaeological Excavation at Monte Polizzo, Sicily, Summer Monte Polizzo, Sicily, Summer

20032003Ian Morris, Director (Stanford University)Ian Morris, Director (Stanford University)

5

Working in the field!Working in the field!

6

Meshed 3D Model (13 scans)Meshed 3D Model (13 scans)

7

We have a 3D site model, We have a 3D site model, now what?now what?

RealReal VirtualVirtual

8

High Resolution Images

DrawingsField Notes

Panoramic Images

Videos

GIS Data

9

High Resolution Images

DrawingsField Notes

Panoramic Images3D Object Models

3D Site Model Videos

GIS Data

Database

10

Two ProblemsTwo Problems

►How to combine all this data in one How to combine all this data in one seamless environment?seamless environment?

►How to make it easy to interact with?How to make it easy to interact with?

11

VITA:VITA: Visual Visual InteractionInteraction Tool Tool forfor

ArchaeologyArchaeology► Multiple usersMultiple users► Multiple displaysMultiple displays

Projected tabletopProjected tabletop HandheldHandheld High-resolution High-resolution

monitormonitor See-through head-See-through head-

wornworn► Multiple interaction Multiple interaction

devicesdevices MERL DiamondTouch MERL DiamondTouch

tabletable EssentialReality P5 EssentialReality P5

glovesgloves Speech inputSpeech input 6DOF tracker6DOF tracker

12

Design ConsiderationsDesign Considerations

►Use the most appropriate display for Use the most appropriate display for the given datathe given data

►Facilitate both human-system and Facilitate both human-system and human-human interactionhuman-human interaction

13

Modular ArchitectureModular Architecture

AR Module

DT ModuleSCREEN Module

HANDHELDModule

AR Module …

DatabaseMessageFacilitator

2D Modules

3D Modules

14

AR Module ComponentsAR Module Components

Head TrackerHand Tracker

(Intersense IS900)

Sony Head-Worn Display(LDI-D100B)

Microphone

P5 Glove(Essential Reality)

DiamondTouch Table Connector

(MERL)

15

Life-size Immersive Life-size Immersive ExplorationExploration

16

3D Multimodal Interaction3D Multimodal Interaction

► Provide natural interaction Provide natural interaction mechanism for our 3D mechanism for our 3D environmentenvironment

► ModalitiesModalities Speech: IBM ViaVoice 10Speech: IBM ViaVoice 10 Gestures: Gestures:

EssentialReality P5 gloveEssentialReality P5 glove Selection statistics: Selection statistics:

SenseShapesSenseShapes

► Focus on selectionFocus on selection Based on collaboration with Phil Based on collaboration with Phil

Cohen et al. (Cohen et al. (ICMI 2003ICMI 2003) and ) and SenseShapes (SenseShapes (ISMAR 2003ISMAR 2003) )

17

VirtualTrayVirtualTray

18

Desktop ComponentsDesktop Components

Touch-sensitive Projective Display

(MERL DiamondTouch)

High-resolutionDisplay

Handheld Display

19

World-In-MiniatureWorld-In-Miniature

20

Harris MatrixHarris Matrix

21

Enhanced Harris Matrix Enhanced Harris Matrix

22

Enhanced Harris MatrixEnhanced Harris Matrix

23

Cross-Dimensional Hybrid Cross-Dimensional Hybrid GesturesGestures

►Synchronized 2D and 3D gesturesSynchronized 2D and 3D gestures►Facilitate seamless transition across Facilitate seamless transition across

dimensionsdimensions

To appear in IEEE VR 2005

24

Cross-Dimensional Hybrid Cross-Dimensional Hybrid GesturesGestures

PullPush

To appear in IEEE VR 2005

25

Cross-Dimensional Hybrid Cross-Dimensional Hybrid GesturesGestures

PullPinDragRotate Push

To appear in IEEE VR 2005

26

Cross-Dimensional Hybrid Cross-Dimensional Hybrid GesturesGestures

PullConnectScaleDisconnect Push

To appear in IEEE VR 2005

27

Handheld Focus-in-Context Handheld Focus-in-Context DisplayDisplay

►Movable high-Movable high-resolution insetresolution inset Tracked by Tracked by

DiamondTouchDiamondTouch Projection Projection

suppressed in its suppressed in its boundsbounds

Physical magic Physical magic lenslens

28

Tabletop InteractionTabletop Interaction

29

User FeedbackUser Feedback

►Overall very positive reactionOverall very positive reaction►Archaeologists benefited from:Archaeologists benefited from:

Temporal–Spatial connectionTemporal–Spatial connection Aggregated collection of all dataAggregated collection of all data Accurate 3D modelAccurate 3D model Simple touch-based interactions Simple touch-based interactions

►Potential for increased collaborationPotential for increased collaboration

30

Room for ImprovementRoom for Improvement► Reduce wiresReduce wires► Reduce weight Reduce weight ► Eye occlusion hinders communication Eye occlusion hinders communication ► Missing data:Missing data:

More objects, features, notes and picturesMore objects, features, notes and pictures More scans during excavation (time-lapse spatial More scans during excavation (time-lapse spatial

record) record) ► Missing features:Missing features:

Virtual scale measure (implemented since)Virtual scale measure (implemented since) Variable site model scalingVariable site model scaling Improved selection in world-in-miniatureImproved selection in world-in-miniature

31

Current and Future WorkCurrent and Future Work

► Larger Site:Larger Site: Summer 2004 - Thulamela, South Summer 2004 - Thulamela, South

AfricaAfrica

► Personalized user experience Personalized user experience based on expertisebased on expertise

► Environment managementEnvironment management

32

AcknowledgmentsAcknowledgments► Special thanks to: Special thanks to:

Peter Allen, Alejandro Troccoli, and Ben Smith (CU Robotics Peter Allen, Alejandro Troccoli, and Ben Smith (CU Robotics Lab)Lab)

Ian Morris and Trinity Jackman (Stanford Archaeology Ian Morris and Trinity Jackman (Stanford Archaeology Center) Center)

Lynn Meskell and James Conlon (CU Dept. of Archaeology) Lynn Meskell and James Conlon (CU Dept. of Archaeology) Sajid Sadi and Avinanindra Utukuri (P5 glove) Sajid Sadi and Avinanindra Utukuri (P5 glove) Shezan BaigShezan Baig Mitsubishi Electric Research Labs (DiamondTouch table)Mitsubishi Electric Research Labs (DiamondTouch table) Alias Systems Alias Systems Microsoft Research Microsoft Research

► Funded in part by NSF ITR Grants IIS-0121239 and Funded in part by NSF ITR Grants IIS-0121239 and IIS-00-82961, and Office of Naval Research IIS-00-82961, and Office of Naval Research Contracts N00014-99-1- 0394, N00014-99-1-0683, Contracts N00014-99-1- 0394, N00014-99-1-0683, N00014-99-1-0249, and N00014-04-1-0005.N00014-99-1-0249, and N00014-04-1-0005.

Questions?Questions?

34