xiangyu wang

Applications of Mixed Reality in Applications of Mixed Reality in ArchitectureArchitecture, Engineering, and

ConConstruction: Specification, truction: Specification,

Prototype, and EvaluationPrototype, and Evaluation

Xiangyu WangXiangyu Wang

Outline

Background Specification Prototype Development System Evaluation Summaries and Conclusions

Background

Mixed Reality (MR)

AugmentedReality (AR)

AugmentedVirtuality (AV)

Reality-Virtuality (RV) Continuum

RealEnvironment

(RE)

VirtualEnvironment

(VE)

Mixed Reality (MR): an environment where real world and virtual world objects are presented together on a single display (Milgram & Kishino 1994; Milgram and Colquhoun 1999).

Background

Goal: To systematically and comprehensively transfer the

available MR-based technology into Architecture, Engineering, and Construction (AEC) arenas.

Objectives To develop a structured specification for mapping the

available MR-based technology to specific tasks in AEC. To develop prototypes named Mixed Reality-based

collaborative virtual environments (MRCVEs) to transform the way of current design review collaboration.

To evaluate the above prototype systems in the aspect of benefits validation and usability engineering.

Classify MR Technology Analyze AEC Tasks

Map Technology to Task

Face to Face Conferencing Scenario

Usability Evaluation

Virtual Space Conferencing Scenario

PROTOTYPE DEVELOPMENT

SPECIFICATION METHODOLOGY

PROTOTYPE EVALUATION

Benefit ValidationFace-to-Face Conferencing Scenario V.S. Current Design Review Meeting

Virtual Space Scenario V.S. Current Web-based Design Collaboration

System ImprovementsEvaluation Methods

Current Collaboration Mechanisms

Groupware Issues

Concurrent Engineering, DVE, CVE

Identify Feasible MRCVE Scenarios

MRCVE Prototype Development

Specification: phase 1

Phase 1- Specify MR Technology: Classify MR (AR) based on four technological components:

Media Representation Input Mechanism Output Mechanism Tracking Technology

Specification covers only major classes of devices.

Results founded and knowledge learnt in phase 1 lays foundation of mapping technology to tasks.


Classifying the Media Representation (as an example of findings in phase 1) Abstract-Concrete, or Schematic to 3D: high-

fidelity representation is not necessarily superior over more abstract one because each type has its own appropriate application area.

Abstract Augmenting Content Continuum

Concrete

Tex

t

2D I

mag

e &

V

ideo

3D O

bje

ct

3D

Wir

efra

me

3D D

ata

Ind

icat

ion

Pla

tfor

m,

Tab

let,

S

cree

n


Other “continuums” for input metaphor, output metaphor, and tracking technology based on human’s cognitive aspect were also developed and elaborated in dissertation. All of these findings were used in mapping technology to AEC tasks (phase 3).


Phase 2 – Analyze AEC Tasks

Factors from task influencing the applicability of MR technological components: Task mental requirements Working environment Physical disposition Hand occupation


Phase 3: Map MR technology to AEC tasks based technological feasibility and usability in terms of physical and mental (human) factors

Influencing Factors(Task Side)

MR System Technological Components

(MR System Side)

Task Mental Requirements Media Representation

Working Environment Input Mechanism

Physical Disposition Output Mechanism

Hands Occupation Tracking Technology

Task Analysis

Wor

king

Enviro

nmen

t

Hand

Occupat

ion

Task Mental

Requirements

Physical

Movem

ent

Track

ing

Tech

nologyIn

pu

t M

ech

anis

m

OutputMechanism

MediaRepresentation

User Level

User Layer

MR Com

ponent

LayerTask Layer

User Layer

Task Layer

A User-Centered Framework of Layer Interactions

Methodology (procedure) for MR System Development Cycle

Task/Operation

Analyze & Breakdown

Analyze Composite Tasks

Cognitive TasksPerceptual Tasks

Information Processing Model

Feasibility

Task Mental Requirement Working Environment Physical Disposition Hand Occupation

Physical Mental

Usability

Media Representation

FeasibilityPhysical Mental

Usability

Input Mechanism


Usability

Output Mechanism


Usability

Tracking Technology

MR System Prototype

Expert Heuristic Evaluation Formative User-centered Evaluation Useful MR System

Site observation, interview etc.

Ste

p 1

Ste

p 2

Ste

p 3


Design specification and guidelines (AEC tasks) Media Representation Input Mechanism Output Mechanism Tracking Technology

Prototype Development

Vision: To explore Mixed Reality (MR)-based tools that can provide the benefit of both 3D modeling and effective real-time collaboration to achieve design coordination objectives.


MRCVE : Mixed Reality – based collaborative virtual environment to realize design review collaboration through face-to-face conferencing or virtual space conferencing.

Technology mapping to review collaboration task for realizing a Mixed Reality Collaborative Virtual Environment (MRCVE) was implemented using the methodology described earlier.


Step 1 — analyze the design review task

Scan, observeInspect, discriminate,

inspectLocate, identify

Encode, estimate, compare, analyze, plan

Write

Perceptual Task

Cognitive Task

Motor Task

Composite Task

ANNOTATE design


Step 1 — analyze the design review task (cont’d)

Factors Analysis

Mental requirements On composite task level, the perceptual tasks such as object inspection, identification and location, and cognitive tasks such as distance and orientation estimation, refer to an egocentric frame of reference. Also cognitive performance demands the design to be rich enough to reach a high degree of realism.

Physical disposition Move around the table either standing or sitting. Although the working space is predominantly around the table, roaming space and working volume are not limited.

Working environments

Indoor and quiet environment

Hands occupation Hands for interaction with digital content by manipulating tracking control device


Step 2 — map the technology to task: Media representation: high-fidelity

representations; Input device: tangible input; Output device: video-based See-through head-

mounted-display: ARvision-stereoscopic HMD with a color video camera attached;

Tracker: large-scale pattern recognition.

Video and audio communication: Commercial Netmeeting software.


Tangible interface

Prototype Development Application Scenarios

1. Face-to-face Scenario 2. Virtual Space Scenario

3. Mixed Scenario 4. Office-to-field Scenario 5. Field-to-office Scenario

Evaluation

Evaluation: Benefits validation through experiments

To validate the benefits MRCVE application scenarios over the prevalent method.

System usability evaluation Implement usability engineering evaluation on current MRCVE

prototype against certain AR design guidelines.

Evaluation: benefits validation Design of experiment 1: Face-to-Face Conferencing

Scenario V.S. Prevalent Design Review Benchmark: Paper-based 3D drawing review collaboration

.

Hypotheses: When compared to traditional paper-based drawing media, Hypotheses 1: MRCD face-to-face scenario will significantly reduce the

amount of time to complete task. Hypotheses 2: MRCD face-to-face scenario will significantly reduce the

workload of design review task.

Methodology: Experiment Post-test Questionnaire: subjects need to fill it in based on their gained

experience from the experiments. NASA task load index (TLX) to measure and compare the workload of using

alternatives. Direct observation or monitor of subjects’ collaborative performance by

experimenter.

Evaluation: benefits validation

Stimulus materials: Large-scale and simple models and corresponding 3D drawings are adapted from real projects of BMW contractor.

Subject: 16 engineering undergraduate and graduate students in Purdue. Every two subjects form a group for each treatment.

Measurement: time of completion and perceived workload.

Procedure: Training session: Subjects were assigned enough time to practice how to

use the different platforms. Pre-experiment setting: Two subjects in one group played with two

different sub-models (A and B) in AutoCAD 3D environment Design error education: Every subject learned 4 design error patterns (3

known in common) Real experiment: Two subjects sat together and started error-identifying in

model C (A and B combined in a certain way) Post-session Questionnaire: Filled in post-test questionnaires and the

NASA TLX rating.


Experimental Treatments:

Paper-based 3D Drawing MRCD Face-to-face Conferencing

Combination 1 Combination 2

Period Method Pipe Model Period Method Pipe Model

Ι Paper P1 Ι Paper P2

ΙΙ MRCD P2 ΙΙ MRCD P1

Combination 3 Combination 4

Period Method Pipe Model Period Method Pipe Model

Ι MRCD P2 Ι MRCD P1

ΙΙ Paper P1 ΙΙ Paper P2

Incomplete Block Design (Single Replication of Four Group – Two Period Crossover Design)


Experimental statistical design

Statistical Model:

• Y = The time of detecting a conflict • M = the direct fixed effect of the nth method• T = the direct fixed effect of the gth pipe model • P = the fixed effect of the jth period. • , random fluctuations which are independent and

normally distributed with mean 0 and variance .

)()()( jgn PTMY


),0( N


Effect of treatments on time of completion

Raw Time Plot for Each Treatment Combination

0:00

4:48

9:36

14:24

19:12

0:00

MRCVE+P1 MRCVE+P2 Paper+P1 Paper+P2

Treatment Combination

Raw

Per

form

ance

Tim

e (m

in:s

ec)

5:24

15:09

0:00

2:24

4:48

7:12

9:36

12:00

14:24

16:48

MRCVE PaperDrawing

Treatment Conditions

Me

an

Pe

rfo

rma

nc

e T

ime

(M

in:S

ec

)


Statistical Results from SAS System

Source DF Mean Square F Value Pr > F (P Value)

Method 1 380.1525063 19.83 0.0008

Model 1 1.7622563 0.09 0.7669

Per 1 0.2376563 0.01 0.9132

Method*Model 20 12.3376563 0.62 0.4465


)()()( jgn PTMY

)(nMY

An F-test was implemented to the model to further validate the simplification. An F-Value 0.052 with corresponding P-value as 0.95 demonstrated insignificance of this simplification.


Combination Mean Value (s) Median Value (s)

MRCVE+P1 5:55 5:43

Paper Drawing+P1 13:56 12:23

MRCVE+P2 4:51 5:06

Paper Drawing+P2 16:21 16:36

)(nMY

A t-test was further implemented to model and yielded an estimated performance difference for these two methods, which is 9.75 mins (with a P-value equaling to 0.0003).

Discussion

Mean and Median Value of Each Combination


Effect of treatments on workload (NASA TLX) F-value is 0.95 and p-value is 0.3385 (insignificant)

10.40211.838

0

5

10

15

20

25

MRCVE Paper Drawing Full Score


Mea

n R

atin

g S

co

re (

20 p

oin

t sc

ale

)

Maximum Possible Rating


NASA TLX Rating Scores

11.9

10.4

5.6

11.212 11.6 11.6

10.4

8.7

13.9

8.7

11.5

6.87.5

0

2

4

6

8

10

12

14

16

Mean

Rati

ng

Sco

re (

20 p

oin

t scale

)

Paper Drawing MRCVE

Rating Categories F value p-value Significance

Mental demand 0.86 0.3623 Insignificant

Physical demand 23.5 <0.0001 Significant

Temporal demand 3.04 0.0913 Insignificant

Effort 0.07 0.7965 Insignificant

Performance 7.78 0.0091 Significant

Frustration level 4.71 0.0381 Significant

Statistical Results for the Each NASA TLX Rating Category

Evaluation: benefits validation Questionnaire Results (Subsection 1): Scale: 1 2 3 4 5 6 poor excellent

Evaluation: benefits validation Questionnaire Results (Subsection 2): Scale: O O O O Totally agree Totally disagree

Q1: I felt that 3D interactivity in the MRCVE system aided design comprehension. 25%; 32%; 37%; 6%. 57%

Q2: Overall, compared with paper drawing, the AR system better facilitates design collaboration tasks. 25%; 37%; 25%; 13%. 62%

Q3: The MRCVE system better facilitated communication. 19%; 12%; 38%; 31%. 69% Q4: The MRCVE system better facilitated creativity. 50%; 50%; 0%; 0%. 100% Q5: The MRCVE system better facilitated problem-solving. 44%; 31%; 19%; 6%. 75% Q6: The AR system increased the overall quality of output from the collaboration. 6%;

38%; 43%; 13%. 44% Q7: The AR system better facilitated the quantity of work I could complete in a given

amount of time. 36%; 32%; 20%; 12%. 68% Q8: The AR system increased the quality of my contribution to the project. 32%; 30%;

32%; 6%. 62% Q9: The MRCVE system increased my satisfaction with the outcome of the collaboration.

19%; 55%; 20%; 6%. 74% Q10: The AR system increased understanding between my collaborator and me. 13%; 38%;

25%; 24%. 50%

Evaluation: benefits validation Design of experiment 2: Virtual Space Conferencing

Scenario V.S. Web-based Design Collaboration

Benchmark: NavisWorks Roamer

Hypotheses: When compared to NavisWorks, Hypotheses 1: MRCD virtual space scenario will significantly reduce time for

performing the design review task. Hypotheses 2: MRCD virtual space scenario will significantly reduce the

workload of design review task.

Methodology: Experiment Questionnaire: subjects need to fill it in based on their gained experience from

the experiments. NASA task load index (TLX) to measure and compare the workload of using

alternatives. Direct observation or monitor of subjects’ collaborative performance by

experimenter.


Stimulus materials: Cluttered 3D models are adapted from real projects of BMW contractor.

Subject: 16 engineering undergraduate and graduate students in Purdue. Every two subjects form a group for each treatment.

Measurement: time of completion and perceived workload.

Procedure: Training session: Subjects were assigned enough time to practice how to use the different

platforms. Pre-experiment setting: Two subjects in one group played with two different sub-models

(A and B) in AutoCAD 3D environment Design error education: Every subject learned 4 design error patterns (3 known in

common) Real experiment: Two subjects sat together and started error-identifying in model C (A

and B combined in a certain way) Post-session Questionnaire: Filled in post-test questionnaires and the NASA TLX rating.


NavisWorks Collaboration Treatment

MRCD Virtual Space Conferencing Treatment


Experimental statistical design The same as for experiment 1.


Effect of treatments on time of completion

8.5

25.6

0

5

10

15

20

25

30

MRCVE NavisWorks


Per

form

ance

Tim

e (m

in)

Raw Time Plot for Each Treatment Combination

05

1015202530354045

MRCVE+P1 MRCVE+P2 NavisWorks+P1 NavisWorks+P2

Treatment Combination

Raw

Per

form

ance

Tim

e (m

ins)


Statistical Results from SAS System

Source DF Mean Square F Value Pr > F (P Value)

Method 1 1183.016025 28.91 0.0002

Model 1 28.143025 0.69 0.4245

Per 1 13.359025 0.33 0.5792

Method*Model 1 4.928400 0.12 0.7351


)()()( jgn PTMY

)(nMY

An F-test was implemented to the model to further validate the simplification. An F-Value 0.547 with corresponding P-value as 0.59 demonstrated insignificance of this simplification.


Combination Mean Value (min) Median Value (min)

MRCVE+P1 7.68 5.93

NavisWorks+P1 23.72 25.72

MRCVE+P2 9.18 8.62

NavisWorks+P2 27.47 27.72

)(nMY

A t-test was further implemented to model and yielded an estimated performance difference for these two methods, which is 17.2 mins (with a P-value equaling to 0.0001).

Discussion

Mean and Median Value of Each Combination


Effect of treatments on workload (NASA TLX) F-value is 4.92 and p-value is 0.047 (Significant)

9.535

12.932

0

5

10

15

20

25

MRCVE NavisWorks Full Score

Mean

Rati

ng

Sco

re (

20 p

oin

ts)

Maximum Possible Rating


Rating Categories F value p-value Significance

Mental demand 0.03 0.855 Insignificant

Physical demand 1.98 0.169 Insignificant

Temporal demand 8.45 0.0068 Significant

Effort 8.75 0.006 Significant

Performance 22.98 <0.0001 Significant

Frustration level 0.53 0.4712 Insignificant

Statistical Results for the Each NASA TLX Rating Category

NASA TLX Rating Scores

12.9

10.4

8.6

11.6

14.7

12.911.3

9.510.8

11.8

6.3

8.9

5.4

9.7

0

2

4

6

8

10

12

14

16

Total

Wor

kload

Men

tal

Physic

al

Tempo

ral

Effort

Perfo

rman

ce

Frustr

ation

Rating Category

Me

an

Ra

tin

g S

co

re (

20

po

ints

sc

ale

) NavisWorks MRCVE

Evaluation: benefits validationQuestionnaire Results (Subsection 1):Scale: 1 2 3 4 5 poor excellent

Evaluation: benefits validation Questionnaire Results (Subsection 2): Scale: O O O O O Totally agree Neutral Totally disagree

Q1: I felt that 3D interactivity in the MRCVE system aided design comprehension better than the 3D interactivity in NavisWorks. 31%; 38%; 13%; 13%; 5%. 69%

Q2: Overall, compared with NavisWorks, the AR system better facilitates design collaboration tasks. 13%; 56%; 0%; 26%; 5%. 69%.

Q3: The MRCVE system better facilitated communication. 19%; 44%; 6%; 19%; 12%. 63%

Q4: The MRCVE system better facilitated creativity. 19%; 44%; 26%; 6%; 5%. 63% Q5: The MRCVE system better facilitated problem-solving. 13%; 52%; 6%; 29%;

0%. 65% Q6: The AR system increased the overall quality of output from the collaboration.

13%; 44%; 13%; 31%; 0%. 57% Q7: The AR system better facilitated the quantity of work I could complete in a given

amount of time. 44%; 26%; 13%; 13%; 4%. 70% Q8: The AR system increased the quality of my contribution to the project. 26%;

44%; 6%; 18%; 6%. 70% Q9: The MRCVE system increased my satisfaction with the outcome of the

collaboration. 13%; 50%; 19%; 13%; 5%. 63% Q10: The AR system increased understanding between my collaborator and me. 13%;

19%; 31%; 26%; 11%. 32%

Evaluation: Usability

Heuristic evaluation AR usability guidelines (Gabbard 1997) Our specification and design guidelines

Formative user-centered evaluation The two experiments mentioned earlier were also implemented

as usability experiments


Issues Mean Ratings Explanation Usability Suggestions

Did you often feel disoriented?

4.0 • Users felt a little disoriented with nothing in the mixed scene to use as navigational cues or landmarks. This was expected from the heuristic evaluation phase.

• The presence of directional cues, such as on-screen compass or, a navigational grid and/or a navigational map may have a positive effect on users' ability to perform navigational tasks.

Is the AR system comfortable for long-term use?

1.69 Very low score demonstrates the most important weakness of the HMD. Bulky HMD and limited cable length compromise the most significant usability issue of MRCVE system.

Easy user fatigue: Tethered by video cabling, limiting user mobility to cable length.

• High-bandwidth wireless HMD or large screen projector.

Is tracking marker lightweight, portable, non-encumbering, and comfortable thereby avoiding issues of limiting your mobility and fatigue?

4.63 Tracking ball appears to have high mobility, being light and portable.

The current size of the ball may occlude virtual objects or the entire virtual display as users are prone to holding the props.

• Reduce the size of the virtual tracking ball

Results and Interpretation of Usability Analysis for Face-to-face Conferencing Scenario. Scale: 1 2 3 4 5 6 (very little) (very much)


Would you be resistant to using face-to-face conferencing scenario system or similar MR systems in the future?

About 81.3% (13) gave negative response.

Would you embrace the opportunity to use the face-to-face conferencing scenario system again in the future?

About 69% (11) gave positive response.


Issues Mean Rating Explanation Usability Suggestions

Did you often feel disoriented?

2.93 • Neutral

Is the AR system comfortable for long-term use?

2.06 • Low score demonstrates the most important weakness of HMD. Bulky HMD.

• Large screen projector.

Is tracking marker lightweight, portable, non-encumbering, and comfortable thereby avoiding issues of limiting your mobility and fatigue?

3.94 • Size of multiple tracking markers is appropriate.

• Current size might not work for even larger models and tall models. In this case, users might have difficulties in moving around, in order to review a dead corner.

• A zooming feature controlled by a hand-held device might help.

• An elevation changing function can be programmed into a hand-held device, which is used to change the elevation

Results and Interpretation of Usability Analysis for Virtual Space Conferencing Scenario. Scale: 1 2 3 4 5 (very little) (very much)


Would you be resistant to using virtual space conferencing scenario or similar MR systems in the future?

About 87.5% (14) gave negative response.

Would you embrace the opportunity to use the virtual space conferencing system again in the future?

About 81.3% (13) gave positive response.

Summaries and Conclusions

Major Original Contributions of Research Work Developed a thorough methodology for mapping

appropriate MR technological components to AEC tasks. Developed usable and intuitive Mixed Reality-based

collaborative virtual environment prototypes: face-to-face conferencing scenario and virtual space conferencing scenario.

Validated the benefits of MRCVE over prevalent methods in realistic environments.

Developed a framework of usability engineering evaluation and implemented heuristic and formative usability evaluation for the two prototype.

Summaries and Conclusions Conclusions:

Experiment 1 (face-to-face vs. 3D paper-based drawing) Face-to-face conferencing scenario enabled subjects to finish same

error detection task with 9.75 mins less than 3D paper-based drawing.

There is no significant difference in the workload between face-to-face and paper-based methods. Subjects felt less frustrated and more satisfied with their performance in

using MR system. Using MR system is much more physically demanding due to the usability

issues inherent in MR system. Experiment 2 (virtual space vs. NavisWoks Roamer)

Virtual space conferencing scenario averagely reduced performance time by 17.2 mins compared against NavisWorks roamer.

There is significant difference in the workload between virtual space scenario and NavisWorks roamer. Subjects felt less time pressure and more satisfied with their performance

in using MR system.


Conclusions (cont.):

Attitude of users towards the effectiveness of MRCVE systems on collaborative work was surveyed. Majority of users would embrace the opportunity to use the MRCVE systems again in the future.

Suggestions for further improvements of the human-machine interface of the MRCVE system were also produced based on the usability evaluation.


Future Work Increase the usability of the MRCVE system

based on the results from usability evaluation. Industry evaluation: industry involvement in

evaluating the further development of MRCVE. The remaining two application scenarios of

MRCVE are to be explored in the future employing other tracking options.

xiangyu wang

Documents

aec tasks phase

available mrbased technology

map mr technology

aec tasksfactors

usability engineering

construction aec arenas

applications of mixed

specific tasks