study results on employing virtual and augmented reality ... · & hololens tracking •...

ESA UNCLASSIFIED - For Official Use

Study Results on Employing Virtual and Augmented Reality for Spacecraft Operations and Astronaut Training at ESOC and EAC

Ruediger Gad (Terma GmbH), Holger Graf (Fraunhofer IGD),Mehran Sarkarati (ESA ESOC), Chris Scott (ESA EAC)

Results of Studies for ESA EAC & ESOCPerformed by Terma GmbH & Fraunhofer IGD

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 2

Outline

• Practical

• High-precision Model-based AR Tracking

• Multi-modal AR Application for Rover Operation

• VR for Astronaut Training

• Integration of AR/VR with ESA Software

• Conceptual

• Agile-like Workflow/Development Processes

• Use Case Identification and Details Development

• Use Case Ideas and Taxonomies

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 3

AR Tracking• “HoloLens Tracking”

• Track “Room”

• Detect “Surfaces”

• Model-based Tracking• Track Objects

• 3D-Model of Object

• “Find Edges”

• Initialisation

• Tracking

• VisionLib(Fraunhofer IGD/Visometry GmbH)

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 4

AR Tracking• “HoloLens Tracking”

• Track “Room”

• Detect “Surfaces”

• Model-based Tracking• Track Objects

• 3D-Model of Object

• “Find Edges”

• Initialisation

• Tracking

• VisionLib(Fraunhofer IGD/Visometry GmbH)

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 5

Interactive Training/Manual PoC – Tracking Precision

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 6

Interactive Training/Manual PoC – Visualisation

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 7

Interactive Training/Manual PoC – Visualisation

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 8

Interactive Training/Manual PoC – Visualisation

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 9

• EAC• Astronaut & Rover on

Celestial Body Surface

• Track Real Rover

• ESOC• No Real Rover at AR User

• Virtual Rover Model& HoloLens Tracking

• Scalable View:Detail & Overview Display

Multi-modal AR Application for Rover Operation

Celestial Body SurfaceSpacecraft Orbiting

Celestial Body

Astronaut

RoverBroker(Space)

MRS(Space)AR

SupportEquipment

HoloLens

TM

TC TM

TC

TM/TC

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 10

Multi-modal AR Application for Rover Operation

Celestial Body Surface

Spacecraft Orbiting Celestial Body Ground Control

Operator

RoverBroker(Space)

MRS(Space)

AR Sup. Equip.

HoloLens

TM

TC

TM

TC TM/TC

• EAC• Astronaut & Rover on

Celestial Body Surface

• Track Real Rover

• ESOC• No Real Rover at AR User

• Virtual Rover Model& HoloLens Tracking

• Scalable View:Detail & Overview Display

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 11

Rover Operation Proof-of-Concept

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 12

Rover Operation Proof-of-Concept

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 13

VR for Astronaut Training• Virtual Lunar Base, Fire Emergency Training

• Users/Roles

• Astronaut

• Operator

• Trainer

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 14

VR for Astronaut Training• Virtual Lunar Base, Fire Emergency Training

• Users/Roles

• AstronautTrainee, in VR

• Operator

• Trainer

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 15

VR for Astronaut Training• Virtual Lunar Base, Fire Emergency Training

• Users/Roles

• Astronaut

• OperatorSupport Astronautvia Mission ControlSystem

• Trainer

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 16

VR for Astronaut Training• Virtual Lunar Base, Fire Emergency Training

• Users/Roles

• Astronaut

• Operator

• TrainerPrepare & Supervise Training

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 17

VR for Astronaut Training Models• Behavioural

• MATLAB (Development)

• SIMSAT (Run-time)

• 3D

• Unity(Develop. & Run-time)

• Elements

• Racks• Buttons, Status Display, …

• Fire Extinguishers• …

• Model IntegrationVR <-> Behavioural

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 18

Integration of VR with ESA Software

Preparation Run-time

Key

VR Run-time(MQTT)

Simulation Run-time(CORBA)

MCSRun-time

(CCSDS-MO)

VR Hardware

Integration Middleware

VR Preparation

Simulation Preparation

VR Application

BehaviouralSimulation

I/OInfo.

Preparation Artefact Deployment Runtime Interaction

I/OInfo.

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 19

Integration of AR with ESA SoftwareA

R Su

ppor

t Eq

uipm

ent

Hol

oLe

ns

Spac

e-cr

aft

Gro

und

Cont

rol

AR Application

Message-oriented Middleware

MOE Adapter

MQTT (JSON UTF-8)

OpenWire (JSON UTF-8)

Broker MRS

Rover(Real/Simulated)AR User

TM TC

Rov.TC

Rov. TM

TM/TC

TM/TC

Visualisation (out)Interaction (in)

MCS

ARTM/TC

HoloLens

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 20

Integration of AR/VR with ESA Software

• Virtual Reality (VR) Study

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 21

Integration of AR/VR with ESA Software

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 22

Integration of AR/VR with ESA Software

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 23

Integration of AR/VR with ESA Software

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 24

Integration of AR/VR with ESA Software

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 25

Integration of AR/VR with ESA Software

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 26

Integration of AR/VR with ESA Software

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 27

Integration of AR/VR with ESA Software

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 28

Integration of AR/VR with ESA Software

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 29

Agile-like Workflow/Development Processes

• Concept Definition (Use Cases / Applications)

• Technologies & Integration Assessment

• Practical Assessment

• -> Proof of Concept (PoC)

TechnologyAssessment

Concept Definition

WorkshopPoC

Development & Assessment

Phase 1 Phase 2

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 30

Agile-like Workflow/Development Processes• Concept Definition (Use Cases / Applications)

• “Open”: Broader Scope than the PoC

• Technologies & Integration

• VR/AR & ESA Systems

• Practical Assessment -> Proof of Concept (PoC)

• Selected Use Case(s)

TechnologyAssessment

Concept Definition

WorkshopPoC

Development & Assessment

Phase 1 Phase 2

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 31

Use Case Identification and Details Development

• Interviews/Mini-workshops

• Two Parts• Ideation

• High-level Ideas for Application Scenarios

• Derivation/Definition• More Details about (Selected) High-level Application Scenarios

• “Open”• Broad Scope

• Not Limited by PoC Considerations

• A Basis for Future Activities

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 32

Ideation “Raw” Results Examples

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 33

Application Scenario “Taxonomy”

• Application Domain

• Mission Control

• Robotic Operation

• Virtual Lunar Base (EVA)

• Goal of the Scenario

• Training

• Planning

• Operations

• …

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 34

High-level Scenarios

• Training

• Interactive Training/Manuals

• Virtual (AR) Representationof Training Instructions/Manual

• Interactively show procedures/steps.

• Overlay instructionswith real world.

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 35

High-level Scenarios (Continued)

• Planning

• Rover Operations Planning

• AR Scene withRover & Environment

• Scientists Mark

• Operators Plan

• Managers Verify

• Multi-user?

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 36

High-level Scenarios (Continued)

• Operations

• Spacecraft Operations

• Analyse Situation

• Plan Next Steps

• Multi-user?

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 37

Conclusion & Outlook• VR

• Immersive Training (“Costly” Scenarios)

• Planning (e.g., Mars/Moon Rover Operations)

• AR

• Interactive Training/Manual (Near Future?)

• Cooperative Scenarios with Multiple Users

• User Experience of Devices

• VR more Mature

• AR Headsets still Somewhat Uncomfortable

• Challenge: Bridge the Gap between Users and Technology.Users Utility Technology

• Further activities are beneficial.

ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 38

End

Thank you very much for your attention.

Questions?

Ruediger Gad for the VR & AR Study Teams

ruga@terma.com