![Page 1: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/1.jpg)
ESA UNCLASSIFIED - For Official Use
Study Results on Employing Virtual and Augmented Reality for Spacecraft Operations and Astronaut Training at ESOC and EAC
Ruediger Gad (Terma GmbH), Holger Graf (Fraunhofer IGD),Mehran Sarkarati (ESA ESOC), Chris Scott (ESA EAC)
Results of Studies for ESA EAC & ESOCPerformed by Terma GmbH & Fraunhofer IGD
![Page 2: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/2.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 2
Outline
• Practical
• High-precision Model-based AR Tracking
• Multi-modal AR Application for Rover Operation
• VR for Astronaut Training
• Integration of AR/VR with ESA Software
• Conceptual
• Agile-like Workflow/Development Processes
• Use Case Identification and Details Development
• Use Case Ideas and Taxonomies
![Page 3: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/3.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 3
AR Tracking• “HoloLens Tracking”
• Track “Room”
• Detect “Surfaces”
• Model-based Tracking• Track Objects
• 3D-Model of Object
• “Find Edges”
• Initialisation
• Tracking
• VisionLib(Fraunhofer IGD/Visometry GmbH)
![Page 4: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/4.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 4
AR Tracking• “HoloLens Tracking”
• Track “Room”
• Detect “Surfaces”
• Model-based Tracking• Track Objects
• 3D-Model of Object
• “Find Edges”
• Initialisation
• Tracking
• VisionLib(Fraunhofer IGD/Visometry GmbH)
![Page 5: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/5.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 5
Interactive Training/Manual PoC – Tracking Precision
![Page 6: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/6.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 6
Interactive Training/Manual PoC – Visualisation
![Page 7: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/7.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 7
Interactive Training/Manual PoC – Visualisation
![Page 8: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/8.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 8
Interactive Training/Manual PoC – Visualisation
![Page 9: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/9.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 9
• EAC• Astronaut & Rover on
Celestial Body Surface
• Track Real Rover
• ESOC• No Real Rover at AR User
• Virtual Rover Model& HoloLens Tracking
• Scalable View:Detail & Overview Display
Multi-modal AR Application for Rover Operation
Celestial Body SurfaceSpacecraft Orbiting
Celestial Body
Astronaut
RoverBroker(Space)
MRS(Space)AR
SupportEquipment
HoloLens
TM
TC TM
TC
TM/TC
![Page 10: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/10.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 10
Multi-modal AR Application for Rover Operation
Celestial Body Surface
Spacecraft Orbiting Celestial Body Ground Control
Operator
RoverBroker(Space)
MRS(Space)
AR Sup. Equip.
HoloLens
TM
TC
TM
TC TM/TC
• EAC• Astronaut & Rover on
Celestial Body Surface
• Track Real Rover
• ESOC• No Real Rover at AR User
• Virtual Rover Model& HoloLens Tracking
• Scalable View:Detail & Overview Display
![Page 11: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/11.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 11
Rover Operation Proof-of-Concept
![Page 12: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/12.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 12
Rover Operation Proof-of-Concept
![Page 13: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/13.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 13
VR for Astronaut Training• Virtual Lunar Base, Fire Emergency Training
• Users/Roles
• Astronaut
• Operator
• Trainer
![Page 14: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/14.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 14
VR for Astronaut Training• Virtual Lunar Base, Fire Emergency Training
• Users/Roles
• AstronautTrainee, in VR
• Operator
• Trainer
![Page 15: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/15.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 15
VR for Astronaut Training• Virtual Lunar Base, Fire Emergency Training
• Users/Roles
• Astronaut
• OperatorSupport Astronautvia Mission ControlSystem
• Trainer
![Page 16: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/16.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 16
VR for Astronaut Training• Virtual Lunar Base, Fire Emergency Training
• Users/Roles
• Astronaut
• Operator
• TrainerPrepare & Supervise Training
![Page 17: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/17.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 17
VR for Astronaut Training Models• Behavioural
• MATLAB (Development)
• SIMSAT (Run-time)
• 3D
• Unity(Develop. & Run-time)
• Elements
• Racks• Buttons, Status Display, …
• Fire Extinguishers• …
• Model IntegrationVR <-> Behavioural
![Page 18: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/18.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 18
Integration of VR with ESA Software
Preparation Run-time
Key
VR Run-time(MQTT)
Simulation Run-time(CORBA)
MCSRun-time
(CCSDS-MO)
VR Hardware
Integration Middleware
VR Preparation
Simulation Preparation
VR Application
BehaviouralSimulation
I/OInfo.
Preparation Artefact Deployment Runtime Interaction
I/OInfo.
![Page 19: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/19.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 19
Integration of AR with ESA SoftwareA
R Su
ppor
t Eq
uipm
ent
Hol
oLe
ns
Spac
e-cr
aft
Gro
und
Cont
rol
AR Application
Message-oriented Middleware
MOE Adapter
MQTT (JSON UTF-8)
OpenWire (JSON UTF-8)
Broker MRS
Rover(Real/Simulated)AR User
TM TC
Rov.TC
Rov. TM
TM/TC
TM/TC
Visualisation (out)Interaction (in)
MCS
ARTM/TC
HoloLens
![Page 20: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/20.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 20
Integration of AR/VR with ESA Software
• Virtual Reality (VR) Study
![Page 21: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/21.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 21
Integration of AR/VR with ESA Software
![Page 22: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/22.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 22
Integration of AR/VR with ESA Software
![Page 23: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/23.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 23
Integration of AR/VR with ESA Software
![Page 24: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/24.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 24
Integration of AR/VR with ESA Software
![Page 25: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/25.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 25
Integration of AR/VR with ESA Software
![Page 26: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/26.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 26
Integration of AR/VR with ESA Software
![Page 27: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/27.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 27
Integration of AR/VR with ESA Software
![Page 28: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/28.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 28
Integration of AR/VR with ESA Software
![Page 29: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/29.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 29
Agile-like Workflow/Development Processes
• Concept Definition (Use Cases / Applications)
• Technologies & Integration Assessment
• Practical Assessment
• -> Proof of Concept (PoC)
TechnologyAssessment
Concept Definition
WorkshopPoC
Development & Assessment
Phase 1 Phase 2
![Page 30: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/30.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 30
Agile-like Workflow/Development Processes• Concept Definition (Use Cases / Applications)
• “Open”: Broader Scope than the PoC
• Technologies & Integration
• VR/AR & ESA Systems
• Practical Assessment -> Proof of Concept (PoC)
• Selected Use Case(s)
TechnologyAssessment
Concept Definition
WorkshopPoC
Development & Assessment
Phase 1 Phase 2
![Page 31: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/31.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 31
Use Case Identification and Details Development
• Interviews/Mini-workshops
• Two Parts• Ideation
• High-level Ideas for Application Scenarios
• Derivation/Definition• More Details about (Selected) High-level Application Scenarios
• “Open”• Broad Scope
• Not Limited by PoC Considerations
• A Basis for Future Activities
![Page 32: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/32.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 32
Ideation “Raw” Results Examples
![Page 33: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/33.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 33
Application Scenario “Taxonomy”
• Application Domain
• Mission Control
• Robotic Operation
• Virtual Lunar Base (EVA)
• Goal of the Scenario
• Training
• Planning
• Operations
• …
![Page 34: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/34.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 34
High-level Scenarios
• Training
• Interactive Training/Manuals
• Virtual (AR) Representationof Training Instructions/Manual
• Interactively show procedures/steps.
• Overlay instructionswith real world.
![Page 35: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/35.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 35
High-level Scenarios (Continued)
• Planning
• Rover Operations Planning
• AR Scene withRover & Environment
• Scientists Mark
• Operators Plan
• Managers Verify
• Multi-user?
![Page 36: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/36.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 36
High-level Scenarios (Continued)
• Operations
• Spacecraft Operations
• Analyse Situation
• Plan Next Steps
• Multi-user?
![Page 37: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/37.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 37
Conclusion & Outlook• VR
• Immersive Training (“Costly” Scenarios)
• Planning (e.g., Mars/Moon Rover Operations)
• AR
• Interactive Training/Manual (Near Future?)
• Cooperative Scenarios with Multiple Users
• User Experience of Devices
• VR more Mature
• AR Headsets still Somewhat Uncomfortable
• Challenge: Bridge the Gap between Users and Technology.Users Utility Technology
• Further activities are beneficial.
![Page 38: Study Results on Employing Virtual and Augmented Reality ... · & HoloLens Tracking • Scalable View: Detail & Overview Display ... Workshop. PoC Development & Assessment. Phase](https://reader034.vdocuments.net/reader034/viewer/2022050515/5f9eecbd996feb328a477666/html5/thumbnails/38.jpg)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 38
End
Thank you very much for your attention.
Questions?
Ruediger Gad for the VR & AR Study Teams