study results on employing virtual and augmented reality ... · & hololens tracking •...
TRANSCRIPT
ESA UNCLASSIFIED - For Official Use
Study Results on Employing Virtual and Augmented Reality for Spacecraft Operations and Astronaut Training at ESOC and EAC
Ruediger Gad (Terma GmbH), Holger Graf (Fraunhofer IGD),Mehran Sarkarati (ESA ESOC), Chris Scott (ESA EAC)
Results of Studies for ESA EAC & ESOCPerformed by Terma GmbH & Fraunhofer IGD
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 2
Outline
• Practical
• High-precision Model-based AR Tracking
• Multi-modal AR Application for Rover Operation
• VR for Astronaut Training
• Integration of AR/VR with ESA Software
• Conceptual
• Agile-like Workflow/Development Processes
• Use Case Identification and Details Development
• Use Case Ideas and Taxonomies
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 3
AR Tracking• “HoloLens Tracking”
• Track “Room”
• Detect “Surfaces”
• Model-based Tracking• Track Objects
• 3D-Model of Object
• “Find Edges”
• Initialisation
• Tracking
• VisionLib(Fraunhofer IGD/Visometry GmbH)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 4
AR Tracking• “HoloLens Tracking”
• Track “Room”
• Detect “Surfaces”
• Model-based Tracking• Track Objects
• 3D-Model of Object
• “Find Edges”
• Initialisation
• Tracking
• VisionLib(Fraunhofer IGD/Visometry GmbH)
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 5
Interactive Training/Manual PoC – Tracking Precision
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 6
Interactive Training/Manual PoC – Visualisation
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 7
Interactive Training/Manual PoC – Visualisation
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 8
Interactive Training/Manual PoC – Visualisation
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 9
• EAC• Astronaut & Rover on
Celestial Body Surface
• Track Real Rover
• ESOC• No Real Rover at AR User
• Virtual Rover Model& HoloLens Tracking
• Scalable View:Detail & Overview Display
Multi-modal AR Application for Rover Operation
Celestial Body SurfaceSpacecraft Orbiting
Celestial Body
Astronaut
RoverBroker(Space)
MRS(Space)AR
SupportEquipment
HoloLens
TM
TC TM
TC
TM/TC
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 10
Multi-modal AR Application for Rover Operation
Celestial Body Surface
Spacecraft Orbiting Celestial Body Ground Control
Operator
RoverBroker(Space)
MRS(Space)
AR Sup. Equip.
HoloLens
TM
TC
TM
TC TM/TC
• EAC• Astronaut & Rover on
Celestial Body Surface
• Track Real Rover
• ESOC• No Real Rover at AR User
• Virtual Rover Model& HoloLens Tracking
• Scalable View:Detail & Overview Display
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 11
Rover Operation Proof-of-Concept
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 12
Rover Operation Proof-of-Concept
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 13
VR for Astronaut Training• Virtual Lunar Base, Fire Emergency Training
• Users/Roles
• Astronaut
• Operator
• Trainer
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 14
VR for Astronaut Training• Virtual Lunar Base, Fire Emergency Training
• Users/Roles
• AstronautTrainee, in VR
• Operator
• Trainer
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 15
VR for Astronaut Training• Virtual Lunar Base, Fire Emergency Training
• Users/Roles
• Astronaut
• OperatorSupport Astronautvia Mission ControlSystem
• Trainer
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 16
VR for Astronaut Training• Virtual Lunar Base, Fire Emergency Training
• Users/Roles
• Astronaut
• Operator
• TrainerPrepare & Supervise Training
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 17
VR for Astronaut Training Models• Behavioural
• MATLAB (Development)
• SIMSAT (Run-time)
• 3D
• Unity(Develop. & Run-time)
• Elements
• Racks• Buttons, Status Display, …
• Fire Extinguishers• …
• Model IntegrationVR <-> Behavioural
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 18
Integration of VR with ESA Software
Preparation Run-time
Key
VR Run-time(MQTT)
Simulation Run-time(CORBA)
MCSRun-time
(CCSDS-MO)
VR Hardware
Integration Middleware
VR Preparation
Simulation Preparation
VR Application
BehaviouralSimulation
I/OInfo.
Preparation Artefact Deployment Runtime Interaction
I/OInfo.
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 19
Integration of AR with ESA SoftwareA
R Su
ppor
t Eq
uipm
ent
Hol
oLe
ns
Spac
e-cr
aft
Gro
und
Cont
rol
AR Application
Message-oriented Middleware
MOE Adapter
MQTT (JSON UTF-8)
OpenWire (JSON UTF-8)
Broker MRS
Rover(Real/Simulated)AR User
TM TC
Rov.TC
Rov. TM
TM/TC
TM/TC
Visualisation (out)Interaction (in)
MCS
ARTM/TC
HoloLens
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 20
Integration of AR/VR with ESA Software
• Virtual Reality (VR) Study
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 21
Integration of AR/VR with ESA Software
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 22
Integration of AR/VR with ESA Software
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 23
Integration of AR/VR with ESA Software
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 24
Integration of AR/VR with ESA Software
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 25
Integration of AR/VR with ESA Software
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 26
Integration of AR/VR with ESA Software
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 27
Integration of AR/VR with ESA Software
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 28
Integration of AR/VR with ESA Software
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 29
Agile-like Workflow/Development Processes
• Concept Definition (Use Cases / Applications)
• Technologies & Integration Assessment
• Practical Assessment
• -> Proof of Concept (PoC)
TechnologyAssessment
Concept Definition
WorkshopPoC
Development & Assessment
Phase 1 Phase 2
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 30
Agile-like Workflow/Development Processes• Concept Definition (Use Cases / Applications)
• “Open”: Broader Scope than the PoC
• Technologies & Integration
• VR/AR & ESA Systems
• Practical Assessment -> Proof of Concept (PoC)
• Selected Use Case(s)
TechnologyAssessment
Concept Definition
WorkshopPoC
Development & Assessment
Phase 1 Phase 2
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 31
Use Case Identification and Details Development
• Interviews/Mini-workshops
• Two Parts• Ideation
• High-level Ideas for Application Scenarios
• Derivation/Definition• More Details about (Selected) High-level Application Scenarios
• “Open”• Broad Scope
• Not Limited by PoC Considerations
• A Basis for Future Activities
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 32
Ideation “Raw” Results Examples
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 33
Application Scenario “Taxonomy”
• Application Domain
• Mission Control
• Robotic Operation
• Virtual Lunar Base (EVA)
• Goal of the Scenario
• Training
• Planning
• Operations
• …
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 34
High-level Scenarios
• Training
• Interactive Training/Manuals
• Virtual (AR) Representationof Training Instructions/Manual
• Interactively show procedures/steps.
• Overlay instructionswith real world.
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 35
High-level Scenarios (Continued)
• Planning
• Rover Operations Planning
• AR Scene withRover & Environment
• Scientists Mark
• Operators Plan
• Managers Verify
• Multi-user?
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 36
High-level Scenarios (Continued)
• Operations
• Spacecraft Operations
• Analyse Situation
• Plan Next Steps
• Multi-user?
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 37
Conclusion & Outlook• VR
• Immersive Training (“Costly” Scenarios)
• Planning (e.g., Mars/Moon Rover Operations)
• AR
• Interactive Training/Manual (Near Future?)
• Cooperative Scenarios with Multiple Users
• User Experience of Devices
• VR more Mature
• AR Headsets still Somewhat Uncomfortable
• Challenge: Bridge the Gap between Users and Technology.Users Utility Technology
• Further activities are beneficial.
ESA UNCLASSIFIED - For Official Use ESA, Terma GmbH, Fraunhofer IGD | 2019-12-02 | Slide 38
End
Thank you very much for your attention.
Questions?
Ruediger Gad for the VR & AR Study Teams