the molecular control toolkit - controlling 3d molecular graphics via gesture and voice
TRANSCRIPT
The Molecular Control Toolkit: Controlling 3D Molecular Graphics via Gesture and Voice
Quan Le HCI Lab
17th Mar, 2015
Sabir, K. ; Garvan Inst. of Med. Res., Australia ; Stolte, C. ; Tabor, B. ; O'Donoghue, S.I. IEEE Symposium on Biological Data Visualization 2013, Atlanta, Georgia, USA
Contents
Introduction
Outstanding
Toolkit Overview
Toolkit Architecture
Toolkit Design
Results
Future Work
Conclusion
2
Fig 1. Molecular Control Toolkit using Leap Motion and Aquaria
Introduction
3
Introduction Outstanding
Toolkit Overview Toolkit Architecture
Toolkit Design Result
Future work Conclusion
- Three-dimensional (3D) molecular graphics systems are widely used in the life sciences.
- Biological systems requires use of molecular graphics to examine atomic-resolution, protein, DNA, …
- Precise control 3D molecular structures can be challenging for occasional users, event experts.
Introduction
4
Fig 3. Aquaria application is accessed by http://aquaria.ws
Fig 2. Molecular Control Toolkit runs as a standalone Java application
Introduction Outstanding
Toolkit Overview Toolkit Architecture
Toolkit Design Result
Future work Conclusion
- Molecular Control Toolkit supports multiple gestures and voice recognition.
- Apply Toolkit to Aquaria molecular graphics system.
Outstanding
- The common interactions: translation, rotation, zooming, selection of specific items, change display mode, …
- Current applications: • In 1996, Pavlovic: using hand gesture and speech interface.
• Multi-touch display: applies six degrees of freedom (6DoF*), only uses finger gestures.
• Sony’s EyeToy, Playstation Eye, uses audio gesture control.
• Nintendo Wii and Sony Move: requires a physical device to manipulate objects in 3D space.
5
* 6DoF: Six degrees of freedom, comprising of three axes for rotation and three for translation.
Introduction Outstanding
Toolkit Overview Toolkit Architecture
Toolkit Design Result
Future work Conclusion
Outstanding
- Molecular Control Toolkit provides a full six degree of freedom.
- Ability to redefine the center of rotation, integrated voice commands.
- Lower cost, more intuitive to learn, interact and communicate.
6
Introduction Outstanding
Toolkit Overview Toolkit Architecture
Toolkit Design Result
Future work Conclusion
Six degrees of freedom (6DoF)
7
Fig 4. The ship movements on the wave
Introduction Outstanding
Toolkit Overview Toolkit Architecture
Toolkit Design Result
Future work Conclusion
The ship movements on the wave:
Translation:
1 – Moving up and down
2 – Moving left and right
3 – Moving forward and backward
Rotation:
4 – Turning left and right
5 – Tilting forward and backward
6 – Tilting side to side
Molecular Control Toolkit
- Supports two gesture devices: Leap Motion and Microsoft Kinect to control and interact with a Java Application.
- Provide voice commands.
8
Fig 5. Molecular Control Toolkit interacts with Aquaria via Leap Motion and MS Kinect
Introduction Outstanding
Toolkit Overview Toolkit Architecture
Toolkit Design Result
Future work Conclusion
Toolkit Architecture
- Gesture Listener
- Device Connectors
- Gesture Dispatcher
- Voice Dispatcher
9
Fig 6. Molecular Control Toolkit architecture
Introduction Outstanding
Toolkit Overview Toolkit Architecture
Toolkit Design Result
Future work Conclusion
Toolkit Listeners
Connectors & Dispatchers
Toolkit Design
• Connector for Kinect gesture recognition:
10
Fig 7. Support gesture controls for Kinect
Introduction Outstanding
Toolkit Overview Toolkit Architecture
Toolkit Design Result
Future work Conclusion
Toolkit Design
• Connector for Leap Motion gesture recognition:
11
Fig 8. Support gesture controls for Leap Motion
Introduction Outstanding
Toolkit Overview Toolkit Architecture
Toolkit Design Result
Future work Conclusion
Toolkit Design
• Connector for Kinect voice recognition:
- Microsoft Kinect Speech API.
- Recognize the words from a custom grammar dictionary.
12
Fig 9. Supported voice commands for Molecular Control Toolkit
Introduction Outstanding
Toolkit Overview Toolkit Architecture
Toolkit Design Result
Future work Conclusion
Results
13
Introduction Outstanding
Toolkit Overview Toolkit Architecture
Toolkit Design Result
Future work Conclusion
Fig 10. Participants background experience
- 18 staff members from Medical Research in Sydney, Australia.
- 20 ~ 30 minutes for training.
- Three tasks: Rotating, Selecting and Zooming.
Results
14
Introduction Outstanding
Toolkit Overview Toolkit Architecture
Toolkit Design Result
Future work Conclusion
Fig 11. Pilot study comparing mouse/keyboard, Leap Motion and Kinect controls for molecular graphics.
Future Work
15
- Voice command:
• The vocabulary should be expanded.
- Support other new gesture devices: Create Interactive Gesture Camera, PrimeSense, …
- Provides gesture tracking algorithms for both Leap Motion and Kinect at the same time.
Introduction Outstanding
Toolkit Overview Toolkit Architecture
Toolkit Design Result
Future work Conclusion
Conclusion
16
Demo: https://www.youtube.com/watch?v=XU_6TWM9ke8
Introduction Outstanding
Toolkit Overview Toolkit Architecture
Toolkit Design Result
Future work Conclusion
- Molecular Control Toolkit is the first system designed to enable life scientists to control molecular graphics using gestures and voice commands.
- Supports Leap Motion and MS Kinect.
Thank you!
17