astronauts and iot: toward true human-autonomy teaming
TRANSCRIPT
![Page 1: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/1.jpg)
Astronauts and IoT: Toward True Human-Autonomy Teaming
Dr. Alonso VeraChief, Human Systems Integration Division
Nordic Digital Business Summit 2016Helsinki, Finland
22 September 2016
![Page 2: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/2.jpg)
05/01/2023 2
OverviewThree Themes- Not designing autonomous systems to interact
with humans increases costs- The need ongoing need for human engagement
with increasingly intelligent systems (IISs)- Both cognitive and physical
- IoT allows IISs to reach into the physical world in an interactive way that has not been possible until now
![Page 3: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/3.jpg)
Machine IntelligenceWe appear to be at an exciting time with respect intelligent machines (again). • Four Related Areas of Development
1. Big Data - volume, velocity and variety2. Deep Learning3. Networked operations and cyber-physical systems4. Moore’s Law (exponential growth, doubling of components on an integrated circuit
every two years): faster, bigger computers driving change with increasing velocity
• Stephen Hawking, Bill Gates and Elon Musk have all recently warned about the potential dangers of AI.
• Also interesting time in terms of self-driving cars and companies with robotic operations/factories like Amazon, Tesla and Toyota
• Big Blue, Watson, Pokerbot• Google DeepMind AI Division beats human at GO (Jan 2016)• First AI investment software hits Wall St. (Feb 2016)
![Page 4: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/4.jpg)
Manpower Reduction: Start with the Human (Not the Technology)The Autonomy Paradox (Blackhurst, Gresham & Stone, 2011)
• Autonomy doesn’t get rid of humans, it changes their roles
• DoD has shifted from Levels-of-Automation to Cognitive Echelons
As machine intelligence advances, the need for better human interfaces increases
The Littoral Combat ShipBuilt to be operated by 45 sailors
Dr. Larry Shattuck, NPS (pg. 13-15) http://human-factors.arc.nasa.gov/workshop/autonomy/download/presentations/Shaddock%20.pdf
![Page 5: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/5.jpg)
Recent DevelopmentsThe Littoral Combat Ship• New, highly autonomous
vehicle
In the end, the ship required 60 sailors, all E5 or above
… and they are still encountering major issues
![Page 6: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/6.jpg)
05/01/2023 6
Apple Research on Teaming of IISs with Humans
http://www.cultofmac.com/444551/why-apple-is-the-new-nasa/?&tc=em
• Characterizing calorie burn during swimming and using learning algorithms to tune the functionality to individual differences
• Developed novel experimental hardware and tested on 700 swimmers• To develop a feature on one app for the new iWatch
![Page 7: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/7.jpg)
Path to Collaborative, Human-in-the-Loop Planning Systems
![Page 8: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/8.jpg)
NEEMO
NASA Extreme Environments Mission Operations (NEEMO): Underwater laboratory off of the coast of Key Largo, Florida.
![Page 9: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/9.jpg)
International Space Station CAST – Crew Autonomous Scheduling
Test: • Playbook Check-Out by Scott Kelly on
Station in August 2015. • Astronaut Peggy Whitson trained on
Playbook in July 2016.• Astronaut self-scheduling study on ISS on
Mission Increment 50/51 Nov 2016.
ISS Mission Control• Three integrated planning
systems: Power, Attitude Control and Crew Activity
• Crew activity includes ESA JAXA and Payloads
![Page 10: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/10.jpg)
05/01/2023 10
What are the Challenges of Working with IISs?• Lack of transparency about intent, state awareness,
risk/confidence posture, etc.
• Part of the challenge is just the reverse though. Given that the IIS does not know what the human is trying to do, it is difficult for the system to know to engage in ways that are useful.
![Page 11: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/11.jpg)
Toward Human-Autonomy Teaming?
Caged Robots
Domesticated Robots
![Page 12: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/12.jpg)
Designed for Teaming
VIDEO REMOVED TO
DECREASE FILE SIZE
![Page 13: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/13.jpg)
05/01/2023 13
IoT and IISs• IoT as a way for increasingly intelligent systems to
have awareness of state in the physical world• Addressing the AI grounding problem• Beyond machine vision
• Applied to manual human tasks• Assembly, maintenance, repair• Reduced training, errors
• Task execution support • For complex tasks• For infrequently performed tasks
![Page 14: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/14.jpg)
05/01/2023 14
Assembly, Maintenance and Repair on International Space Station The Rodent Habitat
![Page 15: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/15.jpg)
05/01/2023 15
How Can IoT Help?Challenge: loss of tools and loss of training• Tools and parts with embedded sensors • Data from sensors transmitted to the network by object• Objects know what they and can also capture and
communicate other information about themselves• Location• Calibration status• History of failures, adjustments, repairs• Motion, torque, force, orientation, relative proximity
May allow intelligent/adaptive procedure execution systems to guide humans though complex tasks with little or no training, as well as detect errors and reduce performance time
![Page 16: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/16.jpg)
Summer 2016 Student Intern Project
![Page 17: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/17.jpg)
Internet of ThingsMicrosoft Hololens
Combine Augmented Reality and Internet-of-Things
Mission
AR-IOT
![Page 18: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/18.jpg)
Existing ProcedureMission
AR-IOT
![Page 19: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/19.jpg)
AR-IOT
![Page 20: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/20.jpg)
Dividing procedure execution research into two parts
Mission
AR-IOT
Task Procedure Path Visualization
![Page 21: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/21.jpg)
AR-IOT
IoT
IoT Pickup Detection
![Page 22: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/22.jpg)
AR-IOT
IOT Technology (server) IoT
![Page 23: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/23.jpg)
AR-IOT
Prototype
![Page 24: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/24.jpg)
Storyboards
AR-IOT
Voice command opens the
selection menu.
Voice or gesture selects the tools option
User selects all wanted tools. Feedback occurs for each selection. Once ready, user selects “Find” to begin path.
1 2 3
![Page 25: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/25.jpg)
Storyboards
AR-IOT
Menu begins to guide user.
Simple HUD cues user to the destination.
Spaced arrows lead user to location. HUD allows user access to a menu to cancel or change a path.
Location has hologram of item over the storage compartment.
4 5 6
![Page 26: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/26.jpg)
Storyboards
AR-IOT
Multiple of similar tools in a
compartment with a hologram over the correct
one. HUD changes to icons
of the tools.
Tool is secured in the compartment and the user picks it up.
Haptic feedback and change of HUD when correct tool is located. Progress indicator fills in the current tool.
7 8 9
![Page 27: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/27.jpg)
Storyboards
AR-IOT
When tool is located the path changes to begin
guide to next location. HUD displays new location goal.
Destination HUD changes to icon HUD.
Line leads directly to the desired tool. A holographic overlay is also placed over the specific tool.
10 11 12
![Page 28: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/28.jpg)
Storyboards
AR-IOT
Haptic and another change of HUD signal found object.
Path changes to lead user to final destination. HUD also indicates location.
Arrow or highlight of doorhandles indicates user to open the door (or module)
13 14 15
![Page 29: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/29.jpg)
Storyboards
AR-IOT
16
Once the user returns to the procedure location there will be indicators of places where they will need to install the tools based off of the IOT sensors and holograms.
Once tools have been placed you
can begin the procedure.
The first step is to place the tools you've found in a secure bag. The user will see a holographic animation that shows you how to do this.
17 18
![Page 30: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/30.jpg)
Step 2 is to attach your bag to the habitat. These animations are similar to the items own movement.
19
Storyboards
AR-IOT
20
When the item is placed
successfully there is a
holographic blinking change
as well as an update in your
heads up display.
Step 3. Open top access hatch. The animation will show the hatch opening and closing.
21
![Page 31: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/31.jpg)
22
Storyboards
AR-IOT
23
Content such as videos or Skype
calls can be displayed on the wall to provide support for the
astronauts while they conduct procedures.
Microsofts “Xray” feature can show objects containing other objects so users view isn't obstructed.
![Page 32: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/32.jpg)
User Test 1B Recording
VIDEO REMOVED TO
DECREASE FILE SIZE
![Page 33: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/33.jpg)
Insights & Opportunities
AR-IOT
Gestures (Select Tools)
Many users said they preferred using voice over hand gesture but during user tests participants tended to use both.“Voice is more comfortable because I wasn’t sure about the depth of my gestures.”
“Instead of navigation through menus there should be a way to simply say, let’s find the flashlight.”
![Page 34: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/34.jpg)
Insights & Opportunities
AR-IOT
Guidance (Path)
Some users felt that they didn’t need such an excessive guidance system which at times could feel unnecessary.“It would be condescending to an astronaut to tell them exactly where to go”
![Page 35: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/35.jpg)
Insights & Opportunities
AR-IOT
Arrow Spacing (Path)
Using calculated arrow spacing along side a linear path gave users an easy guidance system to locate their desired tool.“The arrows were positioned just right so I knew exactly where to turn”
“Following the line was simple and intuitive”
![Page 36: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/36.jpg)
Insights & Opportunities
AR-IOT
Destination (Locate Tool)
Placing a hovering hologram over tools precise location proved to be a successful device to mark the destination.“I noticed the hologram and my immediate reaction was to go there.”
“I saw a star, so it was pretty clear to me that something was there.”
![Page 37: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/37.jpg)
Insights & Opportunities
AR-IOT
Precise Location (Locate Tool)
A single line on the floor leading to a tools exact location was a successful method of marking the destination because the user could follow the line of tape with their eye to the exact destination. “You could clearly see that the highlighted drawer was where the item is”
![Page 38: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/38.jpg)
Insights & Opportunities
AR-IOT
Visual Feedback (Locate Tool)
Visual feedback after task completion was needed after the user found each tool to ensure they were on the right path. “I didn’t have any doubts that I have found the right flashlight but maybe if there was a dozen it would have been more helpful.”
“Visual feedback would definitely be more helpful if there were different types of flashlights”
![Page 39: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/39.jpg)
Insights & Opportunities
AR-IOT
Using Memory (Return)
Many users were able to return to the procedure room while only using their memory and not relying on our guiding system.“To me it was pretty clear because I knew where this room was so I mostly tuned the path out”
![Page 40: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/40.jpg)
Journey Map
AR-IOT
![Page 41: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/41.jpg)
AR-IOT
Procedure Execution
Animations & Procedure
VIDEO REMOVED TO
DECREASE FILE SIZE
![Page 42: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/42.jpg)
AR-IOT
Procedure Execution
Animations & Procedure
Holographic animation
![Page 43: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/43.jpg)
Procedure Execution
AR-IOT
HUDWe wanted an unintrusive heads up display that placed most of the information within the line itself.
![Page 44: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/44.jpg)
Storyboards
Final Test
VIDEO REMOVED TO
DECREASE FILE SIZE
![Page 45: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/45.jpg)
From Cummings, M.L., "Man vs. Machine or Man + Machine?" IEEE Intelligent Systems, (2014) 29(5), p. 62-69.
2015
![Page 46: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/46.jpg)
Architecture based on autonomy performing all skill and rule-based roles, as well as most knowledge-based roles. Manpower reduced by two orders of magnitude with remaining expert humans teaming with machine intelligence to solve complex problem solving under uncertainty. Machine intelligence for airspace management evolves from the outset to support teaming with small set of expert humans to support cooperative problem-solving.
Adaptive
2035
Computers Humans
e.g, current HITL for ATM Next-Gen research
![Page 47: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/47.jpg)
Affordances from the Environment
![Page 48: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/48.jpg)
Using Affordances
• Application of Gibsons’ Ecological Psychology• Alternatives to using
human central attention resource
• A car more like a horse
![Page 49: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/49.jpg)
Teaming of Human and Machine Intelligence• Even as computers get very “intelligent”, it is very likely that the nature
of the their intelligence will be different than that of humans (unless they become omniscient or we program them to function just like humans)
• Humans are particularly good at adaptive problem-solving and discovery, areas where there has been little machine intelligence progress
• Successful efforts going forward will be those that wrap new machine intelligence capabilities around human competencies in order to get the most out of each
Goal: Design the human into the process. Focus on how the system will communicate it’s state to the human so that the human can help in un-anticipated situations, and vice versa. What data and how it is presented to each agent such that each can bring its unique capabilities to bear on it.
![Page 50: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/50.jpg)
The Economics of Human-Centered Automation
• For lower costs, higher efficiencies and overall improved system performance:• Characterize nature of human roles
(skills, rules, knowledge, expertise) and tasks (e.g., proportion of hard and soft constraints)
• Wrap autonomy around remaining human roles from the beginning
Dr. Jon Bornstein, DoD Autonomy Roadmap Autonomy Community of Interest http://www.defenseinnovationmarketplace.mil/resources/AutonomyCOI_NDIA_Briefing20150319.pdf
Critical to shape the autonomy industry• e.g., Apple v. Littoral Combat Ship
![Page 51: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/51.jpg)
05/01/2023 51
Final Thoughts• Humans will remain important components of complex
systems• Use human adaptive expertise as much as possible• Use human cognitive & perceptual system as much as
possible in interactions • Robotics progressing faster than AI• Be aware of areas where you don’t have big data
• Not all problems are associative in nature
• Don’t assume search will solve all problems
![Page 52: Astronauts and IoT: Toward True Human-Autonomy Teaming](https://reader035.vdocuments.net/reader035/viewer/2022062823/58743b041a28ab0e6c8b560d/html5/thumbnails/52.jpg)
Thank you