mapld 2005 computer overload and the apollo 11 landing (an insider s view )

F:MAPLD 2005

MAPLD 2005

Computer Overload and the Apollo 11 Landing

(An Insider s View )Jack Garman

NASA Retired (now Lockheed Martin Information Technology)

Garman 2 F:MAPLD 2005

Introduction

• A “ring-side seat”

• My scope of knowledge

• Parkinson’s Law– Work expands to fill the time available for its

completion • A “proverb” coined by the twentieth-century British

scholar C. Northcote Parkinson, known as Parkinson’s Law.

• It points out that people usually take all the time allotted (and frequently more) to accomplish any task.


Themes• A Story

– “Long, long ago, in galaxies far, far away”

• The Legacy (Observations)– Flexibility

• As in “change” and evolution• Parkinson’s Law Corollary - software requirements

change and expand to exceed all reasonable projections

– Fault Tolerance• Responding to the impossible• Planning for things that just “can’t happen”

– Layers of abstraction are real• No one really sees “the whole”

– Time• Everything is different, but everything is the same


Painting an Image of the Times• No “Internet”, no E-mail, no “graphics”• No “personal” computers

– Word and PowerPoint “by hand”,Excel via (GFE) slide rule

– “Live” CRT’s were a marvel– “The first typewriter”

• Embedded computers– Digital flight control (vs. analog)– Guidance and Control computers

• “Apollo Guidance Computer”

– Technology: • speed, core memory, rope memory, no disk


Painting an Image of the Times (cont’d)• Simulation Concepts

– in other than real time– Voice and data delays

• IT Expertise (relatively small)– Operating systems or “systems software”– Software Engineering

• Amazing for the day – primitive by today’s standards• esp requirements, testing, code inspection

• Youth and naiveté (and politics)– Obedient, smart, dedicated– Politics largely invisible to the troops

• Protected, limited comm - no internet, no “NASAWatch”, no “cup half empty”


Apollo• The Saturn V and IU

• The CSM

• The LM

• The AGC – and CMC and LGC – and AGS

• Software “freeze”


Layers of Abstraction (IT viewpoint) The Astronauts

“Piloting” (and training) The flight control system The guidance system (and orbital mechanics) The flight software (and development systems) The onboard computer and associated gear The spacecrafts (or simulators) and subsystems

The C3 links The ground networks The Mission Control Center The mainframe and associated gear The Mission Control Center (MCC) software

(and development systems) The consoles“Mission Operations” (and simulation/training)

The Staff Support rooms The Flight Controllers and “MOCR”

Me


Flight Computers and Software• The design of the “avionics” (guidance

computer software)– Approach to process mgmt

• Async vs. Sync• Continue in light of the unknown

– Approach to Fault Recovery• Single string, highly reliable, “failure not an

option”– Vs. Multi-string, FOFS, “quit” on hard fail– Nightmare: All quit

• It meant the software had to be able to “restart”– Continue in light of the unknown– Nightmare: Restart loop


The MCC “Consoles”

TV or “digital”

Speaker

Display Select

“Event” lights

“P”-tube

“Voice Loop” Controls

NO KEYBOARDS


Training• Mission Control

– Sci-Fi to a young engineer in the late 60’s– Reading the screens in Mission Control

• Crew Trainers– ISAGC– Simulations

• Types of “training”– Simulations and “Integrated Sim’s”

• Playing both sides

– Pad Tests (MCCH tied to the real vehicle)


Mission Control Center Houston• Very hierarchical (command structure)

– MOCR• “Board of Governors”• Flight Director• Flight Controllers

– Capcom– Comm– The “Trench”– Systems– Medical– Support

• “Simsup”– SSR’s (“the backroom experts”)– Infrastructure Support

• MCCH, computers, networks…– Engineering Support

• NASA engineers and contractors around the country


Reminder: This was 36-years ago…….


The Story• “Setting up” a failure that “can’t

happen”– “playing both sides”

• The Results – and preparation

• The “Real Thing”– 85 + 15 = 100– Alarms, master caution and warning


The Console “Cheat-Sheet”

In retrospect is it somewhat terrifying to me that this was even permitted as a “tool” or procedure for a 24-year-old engineer sitting at a console in Mission Control during the first landing on the Moon


The Story (cont’d)• Two moments of “reality”

– The “echo”– “We’ve got dust…”

• Rich has it all online– http://www.klabs.org/history/apollo_11_alarms/console/


The Legacy(Conclusion)


Flexibility (change and evolution)• Shuttle Legacy

– was/is fly-by-wire• Software fixed hardware development issues• Vehicle development almost “killed” software

development– used a HOL, but a real battle

• “measured” 15% penalty in size• Qualitative gain in reliability• Infinite gain in change (unexpected)

(Parkinson’s Law: Shuttle flight software was “developed” at least three times)

• General (into today)– Extraordinary tools

(and additional layers of abstraction)• Good News: Change is easier• Bad News: Change is easier


Fault Tolerance• Shuttle Avionics Legacy

– FO/FS and “sync points”– The sync/async “wars”– BFS vs. PASS (vs. “BASS”) and S/W fault

tolerance

• Testing– SPF, Crew Trainers, SAIL– Cycle stealing – testing the impossible

• Murphy’s Law– Good: A flight computer failed without impact on

ALT– Bad: The first orbital launch was delayed 48 hrs

due to a PASS/BFS interface 1:67 failure


Layers of Abstraction are Real• No one “knows” the whole

– Horizontal vs. vertical knowledge becoming more and more an issue

– Egocentricity• Downward: Engineering dependency on “lower layer”

black boxes ever increasing– The dilemma of “off the shelf”

– Apollo to Shuttle to Station “flight computer” in degree

• Upward: Knowing where my piece, my “black box”, fits in the whole is more and more difficult

• Anecdote– DoD Software Engineers projection: 1988


The Legacy (concluded)– Flexibility (“change”)– Fault Tolerance (“can’t happen”)– Layers of abstraction are real

• Time and Change– Everything is different today

• but everything is the same!

– If it works, don’t fix it!• BUT - if it works, it’s obsolete!

mapld 2005 computer overload and the apollo 11 landing (an insider s view )

Documents