glast large area telescope: gamma-ray large cost/schedule ... reviews/8.5... · glast lat project...

GLAST LAT Project – Cost and Schedule Review August 5, 2004

AntiCoincidence Detector 1

GLAST Large Area Telescope:GLAST Large Area Telescope:Cost/Schedule ReviewAugust 5, 2004AntiCoincidence Detector (ACD) Subsystem WBS: 4.1.6

David J. ThompsonThomas E. JohnsonNASA Goddard Space Flight CenterSubsystem Manager/Instrument Manager

[email protected]@nasa.gov

GammaGamma--ray Large ray Large Area Space Area Space TelescopeTelescope



Significant AccomplishmentsSignificant Accomplishments

• Completed and delivered all HVBSs• Completed CPT on all flight Front End Electronics Boards.• Corrected mounting hole location error on 11 TDAs. New

holes were drilled and old holes were plugged. TDAs are now being wrapped.

• Completed the assembly of all flight Ribbon Detectors.• Completed the assembly of the clear fiber cables.• Working Photomultiplier Tube glass crack issue. • Received third partial G3/EGSE Test Stand.• Updating grass roots.• Completed assembly of the first Electronics Chassis

(Qual/Spare Unit)



Recent Accomplishments Recent Accomplishments -- Electronic Electronic Component StatusComponent Status

Ready for next assembly

Thermal Testing

Conformal Coating

Functional Testing

Assembly Complete

Quantity RequiredFlight (Spare)

ITEM

2045804580196(40)

Photomultiplier Tube Assembly

282830303024(6)

High Voltage Bias Supply

101114161612(4)

Front End Electronics Boards



Experienced new glass failures during partial life and screening thermal cycling of PMTs

PROBLEM / ISSUE IMPACT ACTIONDATE

ESTAB. COMPLETE

CURRENT STATUS – OPEN

~100 of 250 flight PMTs have been potted/mounted. Potting and PMT sub-assembly work has stopped pending new investigation. Final fight chassis I&T can not be completed without PMTs. This new design already passed previous qualification testing.

7/04 7 or 8/041. Investigate potential causes

including why new design passed initial qualification tests and failed latest tests

2. Determine various solution paths

3. Choose and implement best path

7/04

• Status on Action 1: Root cause has been determined. The mounting design is not robust enough to handle very large variations in material properties and processes. Specifically the Coefficient of Thermal Expansion (CTE) and Poisson’s Ratio of the RTV potting compound vary significantly from batch to batch and even with different cures of the same batch. This results in stresses that can vary by a factor of 25.

• Status on Action 2: There are two fundamental issues. Number one, what should be done with the 100 PMTs that have already been bonded into mechanical housings and secondly how should the remaining flight PMTs be mounted. Multiple solution paths have been developed for both of these issues.

• Status on Action 3: Multiple solutions paths are actively being pursued, others have been but on the “backburner”. We are planning on down selecting the potential solutions on August 17.

ELECTRONICS ELECTRONICS –– Photomultiplier TubesPhotomultiplier Tubes

8/04

7/04 8/15/04

R



Near Term MilestonesNear Term Milestones

Complete!8/3/04AugustComplete Fab of Clear Fiber

Cables

Was completed in July, but rework required on 11 (8 flight + 3 spare) TDAs. Repair complete and TDAsare being wrapped.

8/15/04NovemberComplete Fab of Flight TDAs

Two assembly drawings remain. Designers have been focused on revisions.

8/10/04SeptemberComplete Flight Mechanical

Drawings

Complete!6/1/045/17/04Mechanical Subsystem delivered

to I&T

Unit is assembled and being prepared for testing.8/6/045/20/04Complete Qualification Unit

Electronics Chassis Assembly

First group completed mid-October. See back up material for more information on PMT issue

11/05/04January, 2004Complete PMT Assembly

8/15/04

7/30/04

New Date

All 16 are assembled. 11 ready for assembly into flight electronics chassis.6/30/04Complete Flight FREE Boards

COMPLETE. All 30 are assembled, tested and ready for assembly into Electronics Chassis6/30/04Complete Flight HVBS

Status/NotesDateMilestone Description



ACD MILESTONES vs. TIME1

Variance Explanation: July Variance = -18 ( 14 – PMT Glass, signal wire, capacitor, and Hazardous Material shutdown, 1 – Detectors, 1 – Assembly Drawings, 1 – shippng container, 1 – Micrometeoroid Shield)

0

20

40

60

80

100

120

Nov-0

3Ja

n-04

Mar-04

May-04

Jul-0

4Sep

-04

Nov-0

4Ja

n-05

Mar-05

Cum

mul

ativ

e M

ilest

ones

PLANREPLAN 1/04RECOVERYACTUAL

Sep-03 Oct-03 Nov-03 Dec-03 Jan-04 Feb-04 Mar-04 Apr-04 May-04 Jun-04 Jul-04 Aug-04 Sep-04 Oct-04 Nov-04 Dec-04 Jan-05 Feb-05 Mar-05 Apr-05PLAN 12 21 31 45 58 73 77 85 86 90 92 94 96 99REPLAN 1/04 22 32 43 57 66 79 85 86 89 91 93 96 98 99RECOVERY 68 69 76 82 88 90 93 94 98 99ACTUAL 7 10 15 22 29 34 39 45 56 64 68



InterdependenciesInterdependencies

1. EGSE/G3 – Ongoing development with I&T and Electronics groups. – Last partial EGSE/G3 was received in July. Testing ongoing. – Software updates for software rate counters were received and they are

working. Developing the test scripts.– Full EGSE/G3 delivery planned for August.

2. Delivery of ACD Calibration Unit or subset to LAT I&T – The same electronics chassis being used for G3 commissioning will be used

for the calibration unit. Delivery of calibration unit will occur some time after this testing. Last partial EGSE/G3 was received in July. Testing ongoing.

– Working with LAT I&T to develop a work around that will allow an earlier delivery of a “mini-ACD”.

3. ICD, IDD, etc – Rich Bielawski is helping track a set of needed changes.– Mass CR needs to be released and approved. – Electrical Issues CR needs to be released and approved– IDD (LAT-DS-00309) update. Required for closeout of several open Problem

Reports.4. Funding.

– Recent technical issues are causing cost growth. The new known FY04 liens are provided on the next page.



NEW FY04 KNOWN LIENSNEW FY04 KNOWN LIENS

$32KTDA hole location repair

$38KPMT Purchase (20 PMTs)

$15KPRT contamination issues

$63KVoltage requirement changes (HVBS/GASU)

$37KPMT Issues (screw, wire, and capacitor fixes) plus adding thermal cycling of the PMTs

$507KTOTAL

$260KPMT Glass Crack Recovery

$10KAccelerometer purchase and calibration

$52KTSA Design/Analysis, including instrumentation changes required by LAT Instrumentation Plan

ESTIMATED COSTLIEN



Issues, Concerns, Documentation, & Qual ProgramIssues, Concerns, Documentation, & Qual Program

• Issues– PMTs– Funding

• Concerns– PMTs– Funding– Numerous “minor” technical issues

• Documentation– CM system works great– PR/PFR system working, but since it is a new system team

members are still learning how to use it efficently• Status of PR/PFRs

– 51 PR’s Closed– 21 PR’s Open– 4 PFR’s Open

• Qualification program fully developed



ACD Schedule VariancesACD Schedule Variances

• 4.1.6 – ACD Subsystem (-$425K cum, +$23 current)• 4.1.6.3 - TSA Schedule Variances (-$4K cum, +$5K current)

– Procedures and TDA Assembly and Test• 4.1.6.4 BEA Schedule Variances (-$291K cum, +$20K current)

– ($291K) – Nearly all variance is due to the PMTs.• 4.1.6.7 ACD I&T (-$101K cum, +$1K current)

– ($101K) – Getting started late, but we have begun! Many issues, mainly PMTs, PRTs, TDA’s, and connector polishing have impacted progress.

• 4.1.6.B GSE (-$23K cum, -$3K current)– ($23K) – Shipping container and MMS GSE work being

pushed out to reduce current manpower.



ACD Cost VariancesACD Cost Variances• 4.1.6 ACD (+$189K cum, +453K current period)• 4.1.6.1 ACD Project Management/Sys Eng/Science (+$196K cum,

-$182K current period)– PMCS adjusted for underrun this month– (+$112K) - Labor support lower than planned due to lower than

planned science simulations and test support ($157K), systems engineering being covered by GLAST Project ($48K), Science Support lag in accruals ($65K)

– (-$67K) – Materials. $50K charged to materials instead of lab tax and $17K for software maintenance.

– (+$88K) MPS lower than planned• 4.1.6.2 Safety and Mission Assurance (-$1K cum, -$51K current

period)– Staffing was not sufficient in this area. We have added a QA

person.• 4.1.6.3 Tile Shell Assembly (+$119K cum, +$454K current period)

– CR approved this period– Lag in Fermi invoicing/costing for the TDA’s. Will accrue $182K

in July.



ACD Cost VariancesACD Cost Variances

• 4.1.6.4 Base Electronics Assembly (-$336K cum, +$466K current month)– M&S – (-$17K) PMT issues, (-$63K) HVBS Voltage change, (-$246K) FREE,

HVBS, PMT, and Resistor Network work performed, but have not received credit for work done, (-$46K) SLAC ASIC charges.

• 4.1.6.5 MS/TB (+$23K cum, +$51K current month)– Lag in accruing costs for mechanical designer and MS/TB designer

• 4.1.6.6 ACD Mech Qual and Cal Unit (-$15K cum, -$167K current month)– Final billing for mechanical qualification testing received this month.

PMCS corrected for underrun in this area.• 4.1.6.7 I&T (+$10K cum, -$20K current month)

– (+$10K) - 50/50 earned value• 4.1.6.B Ground Support Equipment (+$146K cum, -$97K current month)

– (+$85K) – Labor. Costs lagging on mechanical design and EGSE support.– (+$59K) – Materials. Not billed for work completed

GLAST LAT ACD Project August 5, 2004


Summary of the ProblemFollowing a series of phototube failures (breakage at low temperature) in January, 2004, the mounting for the tubes was changed, reducing stress on the tube by nearly a factor of 10. The new mounting design was qualified with a set of 9 tubes down to temperatures of -40 C (-60 C for some) with no breakage. All tubes were visually inspected and graded. The new design was put into production. 100 of our 240 phototubes were potted into mechanical mounts (194 are needed for flight).

On July 6, 2004, we found out two out of five tubes subjected to a single thermal cycle down to -30 C broke. Both tubes were “fair” quality. A “good” tube and a “worst” tube in the same test survived.

Over that same weekend, a 12-cycle test (to -30 C) of 11 “worst” tubes had 2 break. After 30 cycles, no additional tubes had broken.

On July 9, a set of 25 “worst” tubes was carefully temperature cycled with checks at different temperatures for breakage. Four tubes broke at -15 C; one tube broke at -30 C.

ACD Phototube Breakage Issue



Find the Root CauseFind the Root Cause

Cause outline – how did we pass the original qualification of the new design but fail the follow-on tests? Summary of possibilities:

• A. The Stresses Increased ––RTV material properties changed

•Test new batches for CTE, modulus, Poisson’s ration•Develop new test to more accurately confirm early poisons ratio tests and data sources•Retest old batch if possible for CTE, Poisson’s ratio, modulus•Compare test results with old test results, data sources and assumptions•Use strain gauge tests to test for strain differences between old and new PMT batches and to correlate stress model

–PMTs that failed were unusual – diameter and glass thickness vary between tubes•Look at new centering step data for diameters

–Less likely causes we must still disprove – large bubbles in RTV causing stress concentrations, Thermal vac chambers in error and we actually went much colder, etc

• B. The photo-multiplier tubes became weaker over time ––Exposure to moisture over time has propagated glass flaws or made flaws much easier to propagate at lower stresses–Handling - very unlikely with new handling procedure–Is there any test of bare PMTs that could help see a weakening?

• C. The stress analysis and /or assumptions were flawed AND we were lucky on what were demanding qualification tests

–RTV properties and property tests were wrong to begin with – compare results of first bullet of A above–Hard to understand how 5 tubes could go to multiple cycles at -60C and 4 tubes could go to -40C without failure. Used as controls but none have failed yet. Retest with very low temperature test. Test 7 PMTs potted with old RTV batches but never cycled.–Review stress model again. Devise and use strain gauge tests mentioned in bullet 1 of A above to correlate model. Add non-linear ability to model.



X

X

X

X

XX

X

X

XXPMT Cracks

Glass Weakness

Excessive Stresses

Stress due to mountingStress due to test

Manufacturing cracks

Inside of tube

Outside of tube

Water

Handling

RTV stress

Curing shrinkage Mixing

Manufacturing

High temperatureLow temperature

Thermal rate of changeTest stand

Mu metalHousing

Model deficiencies

Bonding

Construction

Application

Inspection

Tilt Decentering

Curing

Mechanical properties (CTE, Poisson's ratio, Young's modulus)

Thanks to Art Whipple for starting this fishbone (Ishikawa) diagram.

Priming

Fishbone Diagram – Possible Sources of the Problem

X – source unlikely

Known contributors underlined.



Threats to Schedule and CostThreats to Schedule and Cost

1. PMT Assembly2. GASU/G3 EGSE3. Electronics assembly and test



So• Despite our qualification program, the method for potting ACD

phototubes into housings is still producing breakage of tubes atlow temperatures.

• The number of possible causes of this problem is limited, but noone possibility stands out as most likely. We are investigating all possibilities through a combination of analysis and test.

• Four candidate solution paths have been identified for improvingthe mounting scheme for new tubes and making the already-potted tubes useable for the ACD. Design studies are in progress, but we need more information about cause before selecting one ormore approaches for testing.

ACD Phototube Breakage Issue



PMT Stress as a function of PMT Stress as a function of CTE and PoissonCTE and Poisson’’s Ratios Ratio

• Raw NASTRAN analysis data– ∆T = -60°C (+20 to -40°C)– Exterior element 4959 beneath mounting flange– These values are based on a FEM with RTV modeled beyond

both edges of the glass tube

RTV-566 Poisson's Ratio CTE 233 ppm/deg C CTE 260 ppm/deg C CTE 290 ppm/deg C CTE 310 ppm/deg C CTE 330 ppm/deg C CTE 350 ppm/deg CTube Stress (psi) Tube Stress (psi) Tube Stress (psi) Tube Stress (psi) Tube Stress (psi) Tube Stress (psi)

0.4 161 191 225 248 270 2930.43 238 282 330 362 394 4260.46 427 550 642 703 764 8250.47 571 670 780 853 926 10000.48 847 993 1154 1262 1369 14770.49 1601 1873 2175 2377 2578 27800.499 8303 9699 11250 12284 13317 14351

RTV-566 Poisson's Ratio and CTE Sensitivity Study for PMT (Tube/RTV-566/Al Housing) Assembly



PMT Stress as a function of PMT Stress as a function of CTE and PoissonCTE and Poisson’’s Ratios Ratio



Stress resultsStress results

• Material properties are all over the place

• Improved Tests show changes between RTV batches and between different cures within same batch – in one case within batch differences (Poisson’s Ratio has biggest effect, we have developed a better test for that now) apparently cause more than a order of magnitude difference

• We identified this as a major risk during the peer review of the first solution. However we had many sources for our assumptions and cost and schedule pressures made many believe that the minimum 2 to 3weeks of additional delay and cost of creating and tweaking a new potting material Poisson’s Ratio test was turning leading us into a ‘technology development’ mode we had no time for.

• We have good confidence in the latest material property results (new poisons ratio test we choose not to pursue last time is working well and has been checked for set up and repeatability, etc)



ACD Phototube Breakage Problem – Root CausePhototube, CTE ~3 µm/mºC

RTV, CTE ~300 µm/mºC

Aluminum housing, CTE ~27 µm/mºC

Because the RTV is used to hold the position of the tube in both transverse and longitudinal directions, it must adhere to both the tube and the housing, i.e. be primed.

Because the Coefficient of Thermal Expansion (CTE) of the RTV is much larger than that of either the tube or the housing, at cold temperatures the shrinkage of the RTV pulls on both surfaces.

If the CTE of the RTV were closer to those of the tube or the housing, we would have no problem. BUT – the actual CTE of the RTV is LARGER than the specifications (and from a number of early sample tests) and variable from batch to batch, and, we are learning, from cure to cure within batch.

If the RTV were compressible enough, it would stretch instead of transmitting the stress. BUT – the actual RTV is almost perfectly incompressible, significantly more so than the specifications. We are learning this also appears to vary from batch to batch and between cures within the same batch. Poisson’s ratio is 0.49 in some samples compared to specifications of no more than 0.46 (risk presented in peer review).

RESULT – Combine the above facts with the fact that glass flaws in our particular PMTs make them 50 to 70% weaker than one might expect, at low temperatures the tube is pulled apart by the shrinking RTV, with cracks starting at these flaws. We believe the qual units did not fail because the earlier batches of RTV tended to have much better material properties and there may have been some de-bonding in the early batches.



ACD Phototube Breakage – Choosing a solution

Approach:

• Basic listing of solution options done. Now lay out the options in sufficient detail to understand the basic issues.

• Lay out basic solution decision criteria.

• Distinguish two cases and the possibility of combine solution

• What we do with the 100 tubes already potted

• What we do with the 140 bare tubes

• Establish top level flow and individual flows for each path with dates

• Pursuit of options with most critical criteria related activates given priority. During pursuit of solutions look for critical missing information or “show stoppers”

• Options that begin to look very hard early on could be put on back burner. This allows eliminate options as quickly as possible as initial prime candidates at least by completing the most critical actions.



PMT Top Level Solution Path flowPMT Top Level Solution Path flow

Interpretation of latest data from failures and materials testing

– preliminary conclusion July 30

Develop general set of solution paths from general

set of failure causesJuly 20

Investigate failure July 6-Aug2

Develop general set of failure causes

July 19

Begin pre-mature solution analysis and feasibility tests of

a few promising concepts, (see separate solution flows)

July 22

Lay out more detailed set of solution options, feasibility results and

selection criteriaAug 5

Complete analysis and preliminary testing

Aug 10

PMT selection solution decisionGoal date -Aug 13

See detailed flow for chosen solution

and actions list

Re-asses solutions based on early

feasibility results and failure

cause analysis



PMT Solution Paths PMT Solution Paths

Possible top level Solution Paths• Modified Potting Solution – understand the new variables and stresses in the potting materials

–RTV or more likely an alternate–Must understand exactly how the latest PMTs failed–Slit-potted design. May even be able to do this to already potted PMTs–Others also require removal methods for already potted PMTs

• Thermal control solution – don’t let the PMTs see the stress of lower temperatures. –Heaters to -5 or 0C, Must determine via test what is warm enough–Allows us to fly PMTs that have already been potted

• Mechanical solution – get out of the potting business since it looks like the material properties vary too much for these flawed tubes.

–Partial CTE compensation design. Uses modified existing housings with inserts, does still add some stress but mostly compressive–Quasi kinematic mounts. Various forms. Simple versions may be able to use existing housings with inserts–Bonding release designs. Releases bonding on one side, grooved housings keep PMT from slipping

• Determining Yield in current design – see if there is a screening test that stresses the PMTs in a way that identifies almost all the PMTs that will fail without consuming lifetime of the PMTs that pass or making them more likely to fail.

–Probably have to screen and partial life test large number of PMTs ( which may be tough to then use as flight) to show it could work.

•Any combination of above with combined with some or all new PMTs without so many glass flaws –First 6 units with modified Hamamatsu process are not flawless but are all dramatically better than any of the original tubes we haveNote : Some solutions also require removal methods for already potted PMTs



Salvage potted PMTs

Screen PMT assemblies

New PMT housing design

Heat PMT assemblies

Fix baseline design

Resume flight PMT assembly

PMTs crackduring thermal

testing

Determine root cause of

failures

Correct problemCorrectable?

Yes

No

Requalifydesign

Develop PMT assembly screening process

Screenable? Failure rateacceptable?

YesModify baseline design

Yes

No No

DevelopLow-stress PMT

mounting

Change baseline design

Develop process to

safely remove potted PMTs

Successful?

Remove potted PMTs

Procure more PMTs

Yes

No

Redefine PMT thermal

requirements

Develop thermal control system & assess impacts

Impactsacceptable?

Modify baseline design

Yes

No

Use current flight PMT assm

Use newflight PMT assm



Selection CriteriaSelection Criteria

• STRESS AND STRAIN - Reliably lowers stress to under 1000 psi for tensile stresses with a goal of < 500 psi. Keeps compressive stresses to TBD (< 2500-5000 psi) compressive stresses. Show by analysis and strain test if possible. Higher compressive stresses must be shown to truly be compressive with no variables that could cause associated tensile stresses

• FEASIBILITY - Passes prototype feasibility tests (assemble-able, controllable, seems repeatable)

• MINIMUM PROTOTYPE TESTS - Passes prototype thermal test, light tight and vibration test.

• LOW VARIABLY RISK – low number of variables or that could possibly effect stresses or ACD performance. Low effect on stress on variables we cant control.

• SCHEDULE (& cost). Could be used as tie breaker for solutions that pass above tests would be cost and schedule, however some solutions will be eliminated sooner solely on that basis.



Alternate Design 1Alternate Design 1–– CTE Comp Mechanical MountCTE Comp Mechanical Mount

Design Prototype,Preliminary

Stress AnalysisJuly 9

FabricateParts

July 17 MechanicalFit CheckJuly 21

Strain GagePMT

July 22

1st PrototypeAssembly

Aug 2

AssemblyProcedure

Aug 2

MarginsOK

Detailed Stress

AnalysisAug 9

ModifyDesign

NO

YESPMT solution

DecisionAug 13

1st ThermalAnd

strain Test – Pass?

August 6

Vibrationtest

StartBuild 20

Qual UnitsAugust 31

Finalize Flight Design

August 26

FabricateParts

August 26

QualTesting

September 12

PMT Production(40/week)

September 20

Thermal ScreenSeptember 27

First Batch Ready For Electronic

Chassis Integration Oct 1

glast large area telescope: gamma-ray large cost/schedule ... reviews/8.5... · glast lat project...

Documents