1 microwave integrated retrieval system for npoess preparatory project: mirs npp/atms integration...
TRANSCRIPT
1
Microwave Integrated Retrieval Systemfor NPOESS Preparatory Project:
MiRS NPP/ATMS Integration into NDE
Test Readiness Review
January 19, 2011
Prepared By: Kevin Garrett1
Chris Grassotti1
Sid-Ahmed Boukabara2
Limin Zhao4 Flavio Iturbide-Sanchez1
Wanchun Chen3
Leslie Moy1
1 I.M. Systems Group2 NOAA/NESDIS/STAR
3 Dell4 NOAA/NESDIS/OSPO/SSD
2
Review Agenda
1. Introduction 1:30 – 1:40 K. Garrett
2. CDR Report 1:40 – 2:05 C. Grassotti
3. Software Architecture 2:05 – 2:35Context and System Layer C. Grassotti
4. Unit Test Readiness 2:35 – 3:15MiRS DAP Test Readiness K. Garrett QC DAP Test Readiness C. Grassotti
5. Risks/Actions 3:15 – 3:25 K. Garrett6. Summary and Conclusions 3:25 – 3:30 K. GarrettDiscussion 3:30 – 4:00 All
3
· INTRODUCTION· CDR Report· Software Architecture· Unit Test Readiness
» Test Readiness of Individual Units: MiRS
· Risks/Actions· Summary and Conclusions· Discussion
4
Section 1 – Introduction
Presented by
K. Garrett
5
Project Objectives
· Technical Objective» Adaptation of MiRS to NPP ATMS and integration within NDE and OSPO
· Science Objectives
» Improved temperature and moisture profile retrievals» The extension of the retrieved products to non-standard surfaces including
sea-ice and snow-covered land » The retrieval in all-weather conditions including cloudy and precipitating
conditions» An improved set of retrieved surface properties whose derivation is based on
the retrieved emissivities instead of directly from the brightness temperatures
6
TRR Objectives
· Objectives of the Test Readiness Review» Goal #1: Gather all MiRS stakeholders to review the
overall system integration of MiRS into the NPOESS Data Exploitation (NDE) environment
» Goal #2: Review of MiRS Software Architecture» Goal #3: Review of MiRS Software Unit Test Readiness» Goal #4: Review of CDR action items and actions taken» Goal #5: Identify new or outstanding risks w/mitigation
strategies
· Follow the STAR EPL Guidelines for TRR
7
MiRS Stakeholders
· Development Team» S.-A. Boukabara, K. Garrett, F. Iturbide-Sanchez, C. Grassotti, W. Chen, L. Moy
· OSPO Partners» L. Zhao, J. Zhao, T. Conrad
· NDE Partners» P. MacHarrie, L. Fenichel, D. Powell, J. Silva, G. Goodrum
· MiRS Oversight Board» F. Weng (chair), R. Ferraro (STAR), L. Zhao (OSPO), J. Silva (NDE), T. Schott (OSD)
· Oversight Panels» SPOP, PREPOP, ICAPOP, LSPOP
· MiRS Users» Dan Pisut (NOAA EVL), Tony Reale (STAR), Joe Turk (JPL), Ben Ruston (NRL), Sheldon
Kusselson (SAB), Stan Kidder (CIRA), Kevin Schrab and Andy Adman (NWS), Denise Hobson (AFWA), B. Yan and M. Kim (JCSDA), G. Huffman and W. McCarty (NASA/GSFC), J. Tesmar (FNMOC), P. Wang (Taiwan Weather Bureau), J. Janowiak (UMD), Paul Field (UKMET), K. Okamoto (JMA), M. V. Engel (IAO SB), B. Lambrigtsen (JPL), Peiming Dong, Qui Hong and Hu Yang (CMA), Universities (Graduate School of Chinese Academy of Sciences, Universia degli studi di Roma), Franklin Robertson and Clay Blankenship (NASA/MSFC), Tom Auligne (NCAR), D. Vila (CPTEC, Brazil), W. Han (Chinese Met. Admin.), D. Cimini (IMAA/CNR), M. Itkin (MPI-M, Hamburg), T. Greenwald (SSEC)
8
MiRS Timeline/History(1/3)
· Development Phase Begins (Sep. ’05-Sep. ’06)» Go-ahead with product development (Sep. ’05)» Preliminary Design Review (Oct. ’05)» Critical Design Review (Sep. ’06)
· Pre-Operational Phase Begins (Jul. ’07)» Operational/backup processing capabilities in place. » SPSRB approves product to go operational
· Operational Phase Begins (Aug. ’07)» Operational/backup processing capabilities reach ops status» Code transitions to operations (MiRS Release I)» SPSRB updates product metrics web pages» OSD updates Satellite Products database
· Transitioned to Operations (Aug’07 – Jun’10)» N18, N19, Metop-A, F16, F18» Also running in STAR: NPP/ATMS, AMSR-E, FY3» Recently processing TRMM/TMI, Simulated GPM/GMI
9
MiRS Timeline/History (2/3)
· NPP ATMS Preliminary Design Review (Sept ’09)» Running with proxy data in daily processing at STAR» Identified Risks/Actions
· NPP ATMS Critical Design Review (June ’10)» Provided technical data regarding overall system (operations, requirements,
ATB, system design, architecture, quality assurance)» Identified Risks/Actions
· QC DAP Delivered: Dec 2010» Testing in Progress
· Product Oversight Panel Meetings:» SPOP: S. Boukabara» PREPOP: F. Iturbide» ICAPOP: K. Garrett
• Requirements for MiRS NPP/ATMS and NDE integration outlined in the “Research to Operations Project Level 1 Requirements Document for Microwave Integrated Retrieval System for NPOESS Preparatory Project” (Updated April, 2010)
10
Jul’09 Jul’10Jan’10
Feb’09Integration into NDE
work begins
Mar’09Initial version
of MiRS Integrated
at NDE
May’09Official
version of MiRS
integrated
Jun’09Development
with proxy data begins
Sep’09PDR for NPP
ATMS
Oct’09Process multiple
sample data granules from
IDPS
Nov’09-Dec’10Enhanced MiRS
development (advanced footprint matching, proxy data SDRs, QC
DAP)
Oct’11NPP
scheduled for launch (was
Jan’11)
Oct’11-L+90Post launch activities
CalibrationValidation
FG/bias/preclassifierSRR (Jan ‘12)
etc
L+90Official DAP delivery to
NDE
Jan’11 Oct’11Jan’09
Aug’09Updated version of
MiRS implemented at NDE and STAR
(full ATMS functionality)
Jun’10CDR for NPP
ATMS
MiRS in NDE Timeline (3/3)
Jan’10QC for NPP
Review
Jan’11TRR for NPP
ATMS
Dec’10QC DAP Delivery
Mar ‘11CUTR for NPP
ATMS
11
· Introduction· CDR REPORT· Software Architecture· Unit Test Readiness
» Test Readiness of Individual Units: MiRS
· Risks/Actions· Summary and Conclusions· Discussion
12
Section 2 – TRR Report
Presented by
C. Grassotti
• Review of CDR Action Items• TRR Entry Criteria• TRR Exit Criteria• CDR Summary
13
CDR Action Items
· Total of 7 Action Items
· For each AI, response was drafted (when available) describing the item, the action(s) (to be) taken, and status
· Response sent to AI author(s)
· At present:» 5 Items Closed» 2 Remain Open
14
CDR Action Items
Action Item
Description Author Lead Org Status
1 MiRS Operations Manual in OSPO A.K. Sharma STAR Closed
2Mitigation Plan for Missing/Bad Ancillary Data
A.K. Sharma NDE/STAR Closed
3QC DAP for Trend Analysis and QC Monitoring
A.K.SharmaSTAR Closed
4NDE/OSPO Define Product Tailoring Responsibilities
G. Goodrum NDE/OSPO Open
5 Obtain Risk Analysis Matrix J. Silva STAR Closed
6Data Submission Agreement for MiRS NPP/ATMS in CLASS
T. Schott OSPO Closed
7Feasibility of Implementing Tier-3 Processing in MiRS QC DAP
T. Schott, L. Zhao
OSPO Open
15
Action Item # 1: MiRS Operations Manual
Submitted by: A. K. SharmaDescription: Determine who will be responsible for writing the Operations Manual,
one of the required SPSRB documents. When will this document be delivered to OSPO?
Lead Organization: STARResponse:
(From STAR): Members of STAR, OSPO, NDE attended SPSRB Documentation Guidelines Mtg on 2 Nov 2010. Based on discussions the following guidance was obtained:
» Total of 4 Documents to be generated: Algorithm Theoretical Basis Doc (ATBD), System Maintenance Manual (SMM), Internal Users Manual (optional, depending on need), External Users Manual
» The Operations Manual is no longer a required document, as it provided redundant information already contained in the ESPC Operations Procedures documentation. However, information relevant to the operation of a specific science algorithm (installation, monitoring, etc.) will be contained in the SMM.
» SMM contents to be based on “Document Objects” (section reuse across some documents); SPSRB document guidelines outline which organizations (STAR/OSPO/NDE) are responsible for each document object. http://projects.osd.noaa.gov/spsrb/doc/System_Maintenance_Manual_Guidelines.doc
» STAR is lead organization for all docs up until delivery to operations, but responsibility for individual sections is shared between STAR/OSPO/NDE. After delivery, OSPO/NDE will manage docs
» For MiRS, MS Word may be used, following SPSRB version 2 guidelines. May be ported to Foswiki upon SPSRB approval of the software.
» Document delivery upon promotion of NPP to operational status
Status: CLOSED
16
Action Item # 2: Mitigation for missing or bad ancillary data
Submitted by: A. K. SharmaDescription: Explain the mitigation plan for missing ancillary data or bad ancillary
data (e.g. GFS required for the MIRS QC DAP) in NDE.Lead Organization: NDE/STAR Response: (From STAR): MiRS core retrieval algorithm does not rely on any
ancillary data, however, the QC DAP will require a GFS forecast file for the Tier-2 QC. If the latest GFS forecast file is unavailable, then the NDE system will provide the latest available GFS file. The QC DAP software will have the ability to extract forecast fields valid for the periods around the observation (initial, 6-hr,12-hr forecast etc.), up to a 24-hr forecast. If the GFS data is more than 24 hours old (highly unlikely), then the QC DAP should not be executed due to uncertainty in the forecast fields. If the GFS data are available but are bad for given collocated ATMS Fields-of-View (FOVs), the MiRS will set quality flags, and those FOVs will not be included when generating statistical metrics to monitor. If all GFS data are bad within a whole collocated NPP ATMS granule, processing will still commence, but QC metrics will be set to fill values (-999) signifying to the operator that the ancillary data, and not the retrieval, are bad.
Status: CLOSED
17
Action Item # 3: MiRS QC DAP for trend analysis and QC
monitoringSubmitted by: A. K. SharmaDescription: Explain how the MIRS QC DAP will be useful for trend analysis for
monitoring data quality.Lead Organization: STAR Response: The MiRS QC consists of several components. For trend analysis, the more useful metrics would be: QC
flag monitoring, which looks at rates of QC=0 and QC=2 occurrence, the channel NEDT monitoring, which looks at radiometric noise as a function of channel and time, as well as the radiometric monitoring, which looks at differences between observed and simulated brightness temperatures as a function of channel, scan angle, and time. Time series of these metrics are produced and updated each day within the STAR environment. In the STAR environment, geophysical performance monitoring (part of so-called Tier-3) is also being done in daily processing along with the radiometric QC. If it is deemed to be feasible by OSPO and NDE, and added as a requirement, then this level of QC monitoring, Tier-3, may be included with the QC DAP. This would include geophysical performance time series of temperature and water vapor (bias and standard deviation compared to NWP) at selected atmospheric layers. Slides # 75 and #81 of the MiRS NPP/ATMS CDR show time series examples of NEDT and radiometric bias monitoring, respectively. Many more examples may be found on the STAR MiRS website itself under "Products Monitoring: Data Quality", "Radiometric Performance: Calibration Bias Monitoring", "Sensors Quality: NEDT", and "Geophysical Performance: Performance Time Series". In the context of NDE, the time series for all QC monitoring will be constructed as granules are processed by MiRS and the QC DAP initiated, with the first point of the time series dependent on NDE data retention policies.
Status: CLOSED, Geophysical trend monitoring capability at NDE will be dependent on decisions by OSPO/NDE on whether to implement Tier-3 QC at NDE and add it as a requirement (see AI #7)
18
Action Item # 4: NDE and OSPO product tailoring
responsibilities
Submitted by: Geof Goodrum
Description: NDE and OSPO to coordinate and determine product tailoring responsibility. Meet with NDE and ESPC architects to clarify concepts and answer the following:» What is the ESPC architecture approach to sharing and blending products?» Use NDE distribution or the shared SAN to hand-off products to other OSPO
systems?» What is the schedule for ESPC “To Be” architecture?
Lead Organization: NDE/OSPO Response: (1) Limin, Geof and Joel met and discussed above questions. Data could be made
available to ESPC through NDE DDS, but eventually the data should be made available through the shared SAN in the ESPC “to be architecture”, which no final decision has been made yet. The decision is expected to come out in Dec, 2010 time frame, but no final word yet so far. (2) Limin presented a briefing on the detailed tailoring capability that is provided by ESPC for current POES/Metop/DMSP version of MIRS products, and discussed the cost and benefits based on different tailoring scenario that NDE would provide. NDE team will finalize their responsibility and present their final tailoring solution to STAR/OSPO. (3) As to the blended products that need the MIRS ATMS products, since it involves ingesting products from multi-satellites, NDE will feed the MIRS ATMS products to ESPC to allow the heritage application merge MIRS ATMS products with that from other satellites and continue to generate and distribute the blended products.
Status: OPEN
19
Action Item # 5: Risk Analysis and Probability Matrix
Summary
Submitted by: J. SilvaDescription: MIRS team to get the Risk Analysis Probability and Impact matrix
template from Jim Silva to summarize project risks.Lead Organization: STARResponse: Obtained template for Risk Analysis/Probability Matrix and updated to
reflect MiRS CDR Risk Analysis. This analysis has been added to the risks/actions section of the CDR slides.
Status: CLOSED
20
Action Item # 6: Data Submission Agreement for MiRS NPP/ATMS in CLASS
Submitted by: T. SchottDescription: OSPO to review current ESPC/CLASS Submission Agreement (SA)
with NDE to prepare for updated SA.Lead Organization: OSPO Response: Limin checked with Phil Jones at NCDC, and confirmed that a new
SA is needed for the NDE MIRS products. Options are: (1) the existing MIRS SA can be referenced and/or provided as attachment; (2) write a completely new SA, starting by deleting all legacy references in the existing MiRS DSA. Question as to whether NCDC requires restarting process at Step 1 with a “request for archive”. Given the situation that archive data will be produced by NDE for the first 18 months after launch, and also NDE has information about how and what the products will be made available to CLASS, the SA will need be coordinated between NDE and OSPO. OSPO can take a lead on this, but Limin needs to know whom she can work with in the NDE team on this matter. Action is recommended for closing, as current SA has been reviewed.
Status: CLOSED
21
Action Item # 7: Assess feasibility of implementing Tier-3 QC processing within
operations
Submitted by: T. Schott and L. ZhaoDescription: OSPO to draft RFA to assess feasibility of implementing Tier-3
(science quality) into operations if it was to be included in the MiRS QC DAP.Lead Organization: OSPO Response: OSPO needs detailed information from STAR, regarding effort, cost,
data resources, CPU and storage requirements, to determine if it is feasible to implement the Tier-3 QC monitoring into operation. STAR is available to discuss and provide additional information off-line on Tier-3 QC to OSPO.
Status: OPEN
22
CDR Action Items
Action Item
Description Author Lead Org Status
1 MiRS Operations Manual in OSPO A. Sharma STAR Closed
2Mitigation Plan for Missing/Bad Ancillary Data
A.Sharma NDE/STAR Closed
3QC DAP for Trend Analysis and QC Monitoring
A.SharmaSTAR Closed
4NDE/OSPO Define Product Tailoring Responsibilities
G. Goodrum NDE/OSPO Open
5 Obtain Risk Analysis Matrix J. Silva STAR Closed
6Data Submission Agreement for MiRS NPP/ATMS in CLASS
T. Schott OSPO Closed
7Feasibility of Implementing Tier-3 Processing in MiRS QC DAP
T. Schott, L. Zhao
OSPO Open
23
MiRS for NPP/ATMS TRR Entry Criteria
· Entry # 1 – Review of CDR Report with Action Items, Responses, Status
· Entry # 2 – Review of the TRR for MiRS NPP/ATMS in NDE» Software Architecture» Unit Test Readiness» Risks/Actions
24
MiRS for NPP/ATMS TRR Exit Criteria
· Exit # 1 – Test Readiness Review Report» TRR Report will be compiled and delivered after TRR» TRR Report to contain:
– TRR Presentation– Actions– Comments
25
CDR Report Summary
· This CDR Report closes the CDR· Total of 7 Action Items:
» At present, 5 of 7 items are closed, 2 remain open· 2 TRR Entry Criteria have been established· 1 TRR Exit Criterion has been established
26
· Introduction· CDR Report· SOFTWARE ARCHITECTURE· Unit Test Readiness
» Test Readiness of Individual Units: MiRS
· Risks/Actions· Summary and Conclusions· Discussion
27
Section 3 –Software Architecture
Presented by
C. Grassotti
28
Software Architecture: MiRS Background
· MiRS software already tested, implemented for OSPO (N18,N19, Metop, F16, F18)» High code maturity» A risk reduction for NPP/ATMS» Code unit testing was also done during MiRS development for POES and
DMSP before delivery to OSPO· Also running daily for AMSR-E and TRMM/TMI· Implemented for NPP/ATMS within STAR
» Runs daily using proxy data· Installed and tested using sample and proxy data at NDE
» TBD: integration with “live” data flow, daily processing, retention of multiple days/granules for QC testing.
· Modular design facilitates unit testing down to each major processing unit of the system-level data flow (flow diagram to follow)
· QC DAP components treated separately. MiRS retrieval codes independent from QC, but QC assumes that all MiRS products have been generated
29
Software Architecture:Standards Compliance (1/3)
• ISO Fortran 95 standards adherence is assessed using the Forcheck utility
• Manual Verification that SPSRB F95 coding standards are adhered to as well (standards and guidelines)» http://projects.osd.noaa.gov/spsrb/standards_software_coding.htm
• Standards upheld:» Use Implicit None in all modules and main programs.» Documented well enough. All modules/main programs/subroutines/functions have
a header:– Description of the code– Modules used– Subroutines or functions contained– Data types included– Intrinsic functions used– Argument description (input/output)– Code history
» In the code, major sections have their own comments to describe what the section of the code is doing.
» No hard coded numbers. All Constants are defined in module Const.
30
Software Architecture:Standards Compliance (2/3)
· Standards upheld (continued)» All variables are initialized. » Use free format syntax but the maximum length in each line is less
than 128 ( g95 will cut off any line longer than 128 characters).» No tab characters in all Fortran codes. » No GOTO statements in all codes. » All unused variables are removed.» Use private to ensure proper encapsulation.» No conditional statements using == or /= with floating points.» No numbered DO loops. Labeled DO loops are used.» All intrinsic functions are declared before using them.
31
Software Architecture:Standards Compliance (3/3)
· Use valgrind to help detect any memory leaks. Valgrind is a Linux program used to “detect many memory management and threading bugs, avoiding hours of frustrating bug-hunting, making your programs more stable”. Valgrind detailed information and usage:
http://www.valgrind.org/· Since last version, no memory leaks in MIRS (v7)
» All dynamically allocated memories are deallocated when they are not used anymore.
» No array index out of bounds. Array index out of bounds will usually cause “segmentation fault”.
· All code changes are managed by Subversion: a source version control system
· Full compliment of code review results related to coding standards to be provided in the Code Unit Test Review (March 2011)
32
Software Architecture:Resource Requirements
Minimum*# of CPUs
Minimum RAM
Min. Hard Disk Space
Platform Type
MIRS Requirements
1 1GB 3GB Any
Hardware Requirements
Operating System
C Compiler
F95 Compiler
Commercial Software
Freeware
MIRS Requirements
Linux orIBM AIX
xlC orxlc++ orgcc org++
ifort org95 orgfortran or Xlf95
IDL v6 and Up
BASH
Software RequirementsNot necessary for standard outputs
BUT required for QC DAP
*This is minimum required. Number of CPUs will affect the speed of execution. 8 CPUs are used in STAR to run three sensors daily.
33
Software Architecture:Overview
· The software architecture describes the structure of the system software elements and the external and internal data flows between software elements.
· The MiRS software architecture is described in the MiRS System Description Document created for POES and DMSP and delivered to OSPO. Updated documentation will be delivered by STAR as specified in the CDR Report.
· 3 Layers of design will be presented (STAR EPL Guidelines):» Context Layer - 0: External Interfaces
» System Layer - 1: Flow Between Units
» Unit Layer - 2: Flow Within Units
Section 3 (this section)
Section 4 (next section)
34
The Context-Layer:STAR EPL Guidelines
· The Context-Layer describes the flows between the system and its external interfaces
· An external input is defined as a data source needed by the system that is produced or made available by a process external to the system
· An external output is defined as a data sink that is produced by the system for an external user
· External interfaces must meet standard criteria to be included in the system architecture
MiRS Context Layer:External Interfaces
1
MiRS/QC
SAN
NDE Product Generation Manager
MiRS External Interfaces
Product Generation
Specifications
Working Directory
Systems Configurations
Forensics Repository
Input Files & PCF
InvocationProcess Req.
Rule SetsOutput Files
& PSF
Product Files
PSF (MiRS output)
Return Code
Working Directory Output
NDE Distribution ServerInput Files (HDF5, GRIB)
ESPCInput (HDF5) Files GFS (GRIB)
PCF (MiRS input)
IDPS
DAP Specifications
Data AreasConfigurations InfoMIRS SystemNDE Production Manager
NDE DHS Boundary
DDS ECMWF,GDAS (GRIB)
NWP not necessary for Core Products (QC only)
36
External Interfaces: Assumptions
· Tailoring and (re)formatting done by NDE and OSPO (Context Layer)
· Interactions with operational users and CLASS handled by NDE and OSPO (Context Layer)
· No incompatibility between NDE and OSPO ESPC environment» Single version of MiRS to maintain
37
· File format requirements for NPP ATMS» ATMS level 1b granules/geo formatted in HDF5» PCF ascii file generated by NDE DHS» MiRS product outputs formatted in netCDF4 (CF conventions)» MiRS output PSF ascii file listing output files» MiRS readers and encoders support these formats and have been
tested in NDE· Metadata:
» Current MiRS Collection Level metadata available in ISO 19115 at CLASS (for POES/Metop/DMSP)
» Expectation is for a similar Collection Level file for MIRS NPP ATMS products to be stored at CLASS
» Metadata requirements for MiRS NPP ATMS will be outlined by updated Submission Agreement in the future
» MiRS NPP ATMS Granule Level metadata to be contained inside the MiRS netCDF4 output header (following STAR metadata template)
MiRS External Interfaces
38
External Interfaces: Status
· All external interfaces have been reviewed and approved; no changes since CDR
· No open actions on external interfaces
39
The System-Layer: STAR EPL Guidelines
· The System-Layer data flow expands upon the Context-Layer data flow, showing the first layer of decomposition. » In addition to the System-Layer inputs and outputs, the
major processing units are shown along with their inputs and outputs.
» Each unit is designed as a stand-alone program for ease of testing and integration into a System-Layer scheduler.
40
The MiRS System Layer: Processing Units
· Each major step in the MiRS processing sequence is a stand-alone bash script and a corresponding Fortran 95 executable and namelist file and constitutes a Layer-2 Test Unit
Code Unit Purpose
r2r2tdr Convert raw data records to temperature data records (antenna temperatures); generate sensor NEDT files
tdr2sdr Convert temperature data record to sensor data record (TBs or radiances)
fm Footprint matching
chopp Chop fm files into sub-files
applyRegr (retrRegr) First guess generation using TB-based regression (applied on chopped fm files)
fmsdr2edr (1dvar) 1dvar: converts footprint matched SDRs to EDRs
mergeEdr Merge EDR files into 1 file
vipp Postprocessing converts EDRs to derived environmental parameters (DEPs)
convertMirs2nc Converts files from MiRS binary to netCDF4
41
The MiRS System-Layer:npp_scs.bash
· Some or all units in the system layer may be invoked by the top level driver script npp_scs.bash (in operations all 9 units run from same invocation of driver script).
· The system layer is where NDE will invoke the MiRS software units.· Each unit is a bash script-function that drives a low-level fortran processing program.· When the system’s input data is available (ATMS granule), the NDE PGM will run the
top level driver script.» Execute npp_scs.bash passing the “working directory” path as the argument» The NDE PGM will generate a Process Configuration File (PCF) which contains all input file locations and
parameters required for processing, and is read in by the driver script.
· All code units process sequentially, one ATMS granule at a time. · The NDE DHS must be able to run multiple instances of these units to process
concurrently available granules.· Each instance will produce a PSF file containing a list of output product files if they
were created successfully.· In STAR, the driver script is invoked through crontab and the PCF variables are
specific to the STAR environment.
42
MiRS System-Layer Process Flow: NDE Environment
d
Local Processing Directories(working directory)
NDESAN
(HDF5)
L1
B S
en
so
r D
ata
(H
DF
5)
rdr2tdr
TDRs
TDRs
SDRs
SDRsFMSDRs
FMSDRs
Chop FMSDRs
REGRESS Retr
Chopped FMSDRs
Chopped FMSDRsEDRs
EDRs + Ancillary`
DEPs
SND (netCDF4 EDR)
IMG (netCDF4 DEP)
npp_scs.bashPGM
PCF
Return value to PGM
Process Status File
Log FileLocal Processing Directories
tdr2sdr
fm
chopp
applyRegress
1dvar
mergeEdr
vipp
mirs2nc
Merged EDR
EDRs
EDRs + DEPs
L1B Sensor Data
43
System Layer Architecture: Status
· All System Layer components have been reviewed and approved; no changes since CDR
· No open actions on system layer components
44
· Introduction· CDR Report· Software Architecture· UNIT TEST READINESS
» Test Readiness of Individual Units: MiRS
· Risks/Actions· Summary and Conclusions· Discussion
45
Section 4 –Unit Test Readiness
Presented by
K. Garrett - MiRS DAP C. Grassotti - QC DAP
46
The Unit-Layer: STAR EPL Guidelines
· The Unit-Layer data flow expands upon the System-Layer data flow, showing the second layer of decomposition. » This layer shows the detailed data flow into and out of
each software unit.» And, it shows data flow within the software units (if
applicable).
47
Test Readiness: Environment and
Configuration
· For all unit tests the testing environment will be both the STAR Linux servers and the NDE computational systems
· STAR Environment: Full end to end testing of MiRS DAP (RDRs to SND/IMG netCDF4) and QC DAP using proxy data for ~ 1 day of data and sample data using 1 granule
· NDE Environment:» MiRS DAP: Testing of ~ 1 day of proxy data independent of DHS (minimum);Testing of ~ 1
day of proxy data interfaced with DHS; 1 granule of sample data independent of DHS» QC DAP: Testing of ~ 1 day of proxy data independent of DHS
· For testing in STAR, configuration is the same as that used in the daily processing for all other sensors (e.g. f95, IDL, and bash scripts)
· For testing in NDE, configuration is identical, except for high level process management (interaction with DHS). Output files for all processing steps to be compared against benchmark data produced in STAR test.
48
Test Readiness: Test Data and Truth Data
· Test data will include:» Proxy data generated using MIT/LL software from Metop-A (1 day 2011-01-15)» Proxy data generated using GRAVITE (1 day 2011-01-15)» Sample data generated by IDPS (1 granule TBD)
· All input test data in HDF5 format following NPOESS Common Data Format Control Book (vol. III)
· Truth data: NWP analysis from GDAS as well as EDRs and DEPs obtained from the operational sensors valid at the same time and location (i.e. not all parameters available for validation in NWP data; e.g. sea ice)
49
All Units:Test Method, Sequence
· The test method is to run using ATMS proxy data in stand-alone mode (single execution of bash script and f95/IDL code) with all the required input/output files defined at the script level.
· For many steps the upstream input data files are simply the output generated by the previous required Layer 2 MiRS processing unit (e.g. fm→fmsdr2edr)
· Confirmation of successful test will be determined by verifying creation of intermediate and output data (comparison with benchmark files), using log files, and/or diagnostic graphical tools (IDL) available in the STAR and/or NDE environment (maps, error plots, etc.)
50
Unit Tests
· The following will be presented for each of the 9 units:» Unit Description : Purpose and Function» Process Flow» Test Items» Input/Output Data» Test Risks
51
Unit rdr2tdr: Purpose and Function
· Convert raw sensor data records to temperature data records (antenna temperatures)
· Reformatted into MiRS internal format· Computes sensor radiometric noise values (used to update the
instrument noise matrix used in the 1dvar minimization)· Input: Level 1b in HDF5 format produced by external process at NDE
(current testing with proxy data generator uses HDF5). For NPP ATMS, the Level 1b data may be actual TDRs or SDRs, rather than RDRs.
· Output: TDR and NEDT files in internal format produced by rdr2tdr· It reads in a namelist which specifies operating parameters (passed
from the PCF by npp_scs.bash)
52
Unit rdr2tdr: Process Flow
d
Local Processing Directories
rdr2tdr
Local Processing Directories
Input Files Specified in f95 Namelist
NDE SAN
Input: Level 1b Sensor Data (HDF5)
Output: • TDR Files• NEDT Files (MiRS Internal)
rdr2tdr unit script
Process Control FileExecution from PGM
Log File
Return value to PGM if failure
npp_scs.bash
53
Unit rdr2tdr Test Items
Software Unit Purpose File Input File Output
npp_scs.bash This is a driver script that: (1) Reads in the PCF, parses its contents, and passes necessary information to rdr2tdr.bash. (2) Calls rdr2tdr script(3) Writes out log files, the PSF file, and returns error status to the NDE DHS.
PCF PSFDriver script log file
rdr2tdr.bash Performs local file management , staging, resources file generation (e.g.f95 namelist) and error handling forrdr2tdr.
Script log fileExecutable program log fileExecutable resource input file (F95 namelist)
rdr2tdr.f90 For a single ATMS granule, it must read both the ATMS HDF5 RDR*and Geo files.
* In the case of NPP ATMS, rdr2tdr code can read in either ATMS TDRs or SDRs and convert to MiRS TDR format.
1 ATMS TDR or SDR HDF5 file1 ATMS Geo HDF5 fileNamelist file
1 MiRS TDR_* file1 MiRS NEDT_* file
54
Unit rdr2tdr –Input/Output Test Data Table
Test Data Item Type Filename* Description
1 Input npp_atms_rdr2tdr_2011-01-15.in F95 Namelist file
4 Raw
Sensor data
SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5 NPP HDF5 granule
5 Geo File GATMO_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5 NPP HDF5 Geolocation
2 Output TDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5
MiRS internal format file containing output TDR
data
3 Output npp_atms_nedt_2011_01-15_befFm.dat NEDT file
*All sensor data filenames used in the following tables pertain to proxy data, which mimics IDPS sample data naming conventions.
55
Unit rdr2tdr –Test Risks
· Test Risk #1: Simulated data and data formats may not fully represent the actual data (identified in CDR)
· Mitigation: Use of GRAVITE data sets from IPO or latest sample data from IDPS
56
Unit tdr2sdr: Purpose and Function
· Apply antenna pattern correction (if selected) to convert tdr files to sensor data record (TBs or radiances)
· In operations, no correction applied and the step is simply a reformatting to MiRS internal format (differences accounted for in radiometric bias corrections)
· Input: TDR files in internal format produced by rdr2tdr· Output: SDR files in internal MiRS format· It reads in a namelist which specifies operating
parameters (passed from the PCF by npp_scs.bash)
57
Unit tdr2sdr: Process Flow
d
Local Processing Directories
tdr2sdr
Input Files Specified in f95 Namelist
Input: TDR Files(MiRS Internal)
Output: SDR Files(MiRS Internal)
tdr2sdr unit script
Process Control FileExecution from PGM
Log File
Return value to PGM
Local Processing Directories
Process Status File
npp_scs.bash
58
Unit tdr2sdr Test Items
Software Unit Purpose File Input File Output
npp_scs.bash This is a driver script that: (1) Reads in the PCF, parses its contents, and passes necessary information to tdr2sdr.bash. (2) Calls tdr2sdr script(3) Writes out log files, the PSF file, and returns error status to the NDE DHS.
PCF PSFDriver script log file
tdr2sdr.bash Performs local file management , staging, resources file generation (e.g.f95 namelist) and error handling fortdr2sdr.
Script log fileExecutable program log fileExecutable resource input file (F95 namelist)
tdr2sdr For a single ATMS granule, it must read the ATMS TDR file, apply attenna pattern correction (optional) and write out to SDR file (MiRS internal format).
1 ATMS TDR_* fileNamelist file
1 MiRS SDR_* file
59
Unit tdr2sdr –Input/Output Test Data Table
Test Data Item Type Filename Description
1 Input npp_atms_tdr2sdr_2011-01-15.in F95 Namelist File
2 Input TDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5
MiRS internal format file to containing TDR data
3 Output SDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5
MiRS internal format file containing SDR data
60
Unit tdr2sdr –Test Risks
· Test Risk #1: Proxy data reliability (identified in CDR)
61
Unit fm: Purpose and Function
· Footprint matching to ensure all channels view the same location on the Earth
· Three options:» Use fm code (to be available from IPO) to tailor resolution to available CPU
resources and requirements (Best option)» Use resampled data from IDPS (step would then be a simple copy)» MiRS heritage code: FOV averaging
– High resolution: 96 FOVs per scanline. Each channel measurement retains its value for each FOV
– Low resolution: 3x3 average for each FOV at each channel (3 FOVs x 3 scanlines), 32 FOVs per scanline
· Input: SDR files produced by tdr2sdr· Output: Footprint-matched SDRs (FMSDRs) in the MiRS internal RAD
file format and NEDT file· It reads in a namelist which specifies operating parameters (passed
from the PCF by npp_scs.bash)
62
Unit fm: Process Flow
d
Local Processing Directories
fm
Input Files Specified in f95 Namelist
Input: SDR File(s)(MiRS Internal)
Output: FMSDR File (MiRS Internal)
fm unit script
Process Control FileExecution from PGM
Log File
Return value to PGM
Local Processing Directories
Process Status File
npp_scs.bash
63
Unit fm Test Items
Software Unit Purpose File Input File Output
npp_scs.bash This is a driver script that: (1) Reads in the PCF, parses its contents, and passes necessary information to fm.bash. (2) Calls fm script(3) Writes out log files, the PSF file, and returns error status to the NDE DHS.
PCF PSFDriver script log file
fm.bash Performs local file management , staging, resources file generation (e.g.f95 namelist) and error handling forfm.
Script log fileExecutable program log fileExecutable resource input file (F95 namelist)
fm For a single ATMS granule, it must read in all necessary files, perform footprint matching or averaging and write out to FMSDR file (MiRS internal format).
1 ATMS SDR_* file1 NEDT file (befFm)Namelist file
1 MiRS FMSDR_* file
64
Unit fm –Input/Output Test Data Table
Test Data Item Type Filename Description
1 Input npp_atms_fm_2011-01-15.in F95 Namelist file
2 Input SDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5
MiRS internal format file to containing SDR data
3 Output npp_atms_nedt_2011_01_15_aftFm.dat NEDT File
4 Output FMSDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5
MiRS internal format file containing FMSDR data
65
Unit fm –Test Risks
· Test Risk #1: Proxy data reliability (identified in CDR)· Test Risk #2: Optimal footprint matching codes not
available from IPO for testing within MiRS framework.
66
Unit chopp: Purpose and Function
· The chopp step allows for the simultaneous retrieval of multiple sub-orbits or sub-granules derived from the input file
· Chopping divides the orbit or granule FMSDR files into X number of files, where X is defined in the PCF
· Input: FMSDR files produced by fm step· Output: Chopped FMSDRs· It reads in a namelist which specifies operating parameters (passed
from the PCF by npp_scs.bash)· Note: This step is optional, and may be configured to optimize the
efficiency of the processing environment and take advantage of unused resources.
67
Unit chopp: Process Flow
d
Local Processing Directories
chopp
Input Files Specified in f95 Namelist
Input: FMSDR File(s)(MiRS Internal)
Output: Chopped FMSDR Files (MiRS Internal)
chopp unit script
Process Control FileExecution from PGM
Log File
Return value to PGM
Local Processing Directories
Process Status File
npp_scs.bash
68
Unit chopp Test Items
Software Unit Purpose File Input File Output
npp_scs.bash This is a driver script that: (1) Reads in the PCF, parses its contents, and passes necessary information to chopp.bash. (2) Calls chopp script(3) Writes out log files, the PSF file, and returns error status to the NDE DHS.
PCF PSFDriver script log file
chopp.bash Performs local file management , staging, resources file generation (e.g.f95
namelist) and error handling for chopp.
Script log fileExecutable program log fileExecutable resource input file (F95 namelist)
chopp For a single ATMS granule, it must read in all necessary files in order to chop the FMSDR into sub-granules (MiRS internal format).
1 ATMS FMSDR_* fileNamelist file
X MiRS CHP_FMSDR_*_00X file
69
Unit chopp –Input/Output Test Data Table
Test Data Item Type Filename Description
1 Input npp_atms_chopp_2011-01-15.in F95 Namelist file
2 Input FMSDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5
MiRS internal format file to containing SDR data
3 OutputCHP_FMSDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.
h5_00X
MiRS internal format files containing FMSDR data
70
Unit chopp –Test Risks
· None Identified
71
Unit applyRegress: Purpose and Function
· First guess generation· Needed to start 1dvar minimization (iterative)· Estimate of the geophysical state vector (T,q, Tskin, Emis, etc.)
needed at each fmsdr location· applyRegress: Regression (using real observed TBs based on
off-line training)· Input: FMSDR (or chopped FMSDR) files, tuning files, covariance
matrix files, bias correction files, modeling error file, topography file· Output file format: Same as EDR Scene files (internal MiRS)· It reads in a namelist which specifies operating parameters (passed
from the PCF by npp_scs.bash)
72
Unit applyRegress: Process Flow
d
Local Processing Directories
applyRegress
Input Files Specified in f95 Namelist
Input: • FMSDR File• Regression Coefficients (6
params * 4 sfc types)• Topography• Atm and Sfc bg cov• Tuning file• Rad bias file(MiRS Internal)
Output: REGR File (MiRS Internal EDR)
applyRegress unit script
Process Control FileExecution from PGM
Log File
Return value to PGM
Local Processing Directories
Process Status File
npp_scs.bash
73
Unit applyRegress Test Items
Software Unit Purpose File Input File Output
npp_scs.bash This is a driver script that: (1) Reads in the PCF, parses its contents, and passes necessary information to applyRegress.bash. (2) Calls applyRegress script(3) Writes out log files, the PSF file, and returns error status to the NDE DHS.
PCF PSFDriver script log file
applyRegress.bash Performs local file management , staging, resources file generation
(e.g.f95 namelist) and error handling forapplyRegress.
Script log fileExecutable program log fileExecutable resource input file (F95 namelist)
applyRegress For a single ATMS granule, it must read in all necessary files, perform the regression using FMSDR s(TBs) and write out to REGRESS file (MiRS EDR internal format).
1 ATMS FMSDR_*fileor X CHP_FMSDR_*00XCoefficient filesNamelist file
1 MiRS internal format REGRESS_*file or X REGRESS_*_00X files
74
Unit applyRegress –Input/Output Test Data Table
Test Data Item Type Filename Description
1 Input npp_atms_ApplyRegress_2011-01-15.in F95 Namelist file
2 Input Oc_regressCoeffs_npp_temp.dat* Coefficient Files
3 Input Topography.bin Topography file
4 Input TunParams_npp_atms.in, TunParams_npp_atms_2.inTuning Parameters
files
5 InputCovBkgMatrxTotAtm_all.dat,
CovBkgMatrxTotSfc_all_npp_atms.datCovariance Matrix files
w/background
6 InputbiasCorrec_npp_atms.dat, ModelErrFile_npp_atms.dat
Bias correction and modeling error
* Coefficient files for Ocean, Sea-ice, Land, and Snow surfaces, and for Temperature, Water Vapor,Skin Temperature, Emissivity, and Cloud Liquid Water
75
Unit applyRegress –Input/Output Test Data Table
Test Data Item Type Filename Description
7 Input
FMSDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5
OrCHP_FMSDR_SATMS_npp_d20110115_t0000311_e0000506_b000
15_c20110116000000020002_proxy_pop.h5_00X
MiRS internal format file(s) containing
FMSDR data
8 Output
REGR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5
OrCHP_REGR_SATMS_npp_d20110115_t0000311_e0000506_b00015
_c20110116000000020002_proxy_pop.h5_00X
MiRS internal format file(s) containing EDR
data
76
Unit applyRegress –Test Risks
· Test Risk #1: Proxy data reliability (identified in CDR)
77
Unit fmsdr2edr: Purpose and Function
· 1dvar: converts footprint matched SDRs to EDRs (MiRS core products)· All elements of geophysical state vector retrieved simultaneously: T(p),
q(p), Tskin, Emis, CW(p), RW(p), IW(p)· Variational solution optimal assuming error characteristics of geophysical
background, forward model (CRTM), sensor NEDT, and Jacobians (derivatives) obtained from forward model
· Tuning file can adjust characteristics of retrieval (nEOFs, channels used, etc.)
· QC flags assigned to each EDR location· Input: FMSDRs (MiRS RAD file format) produced by fm code (or
chopped FMSDRs), tuning files, NEDT file, first-guess, covariance matrix files, CRTM files, bias correction files, modeling error file, topography file
· Output: EDR Scene files (internal MiRS)· It reads in a namelist which specifies operating parameters (passed from
the PCF by npp_scs.bash)
78
Unit fmsdr2edr: Process Flow
d
Local Processing Directories
1dvar
Input Files Specified in f95 Namelist
fmsdr2edr unit script
Process Control FileExecution from PGM
Log File
Return value to PGM NDE SAN
Output:• EDR File (Internal format)
Input :• Tuning Files (2)• CRTM Coef Files (3)• Rad Bias• Sfc Topography• BG Mean, Covariance,
Eigenvectors (2)• Fwd Model Error• NEDT• FMSDR(MiRS Internal)
Static
Dynamic
(NetCDF4)
Local Processing Directories
Process Status File
npp_scs.bash
convertMirs2nc
79
Unit fmsdr2edr Test Items
Software Unit Purpose File Input File Output
npp_scs.bash This is a driver script that: (1) Reads in the PCF, parses its contents, and passes necessary information to fmsdr2edr.bash. (2) Calls fmsdr2edr script(3) Writes out log files, the PSF file, and returns error status to the NDE DHS.
PCF PSFDriver script log file
fmsdr2edr.bash Performs local file management , staging, resources file generation (e.g.f95
namelist) and error handling for 1dvar.
Script log fileExecutable program log fileExecutable resource input file (F95 namelist)
1dvar For a single ATMS granule, it must read in all necessary files (dynamic and static), perform the 1dvar retrieval of core products and write out to EDR file (MiRS internal format).
1 ATMS FMSDR_* file(or X chopped FMSDRs)Namelist fileNEDT file (aftFm)Tuning FilesCovariance MatricesCRTM filesTopography fileBias Corr./Modeling ErrorFirst-guess
1 MiRS EDR_* file or X chopped EDR_*_00X
80
Unit fmsdr2edr –Input/Output Test Data Table
(1/2)
Test Data Item
Type Filename Description
1 Input npp_atms_CntrlConfig_1dvar_2011-01-15.in F95 Namelist file
2 InputbiasCorrec_npp_atms.dat, ModelErrFile_npp_atms.dat
Bias correction and modeling error
3 Inputnpp_atms_SpcCoeff.dat, npp_atms_TauCoeff.dat,
mw_cloud_opt.datCRTM files
4 Input npp_atms_nedt_2011_01_15_aftFm.dat NEDT File
5 Input topography.bin Topography file
6 Input TunParams_npp_atms.in, TunParams_npp_atms_2.in Tuning Parameters files
7 Input
REGR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5.LR.ORB
OrCHP_
REGR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5.LR.00X
First-guess file(s)
81
Unit fmsdr2edr –Input/Output Test Data Table
(2/2)
Test Data Item
Type Filename Description
8 InputCovBkgMatrxTotAtm_all.dat,
CovBkgMatrxTotSfc_all_npp_atms.datCovariance Matrix
files
9 Input
FMSDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5
OrCHP_FMSDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c2
0110116000000020002_proxy_pop.h5_00X
MiRS internal format file(s) containing
FMSDR data
10 Output
EDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5.LR.ORB
OrCHP_EDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c201
10116000000020002_proxy_pop.h5.LR.00X
MiRS internal format file(s) containing EDR
data
82
Unit fmsdr2edr –Test Risks
· Test Risk #1: Proxy data reliability (identified in CDR)· Test Risk #2: Radiometric bias generation using proxy data
will require regeneration when sufficient real data become available (identified in CDR)
83
Unit mergeEDR: Purpose and Function
· The mergeEDR step combines all sub-orbit or sub-granule retrieval (EDR) files back into 1 file, if chopping was applied.
· X files are merged together, where X is defined in the PCF· Input: EDR files produced by fmsdr2edr step· Output: 1 Merged EDR file· It reads in a namelist which specifies operating parameters (passed
from the PCF by npp_scs.bash)· Note: This step is optional, and may be configured to optimize the
efficiency of the processing environment and take advantage of unused resources. Must be enabled if the chopping step is applied.
84
Unit mergeEDR: Process Flow
d
Local Processing Directories
mergeEDR
Input Files Specified in f95 Namelist
Input: EDR Files(MiRS Internal)
Output: Merged EDR File (MiRS Internal)
mergeEDR unit script
Process Control FileExecution from PGM
Log File
Return value to PGM
Local Processing Directories
Process Status File
npp_scs.bash
85
Unit mergeEDR Test Items
Software Unit Purpose File Input File Output
npp_scs.bash This is a driver script that: (1) Reads in the PCF, parses its contents, and passes necessary information to mergeEDR.bash. (2) Calls mergeEDR script(3) Writes out log files, the PSF file, and returns error status to the NDE DHS.
PCF PSFDriver script log file
mergeEDR.bash Performs local file management , staging, resources file generation (e.g.f95
namelist) and error handling for merge.
Script log fileExecutable program log fileExecutable resource input file (F95 namelist)
mergeEDR For a single ATMS granule, it must read in all chopped EDR files in order to merge the EDRs into 1 EDR file (MiRS internal format).
X ATMS EDR_*_00X filesNamelist file
1 MiRS EDR_* file
86
Unit mergeEDR –Input/Output Test Data Table
Test Data Item Type Filename Description
1 Input npp_atms_chopp_2011-01-15.in F95 Namelist file
2 Input EDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5_00X
MiRS internal format files containing EDR data
3 Output EDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5
MiRS internal format file containing EDR data
87
Unit mergeEDR –Test Risks
· None Identified
88
Unit vipp: Purpose and Function
· Post-processing converts EDRs to derived environmental parameters (DEPs)· Two types of processing:
» 1. Vertical Integration: – q(p) → TPW, CL(p) → CLW, RW(p) → RWP, IW(p) → IWP
» 2. Post-processing: – Rain Rate: Regression RR=f(CLW,RWP,IWP)– Sea Ice Concentration (FY,MY): Look-up in emissivity catalogs– Snow Water and Grain Size: Minimized cost function (emissivity LUT
based on CRTM physical snow emissivity model, uncertainties in SW and GS).
· Input: EDR Scene files produced by fmsdr2edr or mergeEDR (internal MiRS)· Output: DEP (Derived product) file (internal MiRS)· It reads a PCF, writes a PSF, and returns error handling information to the calling
NDE Data Handling System (via npp_scs.bash).
89
Unit vipp: Process Flow
d
Local Processing Directories
vipp
Input Files Specified in f95 Namelist
vipp unit script
Process Control FileExecution from PGM
Log File
Return value to PGM
Output Files:• DEP File (Internal format)
Input Files:• Sea Ice Emis Catalog• Snow Emis Catalog
• EDR File(s)(MiRS Internal)
Static
Dynamic
(NetCDF4)
Local Processing Directories
Process Status File
npp_scs.bash
NDE SAN
convertMirs2nc
90
Unit vipp Test Items
Software Unit Purpose File Input File Output
npp_scs.bash This is a driver script that: (1) Reads in the PCF, parses its contents, and passes necessary information to vipp.bash. (2) Calls vipp script(3) Writes out log files, the PSF file, and returns error status to the NDE DHS.
PCF PSFDriver script log file
vipp.bash Performs local file management , staging, resources file generation (e.g.f95 namelist) and error handling forvipp.
Script log fileExecutable program log fileExecutable resource input file (F95 namelist)
vipp For a single ATMS granule, it must read in all necessary files (dynamic and static), perform the perform the vertical integration and postprocessing of core products and write out to DEP file (MiRS internal format and netCDF4).
1 ATMS EDR_* file 1 MiRS DEP_* file
91
Unit vipp –Input/Output Test Data Table
Test Data Item
Type Filename Description
1 Input npp_atms_CntrlConfig_1dvar_2011-01-15.in F95 Namelist file
2 Input EDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5.LR.ORB
MiRS internal format file containing EDR
data
3 InputSeaIceEmissCatalog_npp_atms.dat, SnowEmissCatalog_npp_atms.dat
Snow and sea-ice Emissivity Lookup
Tables
4 Output DEP_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5.LR.ORB
MiRS internal format file containing DEP
data
92
Unit vipp –Test Risks
· Test Risk #1: Proxy data reliability (identified in CDR)· Test Risk #2: Catalog files based on proxy data will require
real data for regeneration (identified in CDR)
93
Unit convertMirs2nc: Purpose and Function
· Converts EDR and DEP files from MiRS internal binary to netCDF4 format
· Input: EDR from fmsdr2edr output· Input: DEP from vipp output· Output: SND and IMG product files in netCDF4 format
94
Unit convertMirs2nc: Process Flow
Local Processing Directories
mirs2nc
Input Files Specified in f95 Namelist
Input: EDR or DEP File(s)(MiRS Internal)
Output: EDR (SND) or DEP (IMG) Files (netCDF4)
convertMirs2nc unit script
Process Control FileExecution from PGM
Log File
Return value to PGM
Local Processing Directories
Process Status File
npp_scs.bash
NDE SAN
IMG
/SN
D.n
c
95
Unit convertMirs2nc –Input/Output Test Data Table
Test Data Item
Type Filename Description
1 Input EDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5.LR.ORB
MiRS internal format file containing EDR
data
2 Input DEP_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5.LR.ORB
MiRS internal format file containing DEP
data
3 Output MIRS-SND_v7_NPP_s201101150311_e201101150506_c201101160215.ncSND products in
netCDF4 file
4 Output MIRS-IMG_v7_NPP_s201101150311_e201101150506_c201101160215.ncIMG products in
netCDF4 file
96
Unit convertMirs2nc –Test Risks
· None Identified
97
MiRS Design Assumptions Summary
· The MiRS design assumes the NDE PGM will run each instance of a MiRS unit in a single working directory that is unique to that run.
· The PCF will be made available to each unit driver script locally in the working directory at run time.
· The MiRS processing units (rdr2tdr, tdr2sdr, fm, chopp, applyRegress, fmsdr2edr, mergeEDR, vipp, convertMirs2nc) are designed to process the data for a single ATMS granule set. » The NDE PGM will therefore be able to run multiple instances of the same
MiRS bash script to process several granule sets at a time.
· The NDE DHS will perform all file management for MiRS outside of the working directory where MiRS runs.
· NDE will perform all tailoring and/or reformatting
98
MiRS DAP Software Architecture Summary
· All MiRS products will be generated by 9 software units» rdr2tdr» tdr2sdr» fm» chopp» applyRegress» fmsdr2edr» mergeEdr» vipp» convertMirs2nc
· These units and the layers of data flow will be documented in the MiRS Documentation (System Maintenance Manual, External Users Manual)
· These units will be:» Delivered to NDE in the form of a DAP» Integrated into the NDE DHS by the NDE team» NDE will run MiRS in production mode at NSOF
99
MiRS QC Overview
· Overall Objective: Comprehensive real-time monitoring at various stages of MIRS data processing; selective flagging and notification of anomalies via e-mail
· Run daily in STAR environment; Applied to N-18, N-19, Metop-A, NPP/ATMS, DMSP · Tier-1 (“Autonomous”) QC: Easily implemented in operations
» NEDT monitoring (time series, e-mail alerts) → QC DAP
» QC flag monitoring (time series, e-mail alerts) → QC DAP
» Convergence monitoring (1dVAR) (time series, e-mail alerts) → QC DAP· Tier-2 (“Intermediate”) QC:
» Radiometric performance (time series, e-mail alerts for dynamic biases) → QC DAP· Tier-3 (“Science”) QC: Appropriate for R&D environment
» Geophysical performance (comparisons with GDAS, ECMWF, GFS)» Comparisons w/ RAOBS, field experiments, other sensors/algorithms
· All monitoring results displayed on STAR MIRS website: time series, maps, statistics (+ other assessment plots)
· Off-line tools to assess system performance sensitivity to channel selection, EOFs, noise levels, etc.
Not part of QC DAP
100
MiRS QC: Implementation Status at
NDE
· MiRS QC DAP Prepared and Delivered (Dec 2010) to NDE» Contains all codes and files necessary to run Tier-1 and 2 QC» Requires f95 compiler and IDL v6 license» Currently testing
· What are the NDE requirements for product registration? QC processing assumes that:» Core and derived products have been generated (MiRS binary format
retained)» NWP (GFS) fields are available (for Tier-2 radiometric QC only)» If time series desired, then QC,NEDT,Radiometric bias files must be
retained
101
QC Layer-2 Units: Summary
· Each step in the MiRS Tier-1 and Tier-2 QC processing sequence is a stand-alone bash script and one or more corresponding Fortran 95 executables, or IDL procedures (v6) driven by namelist files
Code Unit Purpose Tier Comment
qcNedt Parses NEDT files (generated by rdr2tdr) and generates an e-mail if thresholds exceeded
1 Autonomous
qcRetrieval Parses QC files (generated by fmsdr2edr) and generates an e-mail if thresholds of QC or conv rate exceeded
1 Autonomous
prepNwp Convert NWP files from GRIB to binary 2 NWP required
nwp Interpolates NWP fields to FMSDR location and time 2 NWP required
fwd Runs forward model on NWP output (generated by nwp) to simulate TBs at each FMSDR location
2 NWP required
biasGen Compares FWD to FMSDR TBs and generates bias files 2 NWP required
qcRadiometricBias Parses bias files (generated by biasGen) and generates an e-mail if thresholds exceeded
2 NWP required
biasMonitor Produces time series of radiometric bias from bias files (generated by biasGen)
2 NWP+product retention
dataQualityMonitor Produces time series of nedt, qc, convergence from NEDT and QC files (generated by rdr2tdr and fmsdr2edr)
1 Product retention required
102
MiRS Tier-1 and 2 QC System-Layer Process Flow: NDE
Environment
dLocal Processing Directories
qcNedt (T1)
Unless specified, all formats are MiRS internal
qcRetrieval (T1)
Local Processing Directories
DDS
NEDT (rdr2tdr)
QC (fmsdr2edr)
NB: Time series require archiving biasCorr, NEDT and QC files
npp_qc.bash
Local Processing Directories
E-Mail if threshold exceeded
E-Mail if thresholds exceeded
T1
T2
fwd (T2)
biasGen (T2)
nwp (T2)
qcRadiometricBias (T2)
FMSDR`(fm)
NWP
biasMonitor (T2)
dataQualityMonitor (T1)
GFS (deGRIBed)
NWP
FWD
FMSDR (fmsdr2edr)+FWD
biasCorr
biasCorr
biasCorrTime series radiometric bias (*.png)
NEDT (rdr2tdr)+QC (fmsdr2edr)Time series qc, conv, nedt (*.png)
E-Mail if threshold exceeded
prepNwp (T2)GFS (GRIB)
GFS (deGRIBed)
103
Unit qcNedt: Process Flow
d
Local Processing Directories
qcNedt.pro
Input Files Specified in IDL Namelist
Input: NEDT File(MiRS Internal)
qcNedt unit script
Process Control FileExecution from PGM
Log File
Return value to PGM
Yes
Local Processing Directories
Process Status File
npp_qc.bash
E-Mail alert !Threshold exceeded?
104
Unit prepNWP: Process Flow
d
Local Processing Directories
prepNWP
Namelist
Input: • 1 NWP file (GFS) Grib format
Output: 2 NWP Files (MiRS Internal Binary)Valid at 2 consecutive forecast times (0, 6, 12, 18 UTC)
prepNWP unit script
Process Control FileExecution from PGM
Log File
Return value to PGM
Local Processing Directories
Process Status File
npp_qc.bash
105
Unit qcRetrieval: Process Flow
d
Local Processing Directories
qcRetrieval.pro
Input Files Specified in IDL Namelist
Input: QC File(MiRS Internal)
qcRetrieval unit script
Process Control FileExecution from PGM
Log File
Return value to PGM
Yes
Local Processing Directories
Process Status File
npp_qc.bash
Thresholds exceeded?
E-Mail alert !
106
Unit qcRadiometricBias: Process Flow
d
Local Processing Directories
checkBias.pro
Input Files Specified in IDL Namelist
Input: biasCorr File(MiRS Internal)
qcRadiometricBias unit script
Process Control FileExecution from PGM
Log File
Return value to PGM
Yes
Local Processing Directories
Process Status File
npp_qc.bash
E-Mail alert !Thresholds exceeded?
107
Unit nwp: Process Flow
d
Local Processing Directories
colocNWPwRad
Input Files Specified in f95 Namelist
Input: • FMSDR file• 2 NWP files (4 valid times: 0, 6,12,18 UTC)
Output: NWP File (MiRS Internal EDR)
nwp unit script
Process Control FileExecution from PGM
Log File
Return value to PGM
Local Processing Directories
Process Status File
npp_qc.bash
108
Unit fwd: Process Flow
d
Local Processing Directories
fwd
Input Files Specified in f95 Namelist
fwd unit script
Process Control FileExecution from PGM
Log File
Return value to PGM
Output:• RAD File (Internal)
Input :• Tuning Files (2)• Instr Config• CRTM Coef Files (3)• NEDT• NWP(MiRS Internal)
Static
Dynamic
Local Processing Directories
Process Status File
npp_qc.bash
109
Unit biasGen: Process Flow
d
Local Processing Directories
Calib_generic_rad.pro
Input Files Specified in IDL Namelist
biasGen unit script
Process Control FileExecution from PGM
Log File
Return value to PGM
Output:• biasCorr File (Internal)
Input :• NWP File• FMSDR• FWD(MiRS Internal)
Dynamic
Local Processing Directories
Process Status File
npp_qc.bash
110
Unit dataQualityMonitor: Process Flow
dLocal Processing Directories
MonitorQC_Mirs.pro
Inputs Specified in IDL Namelists
dataQualityMonitor unit script
Process Control FileExecution from PGM
Log File
Return value to PGM
Outputs:• Time Series Files
(2) (png)
Input :• Dir name containing QC
files• List of NEDT files
MonitorNEDT.pro
Local Processing Directories
Process Status File
npp_qc.bash
111
Unit biasMonitor: Process Flow
dLocal Processing Directories
biasMonitor.pro
Inputs Specified in IDL Namelists
biasMonitor unit script
Process Control FileExecution from PGM
Log File
Return value to PGM
Outputs:• Time Series File (png)
Input :• Dir name containing
biasCorr filesLocal Processing Directories
Process Status File
npp_qc.bash
112
MiRS QC Test Readiness: Summary/Risks
· MiRS QC DAP Prepared and Delivered (Dec 2010) to NDE» Contains all codes and files necessary to run Tier-1 and 2 QC» Requires f95 compiler and IDL v6 license» Currently testing
· Risks: Availability of GFS for Tier-2 testing
113
· Introduction· CDR Report· Software Architecture· Unit Test Readiness
» Test Readiness of Individual Units: MiRS/NDE
· RISKS/ACTIONS· Summary and Conclusions· Discussion
114
Section 5 –Risks and Actions
Presented by
K. Garrett
115
Open CDR Risks and Actions
· Risk #1: Unclear if resources available to run MiRS in high-resolution mode» Risk Mitigation: Already incorporated heritage (MSPPS) algorithms into MiRS
and can be optionally turned on/off to produce high resolution rain rate within MiRS on heritage sensors (at OSPO); Workload scenarios at NDE can determine resource adequacy for MiRS NPP/ATMS.
» Status: Open
· Risk #2: NPP launch date may be postponed beyond Oct 2011; Budget planning assumes minimal schedule slippage.» Risk Mitigation: Project lead to monitor schedule vis-à-vis MiRS project plan for
FY10 and FY11 and, if needed, request additional funding in consultation with MiRS Oversight Board via Annual Review for Satellite Product Development
» Status: Open
116
Open CDR Risks and Actions
· Risk #3: Footprint Matching (FM) codes not integrated» Risk Mitigation: (1) Use resampled data from IDPS; (2) Expected that MiRS will
process highest resolution data available and can do simple FOV averaging.» Status: Open
· Risk #4: Proxy data fidelity; Preclassifier and post-processing algorithms partially dependent on calibration with proxy (or real) data.» Risk Mitigation: MiRS performing well with current nominal values and
parameters. Calibration and tuning to be performed immediately post-launch.» Status: Open
117
Open CDR Risks and Actions
· Risk #5: SciTech Renewal» Risk Mitigation: MiRS IPT expected to remain in place during any transition.
Project will have advance notice and can reallocate resources as necessary to maintain continuity.
» Status: Open
118
MiRS NPP/ATMS Risk Summary(at CDR)
Impact *
5 1
4
3 2
2
1 3,4,5
1 2 3 4 5
Likelihood
Risk No.
(Rank)
Risk Category Risk Risk
LevelRisk Index
1External Risks
If NPP launch is delayed further funds may not be available for post-launch cal/val efforts.
Moderate 15.00
2Project Management Risks
Running MiRS in high-resolution mode requires additional computing resources which may slow data processing time. Low 9.00
3Project Management Risks
If footprint matching codes not available or SDRs not footprint-matched MiRS processing of non-FM SDRs would be sub-optimal. Very Low 3.00
4Project Management Risks
If real NPP/ATMS data has significantly different characteristics than proxy data, additional efforts will be needed post-launch to update MiRS processing algorithms. Very Low 3.00
5External Risks
If SciTech renewal is not awarded to incumbent there may be an impact on MiRS IPT continuity.
Very Low 3.00
Risk Level Likelihood
1 Very Low P < 10%
2 Low 10% ≤ P < 30%
3 Moderate 30% ≤ P < 70%
4 High 70% ≤ P < 90%
5 Very High P ≥ 90%
Impact Level* Schedule Impact (months)
Cost Impact($ x 10K)
1 Very Low < 1 ~ 0
2 Low 1 - 3 1 - 5
3 Moderate 3 - 6 5 - 10
4 High 6 - 12 10 – 50
5 Very High ≥ 12 > 50
*Risk Impact may be on schedule, cost, and/or science product quality and availability. Approximate impact levels for schedule/cost only are in table below. Not all risks will have schedule/cost impacts.
119
CDR Action Items
Action Item
Description Author Lead Org Status
1 MiRS Operations Manual in OSPO A. Sharma STAR Closed
2Mitigation Plan for Missing/Bad Ancillary Data
A.Sharma NDE/STAR Closed
3QC DAP for Trend Analysis and QC Monitoring
A.SharmaSTAR Closed
4NDE/OSPO Define Product Tailoring Responsibilities
G. Goodrum NDE/OSPO Open
5 Obtain Risk Analysis Matrix J. Silva STAR Closed
6Data Submission Agreement for MiRS NPP/ATMS in CLASS
T. Schott OSPO Closed
7Feasibility of Implementing Tier-3 Processing in MiRS QC DAP
T. Schott, L. Zhao
OSPO Open
120
· Risk 1: Simulated data and format may not represent actual NPP/ATMS data (also a CDR risk)
· Risk Assessment (probability): Moderate· Impact: Low· Mitigation: Testing will utilize both STAR generated
proxy data and GRAVITE generated proxy data (official) to ensure consistency. IDPS generated sample data will also be used in testing (current build, and Platinum72 when available), as those should be the true data formats (CDFCB).
· Status: Open
TRR Risk
121
· Risk 2: Coefficient files require real data for training and algorithm tuning (bias, covariances, first-guess, catalogs) (also a CDR risk)
· Risk Assessment (probability): Low· Impact: Low· Mitigation: Post-launch efforts will focus on cal/val;
software and correction coefficients based on proxy data already in place, which is adequate for testing purposes.
· Status: Open
TRR Risk
122
· Risk 3: Available resources in STAR and NDE environments are not consistent (IDL versions, compiler versions, etc.)
· Risk Assessment (probability): Medium· Impact: Low· Mitigation: Identify and address issues during the
code unit testing. Efforts have already been taken to ensure MiRS software is compatible with a variety of operating environments (Linux, AIX, multiple compilers).
· Status: Open
TRR Risk
123
· Risk 4: QC DAP lacks ability to monitor radiometric bias for upper-atmospheric ATMS temperature sounding channels using GFS at input. GFS model profiles only extend to 10 mb, below the sensitivity of ATMS channels 14 and 15 (57 GHz).
· Risk Assessment (probability): High· Impact: Low· Mitigation: Broader instrument issues will be captured
through NEDT, QC, or radiometric monitoring of other ATMS channels. STAR will also perform radiometric monitoring using other data sources which extend to the upper atmosphere where ATMS channels 14 and 15 are sensitive (e.g. ECMWF analysis).
· Status: Open
TRR Risk
124
Review Items Summary
· 5 Risk Items from the CDR were identified, 5 of which are still open
· 7 Action Items from the CDR were identified, 5 were closed, 2 remain open
· 4 TRR Risk Items have been identified , 4 remain open; NB: 2 are already contained in CDR risks
125
· Introduction· CDR Report· Software Architecture· Unit Test Readiness
» Test Readiness of Individual Units: MiRS/NDE
· Risks/Actions· SUMMARY AND CONCLUSIONS· Discussion
126
Section 6Summary and
Conclusions
Presented by
K. Garrett
127
Summary and Conclusions
· Following have been reviewed:» CDR Report» Software Architecture » Unit Test Plans» Risks and Actions
128
Next Steps
· Prepare TRR Report (including action items)· Reviews:
» Code Unit Test Review in late March, 2011 [9.1.4 in the Level 1 Reqs Doc]
» System Readiness Review in January, 2012 [9.1.5]
· Address action items identified in TRR
129
· Introduction· CDR Report· Software Architecture· Unit Test Readiness
» Test Readiness of Individual Units: MiRS/NDE
· Risks/Actions· Summary and Conclusions· DISCUSSION
130
Open Discussion
· The review is now open for discussion
131
Back-up Slides
132
MiRS Official Products
Observational Parameter Imagery Product Sounding Product PackageAtmospheric Temperature profile xAtmospheric Water Vapor profile xTotal Precipitable Water XLand Surface Temperature XSurface Emissivity Spectrum xSea-ice Concentration xSnow Cover Extent xSnow-Water Equivalent xIntegrated Cloud Liquid Water xIntegrated Ice Water Path xIntegrated Rain Water Path xRainfall Rate x
133
Operational Products
Product Proc Format User Coverage Output TypeIMG/SND NDE netCDF4 CLASS/
OSPOGranule Direct
RR/TPW/TB NDE netCDF4 OSPO Orbital Tailored
RR/TPW NDE netCDF4 OSPO AOI* Tailored
TPW/RR/TB Composite
OSPO McIDAS SAB Orbital Tailored
TPW/RR Composite
OSPO AWIPS NWS AOI* Tailored
Blended TPW/RR
OSPO AWIPS/McIDAS
NWS Global/Orbital Tailored
IMG/SND ? HDF-EOS CLASS Granule/Orbital Tailored
*AOIs : HI, AK, Puerto Rico, Super National, Tropical
NDE or OSPO
134
MiRS: Anticipated ATMS Hi-Res Processing
Platform/Sensor Machine Single CPU (core)
Number of CPUs (cores)
Time: 1 orbit(multi CPU)
Number of Profs:1 orbit
Time: 1 granule(single CPU)
Time: 1 granule (multi CPU)
N18/AMSU-MHS orbit272L 2.67 GHz 24 11.0 min 232470 N/A N/A
NPP/ATMS orbit272L 2.67 GHz 24 10.4 min* 224640* 1.28 min* 3.23 sec*
*Based on: • 1 granule = 1152 profiles• 1 orbit = 195 granules• MiRS Outputs: 22 MB• 0.0028 sec/profile
Benchmarks based on current processing capacity
Note: MiRS can distribute processing over multiple CPUs on same node (chopping or not)
Actual Estimated
135
MiRS: Current and Planned Processing Environment in
STAR
Machine Single CPU (core)
Number of CPUs (cores)
Memory (GB)
Disk Storage (GB)
Jobs Running
orbit227L 3.16 GHz 8 16 800 • Daily Global Low-Res: N18,N19,Metop,F16,F18,NPP (30 min/satellite)• Daily Regional High-Res*: N18,Metop,F16,AMSRE
orbit232L 2.83 GHz 4 8 1200 • Development
orbit227L 3.16 GHz 8 16 800 • Daily Global Low-Res: N18,N19,Metop• Daily Regional High-Res: AMSRE
orbit232L 2.83 GHz 4 8 1200 • Development
orbit273L 2.8 GHz 16 32 3000 • Daily Global Low-Res: F16,F18,NPP
orbit272L 2.67 GHz 24 128 4500 • Daily Global High-Res: N18,N19,Metop,F16,F18,NPP
orbit267L 3.46 GHz 16 40 3000 • GPM, development
3 additional servers purchased to enhance MiRS processing capacity within STAR
Current Planned
* Current high-res 15 km daily processing for 4 ROIs/Satellites» N18: Gulf of Mexico» MetopA: U.S. West Coast» F16: China Southeast Sea» AMSR-E: CONUS