observatory control system project management · observatory control system project management...

Project Documentation Document PMCS-0020

Revision A

Advanced Technology Solar Telescope 950 N. Cherry Avenue Tucson, AZ 85719 Phone 520-318-8108 [email protected] http://dkist.nso.edu Fax 520-318-8500

Observatory Control System Project Management

Bret Goodrich Software

September 1, 2015


PMCS-0020, Revision A Page ii

REVISION SUMMARY: 1. Date: September 14, 2011

Revision: DRAFT1 Changes: Created

2. Date: September, 30, 2011 Revision: DRAFT2 Changes: Updates due to internal review

3. Date: Sep 1, 2015 Revision: Rev A Changes: Updated for Baseline.


PMCS-0020, Revision A Page iii

Table of Contents

1. INTRODUCTION ................................................................................................... 1 2. CONSTRUCTION ................................................................................................. 2 2.1 FABRICATION ......................................................................................................... 2 2.2 PURCHASE ............................................................................................................. 2 2.3 INTEGRATION, TESTING, AND COMMISSIONING .......................................................... 2 3. QUALITY CONTROL ............................................................................................ 4 3.1 VERIFICATION ......................................................................................................... 4 3.1.1 Requirements Verification .............................................................................................. 4 3.1.2 Design Verification .......................................................................................................... 4 3.1.3 As-Built Verification ........................................................................................................ 5 3.1.4 Documentation Verification ........................................................................................... 5 3.2 QUALITY ASSURANCE TASKS .................................................................................. 5 3.2.1 Unit Testing ..................................................................................................................... 5 3.2.2 Integration Testing .......................................................................................................... 6 3.2.3 User Acceptance (Verification) Testing ........................................................................ 6 3.3 VERIFICATION TEST PLAN ....................................................................................... 7 3.3.1 Unit Tests ......................................................................................................................... 7 3.3.2 Integration Tests ............................................................................................................. 8 4. RISK ...................................................................................................................... 9 4.1 RISK REGISTER ...................................................................................................... 9 4.2 RISK MITIGATION .................................................................................................... 9 4.2.1 OCS-01: New user interface specifications ................................................................ 10 4.2.2 OCS-02: Engineering archive performance ...................... Error! Bookmark not defined. 4.2.3 OCS-03: Operational mode script performance—OCS cannot run at the rates required by some scripts ............................................................... Error! Bookmark not defined. 4.2.4 OCS-04: Support for Instrument tab developers– developers cannot build their parameter interfaces as required .................................................. Error! Bookmark not defined. 5. COST ESTIMATES ............................................................................................. 11 5.1 COST ESTIMATE ................................................................................................... 11 5.1.1 Construction Labor ....................................................................................................... 11 5.1.2 Integration, Testing, and Commissioning Labor ........................................................ 11 5.1.3 Construction Non-labor ................................................................................................ 12 5.1.4 Contingency .................................................................................................................. 12 6. SCHEDULE ......................................................................................................... 13 7. CONFIGURATION ITEM DATA LIST ................................................................. 15


PMCS-0020, Revision A Page 1 of 15

1. INTRODUCTION

This document discusses the project management of the DKIST Observatory Control System (OCS). It is concerned with construction and development plans, quality control and assurance, risks, costs, and schedule. Each of these areas is covered in sections of this document.



2. CONSTRUCTION

Construction phase planning involves the fabrication, purchase, and integration of the OCS. Planning for each of these areas is described here.

2.1 FABRICATION

Fabrication of the OCS will take place at NSO/Tucson. Fabrication involves the development of the OCS software and the integration of the OCS with the DKIST end-to-end test bed. At the completion of fabrication, the OCS will be fully functional and capable of running science programs on the end-to-end test bed.

The end-to-end test bed is the major test facility for integration testing of DKIST software. Each component of the test bed will be delivered by the relevant developer as a stand-in for the actual hardware, e.g., the telescope mount assembly vendor will deliver the Mount Control System (MCS) simulator, while the enclosure vendor delivers the Enclosure Control System (ECS). As the various control systems are developed, their simulators also mature, from simple interface simulators that only verify the accuracy of properties sent to it, to fully functional simulators that implement the final software components and only simulate the hardware mechanisms.

The OCS will maintain the end-to-end test bed as new systems are added. Regular testing of the test bed occurs with the Quality Assurance System (QAS) nightly regression tests.

2.2 PURCHASE

The OCS will purchase five computers for operations, including one desktop and two large monitors for the control room and four rack mount servers to run the OCS experiment database and facility services. The OCS also will purchase the five servers that operate the Common Services Framework environment (event server, databases, and spare). Additionally, NSO Operations will provide one machine at the summit to replicate the NSO experiment database—this is the major interface between the OCS and Operations. All systems are commodity items, requiring various configurations of RAM, storage, and processing power that are met by currently available systems. The purchase of the operations systems will be delayed until needed at integration, testing, and commissioning in FY2016. Spares will also be purchased for all major computing components.

2.3 INTEGRATION, TESTING, AND COMMISSIONING

OCS site integration begins during the second quarter of 2017. At this time, the Telescope Control System (TCS) should be installed. The first operations of the OCS will use the Instrument Control System simulators to test and commission the OCS. Later integration testing will be performed as each instrument system and telescope system is added to the observatory. Final user testing will likely not occur until shortly before handover to operations.

The DKIST site computer room contains 16 racks, each 48U in height. The racks are fully cooled through the site HVAC system and hot/cold aisles. Each rack is connected to the observatory uninterruptible power supply (UPS) and facility network switch (10 Gbit backbone). The OCS and CSF will use two of these racks.

The DKIST control room will contain three multi-headed computers workstations for operations. These are reserved for the Observer, Instrument Scientist, and Investigator as required by their roles in operations, per the Operational Concepts Definition (SPEC-0036).



The OCS plays a crucial role in the commissioning of the telescope. It is from this system that test scripts can be run to validate both operations and science requirements. Many test plans for other systems, especially instruments, expect to use the OCS for assembly and integration tests.



3. QUALITY CONTROL

Quality assurance and quality control are essential elements in the DKIST construction plan. Many aspects of the QA/QC plan are based upon past history of telescope development, along with modern systems engineering process control. The following definitions apply to the DKIST QA/QC.

Quality control, also known as verification, is a process used to evaluate whether or not a product, service, or system complies with regulations, specs, or conditions imposed at the start of development phase. Another way to think of this process is “Are you building it right?”

Quality assurance, also known as validation, is a process used to establish evidence that provides a high level of confidence that the product, service, or system accomplishes its intended requirements. Another way to think of this process is “Are you building the right thing?”

Change Control Board (CCB) is the name of the DKIST group that meets regularly to approve or reject requested changes in the DKIST specifications and interfaces.

Specification and Interface Control Documents define the specifications for each Work Breakdown Structure element and the public interfaces between each element. Both sets of documents are held under change control and changes must be approved by the CCB.

Design Document is the living document that defines the baseline design of the WBS element. This document is updated through the conceptual, preliminary, and final design phases to reflect the current design approach. During construction this document is updated to reflect the changes in scope or interfaces approved by the CCB. At the completion of construction the document should be a completed theory of operation manual for IT&C and operations.

3.1 VERIFICATION

The following tasks will be performed as part of the QA process. The work package manager and the DKIST QC/QA personnel will perform auditing of these tasks.

3.1.1 Requirements Verification

All new and changed requirements of the system will be verified to ensure the following:

They are directly related to an approved item from the CCB; They are consistent, feasible, and testable; They have been appropriately allocated to the correct mechanical, hardware, software, and

operational processes; and Those that are related to safety, security, and criticality have been verified by the rigorous

processes governing those areas.

The requirements verification process will be conducted by formal review. It will require participation and sign-off by the following team members:

Work package primary investigator; Work package manager; and Work package engineer responsible for the change.

3.1.2 Design Verification

All new and changed design elements will be verified to ensure the following:

Design is traceable to requirements;



Design provides details describing how requirements will be met; and Design implements safety, security, and other critical requirements correctly as shown by

suitably rigorous methods.

The design verification process will be conducted by formal review and requires participation of the following team members:

Work package manager; Work package engineer responsible for the change; and At least one engineering representative from a different DKIST work package.

3.1.3 As-Built Verification

All new and changed components will be verified once construction is complete to ensure the following:

Applicable standards are being followed; Applicable best practices standards are being followed; DKIST software coding and commenting standards are met; and DKIST software best practices are met per SPEC-0005.

The as-built verification process will be conducted by an informal review process such as email. The review process must be performed and signed off by at least one engineer from a different DKIST work package.

3.1.4 Documentation Verification

In conjunction with new or changed component being released the following documentation must be provided/updated:

Detailed design documentation, specifications, ICDs, etc.; Performance benchmarks (for performance critical modules); Test documentation (unit, component, integration, and user acceptance); and Operations document.

The documentation verification process will be conducted by informal review, such as email. The review process must be performed and signed off by the following team members:

DKIST QA/QC representative; and DKIST release manager.

3.2 QUALITY ASSURANCE TASKS

The following tasks shall be performed as part of the quality assurance process. The results of these tasks will be documented and reviewed as part of the “Document Verification” task of the QC process.

3.2.1 Unit Testing

Unit testing involves the testing of individual units of work to ensure they are fit for use. A new or changed component shall be unit tested. This testing shall be performed before proceeding to component level testing. The tasks that must be completed as part of unit testing are as follows:

Prepare new and/or changed unit tests and related documentation; Ensure traceability of new and/or changed tests to requirements;



Execute new, changed, and existing unit tests upon build of component; and Document unit test results.

Unit testing will be performed by the work package engineer responsible for the change or his/her designee.

3.2.2 Integration Testing

Integration testing involves testing an intended release to ensure it integrates correctly with all other DKIST systems for which it interfaces. Integration testing shall be performed in a qualified DKIST test environment that uses mechanical, hardware, and software systems equivalent to the production systems. Integration testing shall be performed before proceeding to user acceptance level testing. The tasks that must be completed as part of integration testing are as follows:

Prepare new and/or changed integration tests and related documentation; Ensure traceability of new and/or changed tests to requirements; Coordinate integration test schedule with test engineer of interfacing systems; Execute new, changed, and existing integration tests; and Document integration test results.

The work package engineer responsible for the change and the designated test engineer for each interfacing DKIST system will perform integration testing. Results shall be reviewed and signed off by the following team members before proceeding to user acceptance testing:

Work package manager; Work package manager(s) for all systems that interface with the released component; Work package engineer responsible for release; and Test engineer for all systems that interface with the released component.

3.2.2.1 Software Specific Tasks

The following tasks are specific to integration testing for software releases.

Any software defects (bugs) identified in testing will be logged in JIRA tracking system; All test cases impacted by the defect must be re-tested once the defect is resolved; Any un-resolved software defects must be approved by the review team before proceeding to

User Acceptance Testing; and Upon successful completion of integration testing the software source code will be tagged in

CVS to indicate it is part of a release. Test documentation will include reference to this release number.

3.2.3 User Acceptance (Verification) Testing

User acceptance testing involves performing tests for which the user will validate the output of the system to determine pass/fail status. User acceptance testing shall be performed in a production environment, or a qualified test environment that is approved by the user. User acceptance testing shall be performed before a release can be made operational for production use. The tasks that must be completed as part of user acceptance testing are as follows:

User to prepare new and/or changed integration tests and related documentation; Ensure traceability of new and/or changed tests to requirements; Coordinate user acceptance test schedule with production or test environments and systems; Execute new, changed, and existing user acceptance tests; and



Document test results.

User acceptance testing will be performed by the user, with the support of the work package engineer responsible for the release, and test engineers from other interfacing systems. Before proceeding to production, the release must be approved by the following team members:

Work package user (i.e., owner or primary investigator); Work package manager(s) for all systems that interface with the released component; Work package engineer responsible for release; Test engineer for all systems that interface with the released component; and DKIST release manager.

3.2.3.1 Software Specific Tasks

The following tasks are specific to user acceptance testing for software releases.

Only software source code from the CVS tag that matches the release number identified in the integration test documentation may be used for user acceptance testing;

Any software defects identified during testing will be logged in JIRA tracking system; All test cases impacted by the defect must be re-tested once the defect is resolved; Any un-resolved software defects must be approved by the review team before proceeding to

production release; and Test documentation should include reference to the CVS tag for this release.

3.3 VERIFICATION TEST PLAN

3.3.1 Unit Tests

Unit tests will be written with the Test Automation Framework (TAF), as described in PROC-0015. All unit tests will test against the OCS compliance matrix. The major DHS system tests are described below.

3.3.1.1 User Support System tests – OCS-supplied widgets

Test procedure: The JES standalone application is started using the OcsJesManager for widget access and all OCS-supplied widgets created and executed in JES' programmed mode. User verifies that widgets are behaving as specified.

Test result: No errors should be reported and behavior should be as expected.

3.3.1.2 User Support System tests – standalone user interfaces

Test procedure: These tests are performed with the cooperation of the OCS EMS and OCS LMS subsystems. Each of the USS-supplied standalone interfaces is operated in JES' live mode, connecting with the EMS and LMS facilities. User verifies that the standalone interfaces are behaving as specified.

Test result: No errors should be reported and behavior should be as expected.

3.3.1.3 Experiment Management System – experiment execution

Test procedure: This test is performed with the cooperation of the DKIST End-to-End test bed running with OCS-supplied principal system simulations. The experiment execution user interface is executed with the end-to-end test bed running. The operation modes created in the test for 3.3.1.3 are executed using the test bed. The user verifies that the correct behavior is exhibited by the end-to-end test bed and the execution status is properly reported by the experiment execution user interface.

Test result: No errors or unexpected behavior should be noted.



3.3.1.4 Lifecycle Management System – Application setup and distribution

Test procedure: The LMS application editor standalone user interface is executed and used to describe the layout of containers and components across several systems when deploying the DKIST end-to-end test bed. The application database is inspected to verify that the correct layout has been recorded.

Test result: No errors should be reported or found on inspection of the application database.

3.3.1.5 Lifecycle Management System – Lifecycle control and health monitoring

Test procedure: This test is performed with the cooperation of the DKIST End-to-End test bed running with OCS-supplied principal system simulations. The lifecycle control and health monitor user interfaces are run. The lifecycle control user interface is used to take the end-to-end test bed through its lifecycle states. The health monitor is checked to verify that it is correctly reporting system health status.

3.3.1.6 Lifecycle Management System – Archiving and Logging

Test procedure: The CSF-supplied tools for testing the engineering archive and log message database performance are run. The results are checked to verify that the performance specifications are met. The Archive and Log standalone user interfaces are then run to examine and extract information from the archive and database. The user verifies that those tools are functioning correctly.

3.3.2 Integration Tests

Integration tests are performed both with the Tucson end-to-end test bed and at the summit.

3.3.2.1 Other Principal Systems

Test procedure: This test is performed with the cooperation of the DKIST end-to-end test bed and the other principal systems. Each of the test beds supplied with each principal system are installed into the end-to-end test bed and the test bed is run. The verification tests specified in 3.3.1 are then repeated. This test is repeated as each principal system's test bed becomes available.

Test result: No errors resulting from incorrect OCS behavior should be produced.



4. RISK

The DKIST Risk Management Plan (SPEC-0037) defines the project approach to risk analysis and mitigation. The project maintains a risk register containing all identified risks at the beginning of the project and throughout the life of the project. The risks are categorized by the likelihood of occurring and the seriousness of impact on the project. Plans for mitigating each high-level risk and the subsequent results of the actions taken are also given.

The risk register exists to accomplish the following risk management goals:

• provide a tool for managing and reducing the risks identified before and during the project; • document risk mitigation strategies being pursued in response to the identified risks and their

grading in terms of likelihood and seriousness; • provide project management with a document from which status can be reported to oversight; • ensure the communication of risk management issues to key stakeholders; • provide a mechanism for seeking and acting on feedback to encourage the involvement of the key

stakeholders; and • identify the mitigation actions required for implementation of the risk management plan and

associated costs.

4.1 RISK REGISTER

Risk Item

Pro

bab

ilit

y

Non

-lab

or

Cos

t

Lab

or C

ost

HLSC-007 Expanded scope for NSO data handling integration 25% $340K $576K

HLSC-073 Scope refinements, specification documents in transition 80% $216K $173K

HLSC-002 Software Communications Infrastructure 5% $0K $72K

4.2 RISK MITIGATION

General risk mitigation strategies are applied to all risks in the OCS These strategies include:

Early Development: The OCS will finish construction in 2017. The IT&C period will be used to address issues arising from later instrument development or unplanned integration needs.

End-to-end test bed: The end-to-end test bed will be used to test the interface to the TCS and ICS. As subassembly element test beds are added (e.g., mount, enclosure, VBI) the end-to-end test bed will be able to test full operational support. In addition, DKIST scientists can begin testing experiments and observations using the test bed.

Interactions with Operations staff in Boulder. Regular communications has been established with the scientific and engineering staff in Boulder to demonstrate prototypes and receive feedback on the OCS displays and functionality.

The following risk mitigation strategies are envisioned for each of the major risks defined in the risk register.



4.2.1 HLSC-007: Expanded scope for NSO data handling integration

Based on HLS PDRs and subsequent discussions, there is some risk that additional implementation details/design changes will be required to integrate into the developing NSO model for both ingestion of scripts (Phase II Proposal preparation) and the export of data products.

The OCS-to-Operations interface has not been finalized. At this time the OCS is assuming its own model for the format of the experiments, based upon the Data Model (SPEC-0122), which also has not been finalized.

Mitigation: Early interaction with the operations group. The HLS developers have met with Operations during preliminary development to ascertain the currently known scope of the DHS to Operations interface. Current operations development plans in this area are still uncertain, but the HLS group will continue to work towards finalizing the interface.

Mitigation: Get the DRD and Data Models approved and align the OCS design with the results.

4.2.2 HLSC-073: Scope refinements, specification documents in transition

There may be additional work or delays in development due to formal specification changes.

The OCS Design Requirements Document (SPEC-0161) has not been finalized. Operator experience with user interface might suggest new interfaces late in development. Some additional interactions may be necessary and are not covered under existing specifications and design. Examples would include experiment and observation management tools and status screens.

Mitigation: Prioritize need for new interfaces and delay/omit implementation of less vital ones.

Mitigation: Provide users with end-to-end simulation tools early to test typical observations and build experiments.

4.2.3 HLSC-002: Software Communications Infrastructure

The required engineering data rates may be too high or large if there are many engineering parameters that need to be simultaneously saved. Current engineering archive hardware is selected on basis of best estimates from incomplete specifications.

Mitigation: Upgrade hardware to either increase performance or split data streams onto separate databases or even networks.

Mitigation: Hire database performance consultant. Database have been tuned only minimally; experience has shown that professional database consultants may be able to dramatically increase performance or make recommendations to operate more efficiently.



5. COST ESTIMATES

The DKIST budget for the OCS is $743K. This escalated cost number was developed by a bottom-up cost analysis performed in March 2009, and assumed hardware technology available at that time. Project contingency applies only to hardware and 5% contingency was set aside.

5.1 COST ESTIMATE

The August 2011 DKIST budget has created separate work packages for labor and materials of each OCS development area. New quotes were obtained all materials required by the OCS and CSF designs. The total cost of the OCS has not changed appreciably from 2009 and is still estimated at $743K. Additional labor costs were incurred with the end-to-end test bed, while hardware costs were lowered in all areas.

S-WASW3-200 Management (ARRA) $53,471.82S-WMSW3-202 OCS – Hardware $89,861.53S-WASW3-205 OCS - Scripting Support $17,484.68S-WASW3-215 OCS – Experiments $20,644.15S-WASW3-225 OCS – Facilities $45,375.07S-WMSW3-210 OCS Final - Scripting Support $82,349.99S-WMSW3-220 OCS Final – Experiments $64,426.05S-WMSW3-230 OCS Final – Facilities $146,697.68S-WMSW3-235 OCS - Integration ICS, TCS, and DHS $96,619.37S-WASW3-226 OCS – End-to-end Testing $74,742.49S-WMSW3-226 OCS – End-to-end Testing $51,540.84 Total $743,213.67

5.1.1 Construction Labor

Labor costs for the OCS were recalculated in August 2011 for the individual schedule items. OCS total labor has been increased to by 0.5 FTE years due to the design changes and overall increase of necessary work. The new design has integrated labor items for the end-to-end test bed.

Resource Work

FY

2011

FY

2012

FY

2013

FY

014

FY

2015

FY

2016

FY

2017

SW.1 End-to-end test bed 50% 33% Experiments 42% 58% 42% Facilities 92% Scripting 8% 100% 42% Integration 58% 58% 25%

Construction labor for the OCS ceases in March 2017, although integration activities will continue through the end of construction under the IT&C budget.

5.1.2 Integration, Testing, and Commissioning Labor

After construction, the OCS labor remains in the project to perform IT&C activities. The IT&C planning for these activities is currently under development. The activities include:



Construction of the summit control room, including operator’s station; Installation of the OCS and CSF in the computer room; Site acceptance testing of the OCS; Integration of the TCS, DHS, ICS, and; Integration of the remaining instruments.

IT&C labor begins April 2017 and completes in July 2019.

5.1.3 Construction Non-labor

Hardware costs for the OCS were recalculated in July 2011 for the latest design. OCS Hardware includes the equipment necessary to deploy CSF services on the summit. All computers are commodity items we expect to purchase shortly before integration. Therefore the configuration and prices are expected to change over the course of the next three to four years. In some areas, such as disk drives, new technology may supplant our baseline hardware. In all cases the current technology meets the OCS requirements.

Total cost for all materials is $90K. This is comprised of both the OCS and CSF equipment.

Item Model # Price Total OCS Equipment Observer Workstation Various 1 $3,000.00 $3,000.00 Inst Sci Workstation Various 1 $3,000.00 $3,000.00 Investigator Workstation

Various 1 $3,000.00 $3,000.00

OCS Container server Various 1 $4,100.00 $4,100.00 Experiment DB server Various 1 $8,500.00 $8,500.00 Log DB server Various 1 $3,000.00 $3,000.00 CSF Equipment Archive DB server Various $8,800.00 $8,800.00 Property DB server Various 1 $8,800.00 $8,800.00 Event server Various 1 $8,500.00 $8,500.00 Header DB server Various 1 $15,000.00 $15,000.00 Spares Various 1 $12,161.53 $12,161.53

$89,861.53

5.1.4 Contingency

DKIST contingency is held by the project management and is not a part of any specific work package. However, to determine the project contingency, the OCS non-labor activities were assessed of possible risk factors and assigned a contingency percentage based upon the criteria of the DKIST Contingency Management Plan. The calculated OCS contingency was 5% of non-labor activities. As mention, the OCS contingency was then rolled up into the project-wide contingency.



6. SCHEDULE

The OCS construction schedule began in October 2009 with the development of the preliminary versions of the experiment, facility, and scripting systems. Beginning October 2011, the development of the end-to-end test bed commenced. In April 2013, the initial development of the experiment, facility, and scripting services commenced. In November 2013, the OCS started integration of all services with the ICS and TCS systems, including the final assembly of the end-to-end test bed.

Continuing work of the OCS includes the final development of the OCS experiment execution through the end of 2015 and the release of the OCS Beta 2. During 2016 the OCS work will focus on the interface to NSO Operations for both experiment ingestion and status reporting. The OCS will also finalize the interface to the telescope with the completion of all major telescope subsystem packages, leading to the OCS Beta 3 release. During 2017 the OCS will provide necessary changes to the user interfaces from input during the IT&C effort.

At the end of OCS construction, the OCS programmer will move to an Integration, Commissioning, and Testing position to assemble the OCS and the major software systems at Haleakela.



7. CONFIGURATION ITEM DATA LIST

The Configuration Item Data List (CIDL) describes the status of all major deliverable software and documentation items. At present, the CIDL for the DHS contains the following:

Source Code Packages

OCS Final Release 1.0

Documentation

SPEC-0015, OCS Specification Document SPEC-0161, OCS Design Requirement Document TN-0064, OCS Critical Design Document CMX-0008, OCS Compliance Matrix MAN-0010, OCS Operations Manual PROC-0017, HLS Software Test Plan ICD-4.2/4.3, OCS to DHS Interface ICD-4.2/4.4, OCS to TCS Interface ICD-4.2/7.0, OCS to Operations Interface OCS Factory Assembly Test Plan