earned value management at nasa: an integrated, lightweight
TRANSCRIPT
1
Earned Value Management at NASA: An Integrated, Lightweight Solution
Peter Putz1, David A. Maluf2, David G. Bell1, Mohana M. Gurram1, Jennifer Hsu3, Hemil N. Patel3, Keith J. Swanson2
1Universities Space Research Association
NASA Ames Research Center Moffett Field, CA 94035
650-604-2137 [email protected] [email protected]
2NASA Ames Research Center
Moffett Field, CA 94035
[email protected] [email protected]
3QSS Group Inc
NASA Ames Research Center Moffett Field, CA 94035
Abstract12—This paper describes a fresh approach to Earned Value Management (EVM) at the U.S. National Aeronautics and Space Administration (NASA). The goal of this approach is to provide a lightweight tool that allows project managers to apply earned value performance measurements with minimal effort in terms of data entry, and without the need to learn the highly specialized jargon that mystifies many EVM solutions. The presented technical and managerial solution addresses the practical challenges of applying EVM in the messy realm of project management. An empirical case study involving five projects at the NASA Ames Research Center illustrates the challenges of creating a consistent performance measurement baseline under the constraints of schedule, budget, and labor requirements, and of matching actual costs with budgeted costs on the level of granularity needed. The case study also highlights the benefits of using the implemented EVM solution in terms of data quality and time savings. The paper concludes with general recommendations for the design and application of EVM tools with the focus on ease of use.
TABLE OF CONTENTS
1. INTRODUCTION......................................................1 2. CREATING A PERFORMANCE BASELINE...............2 3. REPORTING MONTHLY PROGRESS .......................4 4. CUSTOMIZING EVM REPORT FORMATS..............5 5. LESSONS LEARNED ................................................6 6. CONCLUSIONS .......................................................7 REFERENCES .............................................................7 BIOGRAPHY ...............................................................8
1. INTRODUCTION
Earned Value Management (EVM) is a management technique that allows project managers to determine the true cost and schedule variance between plan and material work accomplished of a given project at any time. More
1 1-4244-0525-4/07/$20.00 ©2007 IEEE. 2 IEEEAC paper #1282, Version 2, Updated December 11, 2006
importantly it allows management to predict the total costs at completion and the date of completion. A key benefit of EVM is that it serves as an early-warning system against cost overruns and schedule delays. This is the main reason why NASA and other government agencies—most prominently the Department of Defense—require projects to produce EVM reports on a regular basis (NPR 7120.5C [1]).
Project managers reacted with skepticism when, in 2004, NASA’s newly founded Exploration Systems Mission Directorate (ESMD) mandated that even relatively small projects with a budget from 1 to 10 million dollars had to account for their project performance in the form of a highly specific monthly EVM report. They did not understand the details of the EVM method and, more severely, they were not provided with the necessary tools to integrate their schedule and financial data. After attending multiple-day-long EVM training sessions most of them ended with either creating their own set of Excel spreadsheets to manually calculate the required metrics or struggling with oversized and rigid software packages. These helped in getting the core EVM metrics, but project managers were still not able to automatically produce the customized report formats required by ESMD.
For some reason the practical difficulties of a highly specialized language paired with a lack of adequate tool seem to keep haunting EVM. Fleming and Koppelman [2] complain: “What started out originally as a simple concept on the factory floor has evolved into a sort of vocational cultist confederation in which one must be specifically trained to use a foreign language in order to be a member of the team” (p. 73). Fleming and Koppelman continue on a positive note: “There is nothing difficult or complicated about the earned value concept. It does not require highly trained people to grasp the fundamentals. In fact, many people use the concept in their daily routines and are not even aware they are employing earned value” (p. 73).
This paper describes a novel solution to creating EVM reports in a way that is as easy to understand as the earned
2
value concept itself. After ESMD issued the mandatory EVM requirement, some project managers found that the existing NASA Program Management Tool (PMT) [3], which they were using for project planning and reporting, captured already most of the financial and schedule data necessary for EVM reporting. What was missing was the report itself and some minor changes in the data input templates. Given this assessment, PMT was used to implement a full set of EVM reporting capabilities.
PMT is a NASA-developed program management tool suite that supports program and project managers in all their essential activities, like creating and monitoring annual task plans, analyzing variances between budget plans and actual costs, creating periodic status reports on technical, schedule, budget, and management status, identifying program risks and tracking mitigation strategies, creating aggregated program dashboard views and other customized reports for single subprojects or the entire program.
The PMT software architecture is built around two distinguishing features:
(1) The main user interfaces are standard business documents like spreadsheets, presentation slides, and text documents. The use of standard business document templates not only enables automated exchange of information between machines, but also supports natural and often ad-hoc on-line and off-line workflows between humans gathering data or making decisions.
(2) The backend is a ‘schema-less’ XML3 database, which enables easy data integration and query-based document composition. At the same time it eliminates the need for database administration by automating the integration of information that have diverse schemas.
The following graphical representation illustrates the typical PMT document workflow:
Drag and Drop Email
Web Upload
Decompose to XML
User
Store in XML Database
Enter Context / Content Query
<context> <context>content</context> <É> É </É>
Query result in XML
Compose from XML
3 Extensible Markup Lanaguage – http://www.w3.org/XML/
Figure 1 – PMT Document Workflow
PMT is an ideal software platform for EVM reporting for the following reasons:
(1) Seamless integration of heterogeneous and distributed information: EVM reporting requires the integration of a variety of data which are partly user entered and partly retrieved from existing databases like the financial system, scheduling tools, risk management systems and so on. The XML database allows adding new data elements without the need of changing any predefined schema, in fact, without even touching the database at all.
(2) Automatic composition of analyses and reports: EVM reports are often multi-page documents covering not only core EVM metrics, but also graphical representations of the work breakdown structure, the project master schedule, risk and mitigation status and financial performance. PMT has the capability to automatically produce comprehensive slide decks or multi-page spreadsheet documents. The document composition is built upon an advanced XML query module.
(3) Easy communication of complex information among diverse subject matter experts and stakeholders: The gathering of financial, schedule, and various other project status data is usually an effort involving the collaboration of multiple subject matter experts. With PMT, data entry and report templates can be accessed, distributed, archived in a variety of ways. As standard business documents they can be down-loaded to a local desktop for off-line access, the can be distributed as email-attachments (e.g. to resource analysts who do not even need to aware that a particular template is a PMT input form), and they can be archived in third-party document repositories.
Given those features, PMT seemed to be an ideal software platform for the implementation of an Earned Value Management system. The following sections discuss in detail, how users can setup and use PMT for EVM reporting, including empirical results on the system use in practice.
2. CREATING A PERFORMANCE BASELINE
A consistent project baseline is an indispensable prerequisite for any kind of project performance assessment. For Earned Value Management a performance baseline has to include a work breakdown structure (WBS) for the entire project; a master schedule of work packages with budgeted costs for each work package; and a time-phased budget plan for each WBS element.
3
Figure 2 – Building a Performance Baseline
Creating a work breakdown structure
The work breakdown structure defines the scope of a given project. A WBS is a hierarchical structure of sub-projects, sub-tasks, or sub-products. Graphical representations of a work breakdown structure look very much like an organizational chart, even though its elements are not organizational units like groups, divisions, directorates etc., but sub-tasks.
In PMT the WBS is entered into a configuration spreadsheet in form of a flat list.
Figure 3 – Defining a WBS in PMT
Figure 3 shows the data elements for the WBS structure. Noteworthy are the columns “Level”, “PMT WBS #”, and “BW WBS #”. “Level” indicates the hierarchical position of a particular WBS element, “PMT WBS #” is a string of characters chosen by a project manager to systematically name the WBS elements, and “BW WBS #” links each
WBS element to NASA’s financial system. Fortunately the accounts in NASA’s financial system—or Business Warehouse (BW)—are structured as a work breakdown structure including all current space missions and programs. In most instances this makes it very easy to link a PMT WBS element to an account in the financial system. We will describe in section 4 below how this connection between PMT and the financial system enables an automatic calculation of actual costs for each WBS element.
The users upload the configuration spreadsheet to the PMT server and the system automatically produces three data entry spreadsheets per WBS element: (1) a Taskplan for entering schedule and budget information; (2) a Monthly Report for entering the progress accomplished on each work package; and (3) a Budget Report for analyzing cost plans versus actual costs and for entering subjective cost estimates.
Entering schedule information
Schedule information for work packages—which are called deliverables in PMT—is entered into the Taskplan spreadsheets. Breaking down the total project schedule into work packages per WBS element facilitates the distributed collaboration between sub-project managers who are accountable for their on-time completion.
Figure 4 – Scheduling and Costing Work Packages
Entering time-phased budget information
The Taskplan includes a spreadsheet called “Budget Section” for entering the time-phased budget plan (or phasing plan). This again is done on the level of each WBS element. The phasing plan section is custom designed to match NASA’s accounting system. A phasing plan is typically:
(1) broken down into full cost elements (civil service labor, civil service travel, procurement, etc.)
(2) entered for multiple fiscal years
(3) broken down into the various NASA Centers involved in the sub-project.
4
In the EVM jargon the phasing plan total is called “planned value” or “Budgeted Costs for Work Scheduled (BCWS)”.
Validating the performance baseline
In the context of Earned Value Management it is crucial that the schedule information and the time-phased budget information are integrated into a consistent performance plan. The following graphic shows the connection between schedule and budget data.
In the (messy) reality of project management it is not a trivial task to create a consistent performance baseline. Dependent on the particular project environment the concrete steps for this planning process may vary substantially. Some projects might start by planning in terms of work hours, others in terms of dollar-values; some projects might optimize their schedule around predefined delivery deadlines and assign the required workforce afterwards, others might optimize the schedule for a given level of workforce utilization and move the work packages around to accommodate that. Many other variants are feasible, too.
Figure 5 – Creating a Consistent Performance Baseline
Given the variety of project planning methods, we found it not helpful to provide the project managers with a one-size-fits-all planning tool within PMT. Rather, we left it to the individual project managers to use tools of their own choice. However, once the final schedule and phasing plan information is entered, PMT provides a baseline validation tool.
The baseline validation tool is a spreadsheet that calculates the Schedule Performance Index (SPI) for all WBS elements of a given work breakdown structure under the assumption of the following best-case-scenario:
(1) all work packages are finished on time;
(2) for all periods, the actual costs are equal to the planned costs.
Under these assumptions the SPI and the Cost Performance Index (CPI) are always 1.00 for consistent performance baselines4. However, if schedule and budget information do not match up, the SPI is either bigger or smaller then 1.00. In such a case a project manager needs to go back and revise the plan. Section 5 below will describe our lessons learned this regard.
3. REPORTING MONTHLY PROGRESS
Earned Value Management is nothing more than comparing for a given point in time the planned value (BCWS) with two other numbers: the earned value (BCWP) and the actual costs (ACWP)5.
With PMT the actual costs are loaded automatically from NASA’s financial system (see above) and do not require any manual data input into PMT.
Figure 6 – Entering Percentage Complete
The Earned Value (BCWP) is a dollar value representing the material progress made on a particular work package. 4 In fact, under assumption (2)—budgeted value equals actual cost—CPI and SPI are identical. 5 BCWS stands for Budgeted Cost for Work Scheduled, BCWP for Budgeted Cost for Work Performed, and ACWP for Actual Cost for Work Performed.
5
For each given WBS element it is the sum of the budgeted costs per work package multiplied by the percentage complete. As pointed out above, in PMT the budgeted cost per work package is entered into the schedule section of the Taskplan. Hence the only missing data point is the percentage complete per work package. This is done in a spreadsheet entitled “Monthly Report”.
Figure 6 shows that the individual work packages are graphically depicted as timelines. The percentages complete are updated once a month by the project manager and entered as numbers directly under the timeline for a given work package. In the same graphical display start and end dates for milestones can be moved if necessary.
Note that after the performance baseline is created, at a minimum all a project manager needs to do for EVM
reporting is to access the Monthly Report and update the percentage complete!
In the case that the final EVM report contains additional information besides the core EVM metrics it can also be entered into the “Monthly Report” spreadsheet. E.g., in the EVM report for NASA’s Exploration Systems Mission Directorate required entering project risks.
4. CUSTOMIZING EVM REPORT FORMATS
Figure 7 shows how PMT integrates the performance baseline data and the monthly progress data in order to calculate the core EVM metrics.
6
Figure 7 – Calculation of Core EVM metrics
The system is built such that an EVM report can be run against any individual WBS element. By default PMT calculates the core EVM metrics for the selected WBS element (e.g. the level 1 project) and for the next level down. This level is often called Control Account Plan (CAP), which is simply the management control point where the performance measurement has to take place. PMT dynamically rolls up all lower level schedule, budget and cost information to the CAPs.
Many organizations follow widely accepted standard formats like Contract Performance Report, which was originally mandated for Department of Defense (DoD) acquisition contracts. However, in some other cases, executive management wants to see the EVM metrics in the broader project context including financial reports, risk reports and other status updates. This is a challenge for project managers since it requires time consuming and error-prone manual data manipulation. For this reason PMT was designed to support the production of highly customized EVM reports. The XML database and the XML query protocol used in PMT enable the composition of customized reports with a minimum of software coding efforts.
5. LESSONS LEARNED
The EVM reporting capabilities were implemented in 2004 when NASA’s newly-founded Exploration Systems Mission Directorate mandated earned value project management even for relatively small sized research projects. The software development was done in close cooperation with five ESMD project managers who provided valuable feedback and requirements for a system that is easy to use in a NASA environment.
Inconsistent Baselines
It turned out that challenge number one was the formulation of a consistent performance baseline. In a small case study we analyzed the originally submitted baselines that were produced with a variety of ad-hoc tools—mostly self-created spreadsheets.
We applied a best-case scenario—assuming that a) all work packages were delivered exactly on schedule and that b) the actual costs equal the budgeted costs in any point in time—and calculated the predicted cost and schedule indexes. For a consistent baseline SPI and CPI necessarily need to be 1.00 for all months. However, if the baseline is inconsistent, that is, the schedule for earning value and the time-phased budget do not match—SPI and CPI would be either higher or lower then 1.00.
In accordance with the ESMD EVM format we coded the SPI/CPI numbers with green (0.9 ! index ! 1.1), yellow (1.1 ! index ! 1.2 or 0.8 ! index ! 0.9), and red (index > 1.2 or index < 0.8) color, indicating the amount of the variance from the performance baseline. The following table shows the results:
Figure 8 – Case Study Results
The results indicate that the performance baselines were highly inconsistent. Even under the best case scenario of the project being on budget and on schedule at all times:
(1) not a single project would achieve green each month, meaning not a single project had a consistent baseline
(2) 31.7% of all reported months were in the red or yellow
(3) one particular performance baseline had even 58% of reported months in the red or yellow at project level.
These results proved the urgent need to implement a performance validation tool that project managers could use before they submitted the final plan.
Need for Subjective Cost Estimates
When we showed our EVM pilot implementation to the NASA project managers, their immediate question was: Where do you get the actual costs from? When they learned that we imported the costs directly from NASA’s financial system, they responded: “Then we can’t use it”.
As it turned out the project managers had good reasons for their objection. In fact, under certain circumstances the financial data out of the accounting system are problematic for performance measurement. For example when contractors complete a job it takes up to a few weeks until the contractor receipt is received, reviewed, paid and entered into the accounting system. Other examples are arbitrarily timed assessments of organizational overhead costs or costs charged to the wrong WBS element within a project.
In those cases the accounting system is not “wrong” but the time lag in the accounting data is too great for someone who wants to do project performance measurement. Therefore,
7
project managers need a way to “adjust” the accounting data. With PMT we took the approach to allow project managers to enter 'estimated costs' replacing the system of record data. However, for reasons of data transparency it is essential that the EVM system keeps the numbers of the accounting system and the ‘subjective’ cost estimates logically and visually strictly separated.
Figure 9 – Entering Estimated Costs
The figure above shows the PMT Budget Report. The actual costs out of NASA’s accounting system are depicted as thick vertical bars. The subjective cost estimates are added as thin vertical lines on top of the bars.
6. CONCLUSIONS
The pilot implementation of the described EVM solution at the NASA Ames Research Center was highly successful in terms of time savings and in terms of reliability and consistency of reports across very different projects.
Before project managers started to use the PMT EVM module it took them between one and two weeks to gather the relevant schedule and cost data from all the sub-projects, to manually calculate the EVM metrics, to create the required charts, and to integrate everything into the final slide deck. With PMT this time span came down to one to two days and instead of data gathering and manual data manipulation the project managers could focus on variance analysis and on creating actions where necessary.
We feel that our experiences with the presented approach are encouraging. The positive results are based on the following factors, which can be seen as general recommendations for EVM tools:
(1) Use standard business documents as the main user interface. Project managers live and breathe spreadsheets and slide sets. A tool that uses standard business documents significantly increases user acceptance, reduces the need for training, and allows for complex off-line workflows.
(2) Provide the capability to automatically produce custom report formats. Senior management wants to see high-level project status information in a customized, easy to read format. Since those standards change frequently an EVM tool should provide the flexibility to use any output format required.
(3) Minimize the need for manual data entry. Project managers do not like to enter information twice. A seamless integration with existent financial and scheduling systems is highly recommended.
(4) Provide a baseline validation tool. There are two ways to support project managers in creating a performance measurement baseline. One is to provide a set of predefined input templates that produce automatically a consistent baseline. The other way is to have a baseline validation tool. In a highly heterogeneous project environment it might be more successful to leave it to the project managers to choose their own tools and processes for creating a performance baseline. However, to ensure that the performance baseline is self-consistent, a baseline validation tool has proved highly valuable.
(5) Provide a means for entering “cost estimates”. As shown above corporate accounting systems do not provide the right costs information for performance reporting in all cases. It is frustrating for project managers to explain performance variances that are merely undesired accounting artifacts. One approach to avoid that is to “allow” project managers to enter “cost estimates” based on their accurate knowledge of project activities.
REFERENCES
[1] NASA Program and Project Management Processes and Requirements NPR 7120.5C, Web site: http://www.hq.nasa.gov/office/codeq/doctree/71205.htm
[2] W. Quentin Fleming and Joel M. Koppelman, “The Essence and Evolution of Earned Value,” Transactions of AACE International, 73–79, 1994.
[3] Bell, David G., et al. “The NASA Program Management Tool: A New Vision in Business Intelligence,” 2006 IEEE Aerospace Conference Proceedings, March 4-11, 2006.
8
BIOGRAPHY
Peter Putz is a management scientist with the Research Institute for Advanced Computer Science (RIACS) at the NASA Ames Research Center. Previously he was a member of research staff with the Xerox Palo Alto Research Center (which is now PARC Inc.) where he was doing research on learning and knowledge sharing strategies together with an interdisciplinary group of social scientists in the Knowledge, Interaction and Practice Area. Peter received his Ph.D. for the Johannes Kepler University Linz, Austria. There he was an assistant professor with the Department of Business Information Systems and the Department of Management for more then ten years.
David A. Maluf leads the NASA Advanced Exploration Network laboratory (AEN), a laboratory consisting of 20+ staff with an average of 12 projects/year. He has over 70 technical publications in journals and conference proceedings, over hundreds of presentations at international conferences and is an inventor of numerous patents. He has taught courses on system engineering and databases, and has written two books. He received his PhD from McGill University and conducted post-doctoral research in information integration at Stanford University.
David G. Bell is Director and Senior Scientist at the Research Institute for Advanced Computer Science, located at the NASA Ames Research Center. Prior to working at NASA, David worked for ten years at the Xerox Palo Alto Research Center, and previously held an appointment at MIT where he led a research program in the Center for Innovation in Product Development. David is co-inventor of multiple patent and patent-pending information system technologies, including XML query technologies related to NETMARK and the NASA Program Management Tool, extensible blog technology called Sparrow Web, and distributed knowledge management software called Eureka. David received his Ph.D. from Cornell University with a dissertation on the dynamics of product development processes.
Mohana M. Gurram is a computer scientist for Universities Space Research Association at NASA Ames Research Center. He has worked with Mars Exploration Rover mission and has been presented with awards like TIGR, Honors Award at NASA. His area of interest is Data Visualization especially context-sensitive data. He earned his Masters in Computer & Information Science.
Jennifer Hsu is a Systems Analyst with QSS Group Inc.. She is a member of the Program Management Tool Development Team at Advanced Exploration Networks Laboratory, NASA Ames Research Center. She also worked
on Mars Exploration Rover Mission’s Rover Activity Planner, MAPGEN project. She received her Ph.D. degree in Biochemistry from Cornell University.
Hemil N. Patel is a computer scientist with the QSS Group at the NASA Ames Research Center. Previously he worked on the Aviation Data Integration System (ADIS) where he received numerous awards including NASA's Space Act Board Award. Hemil has a Master's Degree in Electrical Engineering from University of North Carolina at Charlotte.
Keith Swanson is a computer scientist in the Advanced Exploration Networks laboratory of the Intelligent Systems Division at NASA Ames Research Center. He has over 20 years of technology management and development experience in the areas of system health management, planning and scheduling, and knowledge-based systems. Keith has a Master's degree in Computer Science from Stanford University and a Master's Degree in Engineering from UC Berkeley.