the measurable news

THE MEASURABLE NEWS

2013.03The Quarterly Magazine of the College of Performance Management

mycpm.org

INSIDETHISISSUE

11 15 27 37Ensuring Quality in Project Planning, Forecasting, and Execution

By Tom Polen

Earned Schedule – Ten Years After

By Walt Lipke

Agile & EVM Symposium

By Shobha Mahabir

CPM Global Initiatives Continue to Evolve

By Kym Henderson

20 USD/AUD | 15 EUR/GBP

THE MEASURABLE NEWS


mycpm.org

CONTENTS05 Update from the VP Communications

Mark Phillips

06 Project Performance ManagementBarry E. Schuler, Lloyd Carter

07 Laurence Asher “Larry” Stone - A Life WorthwhilePeter Schwarz

11 Ensuring Quality in Project Planning, Forecasting, and ExecutionTom Polen

14 CPM Board Elections 2013

15 Earned Schedule – Ten Years AfterWalt Lipke

24 CPM Awards: The Whys and WhereforesLauren Bone

27 Agile & EVM SymposiumShobha Mahabir

35 EVM World 2013 – The Earned Schedule TrackWalt Lipke

37 CPM Global Initiatives Continue to EvolveKym Henderson

40 Vendors/Services

2013 ISSUE 03

THE COLLEGE OF PERFORMANCE MANAGEMENT

2013 BOARD & STAFFPRESIDENTGary W. Troop 310-365-3876 • [email protected]

EXECUTIVE VICE PRESIDENTRay Stratton 714-318-2231• [email protected]

VICE PRESIDENT OF FINANCEBuddy Everage 202-507-4372 • [email protected]

VICE PRESIDENT OF ADMINISTRATIONLauren Bone UK+44(0)7766974063 • [email protected]

VICE PRESIDENT OF CONFERENCE & EVENTSBarry Schuler571-557-6574 • [email protected]

VICE PRESIDENT OF EDUCATION & CERTIFICATIONKim Hunter301-330-3101• [email protected]

VICE PRESIDENT OF RESEARCH & STANDARDSKym Henderson 61 414 428 537 • [email protected]

VICE PRESIDENT OF COMMUNICATIONSMark Phillips248-914-3774 • [email protected]

PAST PRESIDENTSusan Wood850-585-4830 [email protected]

EXECUTIVE ADMINISTRATORGaile Argiro703-370-7885 • [email protected]

ASSISTANT ADMINISTRATORStephanie Bautista703-370-4301 • [email protected]

THE MEASURABLE NEWS IS AN OFFICIAL PUBLICATION OF THE COLLEGE OF PERFORMANCE MANAGEMENT

EDITORIAL STAFFPublisher: College of Performance ManagementManaging Director: Gaile ArgiroStory Editor: Peter SchwarzDesign: id365.comCommunications VP: Mark Phillips

EDITORIAL COPY Editorial contributions, photos, and miscellaneous inquiries should be addressed and sent to the editor at the College of Performance Management (CPM) headquarters. Please follow the author guidelines posted on the CPM web site. Letters submitted to the editor will be considered for publication unless the writer requests otherwise. Letters are subject to editing for style, accuracy, space, and propriety. All letters must be signed, and initials will be used on request only if you include your name. CPM reserves the right to refuse publication of any letter for any reason. We welcome articles relevant to project management. The Measurable News does not pay for submissions. Articles published in The Measurable News remain the property of the authors.

ADVERTISINGAdvertising inquiries, submissions, and payments (check or money order made payable to the College of Performance Management) should be sent to CPM headquarters. Advertising rates are $1000 for inside front or back cover (full-page ad only), $800 for other full-page ads, $500 for half-page ads, and $300 for quarter-page ads. Issue sponsorships are available at $2500 per issue. Business card ads are available for $100 per issue (or free with full-page ad). Rates are good from January 1, 2013 – December 31, 2013. College of Performance Management reserves the right to refuse publication of any ad for any reason.

SUBSCRIPTIONSAll College of Performance Management publications are produced as a benefit for College of Performance Management members. All change of address or membership inquiries should be directed to: College of Performance Management 101 South Whiting St, Suite 320, Alexandria, VA 22304 Ph 703.370.7885 • Fx 703.370.1757 www.mycpm.org

All articles and letters represent the view of the authors and not necessarily those of College of Performance Management. Advertising content does not signify endorsement by College of Performance Management. Please notify College of Performance Management for single copy or reproduction requests. Appropriate charges will apply.

© 2013 by the College of Performance Management. All rights reserved.

THE MEASURABLE NEWS


mycpm.org

© 2012 SAFRAN NORTH AMERICA, LLC. PROTEUS ENVISION IS A REGISTERED TRADEMARK OF SAFRAN NORTH AMERICA LLC. ALL OTHER TRADEMARKS MENTIONED HEREIN ARE THE PROPERTY OF THEIR RESPECTIVE OWNERS

SAFRAN NORTH AMERICA, LLC Phone: 866-389-6750 l Fax: 866-381-7336 l www.safranna.com

THE PROJECT MANAGEMENT TOOLKIT B� S�� N�� A��

Full functionality from project planning through execution, resource management, costing, performancemanagement and analytics The Fastest Critical Path Method network analysis engine on the market with flexible calendaring Integration of Schedule and Cost from the Ground Up Saving you Time, Money and Labor by eliminatingreconciliation after the fact ANSI/EIA‐748B Earned Value Management compliance Rapid entry of data , designed as an open system Fastest multi‐user environment on the market –Scalable from the desktop to the Enterprise Flexible Setup and structuring of information— Robust Security Visualize cost, schedule, risk and financial data in new ways, transforming raw data into Business Intelligence Integration of disparate systems regardless of source allowing for full integration of all six DCMA Business Systems Customizable and instant Graphics, Analytics, and Reports, including the production of the new UN/CEFACT XMLfile for cost, schedule and financial information submission and inclusion into DoD and federal agency centralrepositories. Built for both On‐site Implementation and Software‐as‐a‐Service (SaaS) Environments Full service Business Solutions that tie Business Processes to Technology to realize improved productivity andensure credibility

From so�ware solu�ons through professional services, let Safran North America be your one‐stop shop for project excellence.

W� �� . S��’� P�� M�� T��, �� S�� P��, P�� E��.

THE MEASURABLE NEWS

The Quarterly Magazine of the College of Performance Management | mycpm.org

2013.03

05

UPDATE FROM THE VP COMMUNICATIONSMark Phillips, PMP

In this issue we are reaping the rewards of the research, discussions, networking and presentations from EVM World 2013.

• There is a summary of the Agile + EVM track that begins the process of aggregating the latest thoughts of how Agile and EVM can work together.

• There is a summary of the Earned Schedule track and a copy of the keynote speech covering the last 10 years of Earned Schedule.

• There is a short but timely article reporting on a discussion session that took place on the need for a Project Performance Management body of knowledge, and what that means.

• Tom Polen, from Acumen (which was recently acquired by Deltek), describes methods for determining and ensuring quality in project planning, forecasting and execution.

• Our VP of Research, Kym Henderson, provides an overview of recent developments in the evolution of CPM’s global initiatives.

EVM World is also a time for us to get together, as a community and reflect. There is a warm tribute to Larry Stone from his family and friends, put together by Peter Schwarz.

I encourage you to spend time with this issue, as a snippet of the numerous activities, advancements and discussions going on in our community. I would also remind you of the great webinars we offer every month. Over the past few months we’ve hosted An Introduction to EVM by Eleanor Haupt and a PARCA webinar on EVM as a Beacon in the Cloud of Program Execution. In the coming months scheduled presentations include A Comparison of EVMS Standards around the globe by Tony Barrett and Glen Alleman on Adding Agile to EVM.

Then, before you know it, we’ll have the opportunity to come together again, as a community, in person at the Integrated Program Management event in the Fall (ipmconference.org).

The Measurable News, our in-person events and our monthly webinars are a few of the ways in which your College of Performance Management is working to add value to our community. As always, if you have further ideas on how we can help or would like to discuss current initiatives, don’t hesitate to reach out to me at [email protected]. We hope you enjoy this issue of The Measurable News.

Mark Phillips, PMPVice President – Communications

© 2012 SAFRAN NORTH AMERICA, LLC. PROTEUS ENVISION IS A REGISTERED TRADEMARK OF SAFRAN NORTH AMERICA LLC. ALL OTHER TRADEMARKS MENTIONED HEREIN ARE THE PROPERTY OF THEIR RESPECTIVE OWNERS

SAFRAN NORTH AMERICA, LLC Phone: 866-389-6750 l Fax: 866-381-7336 l www.safranna.com

THE PROJECT MANAGEMENT TOOLKIT B� S�� N�� A��

Full functionality from project planning through execution, resource management, costing, performancemanagement and analytics The Fastest Critical Path Method network analysis engine on the market with flexible calendaring Integration of Schedule and Cost from the Ground Up Saving you Time, Money and Labor by eliminatingreconciliation after the fact ANSI/EIA‐748B Earned Value Management compliance Rapid entry of data , designed as an open system Fastest multi‐user environment on the market –Scalable from the desktop to the Enterprise Flexible Setup and structuring of information— Robust Security Visualize cost, schedule, risk and financial data in new ways, transforming raw data into Business Intelligence Integration of disparate systems regardless of source allowing for full integration of all six DCMA Business Systems Customizable and instant Graphics, Analytics, and Reports, including the production of the new UN/CEFACT XMLfile for cost, schedule and financial information submission and inclusion into DoD and federal agency centralrepositories. Built for both On‐site Implementation and Software‐as‐a‐Service (SaaS) Environments Full service Business Solutions that tie Business Processes to Technology to realize improved productivity andensure credibility

From so�ware solu�ons through professional services, let Safran North America be your one‐stop shop for project excellence.

W� �� . S��’� P�� M�� T��, �� S�� P��, P�� E��.

THE MEASURABLE NEWS 2013.03

06 The Quarterly Magazine of the College of Performance Management | mycpm.org

The CPM Membership recently approved new CPM Mission and Vision Statements:

MISSION STATEMENTShare, promote, and advance the best of planning, control, and performance management for projects of all sizes and complexity.

VISIONBe globally recognized as the world’s authority on earned value management and other project performance management techniques.

Though the Vision specifically addresses project performance management, this is not a discipline that currently has a body of knowledge associated with it. A search of the internet does not provide a specific definition, however, there are multiple (multiple 10’s) of conferences that address the subject.

At EVM World 2013 we, the authors, held a workshop to discuss project performance management and to see if there is a need to develop a body of knowledge around the subject. At the workshop the authors presented their initial thoughts on project performance management. Lloyd presented his concept that project performance management embrace various disciplines to include Leadership, Cultural Change and Deployment of Systems (e.g. EVM, Balanced Score Card...). Barry presented his concept that project performance management embrace the notion that any disciplined method that sets a baseline and uses that baseline to measure performance can be considered part of the body of knowledge. As part of the workshop the attendees added valuable input to the conversation. Though no definitive conclusions were reached during the workshop several important themes emerged:

Project Performance Management should address:

• Performance measures should focus on prediction of program performance.• Performance measures should be presented as a suite of measures for the program

manager to choose from based on program needs.• Performance measures from programs across the enterprise should be able to be

combined into an enterprise assessment of performance.• Performance measures should encompass pre-project, project-execution, and post-

project assessments

The workshop provided enough information to determine that more work is needed to investigate the need for a body of knowledge in project performance management. The authors will continue to pursue the subject over the next 12 months and look forward to hearing your ideas on the concept. Also, look for more sessions related to project performance management at next year’s EVM World in San Antonio, TX, May 21-23, 2014 – look for information on the CPM website, mycpm.org.

Barry E. [email protected]

Lloyd [email protected]

PROJECT PERFORMANCE MANAGEMENT

THE MEASURABLE NEWS


2013.03

07

OVERVIEW In March of this year, our community lost one of its greatest contributors to the development, growth, and perpetuation of the Earned Value Management (EVM) concept. But, we who knew him understand that we lost much more than that aspect of his life.

EARLY YEARSLarry, as he preferred to be addressed, was born in Boston, Massachusetts on November 22, 1914. He graduated from high school in 1931. He went on to graduate from the Massachusetts Institute of Technology (MIT) in 1935 with a BS, Business and Engineering Administration. (Editor’s note: He once told me that, during his matriculation at MIT, he lived in Newton, MA, and commuted to Cambridge each day by streetcar.) Following graduation from MIT, he was employed in manufacturing near Boston in the areas of product design and production supervision.

AN ARMY OFFICER CAREERLarry was commissioned in the Army Reserve in 1936, and went on active duty in 1939. His first posting, in 1939, was to the Springfield Armory in Springfield, MA. At the time, the Army had just accepted as standard equipment the new Garand rifle developed at Springfield Armory. It was in the process of expanding its manufacturing facilities to mass produce the rifle. For the next seven years, he played key roles in Garand production, which progressed from hand-fashioned prototypes to mass production of more than 1000 rifles per day. Not surprisingly, the Army retained him at the Armory throughout the duration of the Second World War until victory was achieved. (Editor’s note: During one conversation, Larry informed me that although he attempted to get transferred to Europe or the Pacific, he was denied because of the criticality of his assignment to the production of the Garand rifle – the primary infantry weapon in WWII.) During the war, he rose from 1ST Lieutenant to Lieutenant Colonel. He continued his distinguished military career from 1947 to 1965 as an Ordnance Officer, serving in Germany in the Army of Occupation, Hawaii (Editor’s note: Larry said it was payback for 8 years at the Armory), California, Fort Leavenworth, KS, Michigan, and thence to Washington, D.C. After serving for 26 years, he retired from the Army in 1965.

AN ARMY CIVIL SERVANT CAREERHaving spent the first phase of his life devoted to professional, ethical, and dedicated service on active duty, he commenced his career’s second phase in 1965 as an Army Civil Servant at the Army Materiel Command (AMC). In this capacity, he became one of the key pioneers in the development, implementation, refinement and use of the “Cost/Schedule Control Systems Criteria (C/SCSC)” and evolving it to the current “Earned Value Management System.” As an adjunct to his position, he was also the Army representative to the Performance Measurement Joint Executive Group, which was formed to coordinate C/SCSC amongst the armed services. These significant, innovative approaches are used in all major Department of Defense (DoD) acquisitions, and have been adopted for use by the civilian agencies of the U.S. Government. In addition, the world-wide use of these principles has spread at a rapid pace over the past 25 years.

As a result of his dedicated service, Larry received several AMC awards for exceptional and outstanding service including the Department of the Army Commander’s Award for Civilian Service, and the Distinguished Guest Lecturer Award from the Defense Systems Management College.

LAURENCE ASHER “LARRY” STONECOLONEL UNITED STATES ARMY1914 – 2013, A LIFE WORTHWHILEContributions and Recollections by his Family and Friends, Collated and Edited by Peter Schwarz

08 The Measurable News 2013.03 | mycpm.org

REMINISCES FROM LARRY’S ARMY MATERIEL COMMAND YEARSWayne F. Abba: “Larry was the consummate professional and gentleman. He was a role model and mentor when we worked together at HQ AMC in the early 70s and a trusted confidant and advisor after I joined the Office of the Secretary of Defense staff. I was proud to stand with Larry in 1996 when we both received the Driessnack Award from our professional colleagues.

“Larry was excellent at developing high-level policy. Among the objectives he and his fellow EVM pioneers established for EVM was ‘To bring to the attention of, and encourage, DoD contractors to accept and install management control systems and procedures which are most effective in meeting their requirements and controlling contract performance’ – an objective that remains as timely and relevant today as when he helped develop it decades ago.

“But Larry also was precise and attentive to detail. I think of him whenever I see someone make the common mistake of using ‘of’ rather than ‘for’ in Budgeted Cost for Work Scheduled and Budgeted Cost for Work Performed. His contributions to our profession are legion and his influence lives on in those of us who were privileged to work with him.”

Allan J. Bryant: “I first met Larry Stone in 1970 and went to work for him full-time at HQ AMC in 1980, continuing until he retired. From my first contact with Larry, I was impressed by his adherence to the purpose and principles of the C/SCSC while still being flexible to change in order to keep them consistent with needs of the programs at hand. After Larry retired, I was asked to take his place until I retired in 1998. During that time, I tried to maintain Larry’s principles.

I retired from C/SCSC-EVM consulting in 2010, spanning 53 years from my first accounting job in the private sector. During all my working years, both private and public, I never worked for anyone that was more professional, yet more kind, considerate, thoughtful and helpful than Larry Stone. He will always be a giant in my memory.”

Gary C. Humphreys: “Larry was my surrogate father when I was in the Army, far away from home. We could speak about anything on which I needed guidance. He was the most patient boss a young Lieutenant could have. The environment he created allowed for our Cost and Economic Information Office (CEIO) to excel in our obligation to validate the quality of Earned Value Management Systems (then Cost/Schedule Control Systems) while providing a way to fairly adjudicate any differences of opinion between Army personnel and contractors. Larry’s approach and example paved the way for many to a career inside the Government, in industry, or as consultants, dedicated to improving the efficacy of program management.

... Larry, thank you for being all that and more, boss...”

Harvey Mymit: “He would come in with his modest brown bag lunch consisting of a sandwich and a piece of fruit. He also was on one of my teams as the lead for the Organization criteria [of the C/SCSC]. He was the editor in chief for every correspondence going out of the office regarding interpretive guidance for Earned Value, because he was an excellent writer and communicator. He respected the office’s charter so much that he opted to retire rather than lose the position during a Reduction in Force.”

A DEFENSE CONSULTANT CAREERLarry “retired” again in 1992 after a productive 27 years in the Army Materiel Command, but his passion for contributing his talents was not dissipated. He entered the defense consultant field in 1992 to convey his expertise in what was still identified as C/SCSC. During this time, Larry was assigned to support the Strategic Defense Initiative Organization (SDIO), which morphed into the Ballistic Missile Defense Organization (BMDO), and thence into the Missile Defense Agency (MDA). However, his focus remained on the advancement of earned value as a management tool, which was also undergoing a period of transition.

In 1996, Larry also received a significant earned value award, The Hans “Whitey” Driessnack Award, sponsored by the Performance Management Association (PMA) (now College of Performance Management) for his contributions to the betterment of the cost/schedule management system. The award ceremony was held at the PMA conference in Boston. Soon, Defense Department decisions evoked a name change and re-oriented the philosophy in the approach to the management system. The term Cost/Schedule Control Systems Criteria (C/SCSC) was changed to Earned Value Management System (EVMS). An Assistant Secretary of Defense memorandum established the Integrated Baseline Review to provide both industry and Government a better understanding of the intricacies, costs, and schedule

09The Measurable News 2013.03 | mycpm.org

of the work to be accomplished under major contracts. As increasing responsibility for EVMS on major DoD contracts was transferred to industry, joint government-industry meetings were held to discuss and agree to the content of the new EVMS Guidelines, which were to replace the 35 C/SCS Criteria. Eventually, 32 EVMS Guidelines were adopted by mutual agreement. Larry was intimately involved in these actions, as a mentor, a contributor and as an implementer. In addition, he collaborated on many MDA internal EVM directives and handbooks to implement and coincide with existing DoD policy. (Editor’s note: My position as his supervisor and working partner on many of these endeavors enabled me to benefit from Larry’s mentoring and tutelage when he discerned that it was needed. His work ethic and knowledge were unsurpassed in my experience.)

In addition, a multi-service group was formed to compose a new DoD guide for Earned Value Management of which Larry was a principal member. This document became known as the

“Earned Value Management Implementation Guide (EVMIG).” Larry and several colleagues from government and industry received the David L. Packard Award for Excellence in Acquisition in 1998 for implementing “a shift in Earned Value Management ownership and responsibility from government to industry.”

As Larry’s career progressed into the 21st century, he never lost his enthusiasm for his work and his dedication to helping others in understanding the earned value requirements. Although, he moved to Napa, California permanently in 2006, he kept in touch with EVM developments by email. (Editor’s note: I sent him significant developments in EVM, to which he was always responded graciously.)

REMINISCES FROM LARRY’S DEFENSE CONSULTANT YEARS: Peter Schwarz: “Larry personified the word “expert”, a word that is often misused in current times. Whenever we discussed an issue, he always understood the “why” in addition to the dictum. His dedication and work ethic were bolstered by a precise and analytical mind, and a conscientious, thorough approach to the task at hand. He was the ultimate gentleman, always soft-spoken, but firm in his convictions. His mastery of the English language was evident in both his professional and personal writings. If Larry used a word to describe a process, procedure, or even a casual observation, you could be certain that it was the right choice. Whenever Larry created or reviewed a document, he would provide a concise reason for his recommended course of action. Seeing the logic of this approach, I adopted it for my use in similar situations. We collaborated on many projects over our 14 year professional relationship, and I can honestly state that Larry was the best working partner I ever had, including both my Navy and civilian employment.

“Larry was my friend as well as my colleague. He was always considerate and thoughtful. I was honored that he attended my 60th birthday party, hosted by my daughter. I was all the more surprised when he gifted me with a Mensa publication containing a myriad of puzzles and other conundrums. Larry expected too much from me, however, as I have only solved a few over the years.

“In one of my last correspondences with Larry in 2009, I sent him some material on the Springfield Armory that my brother had sent to me with his recommendation that Larry consider writing a book on the Garand rifle production during the war. Larry, as always self-effacing, replied that “probably no one would read it.”

Editor’s note: My thanks to Gary Humphreys, Wayne Abba, Harvey Mymit, Al Bryant, and Major General Robert Rosenberg, US Army (RET) for their help and contributions.

Fall 2013Register Now!Bethesda, MD Basic Proj Sch 09/16-09/17 14 PDUs Basic EVMS 09/18-09/20 20 PDUsBoston, MA Adv Proj Sch 10/07-10/08 14 PDUs Adv EVMS 10/09-10/10 20 PDUsIrvine, CA Basic Proj Sch 10/21-10/22 14 PDUs Basic EVMS 10/23-10/25 20 PDUs Adv Proj Sch 11/04-11/05 14 PDUs Adv EVMS 11/06-11/07 20 PDUs

SEMINAR REGISTRATION www.smawins.com/publicseminars | 949 975 1550 x26918400 Von Karman Avenue, Suite 500 | Irvine, CA 92612

CHOOSE YOUR DESTINATION

PROJECT SCHEDULINGand EVMS SEMINARS

THE MEASURABLE NEWS


2013.03

11

How can you measure the quality of a project plan? Can the quality of the project management process itself be measured? How do I know if the project forecast is reasonable and accurate? If these questions were posed to various project personnel, there will certainly be more than one answer given. Why? Some view “quality” as a simple series of measures and techniques, but in reality, the pursuit of quality for program planning, forecasting, and execution is a multi-front battle. It is a battle worth fighting and one that can be successful.

MEASURING SCHEDULE QUALITYThe first step in measuring schedule quality is understanding and agreeing on the criteria used to define ‘quality’. Several agencies and organizations have established guidelines to measure quality. These agencies, including the Government Accountability Office (GAO), the Defense Contract Management Agency (DCMA), and even software analytics companies such as Acumen, have developed sets of common measures for evaluating just how qualified a schedule is. This trend has been replicated by many organizations internally, creating a company-wide set of standards for evaluation. These standards generally focus on a quantitative approach.

From a technical perspective, a critical path method schedule has compulsory elements. Tasks must have clear descriptions, logical links, and calculated values such as dates and float. Techniques that can undermine the schedule calculation should be minimized, such as hard constraints, unnecessarily complicated logic, overuse of leads and lags, and insufficient detail (high duration) within activities.

Each of the items above can be measured. From an efficiency of time and cost standpoint, they are best measured in software designed for the purpose. In short, quantitative metrics can be measured easily and quickly with the correct tool.

Quantitative vs. QualitativeCan quality end here?

No. Quantitative metrics are the necessary foundation of the quality evaluation process. This metric-based evaluation should be performed early and often in the development of a proper project schedule, rather than end at the point of project baseline. The best method of building quality into the process starts with the early development of the schedule and continues on, rather than delaying such an evaluation until the schedule has been fully constructed.

The qualitative view requires the ability of the project team to answer the following question, “does this schedule represent a meaningful, realistic view of project execution?” Quantitative metrics help with “meaningful,” but measuring “realistic” is a bit trickier.

Risk AssessmentAchieving schedule realism first requires an assessment by program personnel to determine if each step required in the development of the project goals is accurately and comprehensively captured. The next step toward realism is an honest, thorough discovery of specific risks, opportunities, and the probability and potential impact of each. These may include events that can be controlled and mitigated and those that cannot, such as weather events.

While various methods exist to evaluate risk to forecasted schedule completion date and cost, a Monte Carlo simulation performed on the complete, detailed schedule proves to

INTRODUCTION

ENSURING QUALITY IN PROJECT PLANNING, FORECASTING, AND EXECUTIONTom Polen, Solutions Architect, Deltek


be the most reliable indicator. While some approaches to risk focus on the critical path or high-level schedule abstracts, risk evaluation on all paths and all levels of detail is critical. Sequences of activities with low perceived risk, in the midst of a risk workshop, can emerge as program drivers during a comprehensive risk assessment.

Duration Uncertainty and Risk EventsCritical to this process is the ability to assign duration uncertainty to all schedule levels while capturing probable risk events and assigning them to the appropriate scheduled items. Also, weather impacts and other potentially uncontrollable risks must be accounted for and mapped to all tasks that may be affected.

When the risk factors have been identified and assigned, the risk simulation will provide probable outcomes of project execution. Program management will often treat this information in one of two ways. The first approach is to ignore the outcome of the risk exercise and proceed as originally planned. The second approach is to adopt the risk outputs as a probable project outcome and, if necessary, optimize the project schedule to maintain alignment with original cost and schedule delivery goals.

Taking an Impact-based ApproachA diligent risk assessment exercise will often produce probable outcomes that are not in line with the committed project goals. A proactive impact-based approach to risk mitigation is the first step. The simulation will indicate which risks are impacting the project plan the most. The past approach is to mitigate the perceived “big hitters.” The progressive approach is to map the risks to all related tasks, run the simulation, and observe the impact of “stealth risks,” those items that have not been highly discussed, but are revealed as significant threats to project execution during the analysis.

Opportunities should be evaluated alongside threats. It is common for risk workshops to focus on pessimistic aspects of execution; however, attention to innovation and efficiency may produce unexpected yet possible improvements in project delivery time and cost. The opportunities should be tracked in the same format and linked to the schedule in the same fashion as threats.

Finally, when mitigation and opportunities have been identified, mapped to the schedule, and risk simulations performed, the results may indicate a low probability of project success. While it may be viable for the program manager to request schedule and/or cost relief from their sponsor, there is an additional option—project acceleration and optimization.

ACCELERATION AND OPTIMIZATIONRules-based acceleration may assist the project team to meet schedule and cost objectives despite the risks to successful completion. Establishment of ground rules for optimization is critical. For example, the program manager may decide that task durations cannot be reduced, but he may see opportunity to reduce lags and remove hard constraints that are extending the overall project duration. In the event that durations can be reduced, he may choose to limit such reductions of the original values in order to maintain the credibility and achievability of the schedule. Alternatively, the program manager may determine that only tasks of exceptionally long duration will be subject to acceleration.

The crucial aspect of acceleration is the establishment and adherence to the ground rules. It may be very convenient to simply cut all task durations in half; however, the result will be a project plan that cannot be executed! If the result is not a credible, achievable plan, a different optimization strategy may be needed.

THE FUTURE OF MEASURING QUALITYThe landscape of quality definition and action is changing. Several years ago, simply performing quantitative evaluation of quality was a time consuming and unrewarding process. Tools exist today to allow quantitative measures to be reviewed instantly. Leaders within U.S. government program offices and audit agencies agree—the future of program quality, forecast evaluation, and execution measurement begins with automated, repeatable processes performed early and often. The time, cost, and effort required to measure quality has become insignificant while real, actionable results are produced.

The same leaders agree that the time saved gathering quantitative results provides more time to review qualitative, less tangible quality indicators. Additionally, the quantitative information, when gathered early in plan development, can provide fuel for a risk-based approach to qualitative review. In other words, these two approaches of quality can work


CONCLUSION

strongly together, with metrics results providing the impetus for thorough examinations of very specific aspects of the project plan. Before metric-based analysis could be performed instantly, this targeted approach to quality evaluation was not viable.

Ultimately, all components are needed to realize schedule quality. Everything from a solid, well-constructed baseline, a realistic picture of project risk, plan optimization, to a plan that stays relevant throughout project execution are mandatory elements of project success. Absolute perfect quality is rarely achieved, but active management and continual assessment of the schedule, cost, and risk throughout implementation will dramatically increase chances of project success and a satisfied sponsor.

ABOUT THE AUTHORTom Polen, Solutions Architect at Deltek, has 18 years of experience scoping, designing, developing, and implementing scheduling and earned value management systems in the telecommunications, transportation, insurance, and defense industries. Tom’s overall vision is to help the field of project management to focus on analysis rather than development of data and information used to support analysis.

For more information, call 1-512-291-6261 or visit www.projectacumen.com. E-mail inquiries to [email protected].

Acumen’s software products ensure that each and every project and portfolio meets planning

and process requirements, and remains on track for a successful execution time and again.

Acumen Fuse® - Diagnose and resolve shortcomings

Acumen Risk™ - Identify and reduce cost and schedule risk exposure

Acumen 360™ - Accelerate time frames and recover delays

www.projectacumen.com/betterprojectsBetter planning, Better performance, Better projects.... Acumen.

When quality is the only option.Improved program health and performance.

[email protected] | +1 (512) 291-6261



ANNOUNCEMENT The College of Performance Management announces the upcoming election that will start on August 15, 2013 and close September 16, 2013 at 5:00 pm (EST in US). Board positions open for election this year are Executive Vice-President, Vice-President Finance, Vice-President Research & Standards, Vice-President Conference & Events - EVM World, Vice-President - Conference & Events - IPMC and Vice-President Global Outreach. Candidates for these positions are as follows:

Executive Vice President• Wayne Abba• Nick Pisano• Ray Stratton

Vice President of Finance• Yelena Andrianova• Buddy Everage• Jay Patterson

Vice President of Research and Standards• Victor omo Agege• Dale Gillam

Vice President of Global Outreach (NEW)• Marco Buijnsters• Raf Dua• Paul D. Giammalvo• Kym Henderson• Bill Mathis• Fernando Roque• J. Greg Smith

As no candidates were nominated for the following positions, the Governing Board will appoint a member to fill these vacancies:

• Vice-President Conference & Events - EVM World• Vice-President - Conference & Events – IPMC

Resumes and candidate information will be posted to the CPM website. To review this information go to: www.mycpm.org/about-us/elections/

On August 15th you should have received an email from “Elections OnLine” with a link and Login information to vote. If you did not receive the email in your InBox, please be sure to check your Junk mail or Spam filtered folders for this email.

All CPM members current at 31st July are eligible to vote. You are encouraged to participate because the election results will determine your organization’s future leaders and direction for your organization.

For further information and/or questions please contact: Gaile Argiro [email protected] 703-370-7885 or

Lauren Bone [email protected]

CPM BOARD ELECTIONS 2013

THE MEASURABLE NEWS


2013.03

15

ABSTRACT

The article is based on the 31 May keynote presentation at EVM World 2013.

EARNED SCHEDULE – TEN YEARS AFTERBy Walt Lipke, PMI® Oklahoma City Chapter

Earned Schedule is an extension to Earned Value Management. The method provides considerable capability to project managers for analysis of schedule performance. From the time of the public’s first view of Earned Schedule with the publication of “Schedule is Different” in the March 2003 issue of The Measurable News, its propagation and uptake around the world has been extraordinary. This article will cover the capabilities of the method and challenges encountered, progressing through the significant extensions, to its present status.

ORIGIN OF EARNED SCHEDULEThis is the ten year anniversary of Earned Schedule (ES). During its relatively short existence, ES has made a large impact on Earned Value Management (EVM) and project management, as well. Frankly, I have been extremely surprised by its uptake in EVM application and academia, including research.

In my keynote at the recent EVM World conference, I recounted the story of how ES came about. In 2002, I was involved with software process improvement in an organization that had nearly achieved Level 5 of the Software Engineering Institute Capability Maturity Model (SEI CMM®). Only one Key Process Area, Defect Prevention (DP), remained to satisfy the achievement of Level 5.

Software organizations were attempting to satisfy DP by applying Statistical Process Control (SPC) to defect counts from the quality process of software development, i.e. inspections and reviews. My belief was that this approach to achieve DP could possibly cause sub-optimization of the development process. This later was shown to be true.

My thought was to use management indicators that encompassed most of the development process. By encompassing the process, the improvement would more generally be beneficial and more likely have positive impact. We were employing EVM; thus, it seemed reasonable to apply SPC to the cost and schedule performance indexes, CPI and SPI, respectively. After all, EVM was used in the project execution phase, generally eighty percent of the effort.

Before much action was taken to investigate this approach, I attended the College of Performance Management (CPM) 2002 spring conference, at which a presentation was given by Quentin Fleming. In his presentation the statistics based research of CPI by Dr. Christensen was discussed. It was fascinating material, and due to the statistical nature of the research it led me to believe I was on the right track with using CPI to satisfy the DP attribute of the CMM®. However, later that evening after some reflection, I realized Mr. Fleming did not mention anything about SPI.

The next morning as I walked through the courtyard of the hotel, I happened to encounter Mr. Fleming, with his suitcase, on his way to checkout and leave the conference. I told him I had attended his presentation and that I was really interested in the statistical testing and study of CPI behavior. I then asked, “Has any comparable research been accomplished for SPI?” Mr. Fleming responded saying, “No there hasn’t. You do know that SPI fails for late performing projects.” …Yes, I knew that, but sometimes you just don’t make the connection. It was then obvious, with the known failure mode of SPI, reliable statistical analysis of the indicator was not possible and therefore the SEI CMM® Level 5 key process area, Defect Prevention, could not be satisfied using the SPI indicator.

This was my dilemma. I needed a reliable schedule indicator for software process improvement; thus, the impetus for creating ES. Originally, ES had a single purpose; i.e.,


1. For the definitions of PMB, EV, and PV, refer to the Practice Standard for Earned Value Management [PMI, 2011].

provide a path for achievement of CMM® Level 5. There was no intention for ES to be made available to other EVM practitioners. Only after the schedule indicators from ES proved reliable from several months of prototyping on software projects did I realize that the method held potential for the EVM community. This led to the publication of The Measurable News article “Schedule Is Different” [Lipke, 2003].

THEORY AND CAPABILITIESThe fundamental concept of ES is shown in figure 1. As the description reads, “The idea is to determine the time at which the earned value (EV) accrued should have occurred.” The time duration associated with the point on the Performance Measurement Baseline (PMB) where planned value (PV) is equal to EV is Earned Schedule.1 For the EV accrued, ES provides a measure of how much has been earned of the planned duration (PD) of the project.

ES is computed from the simple formula:

ES = C + I

C is determined by comparing EV to the periodic values for PV, i.e., PVn. C is the largest value of n satisfying the condition, EV ≥ PVn. I is an interpolation using the equation:

I = (EV – PVC) / (PVC+1 – PVC)

IndicatorsHaving ES, the time based schedule indicators are formed, Schedule Variance (time) and Schedule Performance Index (time), abbreviated as SV(t) and SPI(t), respectively. The indicators are computed using the following formulas:

SV(t) = ES – AT SPI(t) = ES / AT

where AT is the actual time, i.e. the duration from the start of the project to the time at which EV is measured.

These indicators perform reliably for both late and early performing projects, whereas the EVM schedule indictors fail for late performing projects, as Mr. Fleming related. Furthermore, the time-based indicators always converge to the actual result at project conclusion.

3

where AT is the actual time, i.e. the duration from the start of the project to the time at which EV is measured. These indicators perform reliably for both late and early performing projects, whereas the EVM schedule indictors fail for late performing projects, as Mr. Fleming related. Furthermore, the time-based indicators always converge to the actual result at project conclusion.

Figure 1. Earned Schedule Concept

Forecasting. The SPI(t) indicator has made forecasting duration possible from EVM performance data, using the simple formula [Henderson, 2004]: IEAC(t) = PD / SPI(t) where IEAC(t) is Independent Estimate at Completion (time-based)

From this basic forecasting, very powerful statistical forecasting has evolved. In figure 2 the nominal forecast is the graphed line in the middle. The upper and lower lines are Confidence Limits, determined from SPI(t), variation in the periodic values of SPI(t), and a choice of Confidence Level, usually 90 or 95 percent.c

c The terms, Confidence Limits and Confidence Level, come from the mathematics of statistics. To fully define them and the calculation methods for this application is beyond the scope of the article. The reader, if interested in having a greater understanding, is referred to my book, Earned Schedule [Lipke, 2009].

For the figure, there is a 90 percent probability that the final duration will occur somewhere between the upper and lower graphed lines. As the project progresses, it is observed that the spacing of the two lines narrows, and that they converge along with the nominal forecast to the actual duration.

Figure 1. Earned Schedule Concept

ForecastingThe SPI(t) indicator has made forecasting duration possible from EVM performance data, using the simple formula [Henderson, 2004]:

IEAC(t) = PD / SPI(t)

where IEAC(t) is Independent Estimate at Completion (time-based)


2. The terms, Confidence Limits and Confidence Level, come from the mathematics of statistics. To fully define them and the calculation methods for this application is beyond the scope of the article. The reader, if interested in having a greater understanding, is referred to my book, Earned Schedule [Lipke, 2009].

From this basic forecasting, very powerful statistical forecasting has evolved. In figure 2 the nominal forecast is the graphed line in the middle. The upper and lower lines are Confidence Limits, determined from SPI(t), variation in the periodic values of SPI(t), and a choice of Confidence Level, usually 90 or 95 percent.2 For the figure, there is a 90 percent probability that the final duration will occur somewhere between the upper and lower graphed lines. As the project progresses, it is observed that the spacing of the two lines narrows, and that they converge along with the nominal forecast to the actual duration.

4

Figure 2. Statistical Forecasting

Prediction. For cost analysis, EVM has the To Complete Performance Index (TCPI), an indicator describing the performance efficiency needed for the remainder of the project to meet a specific cost objective. Until ES, a comparable indicator did not exist for schedule analysis. ES facilitates the creation of the To Complete Schedule Performance Index (TSPI), formulated as follows: TSPI = (PD – ES) / (ED – AT) where ED is the estimated or desired project duration. As with TCPI, TSPI can be computed for any desired completion objective. For schedule the durations of interest are, generally, PD and the duration to the customer, derived from the product delivery date. A useful quality of both TCPI and TSPI is that they provide information concerning whether the project objective is achievable and whether a poor performing project is recoverable [Lipke, 2009]. Discrete TSPI values provide management with decision information: TSPI Duration objective is achievable TSPI > 1.10 Duration objective is unachievable 1.00 < TSPI Recovery may be possible

Either TCPI or TSPI can be formulated such that it is dependent upon two variables, the performance index at a specific status point and the fraction complete. For schedule analysis, the formulation is: TSPI = (1 – K) / (R - K/SPI(t)) where K = ES/PD and R = ED/PD

Figure 2. Statistical Forecasting

PredictionFor cost analysis, EVM has the To Complete Performance Index (TCPI), an indicator describing the performance efficiency needed for the remainder of the project to meet a specific cost objective. Until ES, a comparable indicator did not exist for schedule analysis.

ES facilitates the creation of the To Complete Schedule Performance Index (TSPI), formulated as follows:

TSPI = (PD – ES) / (ED – AT)

where ED is the estimated or desired project duration.

As with TCPI, TSPI can be computed for any desired completion objective. For schedule the durations of interest are, generally, PD and the duration to the customer, derived from the product delivery date. A useful quality of both TCPI and TSPI is that they provide information concerning whether the project objective is achievable and whether a poor performing project is recoverable [Lipke, 2009]. Discrete TSPI values provide management with decision information: TSPI ≤ 1.00 Duration objective is achievable TSPI > 1.10 Duration objective is unachievable 1.00 < TSPI ≤ 1.10 Recovery may be possible

Either TCPI or TSPI can be formulated such that it is dependent upon two variables, the performance index at a specific status point and the fraction complete. For schedule analysis, the formulation is:

TSPI = (1 – K) / (R - K/SPI(t))

where K = ES/PD and R = ED/PD

This formulation provides the project manager (PM) with the ability to “look ahead.” For the specific duration objective, the PM can determine when the project becomes unrecoverable and the time available for a corrective management intervention.

Critical PathEspecially for large projects, analysis of Critical Path (CP) performance is aligned with schedule experts, segregated from the EVM analysts. ES provides a method using EVM data to assess CP performance, thereby providing an alternative and cross-check to the schedulers’ assessment [Lipke, 2009].


The technique treats the CP as a separate project. The PMB for the CP is created from the PV in its tasks. Then, using the EV accrued within the CP tasks, ES for CP performance is computed. Continuing, SV(t), SPI(t), and IEAC(t) can be determined for the CP.

The CP values for the indicators and forecast are then used for comparison to the schedulers’ assessment. As well, additional management information is available from the comparison of the CP computed values to the ES analysis for the total project. Inconsistency in compared values is cause for deeper analysis.

From reported application, the forecast from the schedulers’ CP analysis is consistently optimistic because it does not account for the current schedule performance efficiency. The schedulers’ method adds the schedule variance of the completed work to the planned duration, expecting the remaining work to be performed as planned. Because the ES method accounts for efficiency, EVM analysts have indicated earlier identification of CP performance issues.

Schedule AdherenceThe schedule is an embodiment of our best understanding of how to accomplish the project. It follows then that the planned schedule is crucial to project success, and that project managers should do their utmost to ensure project execution conforms to it. The planned schedule is the most efficient path for executing the project; any deviation leads to inefficiency and very likely other problems ….such as constraint reduced production, idle time, skills mismatch and poor quality output, in turn, requiring rework. Therefore, it is not enough to have knowledge of the execution efficiency, SPI(t). Additionally, project managers (PM) need to know how well the process is being followed.

ES provides the capability to determine whether the accomplishment is in agreement with the expectation from the planned schedule [Lipke, 2009]. The value of ES identifies the PV which should have been accomplished in every task. By matching the EV accrued to PV expectation, an indicator of schedule adherence (SA) is formed, termed the P-Factor. When matching is perfect, P equals 1.00 and when there is no correlation, P is 0.00. As the project progresses P tends to increase, concluding at the value 1.00 at completion.

The concept of SA, along with the P-Factor, facilitates very useful analysis. Tasks are pinpointed which may have constraints or impediments hindering project accomplishment. With this information, management has the opportunity to investigate and remove the hindrances. As well, tasks having the potential for future rework are identified. Significantly, the value for project rework can be forecast, thereby providing management with the potential cost impact from the lack of schedule adherence.

Discontinuous PerformanceThere are conditions during project execution, generally for small, short duration projects, that can cause error in the calculated values for the ES indicators and duration forecasts. These conditions are the following:

1. Down Time – periods of performance in which no work has been planned or scheduled2. Stop Work – periods during execution where management has halted performance

ES calculation methods have been developed to accommodate these conditions [Lipke, 2011]. Without the methods, the distorted indicator and forecast values have the potential to cause unneeded and possibly erroneous management action. The ES calculation methods addressing Down Time/Stop Work have been shown to yield improved indicators and forecasts. Improvement is seen in both accuracy and rate of convergence to the actual result.

Longest PathResearch performed by Dr. Mario Vanhoucke has shown that ES forecasting performs best when the topology of the network schedule is serial [Vanhoucke, 2009]. Furthermore, as the network topology becomes more parallel, the research indicates that the forecast values are less reliable.

The most recent evolution of ES forecasting was created to utilize the serial topology research finding. For this methodology, all of the serial paths to completion in the schedule network are identified. For each a PMB is created and used for forecasting, similarly to the method described earlier for CP analysis. The serial path having the longest duration forecast is theorized to best represent the project.


3. The insert was authored by Dr. John Singley and Eleanor Haupt of the College of Performance Management.

6

Discontinuous Performance. There are conditions during project execution, generally for small, short duration projects, that can cause error in the calculated values for the ES indicators and duration forecasts. These conditions are the following:

1) Down Time – periods of performance in which no work has been planned or

scheduled 2) Stop Work – periods during execution where management has halted

performance

ES calculation methods have been developed to accommodate these conditions [Lipke, 2011]. Without the methods, the distorted indicator and forecast values have the potential to cause unneeded and possibly erroneous management action. The ES calculation methods addressing Down Time/Stop Work have been shown to yield improved indicators and forecasts. Improvement is seen in both accuracy and rate of convergence to the actual result.

Longest Path. Research performed by Dr. Mario Vanhoucke has shown that ES forecasting performs best when the topology of the network schedule is serial [Vanhoucke, 2009]. Furthermore, as the network topology becomes more parallel, the research indicates that the forecast values are less reliable.

The most recent evolution of ES forecasting was created to utilize the serial topology research finding. For this methodology, all of the serial paths to completion in the schedule network are identified. For each a PMB is created and used for forecasting, similarly to the method described earlier for CP analysis. The serial path having the longest duration forecast is theorized to best represent the project.

Figure 3. Longest Path Forecasting

Figure 3 illustrates the comparison of forecasts, the normal to longest path (LP). The visual supports the improvement expectation in ES forecasting provided by LP. As

Figure 3. Longest Path Forecasting

Figure 3 illustrates the comparison of forecasts, the normal to longest path (LP). The visual supports the improvement expectation in ES forecasting provided by LP. As observed, the variation of the LP forecast is reasonably uniform around the actual duration, whereas the total project forecast has much more variation in converging to the actual duration.

The results from the notional data example are compelling. However, they are insufficient to say LP forecasting should be adopted and employed without further examination and testing. I am hopeful, with additional confirmation and tools for applying LP, the methodology will provide significant improvement of ES forecasting for network schedules whose topology is highly parallel.

CHALLENGESAlthough ES has become well accepted, it has not always been. After the seminal paper, “Schedule is Different” and the subsequent paper, “Earned Schedule: A Breakthrough Extension to Earned Value Theory?,” by Kym Henderson [Henderson, 2003] were published, an over exuberance developed. There was even some discussion favoring replacement of the EVM schedule indicators with those from ES, SV(t) and SPI(t). The exuberance was quelled; however, an insert to the PMI Practice Standard for EVM, 1st edition, was created to generate awareness of ES and to gain feedback from trial applications [PMI, 2005].3

From that beginning serious resistance to ES developed. Possibly, the initial rejection came from a view that Kym and I had little credibility in the EVM community. From their perspective, we had limited EVM experience and were new members of CPM. Some ridiculed the method and its advocates from the long held position that the only way to perform schedule performance analysis is to directly employ the schedule. Others added the quizzical argument, stating that although EVM has indicators with “schedule” in their description, they were never intended to be used for schedule performance analysis. A few, in the extreme, used intimidation tactics to discourage the use and propagation of ES.

Next came the questioning of the mathematics of the ES calculation. Although the description of the calculation method was made in the seminal article, including application examples, there were assertions that ES is an interpolation made from the linearization of the entire PMB. Of course, it is not. ES does use linear interpolation, but only for a single performance period. After the initial flurry of discussion, the misinterpretation of the ES calculation appears to be resolved. It has not reappeared in several years.

Presently, there remains skepticism about ES, most likely connected to resistance to change. Rigidity is a difficult obstacle to overcome. Logic and reasoning are set aside while opinion, emotion, and political power become the position determinants.

This residual skepticism affects whether ES is accepted by the EVM community and fully incorporated into the EVM practice. Across the globe, there is considerable evidence of increased application of ES, indicating its general acceptance. An eventual preponderance of usage may overcome the skeptics; only time will tell whether ES is adopted.


AFFIRMATIONSignificantly, Kym Henderson’s article verified the performance of the ES indicators from application to real data [Henderson, 2003]. Several advantages were cited for ES:

1. Simplicity of the calculations2. Time-based indicators as opposed to the cost-based EVM indicators3. No additional data required4. Indicators perform reliably for either early or late performing projects5. Indicator values converge and resolve to the actual project result

In his article, Mr. Henderson summarized the examination of ES, saying:

“The retrospective analysis of ES using my own EVM projects’ data, …has confirmed with remarkable precision the accuracy of the ES concept and ES metrics …when compared to their historic EVM counterparts.”

As well, other independent successful trials of the ES method were subsequently reported, most notably is one at Lockheed Martin conducted by Robert Handshuh. Additional to the trials, research was performed which confirmed the performance of ES. Dr. Mario Vanhoucke and Stephan Vandevoorde using methods of simulation compared ES to other EVM based methods of duration forecasting [Vanhoucke, et al., 2007]. They formed the conclusion:

“The results reveal that the earned schedule method outperforms, on the average, all other forecasting methods.” Most recently, Captain Kevin Crumrine and Lieutenant Colonel Jonathon Ritschel of the USAF, using EVM data from 64 major defense projects, examined performance of the schedule indicators from EVM and ES [Crumrine, et al., 2013]. Crumrine and Ritshcel made the comparison using statistical hypothesis testing methods. Their conclusion, with regard to the schedule indicators from each methodology, was made as follows:

“This research finds Earned Schedule to be a more timely and accurate predictor than Earned Value Management.”

Affirmation of ES is noted in several other venues, as well:

1. Positive feedback from application globally2. Incorporation into EVM analysis tools – EV Engine, IPM Pro, Visi Trend, ProTrack, and

Project Flight Deck3. Inclusion in EVM training from various sources4. Inclusion in project management coursework at several universities globally5. Appearance in project management textbooks6. As a topic of interest in several published research articles (academic journals,

occurring globally)

RESOURCESFor those using ES or those who want to begin, there are several resources available. The one, perhaps of the greatest value, is the ES website, www.earnedschedule.com. The website has a considerable amount of material, including news, publications, presentations, calculators, and much more. With very few exceptions the materials are free for download and individual use.

The website began in February 2006. In that first month it received approximately 6000 “hits.” The website has become increasingly more popular and is now receiving approximately 55,000 hits per month, and recently had a count of over 63,000.

Another good source for ES material is the PMI EVM practice standard [PMI, 2011]. The standard includes Appendix D, Schedule Analysis Using EVM Data, which predominantly is a description of ES. And of course my book, Earned Schedule, is available.


ES facilitates considerable schedule performance analysis capability from EVM data. The capabilities offered by ES – time-based indicators, duration forecasting, prediction, critical path, schedule adherence, constraint/impediment identification, rework cost, discontinuous performance, and schedule network topology – previously were not believed possible.

It is my opinion that acceptance of ES should help to popularize EVM. With the inclusion of ES, EVM can make the case that, now, in a single method, project cost and schedule can be managed.

Even without full acceptance the impact of ES can no longer be ignored. It is being used world-wide in many venues – large and small projects, and in academia, as well.

ACKNOWLEDGEMENTThe following people, journals, and organizations are acknowledged for their contribution to the advancement and propagation of Earned Schedule: Kym Henderson, Eleanor Haupt, Dr. John Singley, Ray Stratton, Steve Wake, Alex Davis, Mick Higgins, Dr. Mario Vanhoucke, Stephen Vandevoorde, Robert Van De Velde, Major John Garrett, Norm Brown, Peter Schwarz, Greg Smith, The Measurable News, PM World Today, PM World Journal, Crosstalk, International Journal of Project Management, College of Performance Management, and the Project Management Institute.

RECOMMENDATIONEarned Schedule has endured for ten years. ES has proven itself through application, prototyping, and testing in real and simulated environments. It is propagating world-wide in use, in academic coursework and research. I believe it is time for ES to be normalized to EVM. ES needs to be integrated into EVM guidance; it should become a normal component of EVM training; and, ES capability should be commonplace for EVM analysis tools.

REFERENCESCrumrine, K., J. Ritschel. “A Comparison of Earned Value Management and Earned Schedule as Schedule

Predictors on DOD ACAT 1 Programs,” The Measurable News, 2013 Issue 2: 37-44

Henderson, K. “Earned Schedule: A Breakthrough Extension to Earned Value Theory?,” The Measurable News, Summer 2003: 13-23

Henderson, K. “Further Developments in Earned Schedule,” The Measurable News, Spring 2004: 15-22

Lipke, W. “Schedule is Different,” The Measurable News, March 2003: 31-34

Lipke, W. Earned Schedule, Raleigh, NC: Lulu Publishing 2009

Lipke, W. “Earned Schedule Application to Small Projects,” PM World Today, April 2011: Vol XIII, Issue IV

Lipke, W. “Speculations on Project Duration Forecasting,” The Measurable News, 2012 Issue 3: 1-7 Project Management Institute. Practice Standard for Earned Value Management, 1st Edition, Newtown

Square, PA.: PMI, 2005

Project Management Institute. Practice Standard for Earned Value Management, 2nd Edition, Newtown Square, PA.: PMI, 2011

Vanhoucke, M., S. Vandevoorde. “A Simulation and Evaluation of Earned Value Metrics to Forecast Project Duration,” Journal of the Operations Research Society, October 2007, Vol 58: 1361-1374

Vanhoucke, M. Measuring Time – Improving Project Performance Using Earned Value Management, London: Springer 2009

ABOUT THE AUTHOR Walt Lipke retired in 2005 as deputy chief of the Software Division at Tinker Air Force Base. He has over 35 years of experience in the development, maintenance, and management of software for automated testing of avionics. During his tenure, the division achieved several software process improvement milestones, including the coveted SEI/IEEE award for Software Process Achievement. Mr. Lipke has published several articles and presented at conferences, internationally, on the benefits of software process improvement and the application of earned value management and statistical methods to software projects. He is the creator of the technique Earned Schedule, which extracts schedule information from earned value data. Mr. Lipke is a graduate of the USA DoD course for Program Managers. He is a professional engineer with a master’s degree in physics, and is a member of the physics honor society, Sigma Pi Sigma (SPS). Lipke achieved distinguished academic honors with the selection to Phi Kappa Phi (FKF). During 2007 Mr. Lipke received the PMI Metrics Specific Interest Group Scholar Award. Also in 2007, he received the PMI Eric Jenett Award for Project Management Excellence for his leadership role and contribution to project management resulting from his creation of the Earned Schedule method. Mr. Lipke was selected for the 2010 Who’s Who in the World.

WRAP UP

As you may be aware, the College of Performance Management has been newly transformed as an independent entity, after a mutual and amicable separation from the Project Management Institute (PMI®).

Come join us and get involved in shaping the exciting future of the “new” CPM!!!Membership is $85/year and renewals are $50. Register online at www.mycpm.org.

CPM is the premier organization for earned value management (EVM) and project planning and controls. As an international, non-profit organization, CPM is dedicated to project management and performance management.

CPM Objectives • Promote Earned Value Management and Project Planning and Control: Foster the recognition and use of earned

value management and other project planning and control techniques as integrating processes for project management• Disseminate Information: Provide opportunities for the exchange of ideas, information, solutions and applications• Improve Community: Encourage and enable the advancement of theory and application through research,

standards, and education• Grow Professionals: Provide our diverse membership of project management professionals with growth

opportunities through leadership, education, networking, and other benefits of a professional association• Enhance Membership and Benefits: Improve membership benefits and expand our membership base through

continued development of a professional association

Membership Benefits• Discount to CPM Conferences• Access to latest information on performance management• Networking with other professionals, industry leaders, and academia• Quarterly magazine, The Measurable News • Access to CPM electronic library• Monthly Webinars

Conference• Earn PDUs for Project Management Professionals• Update skills• Network• Participate in EVM Training and Certificate Program

Active members, serving on a committee, as a speaker, volunteer or as a member of the Governing Board, gives you the greatest reward of all – contributing your knowledge and experience to benefit others!

For more information, email [email protected] or call (703) 370-7885

The “New” College of Performance Management (CPM) – JOIN NOW!

Join CPM now at www.mycpm.orgPMI is a registered mark of the Project Management Institute, Inc.

Membership Type: New Member Renewal

Name: Date:

Company/Organization:

Select one: Government Employee Industry

Academic Consulting

Address Type: Work Home

Address:

City: State: Zip/Post Code: Country:

Phone (Daytime): Phone (Evening):

Email Address:

Select one: New - Annual Membership Dues - $85

Renewal - Annual Membership Dues - $50

Payment: Visa Mastercard Check Enclosed

American Express Discover

Card Number:

Exp. Date: Security Code: Billing Address Zip Code:

Signature:

PLEASE RETURN THE COMPLETED APPLICATION WITH PAYMENT BY FAX OR MAIL:

College of Performance Management101 S. Whiting Street, Suite 320

Alexandria, VA 22304

Fax: (703) 370-1757; outside US – Country Code +1-703-370-1757

For more information, contact [email protected] or call (703) 370-7885; outside US – Country Code +1-703-370-7885

BECOME A COLLEGE OF PERFORMANCE MANAGEMENT MEMBERRegister on line at www.mycpm.org



OVERVIEW You probably know that CPM offers several Awards, but did you know what these Awards represent? I’d like to introduce two of these awards and explain a little about the attributes of those for whom they were named. By understanding the people honored by the establishment of the Award and the calibre of those who have been recipients I hope you will be inspired to look at your own professional circle and identify those who are taking Project Control to the next step in its evolutionary journey.

Do you have the vision to spot peers of those described below? If you do see someone you think might be deserving please nominate them for one of these prestigious accolades:

DRIESSNACK DISTINGUISHED SERVICE AWARDLieutenant General Hans H. “Whitey” Driessnack was assistant vice chief of staff, Headquarters U.S. Air Force, Washington, D.C. He also served as senior Air Force representative, U.S. Delegation to the Military Staff Committee, United Nations.

Gen. Driessnack started his career by serving a year in the Navy before entering Syracuse University. Commissioned into the Air Force, he became a fighter pilot and flight commander during the Korean War, flying 25 combat missions. He worked at the Headquarters Systems Command at Andrews Air Force Base and worked his way up to become comptroller of the Air Force in 1978. He was named assistant vice chief of staff in 1981 and also served as the senior Air Force representative to the United Nations.

In July 1963 General Driessnack transferred to Headquarters Air Force Systems Command, Andrews Air Force Base, and was assigned to the Office of the Deputy Chief of Staff, Comptroller, as a research and development officer in the Management Systems Development Division. From May 1964 to June 1968, he was a management system specialist in the Office of the Assistant Secretary of the Air Force (Financial Management). He then returned to Air Force Systems Command headquarters as chief, Cost Information and Management System Division, in the Office of the Deputy Chief of Staff, Comptroller.

Although demonstrating a life-long commitment to learning and increasing responsibility and leadership, it was in the 1970s in particular that “Whitey” Driessnack had the opportunity to build on the seminal EVM work pioneered by A Ernest “Ernie” Fitzgerald. Until then the use of PERT/Cost was predominant in government project control, involving a continually revised Estimate to Complete (ETC) that was proving unreliable. “Whitey”, initially an advocate of PERT/Cost, was introduced to an alternate concept created to redress the unreliability of the ETC, called Earned Value. This was embodied by Fitzgerald into the original Airforce Cost/Schedule Planning and Control Specification (CSpec). Driessnack went on to champion this into becoming adopted by the US Department of Defense in 1967 as the Cost/Schedule Control System Criteria (C/SCSC), the foundation upon which Earned Value Management is based today.

General Driessnack’s accolades include:• graduated from Charles E. Gorton High School in 1945;and received a bachelor of science

degree in civil engineering from Syracuse (N.Y.) University in 1951;• received a master’s degree in business administration from the Air Force Institute of

Technology, Wright-Patterson Air Force Base, Ohio, in 1959; graduated from Squadron Officer School in 1956; graduated from Air Command and Staff College in 1963; graduated from the Naval War College, Newport, R.I., in 1970; and graduated from the advanced management program of the Harvard Graduate School of Business Administration in 1971;

• promoted to lieutenant general Aug. 22, 1978, with same date of rank;• military decorations and awards include the Distinguished Service Medal, Legion of Merit

CPM AWARDS: THE WHYS AND WHEREFORESBy Lauren Bone, Vice President Administration


with two oak leaf clusters, Meritorious Service Medal, Air Medal with oak leaf cluster, Air Force Commendation Medal with two oak leaf clusters, Presidential Unit Citation emblem, Air Force Outstanding Unit Award ribbon, Air Force Organizational Excellence Award ribbon and the Republic of Korea Presidential Unit Citation emblem.

After his military retirement, he worked for six years for a defense contractor, United Technologies, and then retired a second time in 1989. Even then his commitment to his community continued with his involvement in building an Air Force retirement community (Falcons Landing) in the Washington DC area. He formed the Air Force retired officers community board and served as its chairman.

He was considered a leader in project management, so much so that shortly after his second retirement the College of Performance Management named its highest award in his honor.

If you have read any Earned Value literature you will recognise the recipients of this award as a pantheon of the champions of Earned Value Management and Project Controls leadership:

1992 – Robert R. Kemps 1994 – Gary Christle 1995 – Quentin Fleming 1996 – Wayne Abba 1996 – Larry Stone 1997 – Gary Humphreys 2003 – Robert Pattie 2004 – David Christensen 2006 – David Muzio 2009 – Tony Finefield

Is the next deserving recipient someone you know? Here are the attributes required:• has made major contributions to project performance management policy, concepts, and

practices, which have national and international implications; • has provided critical contributions to the evolution of Earned Value Management and

project planning and control standards, research, and education;• has promoted the exchange of theory, development, and application among project

management professionals.

GARY CHRISTLE LEADERSHIP EXCELLENCE AWARDGary Christle was responsible for US Department of Defense (DoD) Acquisition policy as embodied in the DoD 5000 series documents. This responsibility included the role of Defense Acquisition Board executive secretary, establishment of Acquisition Program Baselines, and supervision of the monthly Defense Acquisition Executive Summary process for monitoring the cost, schedule and technical status of major acquisition programs. He was also responsible for policies regarding contractors’ internal cost and schedule control systems (Earned Value Management Systems (EVMS)) and was a member of the DoD Cost Analysis Improvement Group. Gary retired from Federal Service in October 2000 as the Deputy for Acquisition Management, Office of the Under Secretary of Defense (Acquisition, Technology and Logistics).

Subsequently as an independent consultant, Gary has also undertaken program management process assessments of business units of several major Defense contractors. He is currently with the Center for Naval Analyses. As a Senior Fellow with the Center for Naval Analyses Mr. Christle has published reports on Headquarters reorganization, project office organization, program manager training, acquisition metrics, balanced scorecards, earned value implementation, and risk management for executives.

Throughout his distinguished career Gary has consistently worked collaboratively with industry, encouraging both leadership on all sides of the table and the development and ownership of appropriate project control standards to be applied when working with US Government and in particular, US DOD. He initiated the Integrated Performance Management Council which was ultimately responsible for the transformation of C/SCSC (initiated by Driessnack – see above) into Earned Value Management Systems guidelines embodied by the industry sponsored ANSI 748.

Gary’s leadership fostered an extensive body of knowledge. He made available to government researchers, notably the Air Force Institute of Technology under Dr David Christensen’s tutelage, the DoD earned value data base. He encouraged collaboration with friendly foreign countries, leading to creation of a multilateral memorandum of


SUMMARY

understanding among Australia, Canada and US defense organizations and parallel but less formal arrangements with Japan, New Zealand, Sweden and the United Kingdom.Gary Christle’s many awards reflect his distinguished career:

• Distinguished Military Graduate of Northeastern University; and holds a DoD level III certification in Program Management

• recipient of the Northeastern University Outstanding Alumni Award in the Field of Public Service; recipient of the Defense Distinguished Civilian Service Medal (twice); recipient of the Presidential rank designation of Meritorious Executive; recipient of the David Packard Excellence in Acquisition Award; the General Hans H. Driessnack Distinguished Service Award; recipient of the Vice President’s “Hammer” Award.

• When with the Project Management Institute CPM endowed an annual scholarship in his name.• frequent guest lecturer and Honorary Professor of the Defense Acquisition University ; and • CPM has established this leadership award in his name!

There has to date only been one recipient of the Gary Christle Leadership Excellence Award: 2009 – Dr. Robert B. “Bob” Rovinsky

Does someone you know have the attributes to merit this Award? They include:• a leader in his or her community;• will have promoted an environment of cooperation whether it is between customer and

supplier or within the government or a company;• not a one-time effort but ongoing examples of leadership;• has made significant contributions to the development and/or understanding of earned

value management that historically has and will continue to distinguish their work. The contribution should be a definable body of work as demonstrated through one or more of the following: speaking, teaching/training, research, publications, professional practice, etc

Inspired to identify and reward the new leaders in our community? Nomination Forms for either of these Awards can be downloaded from: www.mycpm.org/about-us/awards/2013 Nominations close on 30 September, 2013.

Proposal Support

EVMS StaffAugmentation

EVMS ComplianceReview Preparation

Integrated BaselineReview (IBR)Preparation

EVMSImplementation

EVMS Design andDocumentation

EVMS Requirementsor Gap Analysis

EVMS andProject Management

Training

Humphreys & AssociatesEVMS

Consulting Services

Humphreys & Associates offers a complete range of EVMS consulting services. From proposal preparation and management system gap analysis to mock compliance reviews or contractor self-surveillance and third party validations, H&A is the authority in EVMS.

As the industry recognized leader in EVMS training, H&A offers regularly scheduled Earned Value Management System, Project Scheduling, and Advanced Earned Value Management Techniques public workshops.

H&A now offers a Control Account Manager (CAM) Certification Program – ideal for anyone looking to improve their EVM expertise and distinguish themselves with a professional certification.

Interested in learning more? Visit our web site at www.humphreys-assoc.com or call us today at (714) 685-1730

THE MEASURABLE NEWS


2013.03

27

INTRODUCTION A practice symposium track titled “AGILE+EVM” was presented at EVM World 2013. The interest in the subject was evident by the large number of attendees, at some times >20% of the conference attendees. The track would not have been possible without the support of our presenters and their employers: Scott Dalessio (Grant Thornton), Joy L. Villagomez (Lockheed Martin), Josh Breen (Grant Thornton), Glen Alleman (PrimePM),Pete Zafros (Grant Thornton), Ron Terbush (Lockheed Martin). This article summarizes the six presentations within the track.

BACKGROUNDEarned Value Management has traditionally been applied to the waterfall type of development with fixed scope and fixed technical requirements. The Agile methodology continuously analyzes both the scope and technical requirements to keep pace with the customer’s evolving environment. With Agile, the cost and schedule are fixed; therefore the customer gets the most value achievable at a known cost and on a predetermined date.

INTRODUCTION TO AGILE & SCRUMAgile is an alternative to the traditional and waterfall project management methodologies. The term Agile was introduced in the Agile Manifesto of 2001.1 Agile is based on iterative development used primarily in software development. It focuses on adaptive planning and evolutionary development that is rapid and flexible. The Agile methodology is value - and vision - driven, and it has iterative cycles. It involves poly-skilled teams which foster communication and make sure that they have the required documentation for a project. Similarly, the Office of Management and Budget (OMB) focuses on “modular development”, which is referenced in OMB guidance on Contracting Guidance to Support Modular Development.2

Some aspects of using Agile that need to be understood include:• Agile is NOT a silver bullet for software development problems• Agile does not mean no planning, processes, or documentation• Agile is not appropriate for every type of project

As Scott Dalessio, a presenter at EVM World 2013, stated, “Agile is a silver mirror not silver bullet.” Zac Gery explains this as the foundation of Agile does not solve problems; it helps identify potential obstacles. All decisions whether good or bad are reflected using the methodology.3 Agile is not appropriate for every type of project. It is adaptive and flexible; however, you need to determine what makes sense for the team.

The results of the 7th Annual State of Agile Development Survey4 showed that Scrum was used by over 50% of respondents; it is the most widely adopted agile method. Scrum is an iterative and incremental project delivery framework that relies on organized, self-managing, and cross-functional teams. The Scrum roles include the product owner, the Scrum Master, and the team. The Scrum lifecycle is depicted in Figure 1 below.

AGILE & EVM SYMPOSIUMBy Shobha Mahabir


2

decisions whether good or bad are reflected using the methodology.3

The results of the 7th Annual State of Agile Development Survey

Agile is not appropriate for every type of project. It is adaptive and flexible; however, you need to determine what makes sense for the team.

4

showed that Scrum was used by over 50% of respondents; it is the most widely adopted agile method. Scrum is an iterative and incremental project delivery framework that relies on organized, self-managing, and cross-functional teams. The Scrum roles include the product owner, the Scrum Master, and the team. The Scrum lifecycle is depicted in Figure 1 below.

Figure 1: Scru m Lifecycle 5

Scrum meetings comprise of the following activities:

3 Apply in g Ele m ents of Agile Even if a project is not labelled “Agile”, applying a few elements of the methodology may still be beneficial. The two programs presented at the conference by Ron Terbush and Joy Villagomez illustrated some successes of evolving from waterfall to Agile methodology in selected process. These areas include management of expectations, scope, and risk with a system backlog; implementation of Just in Time planning and risk-based

3 Zac Gery,” Reflections in Scrum’s Silver Mirror”, 16 June 2012. <http://www.codeproject.com/Articles/494380/ReflectionsplusinplusScrum-27splusSilverplusMirror>. 4 7th Annual State of Agile Development Survey: <http://www.versionone.com/pdf/7th-Annual-State-of-Agile-Development-Survey.pdf>. 5 Source: Copyright 2008-2010 Mitch Lacey <http://www.mitchlacey.com>.

Vision

Product Backlog

Sprint Backlog

Daily Scrum

Meetings

2-4 week sprint

Incremental

Functioning

software

Release Planning

• Release kickoff • Project planning • Product backlog • Release dates set

Sprint Planning

• Sprint kickoff • Feature collaboration • Feature estimation • Feature prioritization

Daily Scrum

• Fifteen minutes daily standup • Three questions • Coordination of activities • Identification of risks

Sprint

Review

• End of sprint • Demo

Sprint Retrospective

• Team self-assessment • Continuous process improvement

Figure 1: Scrum Lifecycle5


2

decisions whether good or bad are reflected using the methodology.3

The results of the 7th Annual State of Agile Development Survey

Agile is not appropriate for every type of project. It is adaptive and flexible; however, you need to determine what makes sense for the team.

4

showed that Scrum was used by over 50% of respondents; it is the most widely adopted agile method. Scrum is an iterative and incremental project delivery framework that relies on organized, self-managing, and cross-functional teams. The Scrum roles include the product owner, the Scrum Master, and the team. The Scrum lifecycle is depicted in Figure 1 below.

Figure 1: Scru m Lifecycle 5


3 Apply in g Ele m ents of Agile Even if a project is not labelled “Agile”, applying a few elements of the methodology may still be beneficial. The two programs presented at the conference by Ron Terbush and Joy Villagomez illustrated some successes of evolving from waterfall to Agile methodology in selected process. These areas include management of expectations, scope, and risk with a system backlog; implementation of Just in Time planning and risk-based

3 Zac Gery,” Reflections in Scrum’s Silver Mirror”, 16 June 2012. <http://www.codeproject.com/Articles/494380/ReflectionsplusinplusScrum-27splusSilverplusMirror>. 4 7th Annual State of Agile Development Survey: <http://www.versionone.com/pdf/7th-Annual-State-of-Agile-Development-Survey.pdf>. 5 Source: Copyright 2008-2010 Mitch Lacey <http://www.mitchlacey.com>.

Vision

Product Backlog

Sprint Backlog

Daily Scrum

Meetings

2-4 week sprint

Incremental

Functioning

software

Release Planning

• Release kickoff • Project planning • Product backlog • Release dates set

Sprint Planning

• Sprint kickoff • Feature collaboration • Feature estimation • Feature prioritization

Daily Scrum

• Fifteen minutes daily standup • Three questions • Coordination of activities • Identification of risks

Sprint

Review

• End of sprint • Demo

Sprint Retrospective

• Team self-assessment • Continuous process improvement

APPLYING ELEMENTS OF AGILEEven if a project is not labelled “Agile”, applying a few elements of the methodology may still be beneficial. The two programs presented at the conference by Ron Terbush and Joy Villagomez illustrated some successes of evolving from waterfall to Agile methodology in selected process. These areas include management of expectations, scope, and risk with a system backlog; implementation of Just in Time planning and risk-based statusing; executing cycles of iterative development with continuous integration, verification, and validation; partnering EVM reporting with Agile burn-down charts; and enabling program success by being structured, yet flexible.

During the iterative development cycle, focus is placed on results, not progression. Feedback from user testing sessions can be incorporated into the next cycle through rapid feedback loops. At the iteration level, weekly and monthly percent complete feedback is done for EVM status. Weekly, on-demand feedback using iteration backlog is provided on total percent complete and individual tasks hotspots that provide a quick way to identify, organize and scan what is important. At the cycle level, weekly feedback is provided on the watch list across iterations. This feedback is driven by independent test assessment of requirements and user story satisfaction. The completion trend for the cycle is done at the work package level.

It is important not to lose sight of the backlog and to take a step back to look at the data at various levels. Key areas to focus on are the budget and cycle, together known as iteration


planning – how to map a waterfall budget to the Agile lifecycle. Just in Time planning methodology, where work is planned in near term to incorporate lessons learned from prior cycles, is critical. Continuous verification and validation occurs during each iteration. Shared lessons learned include:

• Don’t wait to integrate. The team is responsible for continuous iteration.• Don’t wait to verify. Test representatives on the team continuously check requirements-

based functionality during the iteration via dry runs.• Don’t wait to validate. The team works to continuously check user acceptability during

the iteration. • Don’t wait to measure verification/validation progress. The team continuously monitors

iteration-level progress via iteration backlog percent done. Test or CMO independently monitors application-level progress via percent done through user stories and/or requirements satisfaction

Some of the challenges with the initiation of Agile Development included organization culture, roles and responsibilities, staffing, and co-location. Having a champion for process improvements is crucial to the success of an Agile program. Members of the team stepped up as coaches, did the research, and coached their team members to understand the philosophy of Agile Scrum. Continuous learning through book clubs assisted with adhering to the process at first, the team then made use of retrospectives to reflect on efficiency. Sprint zero, a preliminary sprint focused on preparing for the first sprint, is a necessity and upfront planning is essential. Short sprints mean more repetitions, which lead to more practice and greater improvement.

METHODS TO APPLY EVM TO AGILETraditional EVM under the waterfall methodology requires up-front planning to implement the requirements of a system. It determines budget and schedule for the entire scope of the work. Progress is tracked to the original budget and schedule. Traditional EVM is less flexible with limited ability to make changes throughout the process.

Agile uses a more dynamic approach. It breaks down high-level features in increments (sprints) after budget and schedule have been determined. This approach allows for continuous baseline revision to reflect the work completed (story points) during sprints. Progress is measured by making allowances for iterative changes.

Adopting EVM to Agile projects uses Agile Scrum processes to produce EVM metrics. It utilizes planned accomplishments, called story points, as a basis for determining Earned Value (EV); however, EV is not taken for partial completion. Story points are earned with the item and accepted by the users or customers. Therefore, the EV technique of 0/100 percent is used. Frequent changes to the baseline are allowed when story points are added or removed, typically the end of each sprint.

Agile terms differ from the traditional EVM due to the integration of sprints and story points. • Budget at Complete (BAC) - the planned budget for the release or project• Planned Percent Complete (PPC) – the number of current sprints divided by the total

number of planned sprints• Actual Percent Complete (APC) – the total number of story points completed divided by

story points planned

The most significant change is in the EVM calculations for Planned Value (PV) and Earned Value (EV).

• Planned Value (PV) [progress against planned sprint schedule] = PPC x BAC• Earned Value (EV) [progress against planned budget] = APC x BAC• Note that Planned Value is defined using sprints, while Earned Value is defined using

story points.

Some leading practices shared by Josh Breen during the Agile + EVM Practice Symposium: • The basis of EVM is an accurate baseline• The baseline in Agile EVM must be established at the beginning of a release • Progress needs to be measured at defined boundaries• The initial data points are needed when planning the baseline:

• Number of planned sprints• Length of the sprints• Number of story points planned for release• Budget planned for the release• Start and end dates


• The planned work in a release is expressed in stories, which are quantified story points• At the start of each sprint, stories are selected from the product backlog and serve as the

baseline for the sprint• Stories may not be added or removed during the sprint, and no value is earned for partial

completion of a story• If an assigned story is not completed during a sprint, it is placed back into the product

backlog to be baselined in a later sprint• Progress and performance are measured at predefined boundaries• After each iteration (sprint or release), the product backlog must be reviewed, and the

future baseline must be adjusted to account for past performance.

INTEGRATING AGILE WITH EVM Whether the methodology is traditional or Agile, the goal is to increase the probability of program success. ANSI-748B is the Earned Value Management standard. While the 32 ANSI-748B criteria are designed to deliver value, touch points identified by Glen Alleman provide the basis for connecting the dots.

ANSI Criteria #

EVM Criteria Agile Approach

1 Define WBS Features and stories define tasks

2 Identify organization Self-organizing team

5 Integrate OBS and WBS Self-organized teams with customer

6 Schedule work Iterations and releases

7 Identify deliverables and milestones

Working software at the end of each iterations

8 Set time phased budget Fixed length iterations and releases

16 Record direct costs Fixed staff = level of effort

23 Determine variances Velocity measures missed features

25 Sum data and variance Missed features moved to next iteration

26 Manage action plans Replan missed features, adjust velocity

28 Incorporate changes Replan missed features, adjust velocity

Table 1: Eleven Critical EVM Criteria with Agile

The Office of Management and Budget (OMB) announced in June 2012 the “Contracting Guidance to Support Modular IT development” that provides guidance to empower agency leadership and Integrated Project Teams to support rapid delivery of workable solutions. From the National Defense Acquisition Act (NDAA) section 804, it seems that the Department of Defense (DoD) is ready to embrace Agile methodologies as depicted in the view of DoD IT Acquisition Lifecycle’s Impact on Agile below.

5

23 Determine variances Velocity measures missed features

25 Sum data and variance Missed features moved to next iteration

2 6 Manage action plans Replan missed features, adjust velocity

2 8 Incorporate changes Replan missed features, adjust velocity

Table 1: Eleven Critical EVM Criteria with Agile

The Office of Management and Budget (OMB) announced in June 2012 the “Contracting Guidance to Support Modular IT development” that provides guidance to empower agency leadership and Integrated Project Teams to support rapid delivery of workable solutions. From the National Defense Acquisition Act (NDAA) section 804, it seems that the Department of Defense (DoD) is ready to embrace Agile methodologies as depicted in the view of DoD IT Acquisition Lifecycle’s Impact on Agile below.

Figure 2: Bu sin e ss Capability Lifecycle (BCL) Acquisitio n Bu sin e ss Model 6

6 Federa l Im ple m ent a tio n a n d Support for Ag ile In the GAO’s study on “Software Development: Effective Practices and Federal Challenges in Applying Agile Methods” 32 practices and approaches were identified as effective for applying Agile software development methods to IT projects. 7

Start with Agile guidance and an Agile adoption strategy.

There were ten practices that were used and found effective by officials from the five federal agencies that were part of the study:

Enhance migration to Agile concepts using Agile terms, such as user stories (used to convey requirements), and Agile examples, such as demonstrating how to write a user story.

Continuously improve Agile adoption at both the project and organizational level.

Seek to identify and address impediments at the organization and project level.

Obtain stakeholder/customer feedback frequently.

Empower small, cross-functional teams.

6 A New Approach for Delivering Information Technology Capabilities in the Department of Defense. OSD 13744 -10 804 Report to Congress. 7 Software Development: Effective Practices and Federal Challenges in Applying Agile Methods, GAO-12-681, Jul 27, 2012. <www.gao.gov/assets/600/593091.pdf >

Figure 2: Business Capability Lifecycle (BCL) Acquisition Business Model6


FEDERAL IMPLEMENTATION AND SUPPORT FOR AGILEIn the GAO’s study on “Software Development: Effective Practices and Federal Challenges in Applying Agile Methods” 32 practices and approaches were identified as effective for applying Agile software development methods to IT projects.7 There were ten practices that were used and found effective by officials from the five federal agencies that were part of the study:

• Start with Agile guidance and an Agile adoption strategy.• Enhance migration to Agile concepts using Agile terms, such as user stories (used to convey

requirements), and Agile examples, such as demonstrating how to write a user story.• Continuously improve Agile adoption at both the project and organizational level.• Seek to identify and address impediments at the organization and project level.• Obtain stakeholder/customer feedback frequently.• Empower small, cross-functional teams.• Include requirements related to security and progress monitoring in your queue of

unfinished work (the backlog).• Gain trust by demonstrating value at the end of each iteration.• Track progress using tools and metrics.• Track progress daily and visibly.

Pete Zafros shared solutions to some of the challenges highlighted in the GAO’s study:

Process Challenge Solution

Scope & Planning

• Scope statements are often too specific to allow for innovation or not focused enough to effectively manage a project

• Scope is defined upfront and all requirements must fit within the scope

• Use business architectures, strategy documentation, process analysis to define the major functions of the systems

• Understanding scope will minimize rework and ensure the system is built in a sound fashion

Budget & Procurement

• Upfront budgeting can force either waterfall requirements processes or inflated budget requests

• Extended procurement processes cause the early on-boarding of staff which leads to periods of low utilization and lack of optimized efforts

• Address budgets iteratively or at the portfolio level to allow for improved fluidity of funding between projects

• Use precise procurements for defined tasks

• Seek extensive skill sets, communication skills and agile experience for less defined tasks

Schedule & Cost Estimation

• Schedule and cost estimation are the most flawed aspects of software development and the most common reason for perceived project failures

• Attempt to make iterative estimates to the greatest extent possible

• Establish consistency in scope and requirements between projects to build historical data that can be used effectively

Requirements • A solid foundation of scope and broad system requirements are not typically established prior to development causing rework

• Staff had difficulty identifying which requirements/issues must be addressed immediately and those that do are not

• Identify scope and broad stroke implementation requirements upfront

• Build detailed requirements according to business priority, common understanding and technical feasibility

• Requirements are not Yes or No, they are Now or Maybe Later

Development • Skill sets may be less broad than the optimal agile developer and lead to bigger teams than usual

• Requirements are not always broken enough to allow for development while unknowns and dependencies exist

• When teams are large focus communications around business context rather than status updates

• Paired programming can prevent dependencies on individuals, provide developers a broader understanding of their efforts and build skill sets


Process Challenge Solution

Testing • Having testers that are distinct from other project roles increases the team size and complexity of communication

• Testers are not used to upfront interaction with developers

• Lengthy test plans and post development testing often cause sprints to become a series of short waterfall cycles

• Small user stories will drive test based environments

• Requirements analysts, product owners and functional staff should be actively involved in testing

Deployment • Lengthy and expensive deployment processes force waterfall delivery and minimize the collection of essential iterative feedback

• Incorporate mandatory deployment testing during development

• Identify external testing groups to use the product in testing environments if frequent deployments are not possible

• Make sure deployment testing environments match production environments

Table 2: Challenges and Recommended Solutions

PANEL DISCUSSIONThe presenters of the Agile + EVM practice symposium were part of a panel discussion at the end of all the sessions in this track. Some of the questions that came from attendees in the room as well as from the virtual transmission are listed below:

Question 1: How is performance taken? Panel Response: The story points are actually the work done or accomplished. At the end of a sprint if the capability is not delivered you will not be able to claim 100% complete. Therefore the work planned and not delivered may drive a cost variance and definitely a schedule variance. The percent complete can be established against the dollar value of the story point to determine the Quantifiable Backup Data (QBD).

Question 2: Explain how the story point becomes part of the calculation. Panel Response: For the story point to become part of the calculation and become a QBD the team needs an objective way to measure performance; the story points equals in-scope tasks. Each story point becomes an assessment of percent complete either 0% or 100% complete. A task contains multiple story points that rolls up to the percent complete of the task at the summary level.

Question 3: What do you base the Basis of Estimate (BOE) on?Panel Response: The BOE is based on flexibility to meet the timeline. Document assumptions to provide the ability to drive to the minimal viable product.

Question 4: Where is the work package (WP) defined at the sprint or release level?Panel Response: The work package is defined at the higher level to encourage the Agile processes. A sprint is a time-boxed event that has content and scope. The release has defined scope

Question 5: Compare your risk management approach between the Traditional versus Agile methodology. Panel Response: Agile keeps the risk process in place and reduces the risks earlier on, therefore you may not have the need for as much management reserve as you would using the traditional approach.

Question 6: If story points are used to measure performance how is the cost per story point determined to establish the budget? Panel Response: One option would be to not create any cost estimates associated with story points. The other option is to work with the team to determine the number of hours associated with the story point to estimate the dollar value. As the team develops their velocity or units of work completed in a certain interval they can use the velocity as a metric to track the actual costs in dollars or hours against the planned costs.


CONCLUSION

Question 7: Is an Integrated Baseline Review (IBR) on an Agile program handled differently than it is on a traditi0nal waterfall program?Panel Response: It is primarily handled the same. From experience shared by one member of the panel, the only sidebar was with the software developers on their processes with the Earned Value Management System (EVMS) Director.

There is a move to adopt Agile in federal agencies with the directives from OMB with regard to modular development and the updated DoD acquisition guidance addressing the use of Agile methods. The Agile + EVM practice symposiums provide an excellent resource to learn about the topics. Please visit the College of Performance Management EVM library for more information.

FOOTNOTES1. Agile Manefesto at www.agilemanifesto.org2. OMB Contracting Guidance to Support Modular Development www.whitehouse.gov/sites/default/

files/omb/procurement/guidance/modular-approaches-for-information-technology.pdf3. Zac Gery,” Reflections in Scrum’s Silver Mirror”, 16 June 2012. www.codeproject.com/

Articles/494380/ReflectionsplusinplusScrum-27splusSilverplusMirror4. 7th Annual State of Agile Development Survey: www.versionone.com/pdf/7th-Annual-State-of-

Agile-Development-Survey.pdf.5. Source: Copyright 2008-2010 Mitch Lacey www.mitchlacey.com6. A New Approach for Delivering Information Technology Capabilities in the Department of Defense.

OSD 13744 -10 804 Report to Congress.7. Software Development: Effective Practices and Federal Challenges in Applying Agile Methods,

GAO-12-681, Jul 27, 2012. www.gao.gov/assets/600/593091.pdf

ABOUT THE AUTHORShobha Mahabir is a consultant with PwC one of the top consulting firms in the USA. She served as a volunteer at EVM World 2013. Ms. Mahabir holds a BA in Economics from the University of Maryland, College Park and a MS in Information Systems Technology from the George Washington University. Ms. Mahabir is a Certified Project Management Professional and Earned Value Professional, and holds certifications in ITIL Foundations and Intermediate. She has contributed on the development of the GAO Cost Estimating and Assessment Guide. Ms. Mahabir has comprehensive experience in technical and project management, possessing more than 15 years of combined IT and project management related work experience. Her experience includes earned value management, capital planning and investment control, scheduling, cost estimating and budgeting, resource planning, project management, contract administration, and design, operation, and maintenance of databases. Ms. Mahabir has also implemented and delivered EVM on multiple large federal IT programs.

Get Involved In CPM:Become A VolunteerMost people agree that the value of membership in any organization is greatly impacted by one’s involvement in that organization. CPM has many opportunities for you to become engaged in your organization.

You can help with our certificate program, become a leader in our two conferences each year or other volunteer opportunities as they appear on the website. Take the time to read each of the job descriptions and ask questions of the point of contact for each if you are not sure what is expected. PDU’s are available for volunteering in CPM.

These opportunities are listed on our website:mycpm.org./ABOUT US/VOLUNTEERS.

Any questions can be directed to: Lauren Bone, Vice-President Administration [email protected] / UK+44(0)7766974063 or Gaile Argiro, CPM Executive Administrator, [email protected] / 703-370-7885.

IPM 2013

I EAA International Cost Estimatingand Analysis Association

Sponsored by…

Plan now for...

The PMP, PMI and R.E.P., and the Registered Education Provider logo are Registered marks of the Project Management Institute, Inc.

The College of Performance Management is a PMI Registered Education Provider

Setting the Baseline

Erin Whittaker • [email protected] 703.938.5090 • fax 703.938.5091 www.iceaaonline.org

Learn from… • Special Guest Speakers • Professional Education Training Seminars • Tools Track Presentations • Topical Workshops • Practice Symposia

For program information…

Socialize and network at… • Newcomers’ Orientation • Speakers-only Reception • All-attendee Reception

Gaile Argiro • [email protected] 703.370.7885 • fax 703.370.1757 www.mycpm.org

For exhibiting information…

The premier conference on Earned Value Management

25th Annual International Integrated Program Management Conference November 18–20, 2013Bethesda North Marriott Hotel & Conference Center • Bethesda, MDLook for more information on www.mycpm.org

THE MEASURABLE NEWS


2013.03

35

OVERVIEW In this tenth year of Earned Schedule (ES), the EVM World 2013 conference had a presentation track for the topic. From my observation, having the ES track was very timely and the presentations exceptional. They covered a wide range of interest - practice, advancements, research, and novel connection to the Agile software development process. Of the six presentations, having a total of eight presenters, two were international – Dr. Robert Van De Velde (Canada), and Mr. Stephen Vandevoorde (Belgium). All of the presentations were well attended, indicating growing interest in ES.

The track began with my presentation, “Advancing Project Duration Forecasting.” A theory was put forth having potential to improve duration forecasting. This was followed by : “Do You Trust Your IMS?,” presented by Yancy Qualls and Alexander Schostag, addressing methods of duration forecasting used at Bell Helicopter. The first day of the ES track concluded with

“AgileES – Earned Schedule for Agile Projects.” This presentation was made by Dr. Robert Van De Velde. He intriguingly demonstrated the equivalence of Agile and ES metrics.

Day two of the track began with USAF Captain Kevin Crumrine’s presentation, “Earned Schedule: Utility in Major U.S. Air Force Acquisition Programs.” Captain Crumrine described his research, which indicates ES forecasting performed well for major defense acquisition projects. Next, “Time Forecasting Using Earned Value/Earned Schedule Methods – the European Experience,” was presented by Stephen Vandevoorde. He described the advancements made through research and the employment of ES. The last presentation was by Dr. Gary Richardson and Saranya Lakshmikanthan, “An Analytical Utility for Earned Value and Earned Schedule Management.” The theme of this presentation was to improve the interpretation of EVM and ES through greater understanding of various aspects of project information.

Finally, from the feedback analysis, the presentations were incredibly well received. For those who did not attend these sessions or the conference, I highly recommend accessing these very informative presentations using the CPM archives.

EVM WORLD 2013 – THE EARNED SCHEDULE TRACKBy Walt Lipke, Track CoordinatorIPM 2013

I EAA International Cost Estimatingand Analysis Association

Sponsored by…

Plan now for...

The PMP, PMI and R.E.P., and the Registered Education Provider logo are Registered marks of the Project Management Institute, Inc.

The College of Performance Management is a PMI Registered Education Provider

Setting the Baseline

Erin Whittaker • [email protected] 703.938.5090 • fax 703.938.5091 www.iceaaonline.org

Learn from… • Special Guest Speakers • Professional Education Training Seminars • Tools Track Presentations • Topical Workshops • Practice Symposia

For program information…

Socialize and network at… • Newcomers’ Orientation • Speakers-only Reception • All-attendee Reception

Gaile Argiro • [email protected] 703.370.7885 • fax 703.370.1757 www.mycpm.org

For exhibiting information…

The premier conference on Earned Value Management

25th Annual International Integrated Program Management Conference November 18–20, 2013Bethesda North Marriott Hotel & Conference Center • Bethesda, MDLook for more information on www.mycpm.org

EVMLibrary.org: The CPM Online Library

A comprehensive, online library of resources about Earned Value Management (EVM) and related project management topics.The CPM library now contains well over 2,100 documents, such as articles and presentations, and is growing everyday. Each document has been catalogued by subject area, author, publication date, keywords, etc.

How You Can Contribute? Do you have topical articles, speeches, or presentations? Contact Gaile Argiro at [email protected] to have your contributions included in this online resource.


website: www.CBTWorkshop.com phone: 888•644•5613

Let The CBT Workshop Assist You!

Staff Augmentation

Tools Implementation and Training

Skilled EVMS Planners, Schedulers, Program Control Analysts, Estimators, and EVMS Consultants

Short or Long Term Assignments Contract-to-hire personnel

We have EVMS tools experts who can implement, train, integrate, and migrate EVMS data in the following tools:

Microsoft Project and Project Server Deltek MPM, wInsight, Cobra, Open Plan Primavera

Everyone is Looking for Skilled EVMS Resources…We Have Them

EVMS Services

Web Based EVMS Training

EVMS Gap Analysis System Description Design and Development Desktop Operating Procedures IBR Preparation and Dispositioning CAM Training and Mock Interviews

EVMS Fundamentals and EVMS Advanced training, testing, and certification.

Call UsToday!

THE MEASURABLE NEWS


2013.03

37

INTRODUCTION The successful 2012 Australian initiative delivered a keynote address to the 2012 PMOZ Conference by Professor Pierre Bonnal from CERN, Geneva Switzerland with supporting presentations by Ms. Lisa Wolf from the USA as well as presentations to the PMI Sydney and Canberra Chapters by Professor Bonnal.

PMGlobal Project Governance and Controls Symposium, Canberra, AustraliaThese outcomes provided a platform of interest which enabled PMGlobal (an Australian not for profit foundation which aims to advance project management) to re-establish a one day Project Governance and Controls Symposium which was held in Canberra on the 10th April 2013.

Highlights of the Symposium included:• Opening Keynote Address by Rear Admiral Michael van Balen RAN, Deputy Chief of the

Royal Australian Navy (on behalf of the Chief of Navy);• Closing Keynote Address by Mr. Gary Troop, President of CPM; and • Remote Plenary Speaker, Mr. Walt Lipke the inventor of the Earned Schedule extension

to Earned Value Management. This was to acknowledge the 10th anniversary of the publication of the seminal paper, “Schedule is Different” in the Measurable News in 2003.

All keynote presentations were very well received, particularly the international perspective on developments in project controls and EVM presented by Gary Troop for CPM. The quality of presentations and re-establishing the Symposium in Canberra after a 6 year break received very positive comment by attendees. In more positive signs for the future, the Symposium was also attended by delegates in leadership roles from both PMI (including PMI Canberra Chapter President) and the Australian Institute of Project Management. In addition to demonstrating CPM’s ability to collaborate with PMI and other professional bodies these developments have provided opportunities for future collaboration.

Another positive outcome from the Symposium was that interest was expressed in the creation of a CPM Australia Chapter with a follow up request for Guidelines on Chapter formation received.

2014 Symposium Hosted by the Australian Defence Force AcademyDue to the very senior level of local support provided to the Symposium in conjunction with the strong international presence, the University of New South Wales at Canberra (UNSW @ Canberra) has agreed to host the 2014 Symposium which will be held on 6-7 May 2014.

UNSW @ Canberra which is located at the Australian Defence Force Academy (ADFA) will be the site of the 2014 Symposium. UNSW Canberra has provided high quality undergraduate and postgraduate education to the Australian Defence Force at ADFA since the Academy opened in 1986. Undergraduate students are mostly all midshipmen of the Royal Australian Navy and officer cadets of the Australian Regular Army and the Royal Australian Air Force. More details are available at http://info.unsw.adfa.edu.au/index.html.

PMI Australia National Conference As an independent initiative which also arose from the 2012 events, Ms. Lisa Wolf from Booz Allen Hamilton (BAH) in the Washington DC area was invited to keynote at the inaugural PMI Australia (PMI AUS) National Conference which was held at the Sydney Convention Centre, Darling Harbor in Sydney on 1-3 May 2013. The PMI AUS Conference is a joint initiative of the six Australian PMI Chapters. Lisa’s very well received presentation was

“Lessons Learned from the Trenches of Project Surveillance” covered her and Booz Allen Hamilton’s experiences of project surveillance on US Government agency programs.

CPM GLOBAL INITIATIVES CONTINUE TO EVOLVEBy Kym Henderson, Vice President of Research and Standards


While funded by PMI AUS this invitation provide the opportunity for Ms. Michelle Jones’s from BAH to attend the Conference where she presented on “EAC Maturity” as well as present to a very appreciative PMI Canberra Chapter “full house” audience on the Booz Allen Hamilton (and National Reconnaissance Office) experience with Earned Schedule.

EVM Europe Conference The EVM Europe Conference also continues to evolve with the EVM Europe 5 conference being held at Ghent University in Belgium on the rescheduled dates of 3-4 December 2013 which avoids the USA Thanksgiving period. In addition to celebrating the milestone of the 5th Conference, it also planning to recognize the 10th anniversary of the publication of the Earned Schedule seminal paper “Schedule is Different” by Walt Lipke in 2003, the outcomes of the first year of the €1.2 million 6 project controls/EVM research project will be presented at this Conference. Details are available from the EVM Europe website at www.evm-europe.eu.

Future DevelopmentsAt the time of writing this article discussions have commenced with the EVM Europe Association on the drafting of a Memorandum of Understanding (MoU) with CPM which will advance the objectives of both organizations and in particular the CPM mission of advancing EVM and other project performance management techniques globally. Similar discussions have commenced with PMGlobal in Australia who are open to assisting with establishing a CPM Chapter in Australia which focuses on meeting the needs of the project control community.

Conclusion While it is still early, the global initiatives undertaken by CPM in Europe and Australia have demonstrated considerable success and many positive signs for future developments. The initiatives have also demonstrated that CPM’s expertise in EVM is respected and we are able to work positively and collaboratively with PMI Chapters on initiatives of mutual benefit to our respective organizations.


2013 PMGLOBAL GOVERNANCE AND CONTROLS SYMPOSIUM HIGHLIGHTS

(L) Opening Keynote AddressRear Admiral Michael van Balen RAN, Deputy Chief of the Royal Australian Navy (on behalf of Chief of Navy)

(R) Remote Plenary Speaker Mr. Walt Lipke, Inventor of the Earned Schedule extension to Earned Value Management

(L) Closing Keynote AddressMr. Gary Troop, President, College of Performance Management

(R) Mr. Peter ColquhounEngineering Service Manager Munitions Branch, DMO Peter’s title slide is of a spectacular controlled demolition of obsolete Explosive Ordnance at Woomera in South Australia

(L) Ms. Lisa WolfMaking a point during her keynote presentation at the inaugural PMI Australia National Conference

(R1) Ms. Michelle JonesDiscussing some of the more technical points about EVM with one of the delegates at the PMI Australia National Conference

(R2) Ms. Michelle JonesPresenting to the PMI Canberra Chapter

THE MEASURABLE NEWS


mycpm.org



VENDORS & SERVICES

P.O. Box 20996Sarasota, FL 34276

Phone: (888) 644-5613

www.CBTWorkshop.com

see our ad on p. 36

Planning every step — Every step as planned™

EVPrep™ Exam Study Guide EVPrep™ Exam Prep Workshops

www.mgmt-technologies.com*Earned Value Professional

Get it now!

Got your ?*www.safranna.com

see our ad on p. 04 www.lexemstrategy.com

www.humphreys-assoc.comsee our ad on p. 26 see our ad on p. 10

www.smawins.com

Your ad could go here.

Contact us today: www.mycpm.org

THE MEASURABLE NEWS


mycpm.org

the measurable news

Documents