quantifying chance of engineering success · quantifying chance of engineering success proceedings...

Proceedings of TheIRES 6th International Conference, Melbourne, Australia, 16th Aug. 2015, ISBN: 978-93-85465-75-8

116

QUANTIFYING CHANCE OF ENGINEERING SUCCESS

1MATTHEW COOK, 2JOHN P.T. MO

1,2BAE Systems Australia, RMIT University E-mail: [email protected], [email protected]

Abstract — Many decisions taken both at the early stages of large and highly complex engineering projects are injudicious due to the poor availability of information. Risky decisions are made leading to budget overruns, schedule impacts/delays, technical failures and ultimately a disappointed customer. This research creates a new method based on a generic enterprise architecture that allows quantification of chance of project success. The method is illustrated by quantifying the risk level of three engineering projects in the naval maritime environment. Index Terms— Complex engineering projects, decision making, chance of success, quantification of risk, 3PE system model. I. INTRODUCTION Risk is a combination of chance and the impact of a failure. When undertaking extensive, highly complex and challenging projects it is essential that any large organisation develops an understanding of the potential risksthat may preclude success. According to the Australian Standard (AS) for Risk Management [1], risk is ‘The effect of uncertainty on objects’. It is critical that a strategy is developed for managing the identified risks to ensure the success of the project. Many organisations face the prospect of attempting to understand, control and mitigate risk throughout a project. Unfortunately, risk is often treated as an afterthought and considered a box ticking exercise, when in fact a sound comprehension of the risks involved holds the key to understanding a project and what levels of success and failure can be achieved. How well this is achieved can make or break the project and even the organisation. Individual companies and businesses tend to develop their own bespoke methods for handling risk. In many cases this is a combination of extant tools and processes which is managed to varying degrees and levels of success. In some organisations, the use of risk management and analysis tools is conducted by dedicated risk engineers who are proficient at using these tools but don’t have detail knowledge of engineering development within the project. In many cases, risk analysis is left to the project manager and/or engineers to try their best to estimate the potential risk. While these individuals are no doubt well aware of possible risks relating to their project, the effect to other parts of the enterprise such as expertise, resources, finance, etc.can be negated. This lack of a broader, holistic enterprise view can lead to a poor demonstration of risk capture and analysis. It can be argued that engineering is essentially dealing with risk and its subsequent mitigation. The primary objective of this research is to explore and propose a risk model that can be used to both visualise and manage risks throughout the project life cycle.

In order to achieve this, an initial understanding of how risks are derived is necessary. It is generally acknowledged that there are two main groups of risk analysis techniques. The first is qualitative risk analysis. This is the process of prioritizing risks by a risk expert or a group of risk practitioners assessing and combining their perception of risk level. This is generally considered subjective. The second method is quantitative risk analysis, which is the process of numerically analysing the effect of potential risks on overall project outcomes. In addition, many of the risk management tools and software currently available are extremely good at identifying risks and other parameters but lack the versatility to manage and visualise the risks throughout the project lifecycle. Cohn [2] explored how to create a risk burndown chart such as Figure 1. Essentially, the chart is built from the probability of risk, size of loss in days which gives the number of days’ exposure to risk. The chart is created by plotting the sum of the risk exposure values.

Figure 1 - The Concept of risk burning

Figure 1presents a visual way of highlighting how differentstrategies in the use of resources, financial investment etc. can reduce or mitigate risk. At the start of the project (left hand side of the graph), the risk is high due to a lot of uncertainties. As time progresses and the project advances, more information and actual development reduce the unknowns in the project. Some risks are eliminated and/or mitigated. This lowers the level of risk level

Quantifying Chance Of Engineering Success


117

until the time when the project completes, i.e. no more risks. It may even highlight that a project is not viable for an organisation to undertake with the current requirements set. The potential development of a relationship between the risk model and a burn down chart, such as in Figure 1, offers a means of associating the identified risks with both their predicted financial and schedule impacts and what effect proposed mitigations will achieve. Unfortunately, computation of the level of risk exposure and the summation of these values is complicated and incomplete. This paper addresses the gap for a quantitative risk model based on a holistic system approach that can identify risk and develop a risk profile that can be presented in a visual format, manage and track residual risks throughout a projects life cycle. II. LITERATURE REVIEW A. Risk concepts In a large engineering project the chief element of risk arises from the fact that many variables can influence the final cost and duration of the project. Every step of the process is laden with risk. In an alliance, as the different players begin to assess their contractual duties, they will try to reallocate risks to the next party. Abi-Karam [3] examined design-build in construction projects and identified the following risks that should be addressed in large engineering projects: Proposal risk Price risk Schedule risk Performance risk Contractual risk Surety/liability risk

However, the research did not attempt to define the exposure values quantitatively. With consideration to reliability, availability, maintainability and supportability (RAMS) Barabadi et al [4] claimed that product issues and failures could be reduced and their consequences could be minimised. The authors highlight the use of analysis tools such as Failure Mode Effects and Critical Analysis (FMECA), Fault Tree analysis (FTA) and event Tree Analysis (ETA) as ways to apply risk analysis to RAMS. They also discussed the idea of the Gated Model. By passing through checks or gates, and ensuring the tasks are evaluated, the project risks could be controlled and reduced. Modarres [5]presented an engineering approach to Probabilistic Risk Analysis (PRA). It emphasizes methods for comprehensive PRA studies, including techniques for risk management. The author drew upon examples from the nuclear industry and provided a best-estimate approach to explain uncertainty characterization. Ayyub [6]explained the fundamental concepts, techniques, and applications of uncertainty, risk

modelling and analysis. The computational algorithms illustrated data needs, sources, and collection. Practical use of the methods presented but required further development for the methods to be applied effectively in large scale engineering projects. Claypool et al [7] conducted surveys with 110 supply chain managers for a product in parallel with designing the new product. They found that when designing a new product, organisations should manage the risks associated with both the product design and the supply chain. They showed that companies could yield high profit if they were able to manage risks, costs and lead times. The paper highlighted the principles to reduce risk in the supply chain but failed to provide further recommendation on what could be done. This is an interesting area to consider as many engineering projects fail due to insufficient emphasis on ensuring a strong supply chain was in place. Assessment of the suppliers to deliver their portion in the final outcome was not possible without a proper methodology.

B. Operational Risk Profiling Process Operational risk profiling (OPR) is a global risk management process that delivers a known level of residual risk to a capability to achieve its operational objectives [9]. The primary goal is to link engineering to operations for cost effective management of defence assets based on critical and residual risk. Ultimately, the mitigation for risks identified will determine the total cost of ownership which can be accepted by the operational function based on the available budget. The development of an OPR process can be seen in Figure 2. A project like Royal Australian Navy (RAN) ANZAC frigate Anti-ship Missile Defence (ASMD) programme is very challenging and complex. Yim et al [8] analysed the impact of the failure of projects and attempted to identify key risk indicators. The aim is to develop an understanding of the impact so as to enable project managers to prepare for the risks early in the lifecycle of a project based on complexity of the project and to subsequently be able to initiate effective mitigation. The paper offers some interesting methodology for obtaining data and how they relate to different complexities of engineering projects. Davis et al [9] found that operational risk management was one of the outstanding action items on most firms’ to-do lists. They concluded that companies should invest in Key Risk Indicator (KRI) which could help improve the ability to convey risk appetite, optimise risk and return, and improve the likelihood of achieving primary business goals. Cornalba and Giudici [10] studied the risks to which a banking organization could be subject. They presented possible approaches based on Bayesian networks to measure and predict operational risks. These researches show that the ability of modelling



118

and quantifying risks is fundamental to managing risks and have been attempted in many industry sectors, but there is virtually no development of such techniques in the complex engineering systems

project sector. This paper approaches the problem from the whole of system perspective in order to develop a workable model for representing the risks in operations.

Figure 2 - OPR development process

C. Assessing Capability of Engineering Enterprise In very large and complex projects, the sheer number of risks that can be generated can quickly become overwhelming and extremely challenging to manage. It is essential to provide some form of logical categorisation that can both identify and simplify the space in which a risk resides. Due to the subjectivity (authoring based) nature of risk assessment, both in the qualitative and quantitative respect, determining a true value for risk becomes very challenging. A brainstorming session can produce one set of views/values which can be radically different from an identical session held with the same people a week later (emotion, time of day, fatigue all play a role here). Such sessions are also massively influenced by the participants involved and their values/motivations. For example, a project manager will have significantly differing views on the risks surrounding items like schedule and cost. Compare this to an Engineer who is primarily concerned with risks relating to technical aspects and achieving qualification requirements. There clearly exists a need to establish some form of baseline that sets the precedent for what is considered a very risky project verses a project where the risk is considered favourable/acceptable. To achieve some form of initial baseline, two strategies were developed, the first was to generate basic risks that could be applicable to the majority of projects coming through the door at BAE Systems – Maritime. The second was to focus on three projects currently within the scope of BAE Systems – Maritime. Details of these projects can be found in section 0. The theory being that one of the projects has been completed and was considered a success, the two other projects are currently being progressed by BAE Systems and are

very different in nature. It should therefore be possible to apply a risk model/analysis to all three projects and develop some meaningful outcomes. Mo [11] describes the use of applying performance measures to assets and support services. Professor Mo goes on to detail a number of calculations that provide both cost and availability measures. This model appears to be somewhat relevant to this project research by modifying and potentially using to assess project risk. The calculations also provide indicators of where a company should increase capacity, effort and expenditure to reduce or mitigate risk. One current method is known as the 3P model, which categories risks into three groups - Product, Process and People. See Figure 3.

PEOPLE

PRODUCT

ENVIRONMENT

PROCESS

Cultural, Human reliability, Training, Health and Safety, Leadership

Change over time, expanding services, renewal, change of usage patterns, social influences

Systems Engineering,

Operations, Project management

Fundamental Engineering Sciences

Figure 3 - 3PE whole of systems approach

As part of the basis for this project, the 3P in the 3PE (environment) model has been used as a fundamental way of categorising the risks that are explored as part of this research.



119

To generate quantitative data that could be subjected to analysis, the established baseline risks were developed into a survey which was promulgated to BAE Systems staff. The results of the survey were then used as a basis for analysis and to explore a risk model that could be used to visualise and manage risks through a project life cycle III. MODELLING PRINCIPLES This research aims to remove the subjectivity of risk assessment and define a baseline or ‘Ideal’ project that is based on 50% probability of success. Risk analysis can then be conducted on new projects in a similar manner and compared to this ‘Ideal’ project to assess what ‘percentage of success’ is possible. This will subsequently allow an organisation to assess whether this is acceptable and what strategy/approach can be taken to improve the percentage of success if necessary. In order to set some form of qualitative baseline

which could then be used for both quantitative assessment and analysis, an investigation into risks surrounding complex engineering projects was undertaken as part of the research. The initial object was to compile a list of risks categorised into product, process and people (3P model). Various sources were used to compile these risks including: Reviewing the risk registers of extant and

current BAE Systems projects; Reviewing standards such as ISO 31000[1]; Risk literature of which some is identified; and A brainstorming session held at BAE Systems. The complied risks were then analysed for repeats and commonality within each category. Over 150 risks were identified. To help focus the research in developing quantification methodology, 10 risks from each of the 3P categories was selected based on their generic nature and applicable to the majority of BAE Systems - Maritime projects.

Table 1 - Product, Process, People evaluation questions



120

IV. QUANTITATIVE RISK In order to ensure the survey participants were not either influenced or mislead by the identified risks. Each of the 30 risks identified above has been reworded so they can be appropriately populated into the survey. Above in Table 1 are the 30 questions that were asked for each of the BAE Systems projects in their appropriate 3P categories. For each of the questions above, it is necessary to establish a quantitative value which could be used for analysis purposes. To ensure that a good spread of data was achieved, a value or metric for each of question (risk) above was a score out of 1 to 10 see Table 2.

Table 2 - Survey metric and scoring value

definitions

In the survey, the established scoring system for each of the questions was based on participants judging the level of importance they associate with the questions content. The majority of risk analysis is achieved by authoring based (people in a room giving a view/brain-storming) methods. This applies to both the initial qualitative development/identification of potential risks and the quantitative method of assigning a numerical value to the risks highlighted. During research into data collection methods, the idea of a questionnaire/survey seemed to be the most feasible way of generating the required data. Some initial research highlighted the possibility of using either some form of survey software or an online survey. Because the survey would be promulgated to BAE Systems staff, issues with installing a software package on to not only a corporate network, but a defence organisation with high level security restrictions was determined to be unfeasible. Enquires into online survey tools (such as www.surveymonkey.com) confirmed that it was possible to forward participants a simple HTML link to the online survey which could be accessed using an internet browser. Initial research into online type survey tools highlighted that while many sites offered free access accounts, these accounts restricted access and limited what a potential survey could achieve. These limitations included: the number of questions, data export abilities and number of participants. In searching the internet and assessing different tools, a website called ‘Eval & Go’ (http://www.evalandgo.com/) was located. This online tool offered a free student account, which provided all the features of an equivalent professional account (which require a monthly access payment). The survey was distributed as group email to BAE

Systems staff and contained instructions and a HTML link to access the online software via a web browser. The survey was password locked to ensure restricted access. The data from the completed survey was downloaded in MS Excel for collating and analysis. V. PROJECTS EVALUATED The three BAE Systems projects detailed below have their own unique challenges and risks to overcome in order to achieve success. They have been chosen as sample projects for the risk modelling research conducted within this report, because they are well known within BAE Systems- Maritime and familiar to the both the Project Management and Engineering Teams. They also offer a good delta in overall financial value and variation in the risk profile. A. MH60R The RAN ANZAC class of Frigates were originally designed for the operation of the Sikorsky S-70B-2 Seahawk helicopter. However,in June 2011, the Australian Government has approved the acquisition of 24 MH-60R Seahawk ‘Romeo’ naval combat helicopters [12]. The ‘Romeo’ helicopter was chosen because it represents the best value for money for taxpayers and was the lowest risk option (Figure 4). The acquisition means that Royal Australian Navy will have the capacity to provide at least eight warships with a combat helicopter at the same time, including Anzac Class frigates. In order to safely operate the new helicopter from the ANZAC platform, a number of modification to the ships are required. This has included: Installation of new support equipment; Changes to the configuration of the hangar and

flight deck area; and Installation of new landing and taking-off

navigation equipment; This project was successfully completed on the first RAN ANZAC ship in late 2014 with a successful landing of the MH60R helicopter being achieved in early 2015. This project is considered medium size and combinedOEM equipment and BAE Systems design and installation.

Figure 4 - Helicopter MH60R Seahawk 'Romeo'



121

B. 1448 4B Phase Array In late 2005, the RAN ANZAC class frigate 1448 2B Anti-Ship Missile Defence (ASMD) programme commenced. This project was tasked with delivering an increased defensive capability to the vessels, with the installation of a newly developed phased array radar (PAR) system for target indication/tracking and mid-course guidance and target illumination of the evolved anti-ship missiles in conjunction with other sensor and combat management system upgrades. Major changes to the ship included a new mast and cupola to house the PAR which was developed by CEA Technologies of Australia (Figure 5).

Figure 5 – RAN ANZAC frigate with ASMD installation

During this programme, the highest risk related to the development by CEA Technologies of a leading-edge phased array radar performance technology, or the product. In addition to the 1448 2B ASMD PAR programme, there is now a pressing need to replace the obsolete long range capability on the RAN ANZAC class frigate. Project SEA 1448 Phase 4B – ANZAC Air Search Radar Replacement has been commenced by the Australian Government. The RAN ANZAC frigates use their air search radar to scan at long ranges for potential threats. The radar is an integral part of a modern warship and important for ensuring the safety of the vessel and other friendly ships in dangerous areas. The current RAN ANZAC radar is old and requires replacement with modern technology to maintain the robust front-line capability provided by these ships. A risk reduction phase of implementing a new technology is currently underwayand CEA Technologies are being considered todesign and developa long range PAR which will most likely be installed on top of the extant ASMD mast. This is considered a major project, with significant risk surrounding the product or new PAR system. C. New Bilge Keel (NBK) Since inception, the RAN ANZAC class frigate has suffered from fatigue cracking of their bilge keels. The origins of the faultcan potentially be linked to operating environments which have seen higher loads than originally design for. The primary function of the

bilge keel is to stabilise the ship and reduce rolling, this is important for the performance of the vessel especially one that operates a helicopter (Figure 6).

Figure 6 - Example of Bilge Keel on RAN ANZAC Frigate

BAE Systems - Maritime has been tasked with the design, manufacture and installation of a new bilge keel set for the RAN ANZAC class of frigate. The project is considered medium with risks surrounding the keel design and installation. VI. DATA ANALYSIS The survey asked questions relating to three projects within BAE Systems – Maritime, these included: MH60R, 1448 4B and Bilge Keel. For each project 30 questions were asked of the participants, with 90 questions in total for the survey. The 30 questions per project were in fact a combination of the 3P model, Product, Production and People, which were randomised for the survey to ensure participants were not influenced. This meant that for one section of the 3P model only 10 questions/answers were generated with 14 answers (participants) per question. This number is very low for a data set and makes any meaningful analysis potentially difficult. To overcome this and develop a model that could provide some useful comparisons between the data, it was assumed that the data is normally distributed. For each of the three projects, the data was separated into the 3P model categories (Product, Process and People). The calculated mean and standard deviation for each projectcan be seen inTable 3.

Table 3 - Data analysis for three projects



122

To visualize the effect of the data, a bell-curve for each of the 3P categories was generated as shown in Figure 7, Figure 8 and Figure 9. The combined 3P distribution is shown in Figure 10. From the perceived understanding of the three projects within BAE Systems - Maritime, it is generally agreed that serious challenges relating to the 1448 4B project need to be overcome and it is considered a ‘risky’ project. MH60R has been completed and considered a success, while the bilge keel task sits somewhere between the two.

Figure 7 - Product Results for 3 BAE Systems Projects

Figure 8 - Process Results for 3 BAE Systems Projects

Figure 9 - People Results for 3 BAE Systems Projects

Figure 10 - 3P Model Results for 3 BAE Systems Projects

From the graphical results, the data does indicate that in product, process and people categories the 1448 4B project sits at a higher mean value than the other two projects. This is especially noticeable in the product graph (Figure 7), which appears to confirm the perceived concerns/risks relating to the complex challenges of designing and developing a new long range phased array radar. It is also interesting to note that across all three categories the standard deviation is lower than the other two projects, indicating the spread of the data or participant’s results from the survey are closer together. From the graphs in Figure 7, Figure 8 and Figure 9, it is also interesting to note that the bilge keel project and the MH60R project achieve very similar results except for product, where MH60R has a higher mean value. This may be explained by the majority of product being purchased outside of BAE Systems, therefore relying on OEMs and suppliers. In comparison, the bilge keel is being fully designed, developed and manufactured by BAE Systems which may result in better control and therefore less perceived risk to the project. To create a complete model of how the data reads across the three projects, the 3P data sets were combined and generated into a combined graph using the same calculation process described above. See graph in Figure 10Error! Reference source not found.. As expected, the 1448 4B project has a higher mean value and therefore sits to the right of the other two projects. It is also interesting that the Bilge keel project and the MH60R project when averaged across the 3P model have almost identical risk profiles. This is an encouraging outcome from the model as the combined graph appears to concur with the theory of the projects nature. It is also interesting to note that no weighting has been applied to these results which could also be used to highlight a high level concern such as with the product in 1448 4B. Such a weighting option is something that could be explored in further work. Another option would be to determine an ideal project that can be used as a benchmark for comparison to other projects and determine a ‘percentage of success’ value.



123

VII. BENCHMARK In order to develop the risk model further, the idea of generating a percentage of success for a given project was explored. The hypothesis being that the two “successful” projects could be used to provide a benchmark for the unknown project. Deviation of the unknown project can then be assessed from the difference distribution. The probability of success of the unknown project is then given by the probability that the unknown projects capabilities are better than the “successful” projects. The combined mean and standard deviation of the two “successful” projects MH60R and the bilge keel task is given by:

If the project 1448 4B is taken as the project to be assessed, the difference distribution is calculated as:

The difference distribution can then be visualized in Figure 11.

Aggregated capability measure

Probability density

1448 4B Project capability measure requirement beyond that of a normal successful project –expect FAILURE 60.4%

1448 B Project capability measure requirement within that of a normal successful project –expect SUCCESS 39.6%

0.0 0.6095 Figure 11 - Assessing the risk of the “unknown” project

This indicates that the 1448 4B project has a 60.4% chance of worse than the success projects. This percentage can then be regarded as the risk of failure. CONCLUSION Understanding what risks exist within a large engineering project is in itself challenging due to their subjective nature and complexity. The challenge increases when attempting to quantify the identified risks and in some way manage these risks throughout the life cycle of a project. There are many extant risks analysis tools available, which offer means of identifying risk, however there seems to be a need for a tool/model that can allow the user to both manage and visualise those risks through the project life cycle.

By developing a survey based on some fairly generic risks, and applying it to three well understood projects, it offered a method of generating quantifiable data. Of the three projects chosen, one has been completed and was considered successful (a baseline), one was considered fairly mainstream and the third was considered challenging and risky. The early stage of a risk model was developed to compare the risk profile of these three projects and the initial results look promising. An attempt to identify an ideal project was proposed and a 50% success rate was set. This was used to compare the other projects against and determined a percentage of success value. While the results appear to follow the perceived nature of the three projects, the risk model is by no means conclusive as a data set of three projects is clearly inadequate.

REFERENCES

[1] ISO, “ISO 31000:2009 Risk Management - Principle

Guidelines”, International Organization for Standardization, 2009.

[2] M. Cohn, “Managing Risk on Agile Projets with the Risk Burndown Chart”, Mountain Goat Software, 2010.

[3] T. Abi-Karam, “Managing Risk in Design-Build”, AACE International Transactions, Chapter 4, Section B-6, 2001.

[4] A Barabadi, J Barabady, T Markeset,“Maintainability analysis considering time-dependent and time-independent covariates”, Reliability Engineering & System Safety, 96(1), pp.210-217

[5] M. Modarres, “Risk Analysis in Engineering: Techniques Tools and Trends”, 1st ed., ISBN: 9781574447941, Taylor & Francis, 2006.

[6] B. Ayyub, “Risk Analysis in Engineering and Economics”, 2nd ed., Chapmann & Hall/CRC, ISBN: 978-1-46-651825-4, 2014.

[7] E. Claypool, B. Norman, K.L. Needy, “Identifying Important Risk Factors in the Design for Supply Chain”, Proceedings of the 2010 Industrial Engineering Research Conference, Cancún, Mexico, pp.1-6, 5-9 June, 2010.

[8] R. Yim, J. Castaneda, T. Doolen, I. Tumer, R. Malak, “Functional Complexity Impact on Engineering Design Project Risk Indicators”, Proceedings of the 2013 Industrial and Systems Engineering Research Conference, Puerto Rico Italy, pp.979-988, 18-22 May, 2013.

[9] J. Davies, M. Finlay, T. McLenaghen, D. Wilson, "Key risk indicators–their role in operational risk management and measurement",ARM and RiskBusiness International, Prague, pp.1-32, 2006.

[10] C. Cornalba, P. Giudici, “Statistical models for operational risk management”,Physica A: Statistical Mechanics and its applications,vol.338, no.1, pp.166-172, 2004.

[11] J. Mo, “Product Services Systems and their Risks,” Asset Management Conference, 2012.

[12] R. Roscoe, “Successful completion RAN Seahawk MH60R ‘Romeo’ First Of Class Flight Trials”, Navy Daily, 10 April 2015.

quantifying chance of engineering success · quantifying chance of engineering success proceedings...

Documents