a development and integration framework for optimisation...

A Development and Integration Framework for Optimisation-basedEnterprise Solutions

Rodrigo Lankaites Pinheiro and Dario Landa-SilvaSchool of Computer Science, The University of Nottingham, Wollaton Road, Nottingham, NG8 1BB, U.K.

{psxrp2, dario.landasilva}@nottingham.ac.uk

Keywords: Optimisation Development Framework, Decision Support Systems, Academia-industry Collaboration,Workforce Scheduling, Routing.

Abstract: The operations research literature includes some papers describing collaborative work between researchers andindustry. However, not much literature exists that outlines methodologies to guide the development of a de-cision support module and its integration into an existing information management system. Here we describea framework to aid the collaborative development of an optimisation solution by researchers and informationsystem developers. The proposed framework also helps in theeffective integration of the information manage-ment system and the decision support module. The framework is divided into three main components: a datamodel, a data extractor and validator, and a solution visualisation and auxiliary platform. We also describeour experience and positive results from applying the proposed development and integration framework to aproject involving the development on an optimisation-based solution for workforce scheduling and optimisa-tion problems. We hope that this contribution would be particularly useful for less experienced researchers andpractitioners who embark on a collaborative development ofa decision support module based on optimisationtechniques.

1 INTRODUCTION

Decision support systems are pieces of software in-tended to aid humans in making decisions regard-ing specific problems in given domains. Over thepast few decades, research and development of suchsystems have increased considerably (Power et al.,2011). Nonetheless, due to their complexity, decisionsupport systems are mostly developed jointly by col-laborative teams. Academics often enter into R&Dprojects in partnerships with practitioners to developdecision support modules – which may consist of oneor more optimisation algorithms, simulation modules,inference engines, etc. – for solving a given problem(Fontana et al., 2006).

As with every software development environment,collaborative development of decision support sys-tems presents its difficulties. The software engineer-ing literature describes many methodologies for soft-ware development such as applying basic softwareparadigms (Pressman, 2010), or adhering to the Ra-tional Unified Process (RUP) (Kruchten, 2004), Ag-ile methodology (Martin, 2003) or Concurrent Engi-neering (CE) (Prasad, 1996). However, the generalscope of these methodologies, often fails to address

the unique challenges of R&D development.

The literature provides few works that proposegeneral methodologies to aid the development ofR&D projects (Barnes et al., 2002; Guarnaschelliet al., 2013). Some commercial tools exist to aid R&Dprojects development, such as the IBM ILOG ODMEnterprise (IBM, 2013). However, adhering to suchtools implies adopting black-box algorithms (whichmay not be in the interest of researchers), limitationsin technology (as compatibility issues arise) and oftena considerable cost.

Here we present a data-centric developmentframework to assist in the collaborative develop-ment of optimisation-based enterprise solutions. Theframework is a tool to facilitate the communicationbetween development teams and to foster progress onthe research of the optimisation problem being tack-led. Moreover, it provides a layer between the in-formation management system and the decision sup-port module being developed. Hence, the practitionerdoes not need to force the use of specific technologiesand methods and the researchers focus on the devel-opment of algorithms. We stress that the proposedframework is not intended to substitute software en-gineering methodologies and frameworks. Instead, it

233

is designed to provide both the researchers and prac-titioners with enough independence while still aid-ing them in the collaborative development. We alsodescribe the application of the proposed frameworkto a real scenario involving the development of anoptimisation-based enterprise solution.

The rest of this paper is structured as follows. Sec-tion 2 outlines the Workforce Scheduling and RoutingProblems Project which is used to illustrate the ap-plication of the proposed framework. Section 2 alsoreviews previous related works. Section 3 presentsthe framework while section 4 presents the obtainedresults. Section 5 provides some insights and the con-clusion of this work.

2 THE WSRP PROJECT ANDRELATED WORKS

Part of the focus in our current research is the develop-ment of algorithmic solutions for workforce schedul-ing and routing problems such as home healthcare ser-vices. Workforce scheduling and routing problems(WSRP) are a class of problems in which workers(nurses, doctors, technicians, etc.) have to be allo-cated to tasks scattered in a geographical area, henceworkers need to travel between locations. Work-ers have their own skills, availability, preferences,etc. while tasks have required skills, associated clientpreferences, etc. This class of problems combinesfeatures from scheduling problems and routing prob-lems. For an overview of WSRP please refer toCastillo-Salazar et al. (2012).

We are engaged in a R&D project in collabora-tion with an industrial partner in order to develop theoptimisation engine for tackling large WSRP scenar-ios. The existing information system collects all theproblem-related data and provides an interface to as-sist human decision makers in the process of assign-ing workers to visits. We are in charge of developingthe decision support module that couples well withthe information management system being developedand maintained by the industrial partner.

The use of methodologies, frameworks and guide-lines aid the process of elaborate software develop-ment and can help to reduce risks and problems. InSection 4) we discuss in detail some of the risks andproblems that the proposed framework has helpedus to address in our project. The literature de-scribes several methodologies and guidelines to helpon the development of both decision support systems(Connors, 1992; Bui and Lee, 1999; Elgarah et al.,2002) and information management systems (Prasad,1996; Martin, 2003; Kruchten, 2004). However, only

few works refer specifically to R&D projects andin our opinion, even fewer could be applicable tooptimisation-based problems. We highlight the fol-lowing ones:

• Several works focus on the management aspectsof R&D projects. They discuss the advantagesof a collaborative project between researchers andpractitioners, how to assess such partnerships andhow to evaluate the benefits. Among them, we canlist the works of Balachandra and Friar (1997),Huchzermeier and Loch (2001), Santiago and Bi-fano (2005) and Fontana et al. (2006).

• There are some case studies of R&D projects thataim to identify characteristics of the software de-velopment process in such R&D projects to thenpropose a few general guidelines. Among theseworks we can highlight those by Pinto and Covin(1989), and Pillai et al. (2002).

• Barnes et al. (2002) presents a general good prac-tice model for R&D projects, but not a methodol-ogy or framework. Hence, we recommend to ad-here to their model while applying our framework,as both methodologies can be fully integrated.Their partner evaluation factors and university-industry issues practises can be addressed prior tothe application of our framework. Our data mod-elling contributes to fulfill several of their projectmanagement topics, such as the clearly definedobjectives and responsibilities, realistic aims, col-laborative agreement, progress monitoring and ef-fective communication. Besides,our data modelhelps in managing the cultural gap issue and ourproposed solution visualisation and auxiliary plat-form aids with the outcome factors they defined.

• Guarnaschelli et al. (2013) proposed a generalmethodology for integrating the development ofdecision models with model driven software de-velopment. They divide the methodology in fourparts, namely the business modelling, the decisionproblem definition, the service modelling and thereference modelling. The framework proposed inthis paper can be integrated with their methodol-ogy since the elements we propose match the el-ements proposed in their suggested timeline. Ourproposed data modelling starts after their sug-gested business modelling and progresses throughthe decision model identification and specifica-tion. Our data extractor and validator contributesto their decision problem realisation and the do-main elements reference modelling. Finally oursolution visualisation and auxiliary platform aidswith the implementation step.

ICORES�2014�-�International�Conference�on�Operations�Research�and�Enterprise�Systems

234

Figure 1: Overview of the development and integration framework.

3 THE PROPOSEDFRAMEWORK

The framework consists of three components to helpwith the communication of teams, the understandingof the problem being tackled and the integration of thealgorithmic solution. The first component is a datamodel understandable to both researchers and practi-tioners. The second component is a data extractionand validation mechanism. The third component is asolution visualisation and auxiliary platform. Figure1 presents an overall diagram of the framework. Alto-gether, these elements create a layer between the prac-titioners and the researchers to assist them in achiev-ing a common understanding of the problem and ob-taining a higher communication level.

3.1 First Component: Data Model

The data model component applies data-centric de-velopment concepts (Bhattacharya et al., 2009; Cohnand Hull, 2009) to optimisation scenarios. The datamodel is a data abstraction in a file format capableof representing an instance of the optimisation prob-lem. It serves as a layer between details of the prob-lem domain and details of the algorithmic solution sothat those building the model approach the problemfrom their own perspective. The purpose of the datamodel building process is that both practitioners and

researchers work in collaboration to create the file for-mat and improve mutual understanding.

3.1.1 Modelling the Data

The data can be modelled into a set of files or a singlefile, depending on the characteristics of the optimisa-tion problem. Ideally, the set of files contains all theinformation needed to run the solving algorithms. Werecommend to have in each dataset all the informa-tion required to solve one problem instance instead ofhaving data common to several instances in one file.This is because the time required for executing the op-timisation algorithms is usually higher than the timerequired for data processing. More importantly, hav-ing all the data for a problem instance available in thisway helps researchers to better understand the prob-lem instance.

The WSRP scenario used to illustrate the frame-work in this paper contains a large dataset, hence wedecided to split an instance into six files, each onecontaining a list of some sort. Figure 2 presents asimplified version of the dataset. Each file contains alist of the element that defines the file. Also, each fileis linked to the mathematical model in order to showthe correlation between the optimisation problem andthe data files.

The use of a data model at the centre of the de-velopment process helps to improve the communi-cation between researchers and practitioners (Cohn

A�Development�and�Integration�Framework�for�Optimisation-based�Enterprise�Solutions

235

Figure 2: The files set for the WSRP project. Each file containsa list of the named elements and each element contains severalfields related to the optimisation problem.

and Hull, 2009). The data files should preferably behuman-readable and as self-explanatory as possible.This also allows researchers to easily alter the data inorder to generate different test cases.

When developing the framework we opted touse eXtensible Markup Language (XML) (Haroldand Means, 2009) as the file format for the model.We have chosen XML because it is easily readable,XML parsers are widely available, and XML offers aschema model. Such schema allows the creation ofXML files that model other XML files and files at-tached to XML schemas are easily validated. Also, inthe case of our WSRP project, both researchers andpractitioners were already familiar with XML. How-ever, an alternative is to use YAML (2013), whichis clearer and more human-readable than XML. Be-sides, several YAML libraries are available for themost common programming languages.

Like in any other software development process,the development of a decision support system maybe subject to many changes during the development.This must be considered when modelling the files, forexample, by using required and optional fields. Therequired fields should be used for very basic data thatmust exist in order to validate the data for a probleminstance. The optional fields are used to capture sec-ondary constraints and objectives such as preferences.

In the modelling stage, the format used to rep-resent a solution should help researchers to verify

that the solution is valid, i.e. meets the problemconstraints and additional requirements. This alsohelps practitioners and researchers to understand eachother early in the process, on what constitutes a good-quality solution.

The final data model obtained is a layer betweenresearchers and practitioners. This means that practi-tioners can work on the information management sys-tem using the data model to interact with the decisionsupport module (even though it may not have been de-veloped yet). Similarly, researchers can work on thealgorithmic techniques without any specific knowl-edge of the information management system. Fur-thermore, this layer helps to ensure that any solutionmethodology developed by the academic team canbe incorporated into the company’s information sys-tem by only handling the corresponding model files.Therefore, the data model plays a crucial role whenintegrating the company’s information system and thedeveloped optimisation algorithms.

3.2 Second Component: DataExtraction and Validation

The sooner the academic team have access to data inthe model format, the sooner research on the solv-ing algorithms can begin. Then, sample datasets or amechanism to efficiently assist in extracting the datawould be useful. Figure 3 shows the schematic for


236

the data extractor and validator developed as part ofthe framework. The figure also shows the interactionof this component with the database in order to gen-erate the file set in the model format.

3.2.1 Data Extractor

The data extractor is the tool that retrieves the datafrom the information system and converts it to a setof files according to the data model. This data extrac-tor plays an important role in the development of thedecision support module. This is because a typical in-formation system is likely to contain a large amountof data that is not needed for the optimisation prob-lem and that should be filtered. Clearly the modelitself helps to eliminate some of such data, but as ex-pressed above, it is highly desirable that each datasetonly contains the data required for a single probleminstance. In our WSRP project, we only extract thevisits/jobs for a given planning period. This helps toeliminate the processing of unnecessary data and im-prove the overall system’s performance. Filtering thedata at the retrieval stage is usually easier, more reli-able and more efficient, specially in a SQL server en-vironment. Figure 3 outlines this process of extractingand collecting raw data, sending it to the data valida-tor, retrieving the validated data and finally generatingthe corresponding file set in the model format.

Figure 3: The data extractor and validator.

3.2.2 Data Validator

After the data extraction, data should be validatedto account for missing and inaccurate entries in thedatabase plus other considerations. In the databasefor our WSRP project, this means for example, fil-tering out staff members that are not available in theplanning period considered, or those whose skills arenot appropriate for the given scenario, or those onlyavailable to work on different geographical locations.We also filter out data about locations and skills notused in the given scenario. Hence, the first step ofthe data validator is to trim data as much as possibleand ensure that data references broken as result of thisdata trimming are also fixed.

Moreover the validator also checks for inconsis-tent data, such as mistyped postcodes, which if notdetected can negatively affect the performance of thesolving algorithm. For example, the data validatorfor the WSRP project checks for wrong postcodes bycomparing them to the postcodes of other locationsin the same geographical area. It also checks for anynumerical value that deviates too much from the ex-pected range of values. Figure 3 presents the proposedflow diagram for the data validator. The extractor andvalidator provides researchers with problem instancesto assist the algorithm development, while also pro-vides practitioners with a guiding mechanism to con-tinue the data gathering and preparation.

3.3 Third Component: SolutionVisualisation and AuxiliaryPlatform

The third component of the framework is a solutionvisualisation and auxiliary platform, a feature that isalmost entirely designed to help researchers in thedevelopment. There are many information systemdevelopment methodologies well consolidated in thesoftware engineering literature. However, these areusally not fully applicable to the development of de-cision support systems, for which well establishedmethodologies and frameworks are not easily identi-fied in the literature.

This third component aims to be a communica-tion layer between researchers and later between re-searchers and practitioners. Figure 4 presents a di-agram to illustrate how the visualisation and auxil-iary platform interacts with the solver and the actorsin the scenario, the company and the academic team.The solutions generated by the algorithmic solver canbe stored into the repository, can be displayed in thevisualisation platform and can be compared to otherstored solutions. Through this visualisation platform,


237

Figure 4: The visualisation and auxiliary platform.

researchers and practitioners can have feedback fromthe solver and can evaluate the progress in the devel-opment of the solving algorithm.

3.3.1 Solution Visualiser

In an algorithm development team where each mem-ber works in a semi-independent manner with its ownknowledge, tools and techniques, having a commonsolution visualiser can help to conduct a fair com-parison between the different solution methods devel-oped. Moreover, a mean to visualise solutions is auseful tool for algorithm developers that can help tosave development time. Furthermore, having a com-mon solution visualiser allows to present results ob-tained by different optimisation algorithms in a stan-dardised format for practitioners to evaluate. There-fore, the visualiser is also a communication asset forthe company and a tool by which researchers canpresent progress in their development.

Ideally the solution visualiser should be platform-free (able to run on any operating system),programming-free (not dependant of a specific pro-gramming language) and based just on the modelledformat (not require extra files, configuration or pa-rameters). To achieve these goals we can resort tosome options which are a standalone application, a

network-based application or a web-application. It isclear that a web-based environment presents many ad-vantages like easy-access, good presentation and flex-ibility. The decision usually depends on the nature ofthe problem and preference of the development team.Furthermore, web-based and network applications arepreferred since they make it easy to integrate with theauxiliary platform. In our scenario, we opted to de-velop a web-based visualisation platform.

3.3.2 Auxiliary Platform

The auxiliary platform is another tool specially con-ceived to aid on the research of the optimisation tech-niques. It consists of a repository of problem in-stances and solutions, plus an automated comparatorof solutions. To compare solutions is fairly easy forsingle-objective problems in which the optimisationobjective is a single scalar value. However, researchon multi-objective approaches has expanded consid-erably and it is now common to apply such approachto real world problems, thus making the comparisonbetween solutions more difficult. The specifics of thecomparison are subject to the problem nature and towhat the decision makers aim to achieve in a solu-tion. The auxiliary platform should preferably pro-vide charts showing strong and weak points of thesolution as well as a detailed table containing usefulinformation. In our WSRP project, the solution com-parator is able to handle a pair of solutions or two setsof solutions. For the WSRP we display the distancetravelled by the workers, the distance time, the travelcost, the payroll cost, and preferences. Also, an inte-gration between the visualiser, the comparator and therepository is highly desirable since the usability of thetools escalates.

4 RESULTS

In this section we present the results obtained by ap-plying the proposed framework to the WSRP project.We highlight the problems found during the develop-ment process and how applying the framework helpedus to tackle these problems.

4.1 Better Collaboration betweenResearchers and Practitioners

Communication Issues. Our experience showed thatcommunication between the actors of the scenariocan potentially be problematic. Information, require-ments and expectations can be different from theperspective of researchers and practitioners, leading


238

to misunderstandings and misconceptions. The datamodel building process played a major role in the re-searchers’ understanding of the optimisation problemdetails. During the process we were able to identifygaps in our conceptualisation and fill them during aphase where changes caused minimal drawbacks. Wewere able to identify more constraints for the optimi-sation problem as well as alter some features to bettersuit the needs of the practitioner.

Knowledge Background Issues. There is obvi-ously difference between the background knowledgeof the company developers and that of the academicteam. Moreover, researchers in the team may havedifferent expertise including maths, management sci-ence, computer science, statistics, etc. This varietycould lead to some difficulties such as different in-terpretation of problem data, unfamiliarity with cer-tain technology and different assessment of results.With the data model we created a structured set offiles in which researchers were able to easily browsethe problem data without knowing any specific tech-nology employed on the information system. Be-sides, the visualisation platform proved to be a valu-able asset to give feedback to the company. Thus,the framework aided the academics on minimising theknowledge-gap difficulties.

Integration Issues. Integrating a decision supportmodule to an information management system can bedifficult and tricky. The first issue is the problem dataretrieval. The second issue is applying the developedalgorithms to an existing information managementsystem. Integrating the solver can be difficult, spe-cially if it uses a specific tool, such as mathematicalsolvers or specific software libraries. The data modelprovided a layer between our developed algorithmsand the information system and the researchers foundthemselves in a comfortable environment where theyemployed different technologies from the ones usedby the company.

4.2 Better Research and DevelopmentProcess

Teamwork. In our example project, the academicgroup was formed by several PhD students, each oneusing the WSRP as a background for its PhD research.The application of the framework helped considerablyto improve communication among academics. Theoptimisation problem was discussed and a commonunderstanding achieved. Besides, building the toolsin the proposed framework provided a good opportu-nity for teamwork and knowledge sharing.

Rework. When developing algorithmic solutionsfor a given optimisation problem, researchers tend to

create their own versions of tools such as file models,data retrieval API’s, solution viewers and representa-tions, etc. It is rarely the case that researchers reusethese types of tools developed by other researcher. In-stead, they tend to reuse algorithms in their quest todevelop improved versions of those. In our WSRPproject, the data extractor and the solution visualisa-tion and auxiliary platform were employed by all aca-demics. Although the deployment of these frameworktools required some time, reusing these tools savedconsiderable time and effort overall by eliminating re-work. Along with the data model we also developed aJava API to handle the XML files and access the data.This API further extended the usefulness of the modelby providing the users with an easy way to access theproblem data.

Benchmarking. In real world problems, a goodbenchmark dataset to evaluate the solutions achievedduring the research stage, is often difficult to ob-tain. Sometimes it is possible to estimate some closeenough targets, but as the complexity of the prob-lem grows, such approaches become less practical.Hence, it is necessary to have a fair and clear way tobenchmark solutions to the real world scenarios, ob-tained with different algorithmic techniques. In ourWSRP project, the auxiliary platform provided theteam with a valuable asset to compare solutions morefairly and improve the overall benchmarking process.

5 CONCLUSIONS

Research and development projects involving univer-sities and companies are common in decision supportsystems development scenarios. Development teamsaround the world usually find their own solutions tosuch problems, based on experience, trial and error.However, despite the frequency of this situation, tothe best of our knowledge few specific methodolo-gies can be found in the literature to help researchersand company alike to efficiently develop decisionsupport systems in such circumstances. In particu-lar, no previous publications seem to exist proposinga methodological approach to develop optimisation-based decision-support systems.

In this paper we have presented a methodologi-cal framework to aid on the development and integra-tion of optimisation-based decision support modulesand information management systems. The proposedframework combines a collection of good practicesalong with a data-centric development. The proposeddevelopment and integration framework helps to: 1)promote communication between the research teamand the company, 2) provide an interface for the inte-


239

gration of the solving algorithms and the informationmanagement system and, 3) provide tools to developand evaluate different algorithmic solutions. More-over, the framework aims to bestow the academicteam full independence to use whatever methods andtechnologies they choose and an easy manner to inte-grate the designed algorithms to the management in-formation system.

We also described our experience on applying theframework to an ongoing project for an optimisation-based decision support system involving workforcescheduling and routing problems (WSRP). By adher-ing to the proposed framework, we were able to iden-tify several features related to the optimisation prob-lem prior to the implementation of algorithmic tech-niques. We also were able to save development timeby avoiding work being redone since the extractor,validator and visualiser tools were available for thedevelopment team. Furthermore, we were able to im-prove communication and collaboration between re-searchers and practitioners and among researchers.

REFERENCES

Balachandra, R. and Friar, J. (1997). Factors for success inramp;d projects and new product innovation: a contextualframework. Engineering Management, IEEE Transac-tions on, 44(3):276 – 287.

Barnes, T., Pashby, I., and Gibbons, A. (2002). Effectiveuniversity industry interaction:: A multi-case evaluationof collaborative r&d projects.European Manage-ment Journal, 20:272 – 285.

Bhattacharya, K., Hull, R., and Su, J. (2009). A data-centricdesign methodology for business processes. InHand-book of Research on Business Process Modeling, chapter23, pages 503 – 531.

Bui, T. and Lee, J. (1999). An agent-based frameworkfor building decision support systems.Decision SupportSystems, 25(3):225 – 237.

Castillo-Salazar, J. A., Landa-Silva, D., and Qu, R. (2012).A survey on workforce scheduling and routing problems.Practice and Theory of Automated Timetabling, pages 29– 31.

Cohn, D. and Hull, R. (2009). Business artifacts: A data-centric approach to modelling business operations andprocesses. InIEEE Data Engineering Bulletin 32.

Connors, D. T. (1992). Software development method-ologies and traditional and modern information systems.SIGSOFT Softw. Eng. Notes, 17(2):43 – 49.

Elgarah, W., Haynes, J., and Courtney, J. (2002). A dialecti-cal methodology for decision support systems design. InSystem Sciences, 2002. HICSS. Proceedings of the 35thAnnual Hawaii International Conference on.

Fontana, R., Geuna, A., and Matt, M. (2006). Factors af-fecting universityindustry r&d projects: The impor-tance of searching, screening and signalling.ResearchPolicy, 35(2):309 – 323.

Guarnaschelli, A., Chiotti, O., and Salomone, H. E. (2013).Decision models as software artifacts - bridging thebusiness-software gap for decision support systems. InICORES 2013 - Proceedings of the 2nd InternationalConference on Operations Research and Enterprise Sys-tems, Barcelona, Spain, 16-18 February, pages 86 – 95.

Harold, E. and Means, W. (2009).XML in a Nutshell. In anutshell. O’Reilly Media.

Huchzermeier, A. and Loch, C. H. (2001). Project man-agement under risk: Using the real options approach toevaluate flexibility in r&d.Manage. Sci., 47:85–101.

IBM (2013). IBM ILOG ODM Enterprise. http://www-03.ibm.com/software/products/en/ibmilogodmente/.

Kruchten, P. (2004).The Rational Unified Process: An In-troduction. The Addison-Wesley object technology se-ries. Addison-Wesley.

Martin, R. (2003). Agile Software Development: Princi-ples, Patterns, and Practices. Alan Apt Series. PrenticeHall/Pearson Education.

Pillai, A., Joshi, A., and Rao, K. (2002). Performance mea-surement of r&d projects in a multi-project, concur-rent engineering environment.International Journal ofProject Management, 20(2):165 – 177.

Pinto, J. K. and Covin, J. G. (1989). Critical factors inproject implementation: a comparison of constructionand r&d projects.Technovation, 9:49 – 62.

Power, D., Burstein, F., and Sharda, R. (2011). Reflectionson the past and future of decision support systems: Per-spective of eleven pioneers. In Schuff, D., Paradice, D.,Burstein, F., Power, D. J., and Sharda, R., editors,Deci-sion Support, volume 14 ofAnnals of Information Sys-tems, pages 25–48. Springer New York.

Prasad, B. (1996).Concurrent Engineering Fundamentals:Integrated product and process organization. Prentice-Hall international series in industrial and systems engi-neering. Prentice Hall PTR.

Pressman, R. (2010).Software Engineering: A Practi-tioner’s Approach. McGraw-Hill series in computer sci-ence. McGraw-Hill Higher Education.

Santiago, L. and Bifano, T. (2005). Management of rd projects under uncertainty: a multidimensional ap-proach to managerial flexibility.Engineering Manage-ment, IEEE Transactions on, 52(2):269–280.

YAML (2013). YAML Aint Markup Language.http://www.yaml.org.


240

a development and integration framework for optimisation...

Documents