Delft University of Technology

A scenario discovery study of the impact of uncertainties in the global container transportsystem on European ports

Halim, Ronald A.; Kwakkel, Jan H.; Tavasszy, Lóránt A.

DOI10.1016/j.futures.2015.09.004Publication date2016Document VersionAccepted author manuscriptPublished inFutures: the journal of policy, planning and futures studies

Citation (APA)Halim, R. A., Kwakkel, J. H., & Tavasszy, L. A. (2016). A scenario discovery study of the impact ofuncertainties in the global container transport system on European ports. Futures: the journal of policy,planning and futures studies, 81, 148-160. https://doi.org/10.1016/j.futures.2015.09.004

Important noteTo cite this publication, please use the final published version (if applicable).Please check the document version above.

CopyrightOther than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consentof the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Takedown policyPlease contact us and provide details if you believe this document breaches copyrights.We will remove access to the work immediately and investigate your claim.

This work is downloaded from Delft University of Technology.For technical reasons the number of authors shown on this cover page is limited to a maximum of 10.

1

AscenariodiscoverystudyoftheimpactofuncertaintiesintheglobalcontainertransportsystemonEuropeanports

Theglobalcontainertransportsystemischangingquickly.Portscanbeseverelyaffectedbythesechanges;thereforeportsneedinsight intohowthesystemmightchangeandwhattheimpact of this will be on their competitive position. Given the intrinsic complexity of thecontainer transport system and the presence of a wide range of deeply uncertain factorsaffectingthesystem,weuseanexploratorymodelingapproachtostudyfuturescenariosfortheglobalcontainernetwork.Usingscenariodiscoveryandworst-casediscovery,weassessthe implications of various uncertain factors on the competitive position of the port ofRotterdam.ItisfoundthatoverallthecompetitivepositionofRotterdamisquiterobustwithrespecttothevariousuncertainfactors.Themainvulnerabilityisthequalityofthehinterlandconnections.Amodestdeteriorationofthequalityofthehinterlandconnections,resultinginincreasedtraveltime,willresultinalossofthroughputforRotterdam.

Keywords:scenariodiscovery,deepuncertainty,containershipping,globalfreightlogistics

1 IntroductionIn the past couple of decades, changes in the global container transport system havehappenedveryrapidly.Thesechangeshaveaffectedvirtuallyallactorsinthesystem,butthebiggest impact has been on maritime ports. Ports have to be adaptive and resilient inresponding to the changes in the global container shipping system. In today’s globalizedeconomy,portsneedtoensurethattheycanfunctionrobustly,bothasatransshipmentnodeand as a gateway node for global trade flows. A failure to respond to changes in a timelymanneroftenresults innegativeconsequences for theport itself,aswellas theeconomyoftheregionorcountrytowhichtheportbelongs.Forexample,therecentcongestionatportsonthewestcoastoftheUSresultedinestimateddamagesfortheeconomyofroughly7billionUSdollar[1].

Preparing aport for awide rangeof possible futuredevelopments is a profound challenge.The global container shipping system is composed of many elements, with stronginterdependencies. Often, small changes cascade quickly through the system, potentiallyresultinginsubstantialchangessomewherequitefarremovedfromtheinitialsmallchange.For example, recently Ultra Large Carrier Vessels have entered themarket. Liner shippingcompanieshavestartedtoformalliancestopursueeconomiesofscale.Thisinturnaffectsthefrequency of port calls, the port rotation schedule, and the container volumes loaded andunloaded at each port. Changes not only take place on the seaside. Developments in thehinterland, such as the construction of new infrastructures such as road, railways, andintermodalfacilitiesaffecttransportcost.

Theongoingchangesonboththeseasideandthelandsizeofseaportschangethespatialflowof containers globally. The operation of the Trans-Siberian railways is an example of theimportanceofhinterlandinfrastructureonglobalcontainerflowsthroughports[2].Asimilar

© 2016 Manuscript version made available under CC-BY-NC-ND 4.0 license https://creativecommons.org/licenses/by-nc-nd/4.0/ Link to formal publication (Elsevier): http://dx.doi.org/10.1016/j.futures.2015.09.004

2

chainedeffectalsotakesplacewhenglobalpoliciesareimposedonports.AnexampleofthisistherecentInternationalMaritimeOrganizationregulationthatlimitstheuseofsulfurinaship’sfuelinmanyportsintheworld[3].Allofthesecomponentsinthesystem,bothphysicalandinstitutional,andthepotentiallinkedchangesthereindemonstratethatportsarepartofacomplex system. In such systems, observation on the components, and their interactionmechanismscanonlybedonepartially, leadingtoa limitedpredictabilityof thebehaviorofthesystemwhencertainchangesoccur.

Asecondreasonwhypreparingaportforpossiblefuturedevelopmentsisimportant,isthatthe future is deeply uncertain [4, 5]. Since there aremany actors with different objectivesinvolvedinthegloballogisticsystem,itisvirtuallyimpossibletopredicthowchangesinanygivencomponentofthesystemwillaffectthesystemasawhole.AnexampleofadisruptivechangeisthelaborstrikecaseonthewesterncoastoftheUSinearly2015.Asaresultofthestrike,theportofLongBeachwasshutdownforseveraldays,causingalosstotheGDPoftheUnitesStatesofAmericaofroughly150Mdollar [6].Anotherexample is therecentdrop intheglobaloilprice,whichhasreducedtransportcostsignificantly.Whenthislowoilpriceissustained fora longperiod(>5years), itwill likely influence trade flowsbetweencountriespositively. Consequently, thiswould result in an increase in the volumeof containers flowstransported globally. Predictingwhen the oil price will return to its 2013 level is a highlychallenging task. Other changesmight include the opening and closing of certainmaritimeroutes such as the Suez Canal and the Northern passage (i.e. the arctic route), changes incompetition strategies of other ports, emergence of political tensions that hamper tradeagreementsbetweencountries,etc.Inshort,thereisamassivenumberofscenariosthatcanbegeneratedtoaccountfortheuncertaintieswithinthegloballogisticsystem.

Because of the intrinsic complexity of the global container shipping system, many portauthorities and government institutions make use of models to help them in designingappropriate policies and strategies for ports and related infrastructure. Given the manychangestakingplacewithintheglobalcontainersystem,however,itishighlyimplausiblethatall these changes can be quantitatively explored using a single model. In fact, there aredifferentmodelingformalismsthatcanbeusedtomodelchangesinthewholesystemanditssub-systems. Examples of those modeling approaches include discrete choice models [7],computable general equilibrium models [8], discrete even simulation models [9] andoptimization models [10]. Experts in each of these areas might argue that their modelingapproach is best suited to represent the system. To complicatematters, different plausiblescenarios canbe specifiedusing thesemodels.Consequently,differentalternativeoutcomesmightemergefromthesedifferentmodelingexercises. Inshort, informationfrommodelstoperformpredictions on such a complex systemwithmanyuncertaintiesmight be seriouslymisleading[11].Specifically,thiscanhappenwhentheirreducibleuncertainties,whichexistin the factors that drive change, are not properly taken into account when calculating theplausible outcomes. Therefore, there is a need for a systematic approach to deal with theuncertaintiesinthecomplexcontainershippingsystem.

3

Theobjectiveofthispaperistoapplyscenariodiscovery[12]inordertoprovideinsightintothemainvulnerabilitiesfortheportofRotterdam.Themainquestionthatweaddressinthispaper is ‘what are the key vulnerabilities for the competitive position of Rotterdam in theBremen – Le Havre range?’ Scenario discovery is an innovative model-based approach toscenariodevelopment.Itisinspiredbythescenariologicapproachtoscenariodevelopment.Scenariodiscoveryusesseriesofcomputationalexperimentstoexploretheconsequencesofvarious unresolved uncertainties. These series of computational experiments aresubsequentlyanalyzedusingstatisticalmachine learningalgorithms inorder to identify thecombinationsofuncertaindevelopmentsthatproducecharacteristicresults.

Scenariodiscoveryreliesontheexploratoryuseofoneormodemodels.Inthispaper,weuseaworldcontainer-shippingmodel[7].Thisisastrategicnetworkchoicemodel.Themodelcanbeusedtoestimatetheflowsofcontainersbetweencountriesgivenassumptionspertainingnetwork structure, port attractiveness, and origin destination data. Time is not explicitlypresentedinthemodel,forthemodelisstatic.Thisimpliesthatthevulnerabilitiesidentifiedthrough scenario discovery are not associated directlywith a particular point in time. Themainuncertain factorsthatwewillbeanalyzinghavebeenderivedfromboth literaturesaswellasfromdiscussionswithvariousexpertsatthePortofRotterdam.Akeycriterionintheselection of uncertain factors was their potential to affect the throughput of ports in theHamburg–LeHavrerange.Methodologically,weapplybothscenariodiscoveryaswellasaninnovativeworst-casediscoverytechnique.

Thispaperisstructuredasfollows.Section2elaboratesonthescenariodiscoveryapproachproposed in thispaper. In section3,wepresenta case study illustrating theapplicationofscenario discovery approach to deal with uncertainties faced by European ports.Furthermore,thissectionalsoprovidesthespecificationoftheworldwidecontainertransportmodel used together with this approach. Section 4 discusses the results of the case studytogether with the analysis done using scenario discovery approach. Section 5 provides theconclusionsofthestudy.

2 Scenariodiscovery:amodelbasedapproachtoscenariodevelopmentScenario discovery is a relatively novel approach for addressing the challenges ofcharacterizingandcommunicatingdeepuncertaintyassociatedwithsimulationmodels[13].Thebasic idea is that the consequencesof thevariousdeepuncertainties associatedwith asimulation model are systematically explored through conducting series of computationalexperiments [14] and that the resulting data set is analyzed to identify regions in theuncertaintyspace thatareof interest [12,15].These identifiedregionscansubsequentlybecommunicated through e.g. narratives to the decision-makers and other actors involved. Inthispaper,wecomplementthisbasicideaofscenariodiscoverywithamoredirectedsearchtechniquethatisusefulforworst-casediscovery.

Amotivation for the use of scenario discovery is that the available literature on evaluatingscenariostudieshas found thatscenariodevelopment isdifficult if the involvedactorshavediverginginterestsandworldviews[12,16].Anothershortcomingidentifiedinthisliterature

4

isthatscenariodevelopmentprocesseshaveatendencytooverlooksurprisingdevelopmentsanddiscontinuities[17-19].Athirdproblemisthatanyscenariodevelopmentapproachthatrelies on thementalmodels of the analystwill strugglewhen facedwith complex systems.Sincementalmodels are typically event based, have an open loop view of causality, ignorefeedback, fail to account for timedelays, and are insensitive to non-linearity [20], essentialelements of dynamics in complex systems, namely feedback, time delays and non-linearity,cannotbeappropriatelydealtwith.Consequently,mentalsimulationsofcomplexsystemsarehighlydefective,somethingthathasalsobeendemonstratedempirically[21-27].

Scenariodiscoveryisamodel-basedapproachthatofferssupportfordecision-makingunderdeepuncertainty.Deepuncertaintyisencounteredwhenthedifferentpartiestoadecisiondonot know or cannot agree on the system model that relates consequences to actions anduncertainmodelinputs[4],orwhendecisionsareadaptedovertime[28].Inthesecases,itispossible to enumerate the possibilities (e.g. sets of model inputs, alternative relationshipsinside amodel, etc.),without ranking thesepossibilities in termsofperceived likelihoodorassigningprobabilitiestothedifferentpossibilities[5].

Whenusingmodelstosupportdecisionmakingunderuncertainty,modelshavetobeusedinan exploratory manner rather than in a predictive manner [14, 29, 30]). In predictivemodeling,modelsareusedtopredictsystembehavioranddevelopedbyconsolidatingknownfactsintoasinglepackage[29].Whenexperimentallyvalidated,thissinglemodelcanbeusedfor analysis as a surrogate for the actual system. In the presence of deep uncertainty andcomplex systems, the construction of a model that may be validly used as a surrogate issimplynotpossible.Thecomplexityof,anddeepuncertaintypertainingthesystemtogetherimplynonlinearityofsystembehavior,dynamiccomplexity,andrivalrepresentationsofthesystemwhich are underdetermined given the available data [31-33]. Exploratorymodelingstarts from this fact of not knowing enough tomakepredictions,while acknowledging thatthere isstillawealthof informationandknowledgeavailable thatcouldbeusedtosupportdecisionmaking[29].

When developing and using models for exploratory purposes, the available information isinsufficient tospecifyasinglemodel thataccuratelydescribessystembehavior. Instead, theinformationcanbeusedtoconstructavarietyofmodels,which,takentogether,areconsistentwith the available information. This ensemble of models typically captures more of theavailableinformationthananyoftheindividualmodels[30].Eachoftheseindividualmodelswill have different implications for potential decisions. A single model drawn from thispotentially infinite set of plausible models is not a prediction. Rather, this model is acomputationalexperimentthatrevealshowtheworldwouldbehaveifthevarioushypothesesencapsulatedinthissinglemodelaboutthevariousunresolvableuncertaintieswerecorrect.

Akeychallengeistodevelopeffectivestrategiesforsearchingthroughtheimplicationsoftheensemble of plausible models for the decision problem at hand. Two families of searchstrategies canbe identified: open exploration anddirected search.Open exploration canbeused tosystematicallyexplore thesetofplausiblemodels.That is,openexplorationaimsatgeneratingasetofcomputationalexperiments thatcovers thespaceofplausiblemodels,or

5

uncertaintyspaceforshort.Thisexplorationreliesonthecarefuldesignofexperimentsandcan use techniques such as Monte Carlo sampling, Latin Hypercube sampling, or factorialmethods. An open exploration can be used to answer questions such as “under whatcircumstanceswould thispolicydowell?”, “underwhat circumstanceswould it likely fail?”,and“whatkindsofdynamicscanthissystemexhibit?”.Anopenexplorationprovidesinsightintothefullrichnessofbehaviorsoftheensembleofmodels.

Open exploration uses series of computational experiments. In order to reason on thisensemble, theresults fromthesecomputationalexperimentshave tobeanalyzed.ThemainalgorithmthatisusedforthisisthePatientRuleInductionMethod(PRIM)[34].PRIMtriestofindcombinationsofvaluesforinputvariablesthatresultinsimilarcharacteristicvaluesforone ormore outcome variables. Specifically, the algorithm seeks a set of hyper rectangularsubspaces of the uncertainty spacewithinwhich the values of a single output variable areconsiderably different from its average values over the entire uncertainty space. PRIMdescribesthesesubspacesintheformofboxesoftheuncertaintyspace.Thisresultsinaveryconciserepresentation,fortypicallyonlyalimitedsetofdimensionsoftheuncertaintyspaceisrestricted.Thatis,asubspaceischaracterizedbyupperand/orlowerlimitsononlyafewinputdimensions.

In contrast to open exploration, directed search is a search strategy for finding particularcases thatareof interest.Directedsearchcanbeusedtoanswerquestionssuchas“what istheworstthatcouldhappen?”“Whatisthebestthatcouldhappen?”“Howbigisthedifferenceinperformancebetweenrivalpolicies?”Adirectedsearchprovidesdetailedinsightsintothedynamicsofspecific locationsinthefullspaceofplausiblemodels.Directedsearchreliesonthe use of optimization techniques, such as genetic algorithms and conjugant gradientmethods.Openexplorationanddirectedsearchcancomplementeachother.Forexample, iftheopenexplorationrevealsthattherearedistinctregionsofmodelbehavior,directedsearchcan be employed to identify more precisely where the boundary is located between thesedistinctregions.

Although scenario discovery can be applied on its own [15, 35, 36], it is also the analyticalcore of Robust Decision Making [13, 37-39]. Robust decision-making is a model-baseddecision support approach for the development of robust policies. That is, policies thatperformsatisfactorilyacrossaverylargeensembleoffutureworlds.Inthiscontext,scenariodiscoveryisusedtoidentifythecombinationofuncertaintiesunderwhichacandidatepolicyperforms poorly, the vulnerabilities of a candidate policy, allowing for the iterativeimprovement of this policy. The use of scenario discovery for Robust Decision Makingsuggeststhatitcouldalsobeusedinotherplanningapproachesthatdesignplansbasedonananalysisoftheconditionsunderwhichaplanfailstomeetitsgoals[40]

3 AglobalcontainershippingmodelInthissection,weintroducetheglobalcontainermodelwhichisthebasisfortheexploratorymodeling,includingtheassociateduncertainfactorsthatwillbeexplored.

6

3.1 AstrategicglobalcontainernetworkchoicemodelTheWorldContainerModelisastrategicnetworkchoicemodelforglobalcontainerflows[7].Themodelconsidersmorethan400majorports,237countries,andmorethan800shippinglines.Givenacountry-to-countryorigindestinationdemandmatrix,themodelcalculateshowcontainer flows are distributed over the global network of shipping lines and through thevariousports.Transportcostsinthemodelarebasedontransporttime,distance,toll,valueoftimeof the goods transported, and cost of handling the containers at theports. The formaldefinitionofthecostmodelisdelineatedbelow:

𝐶! = 𝐴!!∈!

+ 𝑐!!∈!

+ 𝛼( 𝑇!!∈!

+ 𝑡!!∈!

) (1)

where:Cr costsofrouterp portsusedbytheroutel linksusedbytherouteAp totalcostoftransshipmentatportpcl totalcostoftransportationoverlinklTp timespentduringtransshipmentatportptl timespentduringtransportationoverlinklα valueoftransporttime(USD/day/ton)

Themodeoftransport(seaorlandmodes)isembeddedinthenetworkattributesanddoesnot appear in the cost formula. Thismode-abstract formulation enables the use of amoredetailedunderlyingmultimodalnetwork;theaggregateresultisalevelofserviceexpressedintimeandcost.Thevalueof theattractivenessparameter𝐴!and thescalingparameter𝛼areunknownandneedtobeestimated.Thescalingparameter lcapturesthephenomenonthat,although individual shippers and carriers may decide to use one port or another, theiraggregatebehaviorresults intheuseofmorethanonealternativerouteandthattheir jointresponsetopoliciesisasmoothone.

Themodelenumeratesthemajorityofplausibleroutealternativesformajorcountriesintheworldusing thepubliclyavailable service tablesof shipping linesworldwideanda shortestpath algorithm. A route from port of origin to port of destination is determined by firstlookingat theport-callorderofcontainershipping linesworldwideand then,basedon thisorder,usingashortestpathalgorithmto identifythesub-segmentsofthecompleteshortestroute for each port-to-port segment of a shipping line.Hence choice sets are generated forevery country O/D pair using both the physical network and the network of service linesdefinedontopofthisphysicalnetwork.

For instance, a routebetweenanorigin countryOanddestinationcountryD, startingatanoriginportSandadestinationportEisdefinedbyoneormoremaritimeservicesbetweenSandE,withintermediatetransshipmentatportswhereachangeofservicecanbecarriedout.Foreach(outgoing)portStothe(incoming)portEtheshortestpathisaddedtothechoicesetofthiscombinationO–S–E–D(seeFigure1).

7

Figure1ExampleofaroutebetweenOD-countrypair[adaptedfrom7].

Themodelaccountsforbothmaritimeconnectionsbetweentwocountriesaswellasoverlandconnectionsbetweenthesecountries.Therouteandportchoicealgorithmsusesapath-sizedlogitmodelwhich takesoverlapsbetweenthealternativeroutes intoaccountanddescribesthe transport costs associated with these alternatives correctly [41]. The following is theformaldefinitionoftheroutechoicemodel.Therouteprobabilitiesaregivenby:

∑∈

+−

+−

=

CSh

SC

SC

hh

rr

ee

)ln(

)ln(

r P µ

µ

(2)

WiththepathsizeoverlapvariableSdefinedas

ahNr

1zz S

a r

ar ∑

Γ∈⎟⎟⎠

⎞⎜⎜⎝

⎛=

(3)

where:Pr thechoiceprobabilityofrouterC generalizedcostsCS thechoiceseth pathindicatorμ logitscaleparametera linkinrouterSr degreeofpathoverlap

rΓ setoflinksinrouter

az lengthoflinka

rz lengthofrouter

ahN numberoftimeslinkaisfoundinalternativeroutes

8

Calibrationofthemodelwasdoneatanaggregatelevelusingavailableportthroughputandtransshipment statistics.The resultingglobal flowsof containersare shown inFigure2.Thenetwork allows hinterland transportation routeswith differentmodes of transport such astruck,rail,waterwaysorshortseabutisomittedhereforvisualconvenience.Thethicknessofthe lines indicates themagnitudeof the flowsoneach links.Eachport isvisualizedasapiechart,whichshows themagnitudeof throughput (indarkgrey)and transshipment (in lightgrey).

Figure2theglobalmaritimeshippingnetworkintheWCM

3.2 IdentificationofrelevantuncertaintiesInthispaper,weusetheworldcontainermodeltoassesstheimpactofseveralkeyuncertainfactors on the global flowof containers. The following keyuncertainties are treated in theexploratorymodelinganalysis:

Table1KeyUncertaintiesthatwillimpacttheperformanceofportsintheBremen-leHavrerange

Name RangeCostofhinterlandconnectionsinEurope 0%–25%HinterlandcostforRotterdam -25%–+25%TraveltimeofthehinterlandconnectionsofRotterdam -2days–+2daysCosts of the hinterland connection of Mediterraneanports

-25%–0%

HandlingcostsoftheMediterraneanports -25%–0%HandlingcostsofportsintheBremenandleHavrerange -25%–0%TradevolumewithAsiathatisaffectedbyavailabilityofoverland connection or shift of production to EasternEurope

-25%–0%

9

NorthernpassageoverArcticroute {present,absent}SuezCanal {present,absent}

We identified the key uncertainties above by means of brainstorming and discussionsbetweentheexpertsfromvariousorganizations,includingthePortauthority,DelftUniversityofTechnology,andthenationalresearchinstituteTNO.Themainsourceofinputwasagroupdecisionroomsessionwith14experts;11ofwhichwerefromthePortauthority,2werefromDelftUniversityofTechnology,and1wasfromTNO.Allexperts inthegroupdecisionroomsessionweremale.Thesebrainstormsessionsandindepthdiscussionswherefedbyameta-analysis of transport scenarios and forecasts across the globe, from a heterogeneous set ofactors includingportauthorizes,nationalgovernments, international think tanks, andotherknowledge institutes. Visions on possible future developments, forecast studies, visiondocuments from various institutions, and historical data pertaining to each of theseuncertaintieshavebeenusedasinputforthediscussion.SinceweareinterestedtostudytheimpactsofuncertaintiesonEuropeanports in general andRotterdam inparticular, thekeyuncertaintiesthatareidentifiedarelimitedtothosethatwillpotentiallyimpacttheseports.

One of themost plausible uncertain factors that couldmake the ports in Bremen-LeHavrerangetoexperienceadecline incontainer flows is the increaseofhinterlandtransportcost.This increase in cost can be caused by congestion at the terminals, environmental tax,bottlenecks in the inland waterways, etc. A related factor that is subject to change is thehinterland cost of Rotterdam. This change can be caused by congestion or improvement inefficiencyofthetransportnetworksthatconnectRotterdamtothehinterlanddestinations.Inlinewith thescenariodiscoveryapproach,we tookabroadbandwidth intoaccount for thisfactor.We assumed that the change in cost can vary from 25% increase to 25% decreaserespectively. Next, the same presumption is also applied to the travel time of Rotterdam’shinterland connections. Improvement in logistics services at the terminal and throughoutforwardingprocesscanbringreductiontothetotaltimeneededtotransportthecontainers.Wespecifyarangeofplausiblechangeforthisvariablebetween-2to2days,representinganincreaseandadecreaseinthetimeefficiencyrespectively.FortheportsintheBremen–LeHavrerange,achangeoftwodaysamountstoasignificantchangeonvirtuallyallhinterlandconnections.

Furthermore,wealso identifythedevelopments inMediterraneanportsthatcanpotentiallypresent threatsandopportunities forRotterdam.TheMediterraneanportsarecloser to themainsourcesofproductsshipped toEurope inSouthEastAsiaandChina.Froma logisticalpoint of view, this makes these ports quite attractive. A plausible development is theimprovement of hinterland connections of theseports. The improvements can take formofbetterconnectivitydue toavailabilityofbetter infrastructuresuchasrailandroad,and theavailabilityofbetterfreightforwardingservices.Eventually,thisdevelopmentcanbeforeseento reduce the cost of the hinterland connections of these ports. Furthermore, in the face ofcompetition with ports in the Bremen-Le Havre range, Mediterranean ports might reducetheirtariffsothattheycanincreasetheirattractivenessforthefreightforwarders,especially

10

forcontainersthatcanbedirectlyshippedfromandtoSouthernEurope.Theplausiblerangeforthereductionofhinterlandcostduetothisdevelopmentisassumedtoreachamaximumof25%ofthecurrentcost.

TheportsintheBremen–LeHavrerangearequitecompetitiveandpartoftheirstrategycanbea reduction in tariffs.This reduction in tariffwouldalso increase theirattractiveness forlinershippingalliancesthatuseUltraLargeContainerVessels(ULCV)sothattheywouldcallhubports in theBremen-LeHavre range.Again,weuse a range of up to 25% reduction intariffs.

Atthetradelevel,wealsoidentifythattherecanbeareductioninoverseastradevolumewithAsiaduetothepresenceofoverlandconnectionsorthepossibleshiftofproductiontoEasternEuropeancountries.Astheplausiblerangeofthereductionweused25%ofthecurrenttradevolume.Lastbutnotleast,wealsoincludeuncertaintiesintheavailabilityofmaritimeroutessuchas thosevianorthernpassageandSuezCanal.Duetopolitical instability in theMiddleEast, and the presence of pirates in the Gulf of Aden, the Suez Canal might become toodangeroustobeuseforshipping.Duetoclimatechange,slowlybutsteadily,anewpossiblemaritime route is emerging. This northern passage offers an alternative direct route fromChinaandJapan,viatheArctictoEurope.

3.3 ImplementationandComputationThevariousuncertain factorsmayaffect thesamemodelparameter. In thiscase,wehandlethe effects from the different uncertain factors additively. That is, say we have acomputationalexperimentwithanincreaseofcostsonthehinterlandconnectionsinEuropeof20%, incombinationwithachangeof thehinterlandconnectioncosts forRotterdamof -10%,thanthefinalcostswillbetheoriginalcosts+originalcosts*0.2+originalcosts*-0.1.

To explore the consequences of the various uncertain factors in an open exploration, wedefined 10,000 experiments using Latin Hypercube sampling across these 9 uncertainties.This means we generate 10,000 random values for each of the uncertainties within thespecified ranges and combine these values in a random order into 10,000 sets of 9 inputparametersforthemodel.So,wesampleacrossthe9uncertaintiessimultaneously.Next,theworld container model is run for each of the 10,000 scenarios. The experiments whereperformedofa48logicalcoreXeonE5workstation,and192GBofRAM.Runtimewasroughly4hours.Tosupportthecomputationalexperimentation,andthesubsequentanalysis,weusedthe Exploratory Modelling Workbench [42]. This is an open source python project thatfacilitates the entire process of scenario discovery. The World Container Model isimplementedinJavaandconnectedtotheworkbenchusingJPype,aPython-Javabridge.

4 ResultsInthissectionwepresenttheresultsofexploratoryanalysisusingtwodifferentapproaches:open exploration and directed search. By using the open exploration approach,we identifyhowtheuncertainfactorsjointlyimpactthethroughputoftheportsintheBremen-leHavrerange. Subsequently,we use PRIM in order to gain a deeper insight on themeaning of the

11

resultsforPortofRotterdam.Thisanalysisresultsintheidentificationofmainfactors,whichjointly cause Port of Rotterdam to be both vulnerable and flourishing. They are thecombination of input variables that cause the ports to experience decline and gain in theirthroughputrespectively.Next,wealsoapplydirectedsearchtechniqueto identifytheworstpossible scenario that could happen to the port of Rotterdam. This analysis results in theidentificationoftheconditionsunderwhichthisscenariomanifestsitself,andtowhatextentRotterdamwillbenegativelyimpactedbyplausibleuncertainties.

4.1 OpenexplorationandscenariodiscoveryFigure3showsthechangeofthroughputfortheportsintheBremen–LeHavrerangeacrossthe10,000 scenarios. To facilitate interpretation,wedivided the throughput resulting fromthemodelbythebaseflowwhenthereisnouncertaintyintroduced.Thisimpliesthatascorelowerthan1meansalossofflow,whileascoreaboveonemeansanincreaseinflow.Inordertoallowacomparisonacross thedifferentports in theHamburg–LeHavrerange,wehavevisualized theresultsacross the10,000scenariosusingaGaussianKernelDensityEstimate(KDE). The colloquial interpretation of a KDE is that it is a continuous alternative to ahistogram.

So, the figure suggests that Hamburg, Antwerp and Dunkirk are most vulnerable to losingflows.MostofthemassoftheKDEofeachoftheseportsisbelow1.Incontrast,LeHavreandBremen have most of their mass above 1, suggesting that they benefit from the variousuncertainfactors.TheportsofRotterdamandZeebruggeoccupyamiddleposition.Thefigurealso suggests that there is substantial downside risk, and a smaller percentagewiseupsideopportunity:portscanlosecloseto50%oftheirflow,butonlygainamaxofalmost20%.

12

Figure3ImpactsofkeyuncertaintiesonthethroughputofportsintheBremen-LeHavrerange

Looking closer at the result for Port of Rotterdam, we observe that there is a small butsubstantialamountofmassbelow1.Thissuggeststhattherearequiteanumberofscenarioswhere Rotterdam loses throughput as a result of the various uncertain factors. This isindicatedbytherelativelylargeareacoveredbythegraphbetweenroughly0.75and1.Thissuggests that the consequence of the uncertainties can be significant, where Rotterdamsuffers a25%reduction in their throughput.Basedon these findings, it is valuable to get adeeper insight into the combination of uncertain factors that jointly causeRotterdam to bevulnerableandpotentiallylosetheirthroughput.

First,wespecifywhenascenarioisofinterestornotbyestablishingaclassificationrule.Tothisend,weclassifyallcaseswhereRotterdamwitnessesanydeclineinthroughoutcomparedtothereferencecaseasbeingacaseofinterest.Specificallythefollowingequationisusedtoclassifyalltheresults:

𝑓 𝑥 = 1, 𝑖𝑓 𝑥 < 10, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒 (4)

where𝑥isthefactorchangeinthroughputascomparedtothebaseflowforRotterdamasalsousedinFigure3.Next,weusePRIMtoidentifythecombinationsofuncertaintiesthatjointly

13

produceundesirableresults.PRIMreturnsmultipleexplanationsfortheundesirableresults.The analyst can select the explanation that covers most of the undesirable result - this isknownascoverage inscenariodiscovery-whilealsobeingmainlyvalid fortheundesirableresults–thisisknownasdensityinscenariodiscovery[12].Toassesswhethertheinclusionofagivenuncertainfactorinagivenexplanationisstatisticallysignificant,onecanuseaone-sided binomial test - this is sometimes also called the quasi-p value in scenario discovery[12].

Figure4KeyuncertaintiesthatplayasignificantroleinexplainingmajorcaseswithnegativeimpactforRotterdam

Figure4showstheresults fromthePRIManalysis.This figureshowsthat75%of thecaseswithalossofthroughputcanbeexplainedbythecombinationofthreeuncertainfactors:theHamburg – le Havre cost factor, the Rotterdam hinterland travel time factor, and theRotterdamhinterland cost factor.The shaded light greybackground specifies the full rangeforeachof theseuncertain factors,while thebluebarsspecifythesubspaceas identifiedbyPRIM.Betweenbrackets,behindeachlabel,thequasi-pvalueisshown.Ascanbeseen,eachofthethreerestrictionsisstatisticallysignificant.So,whatdoesthisresultimply?Inessence,ifRotterdamexperiences an increase of travel timeon the hinterlandof 0.8 days ormore, incombinationwithasmallreductionincostsfortheportsintheBremen–leHavrerange,andnot an extreme reduction in costs on the hinterland of Rotterdam, Rotterdam will losethroughput. This suggests, that the quality of the hinterland connections of RotterdamstronglydeterminesthecompetitivepositionofRotterdamintheBremen–leHavrerange.

Interestingly, an increased efficiency on the hinterland connections of the Mediterraneanports in combinationwith a reduction in costs for these ports does not cause a significantdeclineinthethroughputofRotterdam.Furthermore,itisalsoremarkablethatthereductionintradewithAsiadoesnotsignificantlyaffectthechangesinthethroughputofRotterdam.

14

4.2 Directedsearchforworstcasediscovery

4.2.1 InsearchofaperfectstormInlightoftheresultsoftheopenexploration,afollow-upquestionwasformulatedbythePortAuthority: is there a perfect storm scenario, where a set of disconnected small changessubstantially affects the throughput and transshipment of the Port? This question can beformulatedasatwoobjectiveoptimizationproblem

maximize 𝐹(𝑙!) =!"!"##$!%&'(!!)!"!"##$!%&'(!!")

, !"!"##$!%&'(!!)!"!"##$!%&!(!!")

(5)

Where𝑙! =

𝑃!"#$%&%𝑃!"#!!"#$%&'(#(𝑃!"#!!"#$%#%&'$(𝑝!"#$%&'(%)'𝑝!"#!!"#$%&'(#𝑃!!"!!"#$#𝑝!"#$!!"#𝑝!"#$

𝑃!"#$%&!!"#$

𝑝!"#$!!"# ∈ 𝑡𝑟𝑢𝑒, 𝑓𝑎𝑙𝑠𝑒

𝑝!"#$ ∈ 𝑡𝑟𝑢𝑒, 𝑓𝑎𝑙𝑠𝑒

subjectto:

𝑐1: 0 ≤ 𝑃!"#$%&% ≤ 0.05 (6)

𝑐2:−0.05 ≤ 𝑃!"#!!"#$%&'(#( ≤ 0 (7)

𝑐3:−0.1 ≤ 𝑃!"#!!"#$%#%&'$( ≤ 0.1 (8)

𝑐4:−0.05 ≤ 𝑝!"#$%&'(%)' ≤ 0 (9)

𝑐5:−0.05 ≤ 𝑝!"#!!"#$%&'(# ≤ 0 (10)

𝑐6:−0.05 ≤ 𝑃!!"!!"#$# ≤ 0 (11)

𝑐7:−0.05 ≤ 𝑃!"#$%&!!"#$ ≤ 0 (12)

where𝑙!" is the performance in the reference case without any uncertain factors,𝑇𝑝 isthroughput, and𝑇𝑠 is transshipment. So, the aim is to find a scenario where the ratiocompared to the reference case isminimized, subject to amax5%changeoneachof the7continuousuncertainfactors.

Tosolvethismultiobjectiveoptimizationproblem,weusedε-NSGA2,astateoftheartgeneticalgorithm for solvingmulti-objective optimization problems [43].We ran the algorithm for150generations,andascanbeseeninFigure5,thealgorithmconvergedoverthecourseof

15

thissimulation.Adetailedinspectionoftheresultsrevealsthatallsolutionsareattheedgesspecified by the constraints. That is, the maximum negative deviations that are possiblespecifytheworst-casescenariosforRotterdam.Theseworst-casescenariosallpointtoalittlebitlessthan5%reductioninthroughputandtransshipmentforRotterdam,givenamaximumof5%changeontheuncertainfactors.Followupanalyseswherewechangetheseboundariesto10%and15%respectively,yieldedessentiallythesameresults.Thelossintransshipmentand throughput is about equal to themaximum percentage change allowed on the variousuncertainfactors.

Figure5.Cumulativeε-progressoverthecourseofrunningε-NSGA2

4.2.2 ImpactofthecompetitionfromMediterraneanportsTheanalysessofaraimedatinvestigatingsituationswhereRotterdamexperienceschangesintheirthroughput,orwhereitscompetitivenesschangesasaresultofchangesinportsintheBremen-LeHavrerange.Basedonthepreviousanalysis,wehavediscoveredthatadecreaseinthecostfactorinMediterraneanportsisnotabletoincreasethecompetitivenessoftheseports relative to Rotterdam. Specifically, the container flows to Rotterdam are not affectedsignificantlywhenportcostfactorsoftheseportsarereduced.

Since there have not been many studies concerning competition between Rotterdam andMediterranean ports, a more thorough investigation on how increased competitiveness ofMediterraneanportswill impactcontainerflowsinRotterdamcangiveavaluable insighttotheportofRotterdam.Hence,inthisanalysis,weaddressthequestion:underwhichchangeinthehinterlandcostsoftheMediterraneanportswillRotterdambeimpactednegatively?Thus,instead of defining a plausible range of variable values in which uncertainties might bepresentandlookingattheoutcomescausedbyoneormoremajorvariables,welookathowbigthechangeininputvariablesneedstobeinordertocausethesystemtobehaveinaveryspecific manner. Specifically we are interested to find combinations of factors, which willminimizethethroughputdifferencebetweenRotterdamandeachoftheMediterraneanports.Thisproblemcanbemodeledasamulti-objectiveoptimizationproblemwherethevaluesofinputvariables,whichminimizethethroughputdifferencessimultaneously,aresearched.Thefollowingistheformulationoftheproblem.

Minimize𝐹(𝑙!) = 𝑓!"#$% , 𝑓!"#$% , 𝑓!"#$% , 𝑓!"#$%% , 𝑓!"#$$ (13)

16

Where𝑙! =

𝑝!"#$%&%𝑝!"#$%&'(%)'𝑝!"#!!"#$%&'(#𝑝!"#$!!"#𝑝!"#$

𝑓! = |𝑇!"##$!%&' − 𝑇!|

𝑖 ∈ 𝐺𝑒𝑛𝑜𝑖,𝐵𝑎𝑟𝑐𝑒,𝑀𝑎𝑟𝑠𝑓, 𝐿𝑎𝑠𝑝𝑖𝑖,𝑉𝑒𝑛𝑖𝑖

𝑝!"#$!!"# ∈ 𝑡𝑟𝑢𝑒, 𝑓𝑎𝑙𝑠𝑒

𝑝!"#$ ∈ 𝑡𝑟𝑢𝑒, 𝑓𝑎𝑙𝑠𝑒

subjectto:

𝑐1: 0 ≤ 𝑝!"#$%&% ≤ 0.25 (14)

𝑐2:−1.0 ≤ 𝑝!"#$%&'(%)') ≤ 0 (15)

𝑐3:−1.0 ≤ 𝑝!"#!!"#$%&'(#( ≤ 0 (16)

Again,ε-NSGA2isusedforsolvingthismulti-objectiveoptimizationproblem.AscanbeseeninFigure6,thealgorithmconvergedoverthecourseof150generations..Figure7showstheresults found by the optimization algorithm. In the two sub figures, we use parallelcoordinateplots.Theseplotscanbeusedtovisualizedatawithmorethanthreedimensions.Thisisachievedbydepictingeachdimensionasaverticalline.Amultidimensionaldatapointis depicted as a line connecting each of the different dimensions. The left hand side figurepresentsthevaluesofeachdecisionvariableforeachsolution,whiletherighthandsidefigurepresentthevaluesoftheobjectivefunctionsoftheproblem.

Figure6ε-progress

17

Figure7Optimizationresults

Ascanbe seen from the righthand side figure inFigure7,when the throughputdifferencebetweenRotterdamandeachoftheMediterraneanportsisminimizedsimultaneously,eachofthe solutions presents a trade-off across the objective functions. This means all of thesesolutions are non-dominated. That is, there is not a solutionwith superior performance incomparisonwithothersolutions.Forexample,virtuallyalllinestradeofftheperformanceofGenoaandBarcelona.IfonehasminimizedthedifferencebetweenGenoaandRotterdam,thedifferencebetweenBarcelonaandRotterdamincreasesandviceversa.

Fromtheresult,itcanbeconcludedthatbothportandhinterlandcostsoftheMediterraneanports need to decline significantly in order for them to be competitive against Rotterdam.Surprisingly, theclosingofSuezCanalandtheopeningofarcticroutealsocontributetotheincreaseofMediterraneanport’scompetitiveness.Thiscanonlyhappenwhenliner-shippingcompaniesthatcallatMediterraneanportsbeforecallingRotterdamdonotchangetheirportrotationscheduleintheeventofsuchdisruptions. Inthiscase,theshortestpathfrommanycountries inFarEast toRotterdam is through thenorthernpassage if this is available.Asaresult,transportcostfromthesecountriestoRotterdamwillincreasesignificantlyduetotheincrease in distance that has to be travelled by these shipping companies. If the northernpassageisnotavailable,theshortestrouteisviaCapeTown.Inamorerealisticscenario,anadaptationoftheshippingroutingisexpectedtotakeplace.Wereturntothispointbelow.

5 ClosingremarksTheglobalcontainertransportnetworkisconstantlychanginginresponsetoawiderangeofdevelopments. It isvirtually impossible tocorrectlyanticipate the futuredynamicsofglobalflows of containers, due to the intrinsic complexity of the network and the wide range ofuncertain factorsaffectingthenetwork.Forports, thisposesa fundamentalchallenge inthelongtermplanningoftheirstrategyandinvestments.

18

To address the combined challenge of complexity and uncertainty,we used a novelmodel-basedapproachtoscenariodevelopment.Ratherthanpredictingfuturecontainerflowsforalimited setof alternativeassumptions,we systematically exploredwhat the container flowscouldbe across10,000 scenarios, covering9uncertain factors.The analysis focusedon theconsequences of uncertainty for the competitive position of the port of Rotterdam in theBremen – le Havre range. Uncertainties that were taken into account included changes inglobal trade flows, changes in the physical network available to ships, and various factorsrelated to transportation costs of ports in Europe and costs on the hinterland connectionswithin Europe. It was found that the critical factor affecting the competitive position ofRotterdamis thequalityof theirhinterlandconnection.Amodest increase in travel timeonthehinterlandconnectionsfromRotterdamwillshiftflowsawaytootherportsintheBremen–leHavrerange.

In two followupanalyses,weusedanoptimizationbasedsearchstrategy inpursuitof twoworst-casescenarios.First,wesearchedforthepresenceofaperfectstormscenario,whereasetofsmallchanges jointlysubstantiallydeterioratesthecompetitivepositionoftheportofRotterdam.Wewerenotabletofindsuchaperfectstorm,whichsuggeststhatthecompetitivepositionisquiterobustwithrespecttosmallchanges.

In a second analysis, we tried to find a scenario where the Mediterranean ports becomeseriouscompetitorsofRotterdam.Wewereable to findsuchascenario,which includes theuse of the northern passage and the closure of the Suez Canal. This counterintuitive resultsuggests twothings.First, itpointsagain to therobustpositionofRotterdamin thecurrentglobalcontainer-shippingnetwork.Second,theresultsoftheseconddirectedsearchpointtooneofthemainlimitationsofthepresentedstudy.Theuncertainfactorsweexploredfocusondifferentaspectsofglobalcontainertransport,buttheyweremainlyfocusedontheportsandthehinterlandconnections.Wehavenotexploredtheimpactofchangesintheshippinglinenetwork itself and how these could affect global trade flows. The analyses that have beenperformedassumedtheglobalshippingservicenetworkstoremainthesameacrossdifferentscenarios.Thisisagapthatneedstobeaddressed,asthischangeinthenetworkspresentsanadditionaluncertaintytotheportsintheBremen-LeHavrerangeandmakesthepositionofthese ports more vulnerable. This is not a trivial challenge. At present, to the best of ourknowledge,nomodelexiststhatrepresentsthedynamicsofallglobalshippinglinenetworksovertime.Theavailable literature, instead, focusesonoptimizingtheroutingandfrequencyforindividualshippinglines,oroccasionallyforanindividualshippingcompany.

Modeling how shipping companies change their network is not simple as there are manyfactors,bothobservable(suchascostoftransport,marketsharecoverage,andinternationalcompetitionpolicy) andunobservable (internal agreementbetween shipping companies toform alliances, political agendas of relevant governments, strategic behavior on part of theshippingcompanies,etc.),thatcontributetotheactualstructureoftheirnetworkatanygivenpoint in time.Hence, inorder tosystematicallydealwith thisuncertainty,aseparatemodelwould be needed. Combining this model with exploratory modeling analysis and scenario

19

discoverywouldgivemorevaluableinsightsonhowuncertaintiesinglobalshippingnetworkwillimpacttheperformanceoftheports.

Theresultsof thispaperdemonstrate thevalueofusingscenariodiscoverywithsimulationmodels toexploreawide rangeofplausible futuresandsummarize the results intoconciseand clear insights. Scenario discovery, both through open exploration and directed searchprovidesasolidforansweringstrategicquestionsthatcomeupinthelong-termplanningofports.

Acknowledgements

TheNextGenerationInfrastructuresFoundationincollaborationwiththePortofRotterdamfundedthisresearch.

References

[1]CNBC,WestCoastports:Retail's$7billionproblem,in,2015.

[2]C.Rastogi,J.-F.Arvis,TheEurasianConnection:Supply-ChainEfficiencyalongtheModernSilkRoutethroughCentralAsia,TheWorldBank,Washington,2014.

[3] I.M.O. (IMO), Ships face lower sulphur fuel requirements in emission control areas from 1January2015,in,2014.

[4] R.J. Lempert, S. Popper, S. Bankes, Shaping theNextOneHundred Years:NewMethods forQuantitative,LongTermPolicyAnalysis,in,RAND,SantaMonica,CA,USA,2003.

[5] J.H.Kwakkel,W.E.Walker,V.A.W.J.Marchau,Classifyingandcommunicatinguncertainties inmodel-based policy analysis, International Journal of Technology, Policy and Management, 10(2010)299-315.

[6]T.Durden,The"CatastrophicShutdownOfAmerica'sSupplyChain"Begins:StunningPhotosOfWestCoastPortCongestion,in,2015.

[7]L.A.Tavasszy,M.Minderhoud,J.-F.Perrin,T.Notteboom,Astrategicnetworkchoicemodelforglobalcontainerflows:specification,estimationandapplication,JournalofTransportGeography,19(2011)1163-1172.

[8] T. Hertel, Future Directions in Global Trade Analysis, in, Center for Global Trade Analysis,PurdueUniversity,Purdue,Indiana,1999.

[9]P.Sinha-Ray, J.Carter,T.Field, J.Marshall, J.W.Polak,K.Schumacher,D.Songs, J.Woods, J.Zhang, ContainerWorld:Global Agent-basedModelling of the Container Transport Business, in:4thInternationalWorkshoponAgent-BasedSimulation,Montpellier,France,2003.

[10] M.G.H. Bell, X. Liu, J. Rioult, P. Angeloudis, A cost-based maritime container assignmentmodel,TransportationResearchPartB:Methodological,58(2013)58-70.

20

[11] J.H. Kwakkel, E. Pruyt, Exploratory Modeling and Analysis, an approach for model-basedforesight under deep uncertainty, Technological Forecasting and Social Change, 80 (2013) 419-431.

[12]B.P.Bryant,R.J.Lempert,ThinkingInsidetheBox:aparticipatorycomputer-assistedapproachtoscenariodiscovery,TechnologicalForecastingandSocialChange,77(2010)34-49.

[13] S.Dalal, B.Han, R. Lempert, A. Jaycocks, A.Hackbarth, Improving ScenarioDiscovery usingOrthogonalRotations,EnvironmentalModelling&Software,48(2013)49-64.

[14]S.C.Bankes,W.E.Walker,J.H.Kwakkel,ExploratoryModelingandAnalysis,in:S.Gass,M.C.Fu(Eds.)EncyclopediaofOperationsResearchandManagementScience,Springer,Berlin,Germany,2013.

[15]J.H.Kwakkel,W.L.Auping,E.Pruyt,Dynamicscenariodiscoveryunderdeepuncertainty:thefutureofcopper,TechnologicalForecastingandSocialChange,80(2013)789-800.

[16] S.A. van ‘t Klooster,M.B.A. van Asselt, Practising the scenario-axes technique, Futures, 38(2006)15-30.

[17]J.Derbyshire,G.Wright,Preparingforthefuture:Developmentofan'antifagile'methodologythatcomplements scenarioplanningbyomittingcausation,TechnologicalForecastingandSocialChange,(2013).

[18]P.W.F.vanNotten,A.M.Sleegers,M.B.A.vanAsselt,Thefutureshocks:ondiscontinuityandscenariodevelopment,TechnologicalForecastingandSocialChange,72(2005)175-194.

[19]T.J.B.M.Postma,F.Liebl,Howtoimprovescenarioanalysisasastrategicmanagementtool?,TechnologicalForecastingandSocialChange,72(2005)161-173.

[20] J.D. Sterman, Learning in and about complex systems, SystemDynamics Review, 10 (1994)291-330.

[21] P.W. Atkins, R.E.Wood, P.J. Rutgers, The effects of feedback format on dynamic decisionmaking,OrganizationalBehaviorandHumanDecisionProcesses,88(2002)587-604.

[22] J.D. Sterman, Modeling managerial behavior: Misperceptions of feedback in a dynamicdecisionmakingexperiment,Managementscience,35(1989)321-339.

[23] L. Sastry, D.R. Boyd, Virtual environments for engineering applications, Virtual Reality, 3(1998)235-244.

[24] D. Dörner, The logic of failure: Recognizing and avoiding error in complex situations, BasicBooks,1996.

[25]B.Brehmer,Dynamicdecisionmaking:Humancontrolofcomplexsystems,Actapsychologica,81(1992)211-241.

[26] B. Kleinmuntz, Computers as clinicians: An update, Computers in biology andmedicine, 22(1992)227-237.

21

[27] E. Diehl, J.D. Sterman, Effects of feedback complexity on dynamic decision making,OrganizationalBehaviorandHumanDecisionProcesses,62(1995)198-215.

[28]S.Hallegatte,A.Shah,R.Lempert,C.Brown,S.Gill,InvestmentDecisionMakingUnderDeepUncertainty:ApplicationtoClimateChange,in,TheWorldBank,2012.

[29]S.Bankes,Exploratorymodelingforpolicyanalysis,Oper.Res.,41(1993)435-449.

[30]S.C.Bankes,Toolsandtechniquesfordevelopingpoliciesforcomplexanduncertainsystems,ProceedingsoftheNationalAcademyofSciences,99(2002)7263-7266.

[31] D. Campbell, D. Farmer, J. Crutchfield, E. Jen, Experimental mathematics: the role ofcomputationinnonlinearscience,Commun.ACM,28(1985)374-384.

[32] N. Oreskes, K. Shrader-Frechette, K. Belitz, Verification, Validation, and Confirmation ofNumericalModelsintheEarthSciences,Science,263(1994)641-646.

[33] J.D. Sterman, Business Dynamics: Systems Thinking and Modeling for a Complex World,McGraw-Hill/Irwin,Boston,MA,2000.

[34]J.H.Friedman,N.I.Fisher,Bumphuntinginhigh-dimensionaldata,StatisticsandComputing,9(1999)123-143.

[35]J.Rozenberg,C.Guivarch,R.J.Lempert,S.Hallegatte,BuildingSSPsforclimatepolicyanalysis:a scenario elicitationmethodology tomap the spaceof possible future challenges tomitigationandadaptation,ClimaticChange,(2013).

[36]M.D.Gerst,P.Wang,M.E.Borsuk,Discoveringplausibleenergyandeconomicfuturesunderglobal change usingmultidimensional scenario discovery, EnvironmentalModelling & Software,(2013).

[37] R.J. Lempert, D.G. Groves, S.W. Popper, S.C. Bankes, A General, Analytic Method forGeneratingRobustStrategiesandNarrativeScenarios,ManagementScience,52(2006)514-528.

[38]R.J.Lempert,M.Collins,ManagingtheRiskofUncertainThresholdResponse:ComparisonofRobust,Optimum,andPrecautionaryApproaches,RiskAnalysis,24(2007)1009-1026.

[39] C. Hamarat, J.H. Kwakkel, E. Pruyt, Adaptive Robust Design under Deep Uncertainty,TechnologicalForecastingandSocialChange,80(2013)408-418.

[40]W.E.Walker,M.Haasnoot, J.H.Kwakkel,AdaptorPerish:AReviewofPlanningApproachesforAdaptationunderDeepUncertainty,Sustainability,5(2013)955-979.

[41]M. Ben-Akiva,M. Bierlaire, Discrete ChoiceMethods and their Applications to Short TermTravelDecisions,in:R.Hall(Ed.),HandbookofTransportationScience,SpringerUS,1999,pp.5-33.

[42] J.H.Kwakkel,E.Pruyt,UsingSystemDynamics forGrandChallenges:TheESDMAApproach,SystemsResearchandBehavioralScience,(2013).

22

[43]P.Reed, J.B.Kollat,V.K.Devireddy,Using interactivearchives inevolutionarymultiobjectiveoptimization:Acasestudyforlong-termgroundwatermonitoringdesign,Environ.Model.Softw.,22(2007)683-692.