distributed hydropower models in stochasticprograms...memory-distributed minimize data passing i...

109
KTH ROYAL INSTITUTE OF TECHNOLOGY Distributed Hydropower Models in StochasticPrograms.jl Martin Biel KTH - Royal Institute of Technology July 30, 2019

Upload: others

Post on 18-Feb-2021

9 views

Category:

Documents


0 download

TRANSCRIPT

  • KTH ROYAL INSTITUTE

    OF TECHNOLOGY

    Distributed Hydropower Models inStochasticPrograms.jl

    Martin Biel

    KTH - Royal Institute of Technology

    July 30, 2019

  • Motivation

    • Simulation of hydro power operations for decision-support

    I Future electricity prices unknownI Irregular power production: solar and windI Nuclear power phase-out

    • Common: trade-off between accuracy and computation time• Aim: provide reliable decision-support in real time

    I Accurate models: optimal model reductionsI Fast computations: scalable algorithms on commodity hardware

    Figure: Manageable models. Figure: Scalable algorithms.

    Martin Biel (KTH) July 30, 2019 2/31

  • Motivation

    • Simulation of hydro power operations for decision-supportI Future electricity prices unknownI Irregular power production: solar and windI Nuclear power phase-out

    • Common: trade-off between accuracy and computation time• Aim: provide reliable decision-support in real time

    I Accurate models: optimal model reductionsI Fast computations: scalable algorithms on commodity hardware

    Figure: Manageable models. Figure: Scalable algorithms.

    Martin Biel (KTH) July 30, 2019 2/31

  • Motivation

    • Simulation of hydro power operations for decision-supportI Future electricity prices unknownI Irregular power production: solar and windI Nuclear power phase-out

    • Common: trade-off between accuracy and computation time

    • Aim: provide reliable decision-support in real timeI Accurate models: optimal model reductionsI Fast computations: scalable algorithms on commodity hardware

    Figure: Manageable models. Figure: Scalable algorithms.

    Martin Biel (KTH) July 30, 2019 2/31

  • Motivation

    • Simulation of hydro power operations for decision-supportI Future electricity prices unknownI Irregular power production: solar and windI Nuclear power phase-out

    • Common: trade-off between accuracy and computation time• Aim: provide reliable decision-support in real time

    I Accurate models: optimal model reductionsI Fast computations: scalable algorithms on commodity hardware

    Figure: Manageable models. Figure: Scalable algorithms.

    Martin Biel (KTH) July 30, 2019 2/31

  • Motivation

    • Simulation of hydro power operations for decision-supportI Future electricity prices unknownI Irregular power production: solar and windI Nuclear power phase-out

    • Common: trade-off between accuracy and computation time• Aim: provide reliable decision-support in real time

    I Accurate models: optimal model reductionsI Fast computations: scalable algorithms on commodity hardware

    Figure: Manageable models. Figure: Scalable algorithms.

    Martin Biel (KTH) July 30, 2019 2/31

  • Motivation

    • Simulation of hydro power operations for decision-supportI Future electricity prices unknownI Irregular power production: solar and windI Nuclear power phase-out

    • Common: trade-off between accuracy and computation time• Aim: provide reliable decision-support in real time

    I Accurate models: optimal model reductionsI Fast computations: scalable algorithms on commodity hardware

    Figure: Manageable models. Figure: Scalable algorithms.

    Martin Biel (KTH) July 30, 2019 2/31

  • Motivation

    Stochastic programming for hydro power operations

    • Optimal orders on the day-ahead market• Maintenance scheduling• Long-term investments• Wind/solar uncertainties• . . .

    In short

    • Accurate decision-support under uncertainty• Variety of parallel decomposition schemes

    Martin Biel (KTH) July 30, 2019 3/31

  • Motivation

    Stochastic programming for hydro power operations

    • Optimal orders on the day-ahead market• Maintenance scheduling• Long-term investments• Wind/solar uncertainties• . . .

    In short

    • Accurate decision-support under uncertainty• Variety of parallel decomposition schemes

    Martin Biel (KTH) July 30, 2019 3/31

  • Contribution

    • StochasticPrograms.jl: framework for stochastic programming

    • Formulate, solve and analyze stochastic models• A collection of structure-exploiting algorithms• Distributed-memory implementation for large-scale models• Collection of (hydroelectric) energy planning models

    Efficiently model and solve stochastic problems using expressive syntax

    Martin Biel (KTH) July 30, 2019 4/31

  • Contribution

    • StochasticPrograms.jl: framework for stochastic programming• Formulate, solve and analyze stochastic models

    • A collection of structure-exploiting algorithms• Distributed-memory implementation for large-scale models• Collection of (hydroelectric) energy planning models

    Efficiently model and solve stochastic problems using expressive syntax

    Martin Biel (KTH) July 30, 2019 4/31

  • Contribution

    • StochasticPrograms.jl: framework for stochastic programming• Formulate, solve and analyze stochastic models• A collection of structure-exploiting algorithms

    • Distributed-memory implementation for large-scale models• Collection of (hydroelectric) energy planning models

    Efficiently model and solve stochastic problems using expressive syntax

    Martin Biel (KTH) July 30, 2019 4/31

  • Contribution

    • StochasticPrograms.jl: framework for stochastic programming• Formulate, solve and analyze stochastic models• A collection of structure-exploiting algorithms• Distributed-memory implementation for large-scale models

    • Collection of (hydroelectric) energy planning models

    Efficiently model and solve stochastic problems using expressive syntax

    Martin Biel (KTH) July 30, 2019 4/31

  • Contribution

    • StochasticPrograms.jl: framework for stochastic programming• Formulate, solve and analyze stochastic models• A collection of structure-exploiting algorithms• Distributed-memory implementation for large-scale models• Collection of (hydroelectric) energy planning models

    Efficiently model and solve stochastic problems using expressive syntax

    Martin Biel (KTH) July 30, 2019 4/31

  • Contribution

    • StochasticPrograms.jl: framework for stochastic programming• Formulate, solve and analyze stochastic models• A collection of structure-exploiting algorithms• Distributed-memory implementation for large-scale models• Collection of (hydroelectric) energy planning models

    Efficiently model and solve stochastic problems using expressive syntax

    Martin Biel (KTH) July 30, 2019 4/31

  • Outline

    • StochasticPrograms.jl showcase• Real-world application: the day-ahead problem• Numerical experiments• Final remarks

    Martin Biel (KTH) July 30, 2019 5/31

  • StochasticPrograms.jl

    • Flexible and expressive problem definition• Deferred model instantiation• Scenario data injection• Variety of tools for analyzing models

    I VSSI EVPII Confidence intervalsI . . .

    • Memory-distributed• Minimize data passing

    I Lightweight sampler objects to generate scenario dataI Lightweight model recipes to generate second stage problems

    • Interface to structure-exploiting (distributed) solver algorithmsI L-shaped variants (LShapedSolvers.jl)I Progressive-hedging variants (ProgressiveHedgingSolvers.jl)

    Martin Biel (KTH) July 30, 2019 6/31

  • StochasticPrograms.jl

    • Flexible and expressive problem definition

    • Deferred model instantiation• Scenario data injection• Variety of tools for analyzing models

    I VSSI EVPII Confidence intervalsI . . .

    • Memory-distributed• Minimize data passing

    I Lightweight sampler objects to generate scenario dataI Lightweight model recipes to generate second stage problems

    • Interface to structure-exploiting (distributed) solver algorithmsI L-shaped variants (LShapedSolvers.jl)I Progressive-hedging variants (ProgressiveHedgingSolvers.jl)

    Martin Biel (KTH) July 30, 2019 6/31

  • StochasticPrograms.jl

    • Flexible and expressive problem definition• Deferred model instantiation• Scenario data injection

    • Variety of tools for analyzing modelsI VSSI EVPII Confidence intervalsI . . .

    • Memory-distributed• Minimize data passing

    I Lightweight sampler objects to generate scenario dataI Lightweight model recipes to generate second stage problems

    • Interface to structure-exploiting (distributed) solver algorithmsI L-shaped variants (LShapedSolvers.jl)I Progressive-hedging variants (ProgressiveHedgingSolvers.jl)

    Martin Biel (KTH) July 30, 2019 6/31

  • StochasticPrograms.jl

    • Flexible and expressive problem definition• Deferred model instantiation• Scenario data injection• Variety of tools for analyzing models

    I VSSI EVPII Confidence intervalsI . . .

    • Memory-distributed• Minimize data passing

    I Lightweight sampler objects to generate scenario dataI Lightweight model recipes to generate second stage problems

    • Interface to structure-exploiting (distributed) solver algorithmsI L-shaped variants (LShapedSolvers.jl)I Progressive-hedging variants (ProgressiveHedgingSolvers.jl)

    Martin Biel (KTH) July 30, 2019 6/31

  • StochasticPrograms.jl

    • Flexible and expressive problem definition• Deferred model instantiation• Scenario data injection• Variety of tools for analyzing models

    I VSSI EVPII Confidence intervalsI . . .

    • Memory-distributed

    • Minimize data passingI Lightweight sampler objects to generate scenario dataI Lightweight model recipes to generate second stage problems

    • Interface to structure-exploiting (distributed) solver algorithmsI L-shaped variants (LShapedSolvers.jl)I Progressive-hedging variants (ProgressiveHedgingSolvers.jl)

    Martin Biel (KTH) July 30, 2019 6/31

  • StochasticPrograms.jl

    • Flexible and expressive problem definition• Deferred model instantiation• Scenario data injection• Variety of tools for analyzing models

    I VSSI EVPII Confidence intervalsI . . .

    • Memory-distributed• Minimize data passing

    I Lightweight sampler objects to generate scenario dataI Lightweight model recipes to generate second stage problems

    • Interface to structure-exploiting (distributed) solver algorithmsI L-shaped variants (LShapedSolvers.jl)I Progressive-hedging variants (ProgressiveHedgingSolvers.jl)

    Martin Biel (KTH) July 30, 2019 6/31

  • StochasticPrograms.jl

    • Flexible and expressive problem definition• Deferred model instantiation• Scenario data injection• Variety of tools for analyzing models

    I VSSI EVPII Confidence intervalsI . . .

    • Memory-distributed• Minimize data passing

    I Lightweight sampler objects to generate scenario dataI Lightweight model recipes to generate second stage problems

    • Interface to structure-exploiting (distributed) solver algorithmsI L-shaped variants (LShapedSolvers.jl)I Progressive-hedging variants (ProgressiveHedgingSolvers.jl)

    Martin Biel (KTH) July 30, 2019 6/31

  • StochasticPrograms.jl - Simple model

    minimizex1,x2∈R

    100x1 + 150x2 + Eω[Q(x1, x2, ξ)]

    s.t. x1 + x2 ≤ 120x1 ≥ 40x2 ≥ 20

    whereQ(x1, x2, ξ) = min

    y1,y2∈Rq1(ξ)y1 + q2(ξ)y2

    s.t. 6y1 + 10y2 ≤ 60x18y1 + 5y2 ≤ 80x20 ≤ y1 ≤ d1(ξ)0 ≤ y2 ≤ d2(ξ)

    Martin Biel (KTH) July 30, 2019 7/31

  • StochasticPrograms.jl - Simple model

    � �simple_model = @stochastic_model begin

    @stage 1 begin@variable(model, x1 >= 40)@variable(model, x2 >= 20)@objective(model, Min, 100*x1 + 150*x2)@constraint(model, x1 + x2

  • StochasticPrograms.jl - Simple model

    � �simple_model = @stochastic_model begin

    @stage 1 begin@variable(model, x1 >= 40)@variable(model, x2 >= 20)@objective(model, Min, 100*x1 + 150*x2)@constraint(model, x1 + x2

  • StochasticPrograms.jl - Simple model

    � �simple_model = @stochastic_model begin

    @stage 1 begin@variable(model, x1 >= 40)@variable(model, x2 >= 20)@objective(model, Min, 100*x1 + 150*x2)@constraint(model, x1 + x2

  • StochasticPrograms.jl - Simple model

    � �simple_model = @stochastic_model begin

    @stage 1 begin@variable(model, x1 >= 40)@variable(model, x2 >= 20)@objective(model, Min, 100*x1 + 150*x2)@constraint(model, x1 + x2

  • StochasticPrograms.jl - Simple model

    � �simple_model = @stochastic_model begin

    @stage 1 begin@variable(model, x1 >= 40)@variable(model, x2 >= 20)@objective(model, Min, 100*x1 + 150*x2)@constraint(model, x1 + x2

  • StochasticPrograms.jl - Simple model

    � �simple_model = @stochastic_model begin

    @stage 1 begin@variable(model, x1 >= 40)@variable(model, x2 >= 20)@objective(model, Min, 100*x1 + 150*x2)@constraint(model, x1 + x2

  • StochasticPrograms.jl - Discrete distribution

    � �s1 = Scenario(q1 = -24.0, q2 = -28.0, d1 = 500.0, d2 = 100.0, probability = 0.4);s2 = Scenario(q1 = -28.0, q2 = -32.0, d1 = 300.0, d2 = 300.0, probability = 0.6);simple_discrete = instantiate(simple_model, [s1,s2])

    Stochastic program with:* 2 decision variables* 2 recourse variables* 2 scenarios of type ScenarioSolver is default solver� �

    Martin Biel (KTH) July 30, 2019 8/31

  • StochasticPrograms.jl - Discrete distribution� �p r i n t ( s i m p l e _ d i s c r e t e )

    F i r s t - s t a g e==============Min 100 x 1 + 150 x 2S u b j e c t t o

    x 1 + x 2 ≤ 120x 1 ≥ 40x 2 ≥ 20

    Second - s t a g e==============S u b p r o b l e m 1 ( p = 0 .4 0 ) :Min - 24 y 1 - 28 y 2S u b j e c t t o

    - 60 x 1 + 6 y 1 + 10 y 2 ≤ 0- 80 x 2 + 8 y 1 + 5 y 2 ≤ 00 ≤ y 1 ≤ 5000 ≤ y 2 ≤ 100

    S u b p r o b l e m 2 ( p = 0 .6 0 ) :Min - 28 y 1 - 32 y 2S u b j e c t t o

    6 y 1 + 10 y 2 - 60 x 1 ≤ 08 y 1 + 5 y 2 - 80 x 2 ≤ 00 ≤ y 1 ≤ 3000 ≤ y 2 ≤ 300� �

    Martin Biel (KTH) July 30, 2019 8/31

  • StochasticPrograms.jl - Discrete distribution

    � �dep = DEP(simple_discrete)print(dep)

    Min 100 x1 + 150 x2 - 9.6 y11 - 11.2 y2 1 - 16.8 y12 - 19.2 y2 2Subject tox1 + x2 ≤ 1206 y11 + 10 y2 1 - 60 x1 ≤ 08 y11 + 5 y2 1 - 80 x2 ≤ 06 y12 + 10 y2 2 - 60 x1 ≤ 08 y12 + 5 y2 2 - 80 x2 ≤ 0x1 ≥ 40x2 ≥ 200 ≤ y11 ≤ 5000 ≤ y2 1 ≤ 1000 ≤ y12 ≤ 3000 ≤ y2 2 ≤ 300� �

    Martin Biel (KTH) July 30, 2019 8/31

  • StochasticPrograms.jl - Discrete distribution

    � �vrp = VRP(simple_discrete, solver = glpk)-855.83

    vss = VSS(simple_discrete, solver = glpk)286.92

    evpi = EVPI(simple, solver = glpk)662.92� �

    Martin Biel (KTH) July 30, 2019 8/31

  • StochasticPrograms.jl - Continuous distribution

    � �@sampler SimpleSampler = begin

    N::MvNormal

    SimpleSampler(µ, Σ) = new(MvNormal(µ, Σ))

    @sample Scenario beginx = rand(sampler.N)return Scenario(q1 = x[1], q2 = x[2], d1 = x[3], d2 = x[4])

    endend

    µ = [-28, -32, 300, 300]Σ = [2 0.5 0 0

    0.5 1 0 00 0 50 200 0 20 30]

    sampler = SimpleSampler(µ, Σ)� �Martin Biel (KTH) July 30, 2019 9/31

  • StochasticPrograms.jl - Continuous distribution

    � �saa = SAA(simple_model, sampler, 100)

    Stochastic program with:* 2 decision variables* 2 recourse variables* 100 scenarios of type ScenarioSolver is default solver� �

    � �confidence_interval(simple_model, sampler; solver = glpk, confidence = 0.95,

    N = 100)Confidence interval (p = 95%): [-2630.44 � -2389.31]� �� �confidence_interval(simple_model, sampler; solver = glpk, confidence = 0.95,

    N = 1000)Confidence interval (p = 95%): [-2568.90 � -2509.78]� �

    Martin Biel (KTH) July 30, 2019 9/31

  • StochasticPrograms.jl - Continuous distribution

    � �saa = SAA(simple_model, sampler, 100)

    Stochastic program with:* 2 decision variables* 2 recourse variables* 100 scenarios of type ScenarioSolver is default solver� �� �confidence_interval(simple_model, sampler; solver = glpk, confidence = 0.95,

    N = 100)Confidence interval (p = 95%): [-2630.44 � -2389.31]� �

    � �confidence_interval(simple_model, sampler; solver = glpk, confidence = 0.95,

    N = 1000)Confidence interval (p = 95%): [-2568.90 � -2509.78]� �

    Martin Biel (KTH) July 30, 2019 8/31

  • StochasticPrograms.jl - Continuous distribution

    � �saa = SAA(simple_model, sampler, 100)

    Stochastic program with:* 2 decision variables* 2 recourse variables* 100 scenarios of type ScenarioSolver is default solver� �� �confidence_interval(simple_model, sampler; solver = glpk, confidence = 0.95,

    N = 100)Confidence interval (p = 95%): [-2630.44 � -2389.31]� �� �confidence_interval(simple_model, sampler; solver = glpk, confidence = 0.95,

    N = 1000)Confidence interval (p = 95%): [-2568.90 � -2509.78]� �

    Martin Biel (KTH) July 30, 2019 7/31

  • Day-ahead problem - Electricity market

    Day-Ahead

    Actor 1

    P1 E1

    Actor 2

    P2 E2

    Market closes

    Actor 1

    Actor 2

    Balance responsible

    PM,E1

    Balance responsible

    PM,E2

    Next day

    Intraday

    P E

    P E

    Martin Biel (KTH) July 30, 2019 8/31

  • Day-ahead problem - Electricity market

    Day-Ahead

    Actor 1

    P1 E1

    Actor 2

    P2 E2

    Market closes

    Actor 1

    Actor 2

    Balance responsible

    PM,E1

    Balance responsible

    PM,E2

    Next day

    Intraday

    P E

    P E

    Martin Biel (KTH) July 30, 2019 8/31

  • Day-ahead problem - Electricity market

    Day-Ahead

    Actor 1

    P1 E1

    Actor 2

    P2 E2

    Market closes

    Actor 1

    Actor 2

    Balance responsible

    PM,E1

    Balance responsible

    PM,E2

    Next day

    Intraday

    P E

    P E

    Martin Biel (KTH) July 30, 2019 8/31

  • Day-ahead problem - Electricity market

    Day-Ahead

    Actor 1

    P1 E1

    Actor 2

    P2 E2

    Market closes

    Actor 1

    Actor 2

    Balance responsible

    PM,E1

    Balance responsible

    PM,E2

    Next day

    Intraday

    P E

    P E

    Martin Biel (KTH) July 30, 2019 8/31

  • Day-ahead problem - Electricity market

    Order Types

    • Single Hourly OrderI Price independentI Price Dependent

    • Block OrderI RegularI Linked

    • Exclusive Group• Flexible Order

    Martin Biel (KTH) July 30, 2019 9/31

  • Day-ahead problem - Electricity market

    Order Types

    • Single Hourly OrderI Price independentI Price Dependent

    • Block OrderI RegularI Linked

    • Exclusive Group• Flexible Order

    Martin Biel (KTH) July 30, 2019 9/31

  • Day-ahead problem - Single order

    0.00 121.73 243.47 365.20 486.93 608.66 730.40Order Volume [MWh/h]

    0.00

    11.85

    23.70

    35.55

    47.40

    59.25

    Price

    [EUR

    /MW

    h]

    Order CurvePrice Independent OrderPrice Dependent Order

    Figure: Single hourly order.

    Martin Biel (KTH) July 30, 2019 10/31

  • Day-ahead problem - Single order

    0.00 121.73 243.47 365.20 486.93 608.66 730.40Order Volume [MWh/h]

    0.00

    11.85

    23.70

    35.55

    47.40

    59.25

    Price

    [EUR

    /MW

    h]

    Order CurvePrice Independent OrderPrice Dependent OrderTrading Outcome

    Figure: Interpolated energy volume for a given market price.

    Martin Biel (KTH) July 30, 2019 10/31

  • Day-ahead problem - Block order

    15 20Hour

    Block Order

    14.93 [EUR/MWh] 500.00 [MWh/h]

    Figure: Block order between 15:00-20:00.

    Martin Biel (KTH) July 30, 2019 11/31

  • Day-ahead problem - Block order

    15 20Hour

    6.71

    9.58

    12.46

    15.33

    18.21

    Price

    [EUR

    /MW

    h]

    Block Order

    14.93 [EUR/MWh] 500.00 [MWh/h]

    Rejected OrderMarket price

    Figure: Rejected after market price settlement.

    Martin Biel (KTH) July 30, 2019 11/31

  • Day-ahead problem - Block order

    15 20Hour

    33.53

    37.13

    40.72

    44.31

    47.91

    51.50

    55.10

    58.69

    62.28

    65.88

    Price

    [EUR

    /MW

    h]

    Block Order

    14.93 [EUR/MWh] 500.00 [MWh/h]

    Accepted OrderMarket price

    Figure: Accepted after market price settlement.

    Martin Biel (KTH) July 30, 2019 11/31

  • Day-ahead problem - Setting

    • Price taking hydropower producer trading in the NordPool market

    • All power stations in the Swedish river Skellefteälven• First stage: hourly electricity volume bids for the upcoming day

    I Single hourly ordersI Block orders

    • Second stage: optimize day-ahead productionI Bid dispatch after market price realizationI Imbalances penalized in intraday marketI Water flow conversation (including water travel time)I Maximize profits in the market and the future value of water

    • Full model defined in HydroModels.jl

    Martin Biel (KTH) July 30, 2019 12/31

  • Day-ahead problem - Setting

    • Price taking hydropower producer trading in the NordPool market• All power stations in the Swedish river Skellefteälven

    • First stage: hourly electricity volume bids for the upcoming dayI Single hourly ordersI Block orders

    • Second stage: optimize day-ahead productionI Bid dispatch after market price realizationI Imbalances penalized in intraday marketI Water flow conversation (including water travel time)I Maximize profits in the market and the future value of water

    • Full model defined in HydroModels.jl

    Martin Biel (KTH) July 30, 2019 12/31

  • Day-ahead problem - Setting

    • Price taking hydropower producer trading in the NordPool market• All power stations in the Swedish river Skellefteälven• First stage: hourly electricity volume bids for the upcoming day

    I Single hourly ordersI Block orders

    • Second stage: optimize day-ahead productionI Bid dispatch after market price realizationI Imbalances penalized in intraday marketI Water flow conversation (including water travel time)I Maximize profits in the market and the future value of water

    • Full model defined in HydroModels.jl

    Martin Biel (KTH) July 30, 2019 12/31

  • Day-ahead problem - Setting

    • Price taking hydropower producer trading in the NordPool market• All power stations in the Swedish river Skellefteälven• First stage: hourly electricity volume bids for the upcoming day

    I Single hourly ordersI Block orders

    • Second stage: optimize day-ahead productionI Bid dispatch after market price realizationI Imbalances penalized in intraday marketI Water flow conversation (including water travel time)I Maximize profits in the market and the future value of water

    • Full model defined in HydroModels.jl

    Martin Biel (KTH) July 30, 2019 12/31

  • Day-ahead problem - Setting

    • Price taking hydropower producer trading in the NordPool market• All power stations in the Swedish river Skellefteälven• First stage: hourly electricity volume bids for the upcoming day

    I Single hourly ordersI Block orders

    • Second stage: optimize day-ahead productionI Bid dispatch after market price realizationI Imbalances penalized in intraday marketI Water flow conversation (including water travel time)I Maximize profits in the market and the future value of water

    • Full model defined in HydroModels.jl

    Martin Biel (KTH) July 30, 2019 12/31

  • Day-ahead problem - Data

    Deterministic

    • Physical parameters for power plants in Skellefteälven• Trade regulations from NordPool

    Uncertain

    • Day-ahead prices from NordPool• Mean water flows in Skellefteälven from SMHI

    Martin Biel (KTH) July 30, 2019 13/31

  • Day-ahead problem - Data

    Figure: Schematic of the power stations in Skellefteälven.

    Martin Biel (KTH) July 30, 2019 13/31

  • Day-ahead problem - Data

    Figure: Historical day-ahead prices 2013-2018 from NordPool.

    Martin Biel (KTH) July 30, 2019 13/31

  • Day-ahead problem - Data

    Figure: Mean water flow in Skellefteälven 1999-2018 from SMHI.

    Martin Biel (KTH) July 30, 2019 13/31

  • Day-ahead problem - Forecasts

    • Recurrent neural networks (GRU)

    • Trained on price data and mean flow data separately• Early stopping to prevent overfitting• Seasonality modeled through separate inputs to the network• Driven by Gaussian noise

    Martin Biel (KTH) July 30, 2019 14/31

  • Day-ahead problem - Forecasts

    • Recurrent neural networks (GRU)• Trained on price data and mean flow data separately

    • Early stopping to prevent overfitting• Seasonality modeled through separate inputs to the network• Driven by Gaussian noise

    Martin Biel (KTH) July 30, 2019 14/31

  • Day-ahead problem - Forecasts

    • Recurrent neural networks (GRU)• Trained on price data and mean flow data separately• Early stopping to prevent overfitting

    • Seasonality modeled through separate inputs to the network• Driven by Gaussian noise

    Martin Biel (KTH) July 30, 2019 14/31

  • Day-ahead problem - Forecasts

    • Recurrent neural networks (GRU)• Trained on price data and mean flow data separately• Early stopping to prevent overfitting• Seasonality modeled through separate inputs to the network

    • Driven by Gaussian noise

    Martin Biel (KTH) July 30, 2019 14/31

  • Day-ahead problem - Forecasts

    • Recurrent neural networks (GRU)• Trained on price data and mean flow data separately• Early stopping to prevent overfitting• Seasonality modeled through separate inputs to the network• Driven by Gaussian noise

    Martin Biel (KTH) July 30, 2019 14/31

  • Day-ahead problem - Forecasts

    5 10 15 202013

    0255075

    100

    5 10 15 202014

    020406080

    5 10 15 202015

    020406080

    5 10 15 202016

    050

    100150200

    5 10 15 202017

    255075

    100125

    5 10 15 202018

    050

    100150200

    Figure: Price forecasts (black) and raw data (colored).

    Martin Biel (KTH) July 30, 2019 15/31

  • Day-ahead problem - Forecasts

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24Hour

    25

    50

    75

    100

    125

    Price

    [Eur

    ]

    Figure: 1000 sampled price curves using RNN.

    Martin Biel (KTH) July 30, 2019 16/31

  • Day-ahead problem - Forecasts

    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov DecMonth

    25

    50

    75

    100

    125

    Price

    [Eur

    ]

    Figure: Price forecasts throughout a year.

    Martin Biel (KTH) July 30, 2019 17/31

  • Day-ahead problem - Forecasts

    1 31 61 91 121 151 181 211 241 271 301 331 3612018

    100

    200

    300

    400

    500

    600

    700

    0 100 200 300

    50

    75

    100

    125

    150

    Figure: Mean flow forecasts (black) and raw data (colored).

    Martin Biel (KTH) July 30, 2019 17/31

  • Day-ahead problem - Sampler

    � �@scenario DayAheadScenario = begin

    ρ::PriceCurve{Float64}Q̃::Vector{Float64}

    end@sampler RecurrentDayAheadSampler = begin

    date::Dateprice_forecaster::Forecaster{:price}flow_forecaster::Forecaster{:flow}

    @sample DayAheadScenario beginprices = forecast(sampler.price_forecaster, month(sampler.date))flows = forecast(sampler.flow_forecaster, week(sampler.date))return DayAheadScenario(PriceCurve(prices), flows[1])

    endend� �

    Martin Biel (KTH) July 30, 2019 18/31

  • Day-ahead problem - Bidlevels

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24Hour

    10

    20

    30

    40

    50

    60

    Price

    [Eur

    ]

    Expected priceBidlevels

    Figure: Available price points for bidding.

    Martin Biel (KTH) July 30, 2019 19/31

  • Day-ahead problem - Value of water

    • Marginal value of water has large impact on optimal dispatch

    • Sometimes optimal to accept imbalance penalty and save water• Naive approach: production from excess water solved at mean price• Leads to crude order strategies

    Martin Biel (KTH) July 30, 2019 20/31

  • Day-ahead problem - Value of water

    • Marginal value of water has large impact on optimal dispatch• Sometimes optimal to accept imbalance penalty and save water

    • Naive approach: production from excess water solved at mean price• Leads to crude order strategies

    Martin Biel (KTH) July 30, 2019 20/31

  • Day-ahead problem - Value of water

    • Marginal value of water has large impact on optimal dispatch• Sometimes optimal to accept imbalance penalty and save water• Naive approach: production from excess water solved at mean price

    • Leads to crude order strategies

    Martin Biel (KTH) July 30, 2019 20/31

  • Day-ahead problem - Value of water

    • Marginal value of water has large impact on optimal dispatch• Sometimes optimal to accept imbalance penalty and save water• Naive approach: production from excess water solved at mean price• Leads to crude order strategies

    Martin Biel (KTH) July 30, 2019 20/31

  • Day-ahead problem - Value of water

    • Solve a dummy stochastic program:I First stage: water content in reservoirsI Second stage: optimize production over the coming weekI Future prices and water inflows are uncertain

    • L-shaped generates a polyhedral objective approximation• Approximation used to model the expected future value of water

    Figure: Polyhedral approximation.

    Martin Biel (KTH) July 30, 2019 21/31

  • Day-ahead problem - Value of water

    • Solve a dummy stochastic program:I First stage: water content in reservoirsI Second stage: optimize production over the coming weekI Future prices and water inflows are uncertain

    • L-shaped generates a polyhedral objective approximation

    • Approximation used to model the expected future value of water

    Figure: Polyhedral approximation.

    Martin Biel (KTH) July 30, 2019 21/31

  • Day-ahead problem - Value of water

    • Solve a dummy stochastic program:I First stage: water content in reservoirsI Second stage: optimize production over the coming weekI Future prices and water inflows are uncertain

    • L-shaped generates a polyhedral objective approximation• Approximation used to model the expected future value of water

    Figure: Polyhedral approximation.

    Martin Biel (KTH) July 30, 2019 21/31

  • Day-ahead problem - Model� �@stage 1 begin

    @parameters beginhorizon = horizonindices = indicesdata = data

    end@unpack hours, plants, bids, blockbids, blocks = indices@unpack hydrodata, regulations = data# Variables# ========================================================@variable(model, xt_i[t = hours] >= 0)@variable(model, xt_d[i = bids, t = hours] >= 0)@variable(model, xb[i = blockbids, b = blocks] >= 0)...

    end� �

    Martin Biel (KTH) July 30, 2019 22/31

  • Day-ahead problem - Model� �@stage 2 begin

    ...@uncertain ρ, V from ξ::DayAheadScenario@decision xt_i xt_d xb# -------------------------------------------------------@variable(model, yt[t = hours] >= 0) # Hourly dispatch@variable(model, yb[b = blocks] >= 0) # Block dispatch@variable(model, z_up[t = hours] >= 0) # Power bought from intraday@variable(model, z_do[t = hours] >= 0) # Power sold to intraday@variable(model, 0 = 0) # Incoming discharge@variable(model, Sf[p = plants, t = hours] >= 0) # Incoming spillage@variable(model, 0

  • Day-ahead problem - Model� �...# Bid-dispatch links@constraint(model, hourlybids[t = hours],

    yt[t] == interpolate(ρ[t], bidlevels, xt_d[t]) + xt_i[t])

    @constraint(model, bidblocks[b = blocks],yb[b] == sum(xb[j,b] for j = accepted_blocks(b))

    )# Hydrological balance@constraint(model, hydro_constraints[p = plants, t = hours],

    # Previous reservoir contentM[p,t] == (t > 1 ? M[p,t-1] : M0[p])# Inflow+ sum(Qf[i,t]+Sf[i,t] for i = upstream_plants[p])# Local inflow+ V[p]# Outflow- (Q[p,t] + S[p,t]))

    ...� �Martin Biel (KTH) July 30, 2019 22/31

  • Day-ahead problem - Model� �...# Production@constraint(model, production[t = hours],

    H[t] == sum(hydrodata[p].µ[s]*Q[p,s,t]for p = plants, s = segments)

    )# Load balance@constraint(model, loadbalance[t = hours],

    yt[t] + sum(yb[b] for b = blocks[t]) - H[t] == z_up[t] - z_do[t])

    ...# Water travel time...# Water value@constraint(model, water_value_approximation[c = 1:ncuts(water_value)],

    sum(water_value[c][p]*M[p,nhours(horizon)]for p in plants)

    + sum(W[i]for i in cut_indices(water_value[c]))

    >= cut_lb(water_value[c]))end� �

    Martin Biel (KTH) July 30, 2019 22/31

  • Day-ahead problem - Algorithm

    • VSS typically low in day-ahead problems

    • Generate tight confidence intervals trough sequential SAA algorithm• Ensure statistically significant VSS• SAA instances of ~2000 scenarios required to reach this bound

    I ~5 million variablesI ~3.3 million constraints

    Martin Biel (KTH) July 30, 2019 23/31

  • Day-ahead problem - Algorithm

    • VSS typically low in day-ahead problems• Generate tight confidence intervals trough sequential SAA algorithm

    • Ensure statistically significant VSS• SAA instances of ~2000 scenarios required to reach this bound

    I ~5 million variablesI ~3.3 million constraints

    Martin Biel (KTH) July 30, 2019 23/31

  • Day-ahead problem - Algorithm

    • VSS typically low in day-ahead problems• Generate tight confidence intervals trough sequential SAA algorithm• Ensure statistically significant VSS

    • SAA instances of ~2000 scenarios required to reach this boundI ~5 million variablesI ~3.3 million constraints

    Martin Biel (KTH) July 30, 2019 23/31

  • Day-ahead problem - Algorithm

    • VSS typically low in day-ahead problems• Generate tight confidence intervals trough sequential SAA algorithm• Ensure statistically significant VSS• SAA instances of ~2000 scenarios required to reach this bound

    I ~5 million variablesI ~3.3 million constraints

    Martin Biel (KTH) July 30, 2019 23/31

  • Day-ahead problem - Algorithm

    Sequential SAA• Lower bound: solve M SAA models of size N• Upper bound: decision evaluation on T SAA models of size Ñ > N• Increase N iteratively until confidence interval is tight enough

    Distributed L-shaped• Regularization

    I Trust-regionsI Level-setsI . . .

    • AggregationI StaticI DynamicI ClusteringI . . .

    Martin Biel (KTH) July 30, 2019 24/31

  • Day-ahead problem - Algorithm

    Sequential SAA• Lower bound: solve M SAA models of size N• Upper bound: decision evaluation on T SAA models of size Ñ > N• Increase N iteratively until confidence interval is tight enough

    Distributed L-shaped• Regularization

    I Trust-regionsI Level-setsI . . .

    • AggregationI StaticI DynamicI ClusteringI . . .

    Martin Biel (KTH) July 30, 2019 24/31

  • Results - Benchmarks

    1 2 4 8 16Number of Cores P

    18.0

    42.5

    66.9

    91.4Co

    mpu

    tatio

    n Ti

    me

    T [s

    ]Distributed L-shaped with trust-regionDistributed L-shaped

    Figure: Distributed L-shaped performance.

    Martin Biel (KTH) July 30, 2019 25/31

  • Results - Benchmarks

    1 10 50 100 125# Cuts in Bundle B

    12.1

    16.1

    20.1

    24.1Co

    mpu

    tatio

    n Ti

    me

    T [s

    ]Distributed bundled L-shaped with trust-region

    Figure: Distributed L-shaped performance using aggregation.

    Martin Biel (KTH) July 30, 2019 25/31

  • Results - Day-ahead

    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov DecMonth

    5.1×10

    5.2×10

    5.3×10

    5.4×10

    5.5×10

    5.6×10

    5.7×10

    Prof

    it [E

    ur]

    Day-aheadDay-ahead VRPDay-ahead EEV

    Figure: Day-ahead profits.

    Martin Biel (KTH) July 30, 2019 26/31

  • Results - Day-ahead

    Apr May Jun Jul AugMonth

    5.1×10

    5.2×10

    Prof

    it [E

    ur]

    Day-aheadDay-ahead VRPDay-ahead EEV

    Figure: Day-ahead profits.

    Martin Biel (KTH) July 30, 2019 26/31

  • Results - Day-ahead

    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov DecMonth

    0.1 %

    0.2 %

    0.3 %

    0.4 %

    Prof

    it [%

    of V

    RP]

    Day-aheadDay-ahead VSS

    Figure: Day-ahead value of stochastic solution.

    Martin Biel (KTH) July 30, 2019 27/31

  • Results - Day-ahead

    Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov DecMonth

    5.0×10³

    1.0×10

    1.5×10

    2.0×10

    2.5×10

    Prof

    it [E

    ur]

    Day-aheadDay-ahead VSS

    Figure: Day-ahead value of stochastic solution.

    Martin Biel (KTH) July 30, 2019 27/31

  • Results - Order strategies

    0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24Hour

    0.0013.2226.4439.6652.8866.1079.32

    Price

    [EUR

    /MW

    h]

    Single Orders

    671.1 642.3 622.2 675.6 692.9696.6

    1.1e+03

    721.3

    736.8

    1.1e+03

    742.8

    1.1e+03

    401.1

    0.0

    398.8

    507.3

    708.6

    520.5

    0.0

    364.7

    797.8

    817.0

    891.3

    867.8 865.6

    440.70.0

    175.4

    45.14

    0.0

    12.56

    17.61

    58.80

    174.6

    508.5

    611.3

    485.9

    611.3

    982.1

    1.1e+03

    1.0e+03

    1.0e+03

    1.0e+03

    1.0e+03 1.0e+03

    Independent Volume [M

    Wh]

    Dependent Volumes [M

    wh]

    0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24Hour

    Block Orders

    13.68 428.6

    14.93 500.0

    23.33 8.03

    23.78 2.39

    52.57 0.80

    Price [EUR/MWh] Volume [MWh/h]

    Figure: Day-ahead order strategy.

    Martin Biel (KTH) July 30, 2019 28/31

  • Results - Order strategies

    0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24Hour

    0.00

    13.22

    26.44

    39.66

    52.88

    66.10

    79.32

    Price

    [EUR

    /MW

    h]

    Single Orders

    671.1 642.3 622.2 675.6 692.9

    696.6

    1.1e+03

    721.3

    736.8

    1.1e+03

    742.8

    1.1e+03

    401.1

    0.0

    398.8

    507.3

    708.6

    520.5

    0.0

    364.7

    797.8

    817.0

    891.3

    867.8 865.6

    440.7

    0.0

    175.4

    45.14

    0.0

    12.56

    17.61

    58.80

    174.6

    508.5

    611.3

    485.9

    611.3

    982.1

    1.1e+03

    1.0e+03

    1.0e+03

    1.0e+03

    1.0e+03 1.0e+03

    Independent Volume [M

    Wh]

    Dependent Volumes [M

    wh]

    Figure: Day-ahead single hourly order strategy.

    Martin Biel (KTH) July 30, 2019 28/31

  • Results - Order strategies

    0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24Hour

    0.00

    12.38

    24.76

    37.13

    49.51

    61.89

    74.27

    Price

    [EUR

    /MW

    h]

    Single Orders

    671.1 642.3 622.2 675.6 692.9699.1

    739.7

    802.3

    401.1

    509.3

    520.5

    364.7 817.0 891.3

    867.8 865.6

    440.7 27.77

    45.14

    16.15

    60.89

    509.8

    505.4 1.0e+03 1.0e+031.0e+03 1.0e+03

    Independent Volume [M

    Wh]

    Dependent Volumes [M

    wh]

    Accepted OrdersRejected OrdersMarket price

    Figure: Result of single order strategy after realized market price.

    Martin Biel (KTH) July 30, 2019 29/31

  • Results - Order strategies

    0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24Hour

    0.00

    12.38

    24.76

    37.13

    49.51

    61.89

    74.27

    Price

    [EUR

    /MW

    h]

    Single Orders

    671.1 642.3 622.2 675.6 692.9 699.1739.7

    802.3

    401.1

    509.3

    520.5

    364.7 817.0 891.3

    867.8 865.6

    440.7 27.77

    45.14

    16.1560.89

    509.8505.4 1.0e+03 1.0e+03 1.0e+03 1.0e+03

    Independent Volume [M

    Wh]

    Dependent Volumes [M

    wh]

    Accepted OrdersRejected OrdersMarket price

    0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24Hour

    33.53

    44.43

    55.33

    66.23

    Price

    [EUR

    /MW

    h]

    Block Orders

    54.97 500.056.92 428.6

    62.59 2.3958.08 8.03

    54.93 0.80

    Price [EUR/MWh] Volume [MWh/h]Accepted OrdersMarket price

    Figure: Result of complete order strategy after realized market price.

    Martin Biel (KTH) July 30, 2019 29/31

  • Results - Order strategies

    0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24Hour

    0.0013.2226.4439.6652.8866.1079.32

    Price

    [EUR

    /MW

    h]

    Single Orders

    674.2

    0.0

    437.9

    0.0

    663.0

    1.1e+03

    0.0

    649.8

    1.1e+03

    0.0

    664.2

    1.1e+03

    0.0

    698.0

    1.1e+03

    706.6

    0.0

    405.5

    734.1

    0.0

    378.0

    0.0

    713.5

    1.1e+03

    926.0 887.5 803.5 932.0

    0.0

    180.1

    852.0

    0.0

    260.1

    853.5

    0.0

    258.6

    862.6 85.37

    0.0

    118.6

    88.17 1.0e+03 980.7 980.4

    0.0

    131.7

    0.0

    1.0e+03

    1.1e+03

    1.0e+03

    0.0

    101.1

    0.0

    1.0e+03

    1.1e+03

    Independent Volume [M

    Wh]

    Dependent Volumes [M

    wh]

    0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24Hour

    Block Orders

    12.18 408.2

    23.33 500.0

    Price [EUR/MWh] Volume [MWh/h]

    Figure: Deterministic order strategy.

    Martin Biel (KTH) July 30, 2019 29/31

  • Final Remarks

    Discussion

    • VSS linked to imbalance penalties in the intraday market

    • Results are only as accurate/useful as the water valuation• Model improvements required.• Proof of concept for large-scale models in StochasticPrograms.jl

    Outlook on future/ongoing work

    • Formulate new energy planning models in StochasticPrograms.jl• Evaluate different aggregation schemes• Sample-based algorithms as an alternative to sequential SAA

    Martin Biel (KTH) July 30, 2019 30/31

  • Final Remarks

    Discussion

    • VSS linked to imbalance penalties in the intraday market• Results are only as accurate/useful as the water valuation

    • Model improvements required.• Proof of concept for large-scale models in StochasticPrograms.jl

    Outlook on future/ongoing work

    • Formulate new energy planning models in StochasticPrograms.jl• Evaluate different aggregation schemes• Sample-based algorithms as an alternative to sequential SAA

    Martin Biel (KTH) July 30, 2019 30/31

  • Final Remarks

    Discussion

    • VSS linked to imbalance penalties in the intraday market• Results are only as accurate/useful as the water valuation• Model improvements required.

    • Proof of concept for large-scale models in StochasticPrograms.jlOutlook on future/ongoing work

    • Formulate new energy planning models in StochasticPrograms.jl• Evaluate different aggregation schemes• Sample-based algorithms as an alternative to sequential SAA

    Martin Biel (KTH) July 30, 2019 30/31

  • Final Remarks

    Discussion

    • VSS linked to imbalance penalties in the intraday market• Results are only as accurate/useful as the water valuation• Model improvements required.• Proof of concept for large-scale models in StochasticPrograms.jl

    Outlook on future/ongoing work

    • Formulate new energy planning models in StochasticPrograms.jl• Evaluate different aggregation schemes• Sample-based algorithms as an alternative to sequential SAA

    Martin Biel (KTH) July 30, 2019 30/31

  • Final Remarks

    Discussion

    • VSS linked to imbalance penalties in the intraday market• Results are only as accurate/useful as the water valuation• Model improvements required.• Proof of concept for large-scale models in StochasticPrograms.jl

    Outlook on future/ongoing work

    • Formulate new energy planning models in StochasticPrograms.jl

    • Evaluate different aggregation schemes• Sample-based algorithms as an alternative to sequential SAA

    Martin Biel (KTH) July 30, 2019 30/31

  • Final Remarks

    Discussion

    • VSS linked to imbalance penalties in the intraday market• Results are only as accurate/useful as the water valuation• Model improvements required.• Proof of concept for large-scale models in StochasticPrograms.jl

    Outlook on future/ongoing work

    • Formulate new energy planning models in StochasticPrograms.jl• Evaluate different aggregation schemes

    • Sample-based algorithms as an alternative to sequential SAA

    Martin Biel (KTH) July 30, 2019 30/31

  • Final Remarks

    Discussion

    • VSS linked to imbalance penalties in the intraday market• Results are only as accurate/useful as the water valuation• Model improvements required.• Proof of concept for large-scale models in StochasticPrograms.jl

    Outlook on future/ongoing work

    • Formulate new energy planning models in StochasticPrograms.jl• Evaluate different aggregation schemes• Sample-based algorithms as an alternative to sequential SAA

    Martin Biel (KTH) July 30, 2019 30/31

  • Final Remarks

    Summary

    • StochasticPrograms.jl: framework for stochastic programming

    • Large-scale day-ahead problem solved on compute cluster• Tight confidence intervals through sequential SAA• StochasticPrograms.jl is a registered Julia package• The full framework is open-source and freely available on Github

    https://github.com/martinbiel

    Martin Biel (KTH) July 30, 2019 31/31

    https://github.com/martinbiel

  • Final Remarks

    Summary

    • StochasticPrograms.jl: framework for stochastic programming• Large-scale day-ahead problem solved on compute cluster

    • Tight confidence intervals through sequential SAA• StochasticPrograms.jl is a registered Julia package• The full framework is open-source and freely available on Github

    https://github.com/martinbiel

    Martin Biel (KTH) July 30, 2019 31/31

    https://github.com/martinbiel

  • Final Remarks

    Summary

    • StochasticPrograms.jl: framework for stochastic programming• Large-scale day-ahead problem solved on compute cluster• Tight confidence intervals through sequential SAA

    • StochasticPrograms.jl is a registered Julia package• The full framework is open-source and freely available on Github

    https://github.com/martinbiel

    Martin Biel (KTH) July 30, 2019 31/31

    https://github.com/martinbiel

  • Final Remarks

    Summary

    • StochasticPrograms.jl: framework for stochastic programming• Large-scale day-ahead problem solved on compute cluster• Tight confidence intervals through sequential SAA• StochasticPrograms.jl is a registered Julia package

    • The full framework is open-source and freely available on Github

    https://github.com/martinbiel

    Martin Biel (KTH) July 30, 2019 31/31

    https://github.com/martinbiel

  • Final Remarks

    Summary

    • StochasticPrograms.jl: framework for stochastic programming• Large-scale day-ahead problem solved on compute cluster• Tight confidence intervals through sequential SAA• StochasticPrograms.jl is a registered Julia package• The full framework is open-source and freely available on Github

    https://github.com/martinbiel

    Martin Biel (KTH) July 30, 2019 31/31

    https://github.com/martinbiel