LLNL-PRES-XXXXXXThis work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC

Power System Co-simulation at LLNL:Lessons Learned and Ongoing Efforts

Nan DuanPower Systems Engineer

Apr. 16, 2021

LLNL-PRES-812739

2LLNL-PRES-xxxxxx

Why Co-simulation?

Reference:http://gridlab-d.shoutwiki.com/wiki/GridLAB-D_Wiki:GridLAB-D_Tutorial_Chapter_3_-_Basic_Distribution_System_Modeling

PSS/EPSLFGridDynGridPACK…

Synergi ElectricCYMEDISTGridLAB-DOpenDSS…

EnergyPlus…

3LLNL-PRES-xxxxxx

Why Co-simulation?

Reference: Young, Marcus, and Alison Silverstein. "Factors affecting pmu installation costs." US Department of Energy-Office of Electricity Delivery and Energy Reliability (2014).

4LLNL-PRES-xxxxxx

Co-simulation of Multiphysics Models

Transmission

Distribution

Co-simulation: couple models and simulators from different domains.

Device

5LLNL-PRES-xxxxxx

Co-simulation of Cyber-physical Models

Communication also plays an important role in power system operation.

6LLNL-PRES-xxxxxx

Overview

• Challenges• Collaborative Autonomy Use Case• Distributed Energy Resource Control Use Case• Advanced Metering Infrastructure Use Case• HPC enabled Transmission Distribution Use Case• Ongoing Efforts• Discussion

7LLNL-PRES-xxxxxx

Challenges

Lessons learned from previous co-simulation projects:

1.Models are difficult to maintain.2.Codebase compatibility between different releases

of software.3.Domain knowledge blind spots.4.Difficulties in building open-source software.5.Constantly evolving model characteristics6.Simulator preference of utility partners.7.Ad hoc coupling configuration between simulators.

8LLNL-PRES-xxxxxx

Overview

• Challenges• Collaborative Autonomy Use Case• Distributed Energy Resource Control Use Case• Advanced Metering Infrastructure Use Case• HPC enabled Transmission Distribution Use Case• Ongoing Efforts• Discussion

9LLNL-PRES-xxxxxx

§ Distributed netload estimation triggered switch action

Distributed Netload Estimation

INV 0

INV 1

INV 2

DMS

SWITCHnetloadestimation

vote

-20000

0

20000

40000

0 50 100 150

measured_real_power

-50000

0

50000

0 50 100 150

measured_real_power

-20000

0

20000

40000

0 50 100 150

measured_real_power2001-01-01 11:00:00 PST OPEN2001-01-01 11:15:00 PST OPEN2001-01-01 11:30:00 PST OPEN2001-01-01 11:45:00 PST OPEN2001-01-01 12:00:00 PST OPEN2001-01-01 12:15:00 PST CLOSED2001-01-01 12:30:00 PST CLOSED2001-01-01 12:45:00 PST CLOSED2001-01-01 13:00:00 PST CLOSED2001-01-01 13:15:00 PST CLOSED2001-01-01 13:30:00 PST CLOSED2001-01-01 13:45:00 PST CLOSED2001-01-01 14:00:00 PST CLOSED

Inverters and loads used in this co-simulation:

10LLNL-PRES-xxxxxx

Distributed Netload Estimation

11LLNL-PRES-xxxxxx

Distributed Netload Estimation

§ Distributed estimation algorithm packets:

Comms/Power co-simulation motivated us to improve our algorithm to address asynchronized progression of iterations. Iteration between 5 and 10 are safeguard iterations

vote from inv 2 sent

vote from inv 1 sent

vote from inv 0 sent

12LLNL-PRES-xxxxxx

Distributed Netload Estimation

For 100 inverters, iteration continued for 63 seconds simulation time. No inverter has reached required iteration count to send out vote. Improvement to the algorithm or communication model is need for large scale consensus. There 10 initial messages held up in the queue. May need advance traffic control.

Message enqueued but not sent

13LLNL-PRES-xxxxxx

For n inverters, with probability 1-δ, the required steps to reduce the relative approximation error to ε is:

Diffusion Speed

Reference: Kempe, David, Alin Dobra, and Johannes Gehrke. "Gossip-based computation of aggregate information." In 44th Annual IEEE Symposium on Foundations of Computer Science, 2003. Proceedings., pp. 482-491. IEEE, 2003.

1 1(log log log )O ne d

+ +

14LLNL-PRES-xxxxxx

Overview

• Challenges• Collaborative Autonomy Use Case• Distributed Energy Resource Control Use Case• Advanced Metering Infrastructure Use Case• HPC enabled Transmission Distribution Use Case• Ongoing Efforts• Discussion

15LLNL-PRES-xxxxxx

Co-simulation Platform Overview

Nan Duan, Nathan Yee, Benjamin Salazar, Jhi-Young Joo, Emma Stewart, Ed Cortez, “Cybersecurity Analysis of Distribution Grid Operation with Distributed Energy Resources via Co-Simulation”, IEEE PES General Meeting 2020.

16LLNL-PRES-xxxxxx

Inverters receive malicious firmware updates and turn on and off around noon for more than 20 min.

Inverters receive malicious firmware updates. However, corrective mitigation reloads correct update quickly to stop the abnormal oscillatory behavior after 6 min.

Compromised DERMS firmware update places malware on device to send dispatch commands to inverters to switch on and off when PV output is high.

atta

ckco

rrec

tive

miti

gatio

npr

even

tativ

em

itiga

tion Preventative mitigation rejects the

malicious update within 2 sec. The inverters outputs are reduced to zero to avoid voltage oscillation.

Choice between preventative and corrective is based on power quality priority. Endure a shortened period of voltage fluctuation but maintain DER output or eliminate voltage fluctuation entirely but reduce DER output.

Mitigation Strategy

17LLNL-PRES-xxxxxx

Overview

• Challenges• Collaborative Autonomy Use Case• Distributed Energy Resource Control Use Case• Advanced Metering Infrastructure Use Case• HPC enabled Transmission Distribution Use Case• Ongoing Efforts• Discussion

18LLNL-PRES-xxxxxx

Smart Meter as An Edge Computing Device

19LLNL-PRES-xxxxxx

Result – end user load

20LLNL-PRES-xxxxxx

Result – voltage stability margin in different seasons

21LLNL-PRES-xxxxxx

Result - communication network performance

22LLNL-PRES-xxxxxx

Overview

• Challenges• Collaborative Autonomy Use Case• Distributed Energy Resource Control Use Case• Advanced Metering Infrastructure Use Case• HPC enabled Transmission Distribution Use Case• Ongoing Efforts• Discussion

23LLNL-PRES-xxxxxx

WECC (179-bus) + 100 feederstime series co-simulation

LLNL HPC

LLNL HPC

Transmission DistributionDistribution

Each HPC compute node runs 4 feeder load flows in parallel using cores within the node

Nan Duan, Chih-Che Sun, Ryan Mast, Pedro Sotorrio, Vaibhav Donde, Wei Ren, Inalvis Alvarez-Fernandez, “Parallel Transmission Distribution Co-simulation Leveraging A Commercial Distribution Simulator”, IEEE ISGT-Europe 2020.

24LLNL-PRES-xxxxxx

WECC (179-bus) + 100 feeders120-days time series with realistic feeders

Name Node Number

kVLL kW kVar

P3U_Sub15 176 230 337.09 52.87P10U_Sub23 436 230 5806.26 1665.45P15U_Sub3 936 230 1396.25 213.27P15U_Sub4 721 230 3496.95 72.75P15U_Sub5 375 230 414.79 79.55

California Feeder Models Converted from OpenDSS format using DiTTo (NREL):

upper bound of speedup:

estimated sequential time tseq,est (sec) 38242.956HPC total time thpc (sec) 647.2676estimated speedup ηest 59.0837maximum time for power flow time series tpfts,max (sec) 529.3244average time for power flow time series tpfts,ave (sec) 381.5424ratio M between tpfts,max and tpfts,ave 1.3873number of CYME instances N 100estimated speedup limit ηmax,est 72.0825

30-day 100-feeder

Theoretical speedup limit depends upon the feeder complexity and # of feeders in parallel

pfts,ave

lau,ave

pfts,avemax,est

lau,max csi,max

pfts,ave

limt

tN

t Nt t MM

t

h®¥

+

= =+

+

Speedup 72.0

25LLNL-PRES-xxxxxx

Parallel Hosting Capacity Analysis for Integrated Transmission and Distribution Planning

Nan Duan, Can Huang, Chih-Che Sun, Vaibhav Donde, “Parallel Hosting Capacity Analysis for Integrated Transmission and Distribution Planning,” IEEE PES General Meeting 2021.

26LLNL-PRES-xxxxxx

§ T & D renewable integration capacity analysis with HPC

Two-level parallelism of GridDyn + CYMDIST co-simulation

Parallel Hosting Capacity Analysis for Integrated Transmission and Distribution Planning

Estimated Sequential Time (s)

Parallel Time (s)

HPC node 1 13242.06 1388.90HPC node 2 12913.85 1358.38HPC node 3 12945.64 1366.63HPC node 4 13207.96 1391.49HPC node 5 13188.11 1390.21HPC node 6 12757.01 1348.78HPC node 7 12819.14 1357.32HPC node 8 13058.89 1374.84HPC node 9 13401.78 1414.51HPC node 10 13190.70 1390.39Total Estimated Sequential Time (s)

130725.14

Overall Parallel Time (s)

1414.51

speedup 92.42

27LLNL-PRES-xxxxxx

Overview

• Challenges• Collaborative Autonomy Use Case• Distributed Energy Resource Control Use Case• Advanced Metering Infrastructure Use Case• HPC enabled Transmission Distribution Use Case• Ongoing Efforts• Discussion

28LLNL-PRES-xxxxxx

ACCESS Core

§ ACCESS Core: a flat solution leveraging meson build system and spack recipes of ns3, HELICS and GridLAB-D to alleviate the pain of building multiple open-source projects.

§ Use Spack to manage the dependencies. Spack is a package management tool designed to support multiple versions and configurations of software on a wide variety of platforms and environments.

https://spack.readthedocs.io/en/latest/

29LLNL-PRES-xxxxxx

Spack Recipes

Example syntax: spack install gridlab-d+helics%gcc@9.2.0

30LLNL-PRES-xxxxxx

GridLab-D minimum co-simulation example

switch transformer

climate

Overhead line

Swing node

meter

inverter solar

load

GridLab-D

Python federate

Switch action/monitor

weather forecast

Meter reading

Flexible load

Inverter control

31LLNL-PRES-xxxxxx

§ Full scale co-simulation from bulk power system to secondary feeder may not be practical without HPC

§ System-level load model reduction is necessary to screen the coupling locations for detailed T&D co-simulation

§ Scenarios considering at least the electromechanical dynamic are desired (most co-simulation use cases focus on power flow).

Transmission Dynamic Load Model Reduction

32LLNL-PRES-xxxxxx

Transmission Dynamic Load Model Reduction

N. Duan, J. Zhao, X. Chen, B. Wang, S. Wang, “Discrete Empirical Interpolation Method based Dynamic Load Model Reduction”, to be presented

at IEEE PES General Meeting 2021.

Load model state space:

Load model nonlinear function:

Linear model reductio still require the full state space in F

Find out the optimal interpolation points to only evaluate a subset of F

33LLNL-PRES-xxxxxx

§ Discrete Empirical Interpolation Method (DEIM) provides a way to identify the best observation points in the state space to preserve the nonlinear behavior of model.

§ We could potentially use this information to find the locations for coupling distribution models pertaining to interested contingencies.

Transmission Dynamic Load Model Reduction

34LLNL-PRES-xxxxxx

Key Takeaways

§ Understanding the interactions among the models from different domains is the key to simulating modern power systems.

§ Develop techniques to couple existing tools and models instead of developing an all-in-one model from scratch is a popular research direction.

§ Co-simulation for power systems is still at its early stage. Many barriers including model confidentiality, software licensing, simulator compatibility, simulation configuration still require the close collaboration between industry and academia on a case-by-case basis.

35LLNL-PRES-xxxxxx

PublicationsN. Duan, C. Huang, C.-C. Sun, V. Donde, “Parallel Hosting Capacity Analysis for Integrated Transmission and Distribution

Planning”, to be presented at IEEE PES General Meeting 2021.

N. Duan, N. Yee, A. Otis, J. Joo, E. M. Stewart, A. Bayles, N. Spiers, E. Cortez, “Mitigation Strategies Against Cyberattacks on

Distributed Energy Resources”, IEEE PES ISGT 2021.

N. Duan, C.-C. Sun, R. Mast, P. Sotorrio, V. Donde, W. Ren, I. Alvarez-Fernandez, “Parallel Transmission Distribution Co-

simulation Leveraging A Commercial Distribution Simulator”, IEEE PES ISGT-Europe 2020.

N. Duan, N. Yee, B. Salazar, J. Joo, E. M. Stewart, E. Cortez, “Cybersecurity Analysis of Distribution Grid Operation with

Distributed Energy Resources via Co-Simulation”, IEEE PES General Meeting 2020.

N. Duan, C. Huang, C.-C. Sun, L. Min, “Smart Meters Enabling Voltage Monitoring and Control: The Last-Mile Voltage

Stability Issue”, IEEE Transactions on Industrial Informatics.

Z. Atkins, C. Vogl, A. Madduri, N. Duan, A. Miedlar, D. Merl, “Distribution System Voltage Prediction from Smart Inverters

using Decentralized Regression”, to be presented at IEEE PES General Meeting 2021.

N. Duan, E. M. Stewart, “Frequency Event Categorization in Power Distribution Systems using Micro PMU Measurements”,

IEEE Transactions on Smart Grid, 2020.

N. Duan, J. Cadena, P. Sotorrio, J. Joo, “Collaborative Inference Of Missing Smart Electric Meter Data For A Building”, 2019

IEEE 29th International Workshop on Machine Learning for Signal Processing.

N. Duan, E. M. Stewart, “Deep-learning-based power distribution network switch action identification leveraging dynamic

features of distributed energy resources”, IET Generation, Transmission & Distribution, 2019.

N. Duan, J. Zhao, X. Chen, B. Wang, S. Wang, “Discrete Empirical Interpolation Method based Dynamic Load Model

Reduction”, to be presented at IEEE PES General Meeting 2021.

Co-simulation

Distribution Network

Monitoring and Control

Reduced-order

transmission load modeling

36LLNL-PRES-xxxxxx

Acknowledgement

This work was supported in part by the U.S. Department of Energy, and in part under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DEAC52-07NA27344.

37LLNL-PRES-xxxxxx

Open Discussion

Q & A

DisclaimerThis document was prepared as an account of work sponsored by an agency of the United States government. Neither the United States government nor Lawrence Livermore National Security, LLC, nor any of their employees makes any warranty, expressed or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States government or Lawrence Livermore National Security, LLC. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States government or Lawrence Livermore National Security, LLC, and shall not be used for advertising or product endorsement purposes.