Operator Training Simulator (OTS) – Grid Reliability

Sreenivas Badri

Manager, EMMS Production Support

09/25/2019

PUBLIC

Agenda

2

• ERCOT Region, Peak Demand Records

• OTS Environments, Architecture and Data flows

• OTS Case Development Process overview

• New Training Facility

• Operator Training Cycles

• Integrated OTS system capabilities

• Control Room Operators Training – Summer Readiness

PUBLIC

The ERCOT Region

3

The interconnected electrical system serving most of Texas, with limited external connections

• 90% of Texas electric load; 75% of Texas land

• 74,531 MW peak, August 12, 2019

• More than 46,500 miles of transmission lines

• 650+ generation units (excluding PUNs)

ERCOT connections to other grids are limited to ~1,250 MW of direct current (DC) ties, which allow control over flow of electricity

600 MW with SPP

30 MW with CENACE

at Eagle Pass

100 MW with CENACE

at Laredo 300 MW with CENACE at McAllen

220 MW with SPP

Texas RE

PUBLIC

Current Records

Peak Demand Record: 74,531 megawatts (MW)*

August 12, 2019, 4-5 p.m.

Weekend Peak Demand Record: 71,864 MW*

Sunday, August 11, 2019, 5-6 p.m.

Winter Peak Demand Record: 65,915 MW

Jan. 17, 2018, 7-8 a.m.

Wind Generation Records (instantaneous)

Output: 19,672 MW

‒ Jan. 21, 2019, 7:19 p.m.

Penetration (load served): 56.16%

‒ January 19, 2019, 3:10 a.m.

‒ Total MW Served by Wind = 17,406 MW

Recent Monthly Peak Demand Records

2019• March: 60,692 MW (March 5, 7-8 a.m.)

• August: 74,531 MW (August 12, 4-5 p.m.)*

2018• January: 65,915 MW (Jan. 17, 7-8 a.m.)

• May: 67,265 MW (May 29, 4-5 p.m.)

• June: 69,123 MW (June 27, 4-5 p.m.)

• July: 73,473 MW (July 19, 4-5 p.m.)

• November: 56,317 MW (Nov. 14, 7-8 a.m.)

*New records are preliminary, subject to change in final settlement

4

PUBLIC

Operator Training Simulator (OTS) – Environments

5

• 6 OTS Environments, each environment contains multiple systems

• 1 – Test, 2- Bastrop Production and 3- Taylor Production environments

EMS

PI OT04OT03OT02OT01-Test OT05

PI Buffer & Archive

PI Buffer & ArchiveWebFG/ETA

DSA/VSAT/TSAT

MMSMA App

MMSMOI Web

EMS

Webfg/ETA

DSA/VSAT/TSAT

MMSMA App

MMSMOI Web

EMS

Webfg/ETA

DSA/VSAT/TSAT

MMSMA App

MMSMOI Web

EMS

Webfg/ETA

DSA/VSAT/TSAT

MMSMA App

MMSMOI Web

EMS

Webfg/ETA

DSA/VSAT/TSAT

MMSMA App

MMSMOI Web

PI AF/Coresight

PI AF/Coresight MMS LDAP

LDAP Servers

PI Buffer & Archive

OT06

EMS

Webfg/ETA

DSA/VSAT/TSAT

MMSMA App

MMSMOI Web

PI AF/Coresight

EMS & MKT Oracle Databases

EMS & MKT Oracle Databases

EMS& MKT Oracle Database

EMS & MKT Oracle Databases

EMS & MKT Oracle Databases

EMS & MKT Oracle Databases

MMSMI/OS

MMSMI/OS

MMSMI/OS

MMSMI/OS

MMSMI/OS

MMSMI/OS

MMAP MMAP MMAP MMAP MMAP MMAP

PUBLIC

Operator Training Simulator (OTS) – Environments

6

o To have flexibility to run 2-3 simulations in parallel in a training cycles

o To troubleshoot any existing training cycle simulator issues

o To develop and prepare next training cycle simulation

o To train control room operators on latest software release changes before

the changes hits production

• All OTS Production environments (except being used for current Training Cycles)

OTS systems & applications software release versions are in sync with actual

production versions.

• OTS environments are treated as critical with production level support and

maintenance considering the significance of Operator Training and back to back

training cycles is being planned throughout the year.

• Test OTS environment is used for OTS software changes testing before deploying

into Production OTS environments

• Why we need 5 Production OTS environments ?

PUBLIC

OTS Systems Architecture

7

EMS & OTS Applications

Data Interface

OTS EMS System - Linux Server

E-terraArchiver Application

Windows Server

EMS

Ora

cle

Dat

abas

e (A

IX

Serv

er)

MM

S O

racl

e D

atab

ase

(AIX

Se

rver

)

Market Operator Interface (MOI) – Display

Server

MMS Application Server(SCED, HRUC, ASM, VDI

etc.)

MMS/OS MI Server(Market/Outage

Scheduler Events)

Tibco

OTS MMS System - AIX

DSA/VSAT - Windows Server

MMAP Windows Sever

PI Buffer & Archive – Windows Server

WebFG Display Server

PI AF/Coresight – Windows Server

Web Based PI Ops Tools

PUBLIC

OTS Applications Data flows

8

DSA/VSAT/TSAT

RTCA

RTNET

QKNET

EMS - Network Applications

TCM SCED

Market Management System

SCADA

LF

RLC

LFC

DTSPSM QSE Simulator

RTOP

Data Interface

EMS/OTS HDR

Event Scenarios

EMS - Generation Applications

OTS - Simulation Engine

MOI Displays – COP/DCTIE User Interface

COP/DCTIE/RUC/ERS/AS Deployments/Outages

(MMS Database)

Outage Scheduler(MMS Database)

HRUC/AS Manager/SASM

MOI Displays – Outages creation/

Update- User Interface

STNET

DTSPSM also mimics TO Role & TO TelemetryQSE Simulator mimics QSE Role & QSE Telemetry

iGridInterface(Reliability Risk

Desk Operations)

RDB Interface

Wind & Solar Forecast

(EMS Database)

OTS Events

PUBLIC

OTS Case Development Process Overview

9

• OTS case development process involves multiple teams, most of the tasks are being

performed by EMMS Production and Operator Training Teams

• End to End Case development, all OTS systems setup takes around 2 weeks

depending on number of simulations are planned in a given training cycle.

• Training Team provides following training cycle requirements and determines new OTS

case is required.

Production Snapshot Date, Peak Demand

Systems and Applications, Tools to be functional

Market Simulation requirements

• EMMS Production develops OTS case, performs following testing on the validated

case to ensure simulator is working as expected before adding OTS events. OTS (DTSPSM, QSE Simulator etc.) applications testing

EMS & MMS application testing and MMS to EMS integration testing

Basic scenario testing (Frequency Recovery, Large Unit Trips, 345KV equipment outages, RRS/Non-

Spin Deployments etc.)

• Tested case is provided to Training Team to develop OTS events and perform case

tuning to simulate required events during operator training cycles.

PUBLIC

OTS Case Development Process Overview

10

High level steps in OTS Case Development• EMS save cases, SCADA HDR files preparation and setup based on Simulation Date and time.

• Run OTS Save Case Preparation Application to do following

o EMS Save Cases load

o Validate all Model Clones

o Load validated save cases into Real Time Clones

• Case updates and tuning – run scripts to populate OTS specific data and set OTS specific parameters

• Bring up OTS system and Verify OTS applications specifically QSE Simulator and DTSPSM

• Ensure that DSA/VSAT runs and transfer Stability Limits to EMS

• Perform Market Database Refresh based on Simulation date and time.

• Perform CIM Model Load in Market System based on Simulation date

• Setup EMS/MMS Interfaces and ensure EMS to MMS dataflow is functional

• Enable OTS Market Integration, ensure Market Management System (MMS) applications SCED/HRUC/ASM/VDI

are working as expected based on basic scenario testing.

• Perform OTS Integration Testing

• Prepare PI data and ensure PI Archive, PI AF/CoreSite, PI Ops Tools, Control Room Spreadsheets are functional

based on Simulation time.

• Ensure Macomber Map is setup and integrated with EMS.

PUBLIC

OTS Case Development Lifecycle

11

Simulation Case Development Lifecycle

EM

MS

Pro

du

ctio

n S

upp

ort

Sys

tem

Ope

rati

ons

Tra

inin

gE

MM

S

De

velo

pm

ent

Da

tab

ase

Ho

stin

g

Ser

vice

s

Sys

tem

Op

era

tions

Overall development timeline depends on accurate estimate of required work in each department

Start

Complete

Simulation Case

Preparation

Checklist and

submit to EMMS

Test case Fixes required?

Evaluate case

Simulation

performance

acceptable?

No

RFC for code

changes

required?

Create code or

tuning changes

and deliver to

EMMS PS

No

Event

performance

acceptable?

Yes Yes

Yes

Sim dev

process

efficiencies

identified?

Yes

Update Simulation

Case Preparation

Checklist

End

Highlighted boxes represent individual department-specific documented processes/procedures/controls which require

department commitments

Red arrows represent the optimal path

Training/

Performance

simulation

requested

Create simulation

training script

Ops

procedures/

tool issues

identified?

Submit to

Operations

Improvement for

analysis

No

No

Yes

YesCreate RFC for

Dev

Conduct training

and support

training

Simulation

indications

acceptable?

Yes

NoDevelop case

Yes

Perform Market

data refresh

Market data

refresh req’d?No

No

No

Create crew

performance

feedback report

PUBLIC

Taylor Training Facility

12

• Black Start and Operator

Training Seminars will

take place in the new

building in 2019.

• ERCOT’s new training center

officially opened in January.

PUBLIC

Operator Training Facility

13

• Operator Training Facility Control Room is replica of actual Control

Room

• Operators from every desk are trained in OTS environment on all the

EMS/VSAT/MMS/PI/MMAP etc. applications and situational awareness

tools they use in actual control room.

PUBLIC

Operator Training Cycles

14

• 6 Operator Training Cycles every year

Black Start Training with Market Participants every year - Cycle 2 (March – April) –

800 Operators (including ERCOT Operators) attending this year Black Start

training this year.

Summer Readiness Training Cycles – Cycle 3 and Cycle 4 (May to August) It also includes SOLs & IROLs, EEA and Loss of Primary Control Center related

training simulations

Operator Training Seminar – Cycle 5 (OTS is not used)

Emergency Weather Drills (Alternate between Hurricane and Winter Storm) –

Cycle 6 (October – November)

Grid Ex – Cycle 6

• Operators are trained in OTS environment on EMS/MMS/PI Ops Tools, Situational

Awareness tools changes that go into production that impacts control room

operations, it includes defects, enhancements and functionality upgrade &

technology upgrade projects.

• Each Training Cycle duration is 6 weeks to train 6 operator crews

PUBLIC

Integrated OTS System Capabilities

15

• Integrated OTS system has capability to simulate various type

of simulations

Simulate Grid Events or EMMS Applications failure scenarios (ICCP/SCADA Telemetry failures,

State Estimator, CA, LFC/RLC/SCED, VSAT etc. failures) and Applications functionality

changes (Nodal Protocol changes, System changes, Enhancements etc.)

Replay Grid and Real-Time Market Events

Standalone mode without Market Integration

Black Start Training with Market Participants

Market Participants Training (Winter Storm Drill, Hurricane Drill)

Simulation based on Future Models (60, 90 days ahead CIM Models)

• Integrated OTS system has capability to run simulation for 48 hours or

more in integration with all other systems including Market System.

PUBLIC

Operator Training – Summer Readiness

16

• Developed OTS case based on 2019 March Production EMS Snapshot and Market

data, overlaid summer peak load pattern.

• OTS Market System data (AS Qualification, COP, Offers) was prepared to setup Market

applications to run SCED/HRUC, deploy Offline Non-Spin, Load Resources RRS, ERS

and dispatch switchable generators during EEA event.

• Training team added events and tuned OTS case to simulate following events to train

Operators for summer events

Energy Emergency Alert (EEA)

Switchable Generators

Real-Time Assessment (RTA)

IROLs and SOLs Exceedances

• All required OTS systems and application were setup to run simulator and simulate

above events during Cycle 3 training cycle (May – June).

PUBLIC

Operator Training – EEA

17

• EEA Trigger Conditions

EEA1 - must be declared when PRC < 2300 MW & it is not projected to be recovered above

2300 MW within 30 minutes

EEA 2 - must be declared when PRC < 1750 MW & & it is not projected to be recovered above

1750 within 30 minutes or may be declared EEA 2 if System frequency below 59.91 for 15

minutes

EEA3 - must be declared when PRC <= 1375 or must go to EEA3 if system frequency is below

59.91 for 25 minutes or may declare EEA3 if system frequency is below 59.91 for 20 minutes

• Energy Emergency Alert (EEA) Days

EEA days this year – 08/13/2019 & 08/15/2019

Last EEA Days – 01/06/2014 & 01/18/2014

• Constant EEA training is required for operators in Operator Training

Simulation environment because

EEA is rare event

EEA situational awareness tools, EMS/MMS applications, processes

and procedures changes constantly

Every operator crew requires to go through real-time EEA events

simulation

o ERS, Offline Non-Spin and Load Resources RRS deployment related MMS

Applications are used mainly during EEA events other than qualification

testing.

PUBLIC

Operator Training – EEA

18

• Operators followed the EEA procedure to maintain Grid reliability during EEA and

brought back the Grid to normal conditions. Following are some of the key actions

performed by Operators during EEA event using OTS EMS and MMS applications

Manual RRS Deployment through LFC

Offline Non-Spin Deployment

ERS Deployments

Load Resource RRS deployment

RUC Commitments through VDI

Firm Load shed through OTS Events

• All Operator crews were trained on EEA event in OTS environment by end of June that

includes all the situational awareness tools changes and EEA event

processes/procedures.

• Operators were also trained on Market System Emergency Response Service (ERS)

and Non-Spin Display enhancements during EEA event training in OTS environment.

• Control Room Shift Supervisors and Operators were extremely happy on how OTS

training on EEA helped them to handle this summer EEA events effectively.

PUBLIC

Operator Training – EEA Dashboard

19

PUBLIC

Operator Training – Switchable Generators

20

• Switchable Generators (SWGRs) changes were brought into Production EMS and MMS

systems end of May this year in preparation to handle peak demand this summer.

• A new telemetered resource status code EMRSWGR is made available to represent

Emergency Switchable Generation resources.

• When Switchable Generation qualified resources are switched from other region and

ready to operate in ERCOT region, QSE changes resource status telemetry from

EMRSWGR to ON, SCED dispatches the units.

• OTS Market System data was prepared to simulate Switchable Generators (SWGRs)

following behavior in OTS environment the way it happens in real-time

Showing SWGRs current and future operating status, capacity in Market System

new Displays

SWGRs Resource Status Telemetry changes in EMS

SCED dispatching SWGR units

• Operators were trained on Switchable Generators operations in OTS environment before

changes were deployed in Production before start of this summer.

• Operators were able perform Switchable Generators Operations effectively during this

summer EEA days.

PUBLIC

Operator Training - RTA Failure Scenarios

21

RTA failure scenarios

Partial Loss of ICCP

Complete Loss of ICCP

Loss of an EMS

Application(RTNET/

RTCA/RTDCP)

Loss of Real Time (RT)

EMS System

Loss of RT & Study (ST)

EMS systems

Perform RTA using

Powerworld / Offline VSAT

using last successful RTA

solution exported in

the applicable format

Use Study EMS system with

last successful RTA solution

available

Perform Manual RTA

Using Enhanced

STNET Online Powerflow in

RT EMS

Perform Manual RTA

Using Enhanced

STNET Online Powerflow in

RT EMS

Use SCADA/RTNET

enhancements to help solve

RTNET

• RTA (Real-Time Assessment) improvements that were put in place in EMS after past

incidents helped significantly ERCOT RTA process running through any TSP telemetry

failures.

• Rigorous Shift Engineer and Operator Training was provided during this summer in OTS

environment by simulating RTA failure scenarios that helped them significantly in

understanding the RTA details and performing manual RTA within 30 minutes.

PUBLIC

Questions?

22

PUBLIC

Thank You

23