PJM©2018

Data Exchange Infrastructure & TestingTOP-001-4 R20, R21, R23, & R24

Srinivas Kappagantula, PJMMonitoring and Situational Awareness ConferenceOctober, 2018

www.pjm.com

PJM©20182

Agenda

www.pjm.com

Background Solution Scope Progress

Problem Team Timeline

PJM©20183

BackgroundHow did it all start for current versions of TOP and IRO standards?

www.pjm.com

TOP-001-4 & IRO-002-5

2• Retired or superseded 18 TOP/IRO

standards

Project 2014-03: Mods. to TOP/IRO

1Key Factors Leading to TOP/IRO Revisions

• Southwest Outage• Independent Experts Review

• Stakeholder technical conferences

4

• Project 2016-01: More revisions to TOP/IRO Standards

• Added TOP-001-4 R20, R21, R23, and R24

• Added IRO-002-5 R2 and R3• FERC approved revisions on April 17, 2017

NERC Reaction & Result

3FERC Concerns• Order 817 – approved new TOP/IRO standards

but raised reliability concerns• Monitoring non-BES facilities for SOLs• Redundancy and diverse routing not

addressed• Testing of alternative data exchange

capabilities not addressed

PJM©20184

FERC Concerns

• Monitoring non-Bulk Electric System facilities. The Commission noted that "in some instances the absence of real-time monitoring of non-BES facilities by the transmission operator within and outside its TOP area as necessary for determining SOL exceedances in proposed TOP-001-3, Requirement R10 creates a reliability gap." (P.35)

• Redundancy and Diverse Routing of Data Exchange Capabilities. The Commission determined that, with respect to data exchange capabilities, the TOP and IRO standards requirements for Reliability Coordinators (RCs), Transmission Operators (TOPs), and Balancing Authorities (BAs) "do not clearly address redundancy and diverse routing so that registered entities will unambiguously recognize that they have an obligation to address redundancy and diverse routing as part of their TOP and IRO compliance obligations." (P. 47)

• Testing of the Alternate or Less Frequently Used Data Exchange Capability. The Commission determined that existing requirements do not establish a clear obligation for RCs, TOPs, and BAs to test alternative data exchange capabilities (P. 51).

www.pjm.com

PJM©20185

TOP-001-4 & IRO-002-5

www.pjm.com

• R20: Each Transmission Operator shall have data exchange capabilities, with redundant and diversely routed data exchange infrastructure within the Transmission Operator's primary Control Center, for the exchange of Real-time data with its Reliability Coordinator, Balancing Authority, and the entities it has identified it needs data from in order for it to perform its Real-time monitoring and Real-time Assessments.

• R21: Each Transmission Operator shall test its primary Control Center data exchange capabilities specified in Requirement R20 for redundant functionality at least once every 90 calendar days. If the test is unsuccessful, the Transmission Operator shall initiate action within two hours to restore redundant functionality.

TOP-001-4 R20 & R21

1. TOP-001-4

2. TOP-001-4

3. TOP-001-4

4. TOP-001-4

5. IRO-002-5

6. IRO-002-5

StandardR20

R21

R23

R24

R2

R3

Req.#7/1/2018

7/1/2018

7/1/2018

7/1/2018

10/1/2017

10/1/2017

Enforceable

• Same requirements applicable to the Balancing Authorities and Reliability Coordinators

R23, R24 and IRO-002-5 R2, R3

PJM©20186

Illustrative Example – For Discussion Purposes Only

www.pjm.com

RING NETWORK

EMS EMS EMS

R20• Focuses on not having a single point of

failure for halting the flow of Real-time data within primary control center

• Requires redundant and diversely routed exchange infrastructure inside the TOP’s primary control center

• Anything external to the brick walls is not addressed by R20 – See rationale in TOP-001-4

R21• Taking out a piece of the ring network or

one of the data feeds and still functioning may constitute a test

• Not the only way to conduct a test

Primary Control Center

Data Feeds Entering

PJM©20187

Problem

www.pjm.com

• Rationale section in the standard useful but not enforceable • What constitutes “data exchange infrastructure” – not defined• Where does a registered entity draw the line for redundant and

diverse routing? • What is a test? What equipment needs to be tested?

No NERC defined terms in the standard

• Industry as well as Regional Entities have varied interpretations or opinions of these requirements

• No idea how an auditor would interpret these requirements• How do registered entities demonstrate compliance – moving

target!!

Inconsistent interpretations of TOP-001-4 R20 & R21

• NERC published a practice guide on these requirements• Gives registered entities some approaches but leaves lot of

room for an auditor to interpret • Doesn’t address all industry concerns• Doesn’t provide compliance examples for industry use

CMEP Practice Guide useful but not enough

PJM©20188

Solution

www.pjm.com

EAS brought the concerns to NERC OC at the March 2018 meeting of the NERC OC

NERC OC assigned the EAS the task to develop Compliance Implementation Guidance for these requirements

EAS formed a task force (DEIRTF) to identify and address related concerns by developing compliance examples for industry use

PJM©20189

Team

www.pjm.com

NAME TITLE COMPANY

Benny J. Naas SCADA Engineer Vectren

Hassan Hamdar Manager, Reliability Performance and Standards Development FRCC

Jagan Mandavilli Sr. Reliability Engineer TRE

Jeff Fuller Director, Infrastructure Security AES

Jodi A. Jensen Senior SCADA Specialist WAPA

Joe Randazzo Director, Networks & Info Security ITC

Mike Fitzpatrick Manager, EMS Operations OPPD

Phil Hoffer Manager, EMS Applications AEP

Srinivas Kappagantula Sr. Analyst, Reliability Compliance PJM

Wei Qiu Sr. Reliability Engineer NERC

PJM©201810

Scope

www.pjm.com

• Review industry issues/concerns and identify examples registered entities may rely on to demonstrate compliance

• DEIRTF activity includes the following topics– Clarify what constitutes data exchange infrastructure– Clarify the concept of redundant and diversely routed data exchange

infrastructure– Provide clarification of “within the primary Control Center”– Investigate acceptable methods/ways an entity can utilize to test the data

exchange capabilities for redundancy– Actions a registered entity (BA/TOP) should initiate within two hours if the

test for redundancy is unsuccessful

PJM©201811

Scope & Industry Concerns Mapping

www.pjm.com

Scope #1 Scope #2 Scope #3 Scope #4 Scope #5

1. Data in scope• RTA• Internal• External

2. TOP-003-3 Data Specification3. Infrastructure within primary Control Center4. Equipment not owned by Registered Entity

Data Exchange Infrastructure1. One primary one backup case2. Data feeds from backup coming into primary

• EOP-008-1 R6 in play?3. Multiple primaries (hot-hot) case4. Infrastructure connecting data center with primary in

scope5. Infrastructure not within the primary Control Center

not addressed by R20

Within primary Control Center1. Initiate within 2 hours2. Notify entities?3. How long to restore

redundancy

Actions when Test Fails

1. Risks to reliability from physical testing• Production resources out of service• Servers & cables disconnected• Secondary interface outage can occur during testing

2. Testing across Control Centers acceptable?3. Rationale suggests testing all failure modes 4. What if testing causes redundant link to fail?

Testing1. Equipment Diversity

• Fiber, cable?• Servers, racks?

2. Physical Separation3. Pass “sawzall” test

• Cables in same tray/conduit?• Servers in same rack?

4. Components routed through same infrastructure5. Logical Separation enough?

Diversity & Redundancy

PJM©201812

Tentative Timeline

www.pjm.com

2018 2019March April May June July August September October November

DEIRTF Finalizes Draft CIG & Sends to EAS

Drafting Phase• ~Bi-Weekly Meetings of DEIRTF• Bi-Weekly Updates to EAS

December January February

Issue Raised at EAS & OCOC Assigned EAS Action ItemMarch 5 & 6, 2018

DEIRTF FormedReports to the EAS NERC & RE

ApprovalDEIRTF In-Person Meeting (11/1-11/2)

EAS & OC Approval

March April May

DEIR-CIG PostingFor Industry Use

NERC & RE Review Window

PJM©201813

Progress To Date

• Finalized the scope for project• Met with a Standard Drafting Member to gain better

understanding of the SDT and FERC intents• Reviewed the CMEP Practice Guide on TOP-001-4 R20 and

ensuring consistency to the extent possible• Began drafting the compliance implementation guidance

– Data in scope– Fleshing out some examples of data exchange infrastructure

configuration• Created a project schedule

www.pjm.com

PJM©201814www.pjm.com