c copyright 2013 cigre australia notice changes introduced ...(similar to the way 9-2le limits the...

33
This is the author’s version of a work that was submitted/accepted for pub- lication in the following source: Ingram, David M.E., Schaub, Pascal, Campbell, Duncan A.,& Taylor, Richard R. (2013) Assessment of real-time networks and timing for pro- cess bus applications. In Cigré SEAPAC 2013 South East Asia Protection and Automation Conference, 12-13 March 2013, Brisbane, Queensland, Australia. This file was downloaded from: c Copyright 2013 CIGRE Australia Notice: Changes introduced as a result of publishing processes such as copy-editing and formatting may not be reflected in this document. For a definitive version of this work, please refer to the published source:

Upload: others

Post on 24-Jan-2021

1 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

This is the author’s version of a work that was submitted/accepted for pub-lication in the following source:

Ingram, David M.E., Schaub, Pascal, Campbell, Duncan A., & Taylor,Richard R. (2013) Assessment of real-time networks and timing for pro-cess bus applications. In Cigré SEAPAC 2013 South East Asia Protectionand Automation Conference, 12-13 March 2013, Brisbane, Queensland,Australia.

This file was downloaded from: http://eprints.qut.edu.au/58568/

c© Copyright 2013 CIGRE Australia

Notice: Changes introduced as a result of publishing processes such ascopy-editing and formatting may not be reflected in this document. For adefinitive version of this work, please refer to the published source:

Page 2: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5

SEAPAC 2013

Brisbane 12-13 March

© CIGRE Australia

Page 1 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

David Ingram Queensland University of Technology [email protected]

Pascal Schaub QGC Pty Ltd [email protected]

Duncan Campbell Queensland University of Technology [email protected]

Richard Taylor Queensland University of Technology [email protected]

Keywords: Ethernet networks, IEC 61850, IEEE 1588, industrial networks, performance evaluation, process bus, protective relaying, smart grid, substation automation

Contents

Abstract 2 1  Introduction 2 

1.1  Non-Conventional instrument transformers & their benefits 2 1.2  The need for process bus networks 3 1.3  New substation applications enabled by process buses 3 1.4  Precision Time Protocol (PTP) and the PTP Power System Profile 4 

2  Process bus test bed 5 2.1  Real-time power system simulation 5 2.2  Use of Precision Time Protocol (PTP) to synchronise merging units 7 2.3  “Live” hardware in the test bed 7 2.4  Network testing tools 7 

3  Use of Ethernet networks for real-time communication in substations 7 3.1  Direct measurement of merging unit publishing times 8 3.2  Ethernet taps and hardware accelerated Ethernet capture cards 9 3.3  Sampled value (IEC 61850-9-2) and GOOSE (IEC 61850-8-1) interaction 10 

4  Performance of Precision Time Protocol for merging unit synchronisation 12 4.1  Comparison of grandmaster and slave clocks 13 4.2  Effect of grandmaster drift 13 4.3  Transparent clock performance measurement 16 4.4  Sampled values (IEC 61850-9-2) and PTP (IEEE Std 1588/C37.238) interaction 18 

5  Protection system performance 21 5.1  Performance Measures 21 5.2  Method 21 5.3  Protection response with conventional and Ethernet connections 22 5.4  Repetitive testing with the RTDS 25 5.5  Merging unit synchronising error 26 5.6  Effect of background network traffic on protection response 27 

6  Discussion 28 7  Conclusion 29 References 29 List of Figures 31 List of Tables 32 Biography: David Ingram 32 Biography: Pascal Schaub 32 Biography: Duncan Campbell 32 Biography: Richard Taylor 32 

Page 3: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 2 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

Abstract

Widespread adoption by electricity utilities of Non-Conventional Instrument Transformers, such as optical or capacitive transducers, has been limited due to the lack of a standardised interface and multi-vendor interoperability. Low power analogue interfaces are being replaced by IEC 61850-9-2 and IEC 61869-9 digital interfaces that use Ethernet networks for communication. These ‘process bus’ connections achieve significant cost savings by simplifying connections between switchyard and control rooms; however the in-service performance when these standards are employed is largely unknown.

The performance of real-time Ethernet networks and time synchronisation was assessed using a scale model of a substation automation system. The test bed was constructed from commercially available timing and protection equipment supplied by a range of vendors. Test protocols have been developed to thoroughly evaluate the performance of Ethernet networks and network based time synchronisation.

The suitability of IEEE Std 1588 Precision Time Protocol (PTP) as a synchronising system for sampled values was tested in the steady state and under transient conditions. Similarly, the performance of hardened Ethernet switches designed for substation use was assessed under a range of network operating conditions. This paper presents test methods that use a precision Ethernet capture card to accurately measure PTP and network performance. These methods can be used for product selection and to assess ongoing system performance as substations age.

Key findings on the behaviour of multi-function process bus networks are presented. System level tests were performed using a Real Time Digital Simulator and transformer protection relay with sampled value and Generic Object Oriented Substation Events (GOOSE) capability. These include the interactions between sampled values, PTP and GOOSE messages.

Our research has demonstrated that several protocols can be used on a shared process bus, even with very high network loads. This should provide confidence that this technology is suitable for transmission substations.

1 Introduction

Non-Conventional Instrument Transformers (NCITs) that are safer and pose less risk to the environment are now commercially available. Examples of NCITs include capacitive voltage sensors and optical current transformers. Digital transmission of current and voltage signals from NCITs in the switchyard to substation control rooms significantly reduces the cabling required, with a single fibre optic cable capable of replacing more than 100 copper circuits. Ethernet process bus networks simplify system-wide automated testing, and facilitate innovative test methodologies such as the testing of in-service protection relays.

The IEC 61850 family of standards were released in the early 2000s to standardise substation communication systems. These standards include Ethernet based process-level connections between switchyards and control rooms; however their in-service performance is largely unknown. High voltage power systems are critical infrastructure, and therefore substation automation systems must be extremely reliable and proven to meet performance requirements before going into service.

This research sits at the intersection of three fields: power system automation, precision timing and real-time data networks.

1.1 Non-Conventional instrument transformers & their benefits

The term NCIT is used to describe any measurement transformer that does not use magnetic induction and iron cores. The two most common types of current measurement NCIT are the “Rogowski Coil” air-cored transformer [1] and the optical current transformer (OCT) first described by Saito [2]. Capacitive voltage dividers are widely used in gas insulated switchgear (GIS), often in combination with a Rogowski coil for current measurement [3].

NCITs offer two significant benefits over conventional inductive transformers:

1. Instrument transformers can be either integrated into switchgear or supported by switchgear. This significantly reduces the footprint of high voltage substations.

Page 4: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 3 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

2. OCTs and some Rogowski coils are air-insulated instead of using oil or sulphur hexafluoride (SF6) gas for high voltage insulation.

Many utilities avoid using SF6 where possible, as it is a very potent greenhouse gas, having a 100 year warming potential 22 800 times that of carbon dioxide.

The low power output signal from NCITs has been an impediment to their adoption for substation automation. Despite international standards for analogue low-power connections [4] and digital connections [5] being in place for more than a decade, most manufacturers have not incorporated these interfaces into protection relays.

1.2 The need for process bus networks

Digital process connections are of interest to utilities for a number of reasons, and can be used by “merging units” for conventional instrument transformers as well as for NCITs. The amount of cabling required in a substation is significantly reduced, as many signals can be multiplexed onto a fibre optic cable. A single 100 Mb/s Ethernet connection is capable of transmitting the equivalent of twenty three-phase sets of measurements, which is ten of the eight-core cable or twenty of the four-core cable shown in Figure 1.

Figure 1: Multicore cables used for conventional secondary connections between the switchyard and control room.

IEC 61850 standards provide a great deal of flexibility, and while this works towards future-proofing the standards, it poses its own challenges when manufacturers choose different options and inter-operability cannot be achieved. This was the case with IEC 61850-9-2 sampled values [6], and a reduced set of options based around the IEC 60044-8 dataset were selected to create an industry guideline. This guideline, published by the UCA International Users Group (UCAIug) is known as “9-2 Light Edition” or “9-2LE” [7]. All of the sampled value experimentation presented in this paper are based on 9-2LE. Figure 2 shows how simple the connections to protection relays can be with sampled values and IEC 61850-8-1 Generic Object Oriented Substation Event (GOOSE) messaging [8]. Other than power, a sampled value protection relay only requires Ethernet connections to the Station Bus and Process Bus. If a relay accepts conventional inputs (as is the case for the lower relay in Figure 2), a One Pulse Per Second (1-PPS) signal is required to synchronise the analogue inputs to the 9-2LE data for comparison purposes (such as transformer differential protection).

1.3 New substation applications enabled by process buses

The benefits achieved with digital process buses extend beyond the immediate cost savings and safety improvements. Sampled value data is readily aggregated and distributed by Ethernet switches. This simplifies the connections required for centralised substation automation functions, including disturbance recording, bus protection, power quality monitoring, and synchrophasor observations.

Page 5: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 4 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

Figure 2: The simplicity of Ethernet relay connections.

The reduction in the cabling that is installed (in terms of quantity and physical size) and relaxation of cable distances (as voltage drop is not a concern) allows the design of substation control rooms to be rethought. One such example is centralised substation protection with distributed measurement (e.g. merging units or NCITs) and a protection computer instead of protection relays. This type of protection architecture has been implemented in distribution substations in the Netherlands, albeit using proprietary protocols rather than IEC 61850 [9]. The same principles could be applied to an IEC 61850 process bus based on GOOSE and sampled values, and may ultimately reduce a substation automation system to a pair (to meet redundancy requirements) of substation automation servers.

1.4 Precision Time Protocol (PTP) and the PTP Power System Profile

The Precision Time Protocol (PTP), IEEE Std 1588, is a bidirectional message-passing protocol that achieves higher accuracy than the Network Time Protocol (NTP) through the use of specialised networking hardware [10]. PTP is used primarily over Ethernet with special hardware that provide accurate time stamps of when each PTP message arrives. This eliminates CPU time-stamping jitter as a source of error. PTP is a very flexible protocol, however restrictions on the available options can be imposed through the use of ‘profiles’ (similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed a “Power System Profile” (IEEE Std C37.238 [11]) intended to provide synchronising accuracy better than 1 µs over 16 network hops [12]. The message passing intervals are fixed by C37.238 to one second for all PTP message types.

A key feature of the Power Profile is the mandatory use of the peer-delay mechanism. This requires all network devices in the timing chain, including Ethernet switches, to be PTP-aware and to measure the path delay between devices. The Ethernet switches must support “transparent clock” capability, where the switch measures the time a PTP message takes to pass through the switch and passes that information onto downstream devices. This allows PTP ‘consumers’ (slave clocks) to correct for queuing latency, even with high network loads. This “correction” can either be inserted into the Sync message as it is transmitted (one-step operation) or transmitted in a separate Follow Up message (two-step operation).

Relay #1

Relay #2

Page 6: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 5 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

The three main device types in a PTP timing network are:

Grandmaster clock. This is the primary time reference for PTP, and generally includes a GPS receiver.

Slave clock. The end-user of PTP, which may be a merging unit (if it has native PTP support), or a stand-alone clock that regenerates 1-PPS or IRIG-B timing signals for existing applications.

Transparent clocks. PTP-aware Ethernet switches provide correction updates on the basis of path delay and residence time to downstream devices.

The peer-delay mechanism reduces network traffic on the grandmaster clock as each device only communicates with its neighbours. Other PTP systems (such as used in telecommunications) require each slave device to pass messages back and forth with the grandmaster.

2 Process bus test bed This research was a performed using a scale model of a substation automation system (SAS). Real time simulation, protection relays, PTP time clocks and artificial network impairment were used to investigate the behaviour of sampled values, GOOSE and PTP in a shared network environment. Figure 3 shows the schematic of this test bed, and the following sections describe the components in more detail.

Figure 3: Schematic of the process bus test and evaluation system.

2.1 Real-time power system simulation

The Real Time Digital Simulator (RTDS) is a multi-processor simulation system that is running electromagnetic transient program (EMTP) simulations of power systems in real time. Power system models are created using a schematic interface, compiled and then executed on the RTDS hardware in a ‘run-time’ environment. The fast execution speed allows the EMTP model to respond to external stimuli and for hardware (such as protection IEDs) to interact with the simulation. This is a significant improvement over playback of pre-calculated faults, as the response of the IEDs changes the outcome of the simulation. GTNET cards enable the RTDS to send and receive GOOSE messages (to take the place of digital IO) and to send sampled value messages (which act as analogue outputs) over Ethernet. The RTDS used in this test bed has a total of 28 processors and three GTNET cards.

Scripting in the RTDS run-time varies the location and impedance of faults, which enables exhaustive testing under a variety of network and synchronising conditions to take place without operator intervention. Conventional protection testing using secondary injection verifies that the protection settings have been correctly entered into the IED, but testing with the RTDS demonstrates that the protection design is correct.

Page 7: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 6 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

Figure 4 shows the ‘draft’ schematic and Figure 5 shows the ‘run-time’ user interface for protection testing using this test bed. Push buttons, toggle switches and ‘sliders’ control the power system model in real-time, and can be changed with a mouse or through scripting. The run-time is used to display measurements and waveforms during the simulation. Having the RTDS subscribe to analogue GOOSE messages allows the RTDS to acts as the human-machine interface for the protection relays in the test bed.

Figure 4: RSCAD ‘draft’ schematic of the power system model used for transformer protection assessment.

Figure 5: The RTDS ‘run-time’ interface for transformer protection testing, with a fault applied to the high voltage transformer terminal.

HVCBa

HVCBc

HVCBb

HV

HVcHVbHVa1.00000 /_ 0.00000

LVCBa

LVCBc

LVCBb

LV

LVcLVbLVa0.98730 /_ -13.64917

HVSRC

RRL

RRL

RRL

A

B

C

AC Type

Ell= 275 kV

R//L

Current Transform ersHVCTa

HVCTbHVCTc

T1TapDn1

0

T1TapUp1

0

T1_TapPos

LVSRCa

LVSRCc

LVSRCb

LVCTbLVCTc

Current Transform ersLVCTa

LVSRCCTbLVSRCCTc

Current Transform ersLVSRCCTa

LVS

LVSRCVcLVSRCVbLVSRCVa1.0 /_ 1.0

LVSRC

RRL

RRL

RRLA

B

C

AC Type

Ell=110kV

R//L

DiffCBctrl

Dyn

amic

Loa

d (R

-X)

LDbi

as

AB

C

RIS

C

HVDifCBa

HVDifCBc

HVDifCBb

DiffCBa

DiffCBc

DiffCBb

Dyna

mic

Loa

d (R

-X)

LDdi

ff

AB

CR

ISC

PBLVDiff1

0

A

B

C

A

B

C

Tmva = 375 MVA

275.00 110

Trf = T1

#1 #2I

tapupdn

pos

RISC

HVDifCBctrPBHVDiff1

0

HVCBctrl

HVCBpos1

0 LVCBctrl

LVCBpos1

0

LVSRCctrl

LVSRCpos10

Draft VariableRon

0.1

A

B

C

L-L FAULT POINT

A

B

C

L-G FAULT POINT

Dyn

amic

Lo

ad (R

-X)

HVLD

diff

AB

C

RIS

C

A

B

C

L-G FAULT POINT

A

B

C

L-L FAULT POINT

A

B

C

L-G FAULT POINT

A

B

C

L-L FAULT POINT

PBTFLVLGfl1

0

PBTFLVLLfl1

0

LVTFLGFctr

LVTFLLFctr

Low Voltage Bus

Through Faults

Low Voltage Transform erInternal Faults

High Voltage Transform erInternal Faults

PBHVLLflt1

0

HVLLFctrl PBHVLGflt1

0

HVLGFctrl

PBLVLLflt1

0

LVLLFctrl

PBLVLGflt1

0

LVLGFctrl

Draft VariableRsrc1

8.5

Draft VariableRsrc0

17

Low Voltage Bus110 kV

High Voltage Bus275 kV

Page 8: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 7 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

2.2 Use of Precision Time Protocol (PTP) to synchronise merging units

The process bus test bed was constructed in two parts: the “switchyard” and the “control room”. The only connections between the two parts were four fibre optic cables, and all communication over the fibre was with Ethernet.

The GTNET cards in the RTDS required a 1-PPS signal to properly time-align the 9-2LE messages. This was supplied by two PTP slave clocks that each generated the required signal.

2.3 “Live” hardware in the test bed

The core automation equipment in the test bed was purchased on a commercial basis, donated, or lent by manufacturers. This enabled vendor-independent testing to take place. The following equipment was installed:

Three models of PTP grandmaster clocks Three models PTP slave clocks Four models PTP transparent clocks (managed Ethernet switches) for the process bus. Three models of sampled value protection relay (distance, transformer differential and feeder

differential) One managed Ethernet switch for the station bus.

2.4 Network testing tools

IEC 61850-5 defines the maximum total transmission time for Type 1A (trip message) and Type 4 (raw data messages) in P2 (transmission) substations to be 3 ms. Conformance testing requirements in IEC 61850-10 allocate of 20% the total transfer time to network delays, with 40% each allocated to the publishing and subscribing network stacks. This results in a requirement to keep network delays under 600 µs; however measuring latency accurately at this time-scale requires the use of specialist network tools.

A four-port 100/1000 Mb/s hardware accelerated network card (Endace DAG7.5G4 [13]) was used to capture and transmit data at up to 4 Gb/s without imposing any load on the host computer’s CPU. This network capture card has a time stamping unit that can be synchronised to an external 1-PPS source for absolute time and to improve the accuracy of its interval counter. As each Ethernet frame arrived at the card a time stamp was pre-pended to the message, and the same clock was used for all four ports. This resulted in a maximum time stamp error between ports of ±8 ns and a maximum synchronising error between the card and its 1-PPS source of ±100 ns.

The time sensitive nature of the protocols, in particular sampled values and PTP, precluded the use of mirror ports on Ethernet switches. An Ethernet tap (NetOptics 10/100/1000 [14]) was used to copy Ethernet frames as they passed through the tap. This introduced a fixed delay of approximately 100 ns that did not affect the accuracy of PTP time synchronisation, but could be observed in the path delay measurements.

Network emulation is a technique where a device simulates communication network impairment, but in a controlled and repeatable manner. Common impairments include packet delay variation, packet loss and packet corruption. A Data Link Layer (OSI layer 2) emulator, the Simena NE1000, was selected as sampled values, GOOSE and PTP use OSI Layer 2 Ethernet frames. The emulator has the ability to selectively filter or modify frames based on source or destination address, payload type and Virtual Local Area Network (VLAN) ID. The selective nature of filtering allows the evaluation system to increase bit error rates for selected protocols and to drop individual devices from the network to test fail-over schemes.

3 Use of Ethernet networks for real-time communication in substations Communication networks are critical for smart grid applications, however much of the focus on smart grid communication has been on distribution networks or synchrophasors. The network traffic characteristics of a sampled value process bus local area network, with high data rates and strict performance requirements, are presented in this section.

Page 9: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 8 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

3.1 Direct measurement of merging unit publishing times

Each IEC 61850-9-2 sampled value message includes a sample counter, SmpCnt. This counter is reset to zero when the time synchronising signal occurs. The SmpCnt counter will range from 0 to 3999 (50 Hz) or 4799 (60 Hz) for applications using 9-2LE with 80 samples per nominal power system cycle (protection mode). Synchronising the source merging unit to the capture card enables the time required for the merging unit to publish a sample to he calculated. This information can then be used to assess the effect of Ethernet switches and network traffic on latency with network captures at a single point in the network. Differential timing measurements are not feasible when the source (merging unit) is several hundred metres away from the destination (protection relay).

3.1.1 Method

The source of 1-PPS signals used to synchronise sampling of the merging unit were used to synchronise the time stamping unit in the Ethernet capture card. The propagation delay of the synchronising signal through cables and any media converters was quantified and compensated for, and is typically less than 200 ns for cable lengths under 20 m.

Precise delay measurements were taken on the SmpCnt=0 event, as this was the only sample that is precisely aligned to the synchronisation source. A network capture was initiated and SmpCnt=0 frames were extracted for further analysis. Figure 6 shows how the DAG7.5G4 card can be synchronised to a merging unit. FO/Cat5 is an Ethernet media converter, and FO/TTL and TTL/422 are media converters for the 1-PPS signal.

Figure 6: Latency measurement using externally synchronised Ethernet capture card.

Testing was performed in a live substation, with the merging unit providing the 1-PPS reference over a fibre optic cable and the sampled values over 100BASE-FX Ethernet. The same fibre optic cables were used for all tests to ensure constant path delay. Each physical merging unit contained three logical merging units (each connected to a different set of three-phase current and voltage sensors) and an integrated Ethernet switch.

3.1.2 Results

Figure 7 shows the variation in frame publishing time from the seven physical merging units. The 0 µs reference represents the average arrival time for all seven merging units, which is commercially sensitive.

99.97% of frames are spaced between 248 µs and 252 µs, with inter-arrival times bounded by 235 µs and 264 µs. This confirms that the data transmission is regular. The combination of frame transmission occurring at the same point in time (synchronisation) and at the same rate (syntonisation) means that the merging unit transmissions can be considered coherent transmissions, using terminology analogous to that of coherent light (light that has the same wavelength and phase).

This test was conducted with merging units from one manufacturer, however these results show that coherent transmission is possible with commercially available merging units, and this is the worst case as the results will show. As a result, network designers need to allow for the simultaneous arrival of frames when specifying Ethernet switches.

Page 10: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 9 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

(a) (b)

Figure 7: Distributions of (a) variation in frame arrival time from the average arrival time and (b) the time spacing of sampled value frames.

3.2 Ethernet taps and hardware accelerated Ethernet capture cards

The synthetic data was based upon standard 9-2LE frames and was created with a custom application that allows key parameters to be varied. Synthetic data avoids reproducing variations in inter-frame time that may occur with a real merging unit and this provided consistency between tests.

3.2.1 Method

The test frames were injected into switches under test via the NetOptics tap, as shown in Figure 8. The tap output was captured with the DAG card, providing accurate switch ingress time-stamps. A second capture port on the DAG card captured the frames leaving the switch, and from this the residence time, or latency, was calculated. The DAG card used a common clock to time-stamp all frames entering the card, and the resolution of this clock was 7.5 ns.

Figure 8: Schematic showing the use of the Ethernet tap and multi-port capture card to measure Ethernet switch latency.

Network traffic was created for the switches under test that reflected this environment. Six synthetic sampled value “streams” were created, with each merging unit offset from the previous merging unit by a fixed time to ensure consistency when switching. The synthetic data was injected into the switch under test at 1000 Mb/s to simulate the near simultaneous arrival of frames from six merging units. The accuracy of frame transmission times was verified by capturing the DAG output with a Fluke OptiView II network analyser.

3.2.2 Results

The spacing of frame arrivals has a significant effect on the latency that is introduced. Figure 9 shows the cumulative probability of latency for two configurations. The “bunched” case has the messages from the six merging units arriving at 2 µs intervals, while the “spaced” data arrives at 42 µs intervals (the 250 µs

-2 -1 0 1 2

02

04

06

08

010

0

Frame Appearance Delay – 1st Logical Merging unit

Offset from 1PPS (µs)

Co

un

t

MU #1MU #2MU #3MU #4MU #5MU #6MU #7

Inter-frame Arrival Time

Time (µs)

Co

un

t

230 240 250 260 270230 240 250 260 270

11

00

104

10

6

Page 11: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 10 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

sampling period divided by six). The output queuing experienced by the bunched data is apparent, with the last frame of the bunch having an additional 55 µs latency. The spaced merging unit transmissions all experience the same latency as there is no queuing.

Once the bunched frames have passed through one switch they are serialised into a single flow, and as a consequence pass through subsequent switches with minimum additional latency. Figure 9 shows observed latency for bunched and spaced sampled value frames that have passed through five Ethernet switches in series (with no additional traffic introduced). This is a significant result as a fixed 15 µs latency, rather than load dependent latencies (of up to 250 µs), are introduced by each switch.

Figure 9: Six sampled value streams, showing effect of frame spacing and number of switches.

3.3 Sampled value (IEC 61850-9-2) and GOOSE (IEC 61850-8-1) interaction

A multi-function process bus uses a shared Ethernet network to exchange messages (sampled values, GOOSE and PTP). PTP messages generate a low volume of traffic, typically 300 bytes per second, and therefore do not affect the operation of sampled values or GOOSE. The impact of sampled value messages on PTP performance is discussed in Section 4.4 .

3.3.1 Method

The effect of background traffic was assessed using the Ethernet tap and capture card shown in Figure 8, with the addition of traffic generation using the tcpreplay tool running on Ubuntu Linux. Transmissions from tcpreplay were captured with the DAG card to confirm that all frames were sent, and at the correct rate (±1 µs). This was required as the packet timing was software used a CPU intensive routine.

Five Ethernet switches (four transparent clocks and one conventional switch) was used to evaluate protocol interactions, and is shown schematically in Figure 10. The “process bus” component operated at 100 Mb/s, using a combination of 100BASE-TX and 100BASE-FX media. The “control” component, which introduced the traffic to the network, operated at 1 Gb/s. Gigabit Ethernet was used to simulate the simultaneous arrival of frames from multiple sources. Switch S is the station bus root, switch P is the process bus root, and switches F1-F3 are field switches. All Ethernet switches were configured to enforce strict priority queuing, with 0 being the lowest priority and 7 the highest priority.

0 20 40 60 80 100 120

0.0

0.2

0.4

0.6

0.8

1.0

Bunched Data — One Switch

Latency (µs)

Cu

mu

lati

ve P

rob

abili

ty

MU 1MU 2MU 3MU 4MU 5MU 6

0 20 40 60 80 100 120

0.0

0.2

0.4

0.6

0.8

1.0

Bunched Data — Five Switches

Latency (µs)

Cu

mu

lati

ve P

rob

abili

ty

MU 1MU 2MU 3MU 4MU 5MU 6

0 20 40 60 80 100 120

0.0

0.2

0.4

0.6

0.8

1.0

Spaced Data — One Switch

Latency (µs)

Cu

mu

lati

ve P

rob

abili

ty

MU 1MU 2MU 3MU 4MU 5MU 6

0 20 40 60 80 100 120

0.0

0.2

0.4

0.6

0.8

1.0

Spaced Data — Five Switches

Latency (µs)

Cu

mu

lati

ve P

rob

abili

ty

MU 1MU 2MU 3MU 4MU 5MU 6

Page 12: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 11 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

Figure 10: The test network used to investigate interactions between GOOSE and sampled value messages.

The effect of high volume sampled value traffic on GOOSE signalling was investigated, along with the effect (if any) of GOOSE on sampled values. Synthetic GOOSE messages were transmitted at 100 ms intervals. Each GOOSE frame was 146 bytes long and contained six entries in the transmitted dataset. GOOSE messages with three priorities were transmitted using the 802.1Q header (priority 0, 4 and 7).

3.3.2 Results

Figure 11 shows that outbound GOOSE messages exhibited the same latency regardless of sampled value background traffic, while sampled value traffic levels and prioritisation affected inbound GOOSE messages. “Box plots” have been used to show the statistical distribution of the measurements. Outlying results are significant for protection systems, and these are not captured by mean and standard deviation. The box represents the Inter-quartile Range (IQR), which is the 25th to 75th percentile, and the bar is the mean value of all observations. The “whiskers” extend to the minimum and maximum values, provided these are within two times the IQR from the upper or lower limits of the box. Any points beyond this are outliers, and are shown as hollow circles if need be.

Figure 11: Latency of outbound (SF2) and inbound (F2S) GOOSE messages.

There is a 28 ns difference in mean latency of GOOSE messages between no background traffic and that with twenty synthetic merging units. This shows that GOOSE tripping of circuit breakers (an “outbound” message) via a process bus will not affect the flow of sampled values measurements from the switchyard

35 36 37 38 39 40 41 42

050

100

150

200

25

0

Outgoing Goose Latency

Latency (µs)

Cou

nt

GOOSE P0, 0 SVGOOSE P4, 0 SVGOOSE P7, 0 SVGOOSE P0, 12 SVGOOSE P4, 12 SVGOOSE P7, 12 SVGOOSE P0, 20 SVGOOSE P4, 20 SVGOOSE P7, 20 SV

P0

GO

OS

E0S

V

P4

GO

OS

E0S

V

P7

GO

OS

E0S

V

P0

GO

OS

E12

SV

P4

GO

OS

E12

SV

P7

GO

OS

E12

SV

P0

GO

OS

E20

SV

P4

GO

OS

E20

SV

P7

GO

OS

E20

SV

5010

015

020

025

0

Incoming GOOSE Latency

GO

OS

E L

ate

ncy

(µs)

Page 13: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 12 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

back to the control room. “Inbound” GOOSE is affected by sampled values, but applications such as tap-changer position reports are not as time-sensitive as tripping. If there is a particularly time-sensitive application then increasing the priority of the GOOSE message with 802.1Q tagging is shown to be effective.

Outgoing GOOSE messages had no impact on sampled value latency. Inbound GOOSE messages did increase the maximum latency of sampled values slightly, but the IQR did not increase.

Figure 12: Latency of sampled value (SV) messages from the first and last merging units of 20, with no traffic, “outbound” GOOSE, and “inbound” GOOSE traffic.

The most significant finding is that GOOSE messages (at a rate of ten per second) and sampled value data (20 merging units) can share a process bus without adverse interactions. The sampled value load is at the upper limit, and therefore operating the process bus with a more realistic load will provide greater capability of handling unexpected traffic. The only interactions that were apparent were when the messages travelled in the same direction on the same path, which resulted in additional queuing delays.

4 Performance of Precision Time Protocol for merging unit synchronisation Tests were performed with commercially available PTP clocks to determine whether PTP was a viable source of 1-PPS timing signals for merging units. These tests examined the steady-state and dynamic performance of slave clocks, with particular emphasis on recovery from contingencies. Figure 13 illustrates the equipment used to measure the jitter and drift of 1-PPS outputs from a slave clock directly connected to a grandmaster, representing the best case scenario.

Figure 13: Experimental arrangement to assess performance of PTP with directly connected grandmaster and slave.

No

GO

OS

EM

U 1

No

GO

OS

EM

U 2

0

Pri

0 G

OO

SE

MU

1

Pri

4 G

OO

SE

MU

1

Pri

7 G

OO

SE

MU

1

Pri

0 G

OO

SE

MU

20

Pri

4 G

OO

SE

MU

20

Pri

7 G

OO

SE

MU

20

5010

015

020

0

Effect of Outbound GOOSE on Sampled Values

SV

La

ten

cy (

µs)

No

GO

OS

EM

U 1

No

GO

OS

EM

U 2

0

Pri

0 G

OO

SE

MU

1

Pri

4 G

OO

SE

MU

1

Pri

7 G

OO

SE

MU

1

Pri

0 G

OO

SE

MU

20

Pri

4 G

OO

SE

MU

20

Pri

7 G

OO

SE

MU

20

5010

015

020

025

0

Effect of Inbound GOOSE on Sampled Values

SV

La

ten

cy (

µs)

Page 14: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 13 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

Process buses based on IEC 61850-9-2 must meet sampling accuracy requirements specified by IEC 61850-5. Table 1 lists the timing classes from edition 1 of this standard that are relevant to process bus networks, along with the proposed classes in the draft of edition 2. Protection class P2 is intended for transmission substation bays, and class P3 for transmission substation bays with high accuracy requirements. Class P1 is for distribution substations.

Table 1: Sampled value time accuracy classes from IEC 61850-5.

Protection Class

Required Accuracy

Edition 1 Timing Class

Edition 2 Timing Class

P1 ±25 µs T3 TS3

P2 ±4 µs T4 TS4

P3 ±1 µs T5 TS5

9-2LE specifies that sample synchronisation use 1-PPS signals, and that these have an accuracy that is better than 1 µs. This, in combination with pulse propagation delays and sampling errors, ensures that overall sampling error is within the ±4 µs required by timing class T4/TS4.

The PTP parameters specified in Table 1 of the PTP Power System Profile were used for these tests. The key parameters were 1 s update rates for Sync, Announce and PathDelay messages. Layer 2 multicast messages were used as the transport and the network speed was fixed at 100 Mb/s. The peer delay mechanism was used for path delay measurement. The grandmaster and slave clocks are represented by host names (PTPx), and the Ethernet switches are represented by a code letter (H, M, N and O). A total of three grandmasters, four slave clocks and four Ethernet switches (capable of transparent and boundary clock operation) were used in these tests.

4.1 Comparison of grandmaster and slave clocks

The test method used to assess synchronising performance is an established method, and uses the 1-PPS electrical outputs of the master and slave clocks. A digital oscilloscope (Tektronix DPO2014) sampling at 109 samples/s calculated the time difference (which is referred to as “delay” in these results) between the reference (grandmaster) and slave clock over a 30 minute period. A computer recorded each measurement (1800 in total for each test) for statistical analysis.

Time series plots for each grandmaster and slave clock combination are shown in Figure 14, with a common y-axis range for all plots. The rows represent grandmaster clocks and the columns represent slave clocks in the figure. The results show that PTP devices intended for power system use are interoperable, as each grandmaster and slave clock combination successfully synchronised. The worst performing combination of grandmaster and slave clock, PTPD/PTPA, had jitter ten times worse than that of best performing combination, PTPC/PTPF. The clocks selected for a substation timing system influence its performance.

4.2 Effect of grandmaster drift

A clear view of the sky is required for optimum GPS reception (or for any other global navigation satellite system, such as GLONASS1 or BDS2) as the satellites move in low earth orbit. There are times where building shading that reduces the clear view of the sky may result in a GPS receiver losing synchronisation to International Atomic Time (TAI). The internal oscillator will then drift from TAI, with the drift rate dependent upon the oscillator's stability [15]. Loss of lock between the grandmaster and the GPS system was identified as a problem during this investigation when the 1-PPS output of the slave clock exhibited large excursions for no obvious reason. Periodic loss of lock was due to GPS antenna shading from a steel building, as shown in Figure 15. This effect was recreated by disconnecting the GPS antenna on the grandmaster and observing the drift between its 1-PPS output and that of a reference GPS.

1 Globalnaya Navigatsionnaya Sputnikovaya Sistema, the Russian GNSS. 2 BeiDou Navigation Satellite System, the Chinese GNSS previously known as Compass.

Page 15: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 14 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

Figure 14: Thirty minute time series of the delay between 1-PPS outputs for each combination of directly connected grandmaster and slave clock.

Figure 15: GPS antenna ‘farm’ located one third the way up a 25 m high steel building.

Figure 16 shows the behaviour slave clocks when the grandmaster recovers synchronisation with TAI after a drift of 1 µs with two makes of slave clock, and with two grandmasters. The step and oscillation in synchronism are not acceptable, as the 1 µs accuracy requirement of 9-2LE will not be met if the grandmaster drifts more than 1 µs. The difference in response between manufacturers is a major concern, however there is no standard servo response defined in IEEE Std 1588.

Page 16: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 15 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

Figure 16: PTP slave clock response to grandmaster corrections.

One solution to this problem is to use a highly stable internal oscillator in the grandmaster, such as an oven controlled crystal oscillator (OCXO) or temperature compensated rubidium (Rb) cell, to reduce the drift from TAI when synchronisation with the GPS system is lost. These typically have four (OCXO) or six (Rb) orders of magnitude better stability than uncompensated crystal oscillators [16]. Another solution is to use multiple grandmaster clocks on the one network. The Best Master Clock Algorithm (BMCA) specified in section 9.2 of IEEE Std 1588 selects the active grandmaster using a defined hierarchy that includes clock class and quality. This enables a backup grandmaster to take over should the primary clock fail or experience degraded performance.

These tests further investigate this effect with grandmasters from different manufacturers, each with a different class of internal oscillator. The oscillators used were an uncompensated crystal oscillator (XO), a temperature compensated crystal oscillator (TCXO) and an OCXO. In each test the antenna was disconnected from the grandmaster under test and the deviation from the other grandmaster (which remained synchronised to GPS) was monitored. The drift was allowed to reach 1 µs and 2 µs, and then the antenna was reconnected.

Figure 17 shows the slave and grandmaster 1-PPS performance, as well as the active source of Sync messages, when a backup grandmaster is added to the timing system. The test proceeded as follows:

1. The GPS antenna on PTPA was disconnected at 17h49m02s, however Sync and Announce messages continued to be sent by PTPA.

2. PTPC started transmitting Sync and Announce messages at 17h49m16s when it determined that it had a higher quality clock than PTPA.

3. PTPA stopped transmitting PTP messages after three Announce messages from PTPC were received. 4. The internal clock of PTPA drift at 4.5 ns/s. 5. The GPS antenna was reconnected to PTPA when the drift was approximately –1 µs at 17h53m00s. 6. After 3 s elapsed PTPA started transmitting Sync and Announce messages. 7. PTPC ceased transmission after reception of one Announce message from PTPA and the slave clocks

resynchronised to PTPA after 5 s.

0 20 40 60 80

-0.5

0.0

0.5

1.0

1.5

PTPA Correction

Off

set

(µs)

PTPA GPS Drift

0 20 40 60 80

-1.5

-1.0

-0.5

0.0

0.5

Time (s)

Off

set

(µs)

PTPB SlavePTPF Slave

0 20 40 60 80

-1.5

-1.0

-0.5

0.0

0.5

PTPC Correction

Off

set

(µs)

PTPC GPS Drift

0 20 40 60 80-0

.50.

00.

51.

01.

5

Time (s)

Off

set

(µs)

PTPB SlavePTPF Slave

Page 17: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 16 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

Figure 17: Slave clock 1-PPS response to a grandmaster (GM) losing GPS synchronisation, with a backup grandmaster available.

Figure 17 shows that the slave clocks synchronised to the alternative grandmaster rather than tracking the faulty grandmaster, and resynchronised with the primary grandmaster when its time quality improved.

4.3 Transparent clock performance measurement

4.3.1 Method

Rather than use a packet injection test set or specialised slave clock, the approach taken was to simultaneously capture the output of a conventional grandmaster (with the Ethernet tap) and the output of the transparent clock under test with the DAG precision Ethernet capture card, as shown in Figure 18. Syntonising (frequency locking) the DAG card to a 1-PPS signal from a GPS improved the DAG card's time-stamping accuracy.

A third Ethernet port on the DAG card was used to inject multicast traffic into the transparent clock to simulate other network traffic. VLAN filtering was used to protect the grandmaster from the multicast traffic, and this is recommended practice with multiple multicast protocols in a process bus [17]. The multicast traffic was injected into the switches at 1000 Mb/s to simulate the simultaneous arrival of sampled value frames. This increased the latency but the total load did not exceed 100 Mb/s.

Figure 18: Experimental equipment used to measure PTP Sync message residence time, based on an Ethernet tap and an Endace DAG7.5G4 capture card.

“Correction Factor Error” (CFE) was defined by Burch et al. to be the difference between the actual residence time and the Correction field value [18]. A key observation was that a CFE that varies with latency

-100

0-5

000

500

GPS Antenna Failure on Grandmaster PTPA

Off

set

(ns)

17:46:00 17:48:00 17:50:00 17:52:00 17:54:00 17:56:00D

isco

nnec

t

Rec

onne

ct

Slave c.f. PTPC

PTPBPTPF

PTPA c.f. PTPC

PTPA

Time

G'm

aste

r

17:46:00 17:48:00 17:50:00 17:52:00 17:54:00 17:56:00

CA

Background traffic Normal PTP messages sent by grandmaster

Page 18: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 17 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

indicates an error in the transparent clock's estimate of the frame residence time. This metric is used in this paper to determine the performance of transparent clocks under a variety of network load conditions.

The DAG card combines frames captured from all ports into one “ERF” file. The PTP Correction field contents were extracted from PTP Sync and Follow-Up frames entering and leaving the transparent clock. The change in the Correction field values were calculated using the Wireshark network analysis tool. This allowed for transparent clocks to be installed between the grandmaster and transparent clock under test, and for one- or two-step transparent clocks to be used.

Four different transparent clocks were tested, and five multicast network loads were applied to each transparent clock: no background traffic, six 9-2LE merging units, 21 9-2LE merging units, 25 Mb/s random length frames and 95 Mb/s random length frames. The length of the random frames was uniformly distributed between 64 and 1500 bytes. Approximately 4880 PTP Sync and Follow-Up messages were captured for each test. The sync rate was faster than that specified in C37.238, but enabled a greater sample size to be collected in a reasonable time.

Experiments were conducted that examined the effect of transparent clocks on overall synchronising performance. The influence of each transparent clock was assessed by placing the Ethernet switch under test between the grandmaster and slave clocks, in place of the cross-over cable. No other network traffic was introduced to the switch, and switch management links were disconnected for the duration of each test. Each test ran for fifteen minutes, generating 900 1-PPS delay measurements. The PTPC grandmaster and PTPF slave clock were used for these tests as this pair gave the best synchronising performance when directly connected, as shown in Figure 14.

4.3.2 Results

The four network loads (six merging units, 21 merging units, 25 Mb/s random length frames and 95 Mb/s random length frames) increased the Sync message switch residence times. The 25 Mb/s random length frames case was selected to best demonstrate the effect of background traffic on latency and CFE. The other cases give similar results, with different ranges for latency. The random length frames represent TCP traffic on the process bus, which may be from a variety of sources. 25 Mb/s is equivalent to approximately six merging units. Figure 19 shows plots of CFE versus latency for the four transparent clocks.

The mean and range of CFE did not vary when background traffic was added for switches H, M and N; however switch O's CFE has increased range when the background traffic was applied. The CFE for switch O is dependent on latency, with the negative slope apparent in Figure 19. This suggested the transparent clock was over-estimating the frame residence time (Correction exceeds the actual residence time). The internal oscillator frequency error was estimated to be 18 parts per million.

The residence time of Sync messages varies between the transparent clocks, which contrasts with latency observations of sampled value and GOOSE traffic (the traffic most likely to be on a process bus), where latency is similar between Ethernet switches. This is due to variations in the PTP implementation of the transparent clocks, where switches H, N and O have software stacks (millisecond latencies) and switch M has a hardware stack (microsecond latencies).

The majority of transparent clocks did not increase the jitter in the measured delay, but all transparent clocks other than switch M introduced an offset to the 1-PPS delay. Figure 20 shows the distribution of 1-PPS delay for the four transparent clocks, with the direct-connect (cross-over cable) result included for comparison. Switch O introduced additional jitter to the distribution of delay. Selection of a grandmaster and slave clock that gave a 1-PPS delay that has low noise greatly assists the identification of perturbations introduced by transparent clocks.

The poor performance of switch O in Figure 20 shows that error in the Correction estimate shown in Figure 19 does degrade the performance of a PTP timing systems, and that independent performance testing should be undertaken when selecting products for substation automation systems.

Page 19: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 18 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

Figure 19: Transparent clock Correction field accuracy with 25 Mb/s random length frames background traffic.

Figure 20: Sample distributions showing the effect of transparent clocks on the best performing grandmaster/slave clock combination.

4.4 Sampled values (IEC 61850-9-2) and PTP (IEEE Std 1588/C37.238) interaction

Results from testing the transparent clocks individually show that three of the four transparent clocks accurately estimated switch residence time.

4 6 8 10 12 14 16

-300

-250

-200

-150

-100

Switch H25% Load, Random Length

Latency (ms)

Err

or (

ns)

20 40 60 80 100 120

-650

-600

-550

-500

Switch M25% Load, Random Length

Latency (µs)

Err

or (

ns)

0.5 1.0 1.5 2.0 2.5 3.0

-600

-550

-500

-450

Switch N25% Load, Random Length

Latency (ms)

Err

or (

ns)

10 15 20 25 30 35 40

-900

-800

-700

-600

-500

-400

-300

Switch O25% Load, Random Length

Latency (ms)

Err

or (

ns)

-100 -50 0 50 100

0.0

00

.01

0.0

20

.03

0.0

40

.05

0.0

6

Transparent Clock Effect on Delay

Grandmaster-Slave Offset (ns)

De

ns

ity

Switch

HMNOnone

Page 20: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 19 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

4.4.1 Method

Synthetic sampled value traffic was injected into the first transparent clock at 1 Gb/s to simulate the simultaneous arrival of frames from multiple merging units. Figure 21 shows the arrangement of devices. 1-PPS delays between the grandmaster and slave clock were recorded for 15 minutes (900 samples for each test). The transparent clocks that correctly estimated residence time were placed in series to represent a substation network topology, with bay level, voltage level, and core process bus switches.

Figure 21: Connection of three transparent clocks with sampled value traffic injection to simulate a loaded process bus network.

Prioritisation and VLAN separation using IEEE Std 802.1Q tagging was used, with sampled value and PTP frames placed in separate VLANs. Sampled value frames were assigned a priority of 4 for all tests. Two sets of experiments were conducted to determine the effect of load and priority on synchronising performance:

1. The effect of sampled value traffic on PTP performance was assessed by injecting six levels of sampled value traffic into the test system: no traffic, 1 merging unit (MU), 3 MUs, 6 MUs, 12 MUs and 21 MUs. Previous testing has shown that 21 sampled value transmissions are the maximum that 100 Mb/s Ethernet can accommodate without dropping frames in a 50 Hz power system [19]. PTP frames had a fixed priority of 4 for the loading tests.

2. The effect of prioritisation on PTP performance was examined by varying the 802.1Q priority of PTP frames while keeping the sampled value frame priority fixed at 4. Two levels of sampled value traffic were injected (12 MUs and 21 MUs) for each of the three PTP priorities: 2, 4 and 7.

4.4.2 Results

Figure 22 shows there is little variation in PTP performance as sampled value traffic levels increased from one to 21 merging units, and the observed differences are more likely to be natural variation in clock performance. It is significant that the “none” and “21x MU”' sample distributions are the most similar, despite having the largest difference in sampled value traffic levels. Table 2 lists the total latency experienced by the PTP Sync message after passing through all three transparent clocks. Three outlier delays occurred, with all other latencies under 23 ms. The Correction field remained accurate, even at 657 ms, with a CFE of –484 ns.

Figure 23 shows that at moderate load (12 MU, approximately 48 Mb/s) priority did not affect performance. However, at high load (21 MU) higher priority PTP messages did yield slightly improved performance (an improvement of 5 ns over Priority 4), and the low priority case had marginally degraded performance (by approximately 18 ns). This result is contrary to established practice where PTP messages are thought to require high priority handling; however this does require the use of Ethernet switches that support PTP and the peer-delay mechanism.

Page 21: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 20 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

Figure 22: Sample distributions for three transparent clocks in series, with six different 9-2LE sampled value (SV) traffic levels.

Table 2: Latency experienced by PTP Sync frames passing through three transparent clocks.

Sampled Value Traffic

Latency

Min Max

None 2.59 ms 16.1 ms

1 MU 2.59 ms 364 ms

3 MU 2.58 ms 187 ms

6 MU 2.57 ms 19.5 ms

12 MU 2.58 ms 15.3 ms

21 MU 2.59 ms 657 ms

Figure 23: Sample distributions for three transparent clocks in series, with two different sampled value traffic levels and three PTP 802.1Q priorities .

60 80 100 120 140 160 180

0.00

0.01

0.02

0.03

0.04

0.05

Effect of Sampled Value Traffic on PTP Performance

Offset (ns)

Den

sity

SV Traffic

none1x MU3x MU6x MU12x MU 21x MU

40 60 80 100 120 140

0.00

0.01

0.02

0.03

0.04

Effect of Prioritisation – 12x SV MU

Offset (ns)

Den

sity

PTP Priority

247

40 60 80 100 120 140

0.00

0.01

0.02

0.03

0.04

0.05

0.06

Effect of Prioritisation – 21x SV MU

Offset (ns)

Den

sity

PTP Priority

247

Page 22: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 21 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

5 Protection system performance The primary function of a protection system in a substation is to protect plant from damage when faults occur. Utilities must ensure that their protection systems, using conventional or process bus connections, will clear faults within the time limits specified in their jurisdictional grid code. The previous sections in this paper have assessed the performance of synchronising systems and real-time networks with a “bottom up” approach. This section presents the result of testing of the entire protection system from the “top down”. Transformer differential protection was used in the system tests as it required multiple sampled value streams that were synchronised to each other, which requires merging unit synchronisation.

Experiments were conducted to validate accuracy and consistency of the measurement system. This was achieved by measuring protection performance with the Real Time Digital Simulator and an independent system. The performance of process bus protection was compared to traditional protection connections (current transformer and relay trip contacts) using the same protection relay, quantifying the performance advantage or penalty of a digital process bus.

5.1 Performance Measures

Australia's National Electricity Rules [20] and the UK Grid Code [21] specify the clearing times listed in Table 3. Australia and the UK operate 50 Hz power systems, and therefore the clearance times range from four to six power frequency cycles.

Table 3: Protection clearance times for power system faults.

Australia United Kingdom Clearance Time

≥ 400 kV 400 kV 80 ms

250 kV – 400 kV 275 kV 100 ms

100 kV – 250 kV 132 kV 120 ms

The operating time of high voltage circuit breakers generally range from two power frequency cycles (400 kV and some 275 kV breakers) to three cycles (some 275 kV and most 110/132 kV breakers). As a result, protection relay response times must be less than 40 ms at 400 kV and be less than 60 ms for other operating voltages. 45 ms was selected as the protection response time benchmark, which is consistent with utility protection testing methods.

5.2 Method

The transformer differential protection relay used for this testing was an ABB RET670 with sampled value (9-2LE) inputs and conventional analogue current transformer (CT) and voltage transformer (VT) inputs (1 A and 110 V respectively). The relay had digital inputs and dry contact outputs, along with IEC 61850-8-1 station bus connectivity. The simulated transformer was a 375 MVA 275/110 kV auto transformer. The protection settings were the factory default settings. The restraint curve is shown graphically in Figure 24. Unrestrained operation was set to 10 per unit (p.u.) differential current (ID). The bias current in the RET670 is the maximum of the primary or secondary current in per unit terms.

The RTDS had three GTNET cards fitted, two with sampled values capability (which are referred to as GTNET-SV) and one with GOOSE capability (referred to as GTNET-GSE). The general connection arrangements are shown in Figure 25. Sampled value and GOOSE network traffic was generated by the DAG Ethernet card.

Page 23: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 22 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

Figure 24: RET670 three-section restraint curve with factory settings.

Figure 25: Connection arrangements to assess performance of RET670.

5.3 Protection response with conventional and Ethernet connections

The first series of tests performed validated the response measurement system, and assessed the variation in response time under ideal network and synchronising conditions.

5.3.1 Method

The performance of the RTDS was cross-checked with an OMICRON CMC256-6 protection test set that had sampled value and GOOSE capability. This test was used to inject conventional currents (1 A nominal) to compare sampled value performance to that with conventional CT and VT connections. A conventional relay contact and a GOOSE message were used by the RET670 to indicate to the RTDS or OMICRON that a fault was detected.

Page 24: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 23 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

(a)

(b)

Figure 26: Sources for protection testing. (a) RTDS and (b) OMICRON test set.

This gave the six configurations listed in Table 4 for comparison purposes.

Table 4: List of measurement configurations for protection relay testing.

Source Relay Input Tripping

RTDS GTNET-SV Sampled Values GOOSE

RTDS GTNET-SV Sampled Values Relay

CMC256-6 Sampled Values GOOSE

CMC256-6 Sampled Values Relay

CMC256-6 Conventional 1 A GOOSE

CMC256-6 Conventional 1 A Relay

High fault current (20–25 p.u.) three phase faults on the high voltage (HV) terminals of the transformer and medium fault current (3–4 p.u.) line to ground faults on the low voltage (LV) transformer terminals were used to test unrestrained and restrained operation of the relay.

5.3.2 Results

The majority of tests were conducted with 1000 faults applied, as per the requirements of IEC 61850-10. Figure 27(a) shows the protection response time for three phase faults on the HV side of the transformer, which resulted in a fault current of 24 p.u. (approximately 19 kA). Figure 27 (b) is the corresponding LV phase to ground fault response time, with fault currents of approximately 3.5 p.u. (7 kA).

Page 25: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 24 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

(a)

(b)

Figure 27: Comparison of protection response times for the RTDS and OMICRON CMC256-6 with (a) HV three phase faults and (b) LV phase to ground faults.

The response time of the differential function in the RET670 to 100 restrained and 100 unrestrained faults is shown in Figure 28. These response times were calculated from disturbance records downloaded from the RET670 that had a time resolution of 1 ms.

Figure 28: T3WPDIF protection response time based on disturbance records.

The mean response for restrained faults was 25.1 ms and for restrained faults was 11.9 ms. The variation in the unrestrained response was half that of the restrained response, with a standard deviation of 0.71 ms. These results show that the variation in response measured by the RTDS is mostly due to response variation

RT

DS

GO

OS

E

OM

ICR

ON

SV

/GO

OS

E

OM

ICR

ON

CT

/GO

OS

E

RT

DS

Rel

ay

OM

ICR

ON

SV

/Rel

ay

OM

ICR

ON

CT

/Rel

ay

1012

1416

1820

Three-phase HV Faults

Re

spo

nse

Tim

e (

ms)

RTDSOMICRON SVOMICRON CTGOOSERelay

RT

DS

GO

OS

E

OM

ICR

ON

SV

/GO

OS

E

OM

ICR

ON

CT

/GO

OS

E

RT

DS

Rel

ay

OM

ICR

ON

SV

/Rel

ay

OM

ICR

ON

CT

/Rel

ay

2224

2628

3032

3436

Line-ground LV Faults

Re

spo

nse

Tim

e (

ms)

RTDSOMICRON SVOMICRON CTGOOSERelay

0 20 40 60 80 100

05

10

15

20

25

30

Differential Function Response

Fault Number

PD

IF T

ime

(m

s)

Fault Type

RestrainedUnrestrained

Page 26: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 25 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

in the time required for the differential protection function to detect the fault. Second harmonic blocking, which avoids tripping due to transformer in-rush current, was the dominant restraint signal.

5.4 Repetitive testing with the RTDS

The scripting function of the RTDS run-time was used to map out the differential protection restraint curves, while recording the response time. The restraint characteristic (shown in Figure 24) was “walked” with bias currents (IB) from 0.5–5.0 p.u. and differential currents (ID) ranging from 0.0 p.u. to IB (the maximum differential current that can be made to flow through the transformer). IB was set with a variable load on the LV bus and ID was set with a load on the HV or LV terminals of the transformer (inside the protected zone). These loads are shown in Figure 4. An extract of the RTDS script that performed the assessment is shown in Figure 29.

Figure 29: RTDS script that walks the relay restraint characteristic.

Figure 30 shows the protection response times for (a) HV faults and (b) LV faults. The colours represent the response time, which ranges from 17–33 ms for HV faults and 21–36 ms for LV faults. The observed response matches the ideal restraint curve, and there is no slowing down of response on the curve as the trip/restrain decision is made.

for (ibias=IB_START; ibias < IB_STOP; ibias+=IB_STEP) {

fprintf(stdmsg, "iB=%f\n", ibias);

for (idiff=0; idiff <= ibias; idiff+=ID_STEP) {

// 275kV primary and 132kV secondary pdiff = idiff * 787 * 275000 * 1.7320508 / 1e6; pbias = (ibias-idiff) * 1964 * 110000 * 1.7320508 / 1e6;

if (pdiff < 0.01) { pdiff = 0.01;

}

SetSlider "Subsystem #1 : Loads : LDbias : Pset" = pbias; SetSlider "Subsystem #1 : Loads : HVLDdiff : Pset" = pdiff;

// Let loads stabilise SUSPEND 0.2; PushButton "Subsystem #1 : CTLs : Inputs : PBHVDiff"; SUSPEND 1.8;

response_time = MeterCapture("PDIF_TIME"); tap_pos = MeterCapture("T1_TapPos"); ret670_ib = MeterCapture("RET670.IB"); ret670_id = MeterCapture("RET670.ID");

fprintf(stdout,"%f,%f,%f,%f,%f,%f,%f,%d\n", ibias, idiff, pbias, pdiff, response_time,

ret670_ib, ret670_id, tap_pos); fprintf(stdmsg,"%f,%f,%f,%f,%f,%f,%f,%d\n", ibias, idiff, pbias, pdiff, response_time,

ret670_ib, ret670_id, tap_pos);

ReleaseButton "Subsystem #1 : CTLs : Inputs : PBHVDiff";

SUSPEND 1.5;

} }

Page 27: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 26 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

(a)

(b)

Figure 30: Protection response time for (a) HV and (b) LV faults of differing magnitudes.

The RTDS using sampled values is well suited to these tests as there is no heating due to high secondary currents during each simulated fault. Each mapping sequence in Figure 30 required the application of 1325 faults at 3.5 s intervals. The slight divergence from the curve in Figure 30(b) is due to the transformer impedance and very high load currents. When the tests were repeated with a very low impedance transformer the curves overlapped. This is a major difference between secondary injection (with ideal currents) and real-time simulation where currents are dependent upon power system objects.

5.5 Merging unit synchronising error

The effect of errors in the 1-PPS signal used to synchronise the sampling of GTNET-SV merging units was evaluated by introducing controlled delays into one GTNET-SV card. The actual delay was configured through digital inputs to the a custom built time delay generator, and these were controlled by digital outputs on the RTDS. The time delays were verified with a digital oscilloscope (Tektronix DPO2014). The mean error, with 1000 samples at each delay (2–1000 µs) was 65 ns and the standard deviation was 0.2 ns.

Spill current from synchronising error changed the point at which the relay tripped. The characteristic was mapped by applying bias (IB) currents in 0.2 p.u. steps from 0.5--5.0 p.u., and differential currents (ID) from 0.8 p.u. below the expected curve to 0.4 p.u. above the curve, in 0.02 p.u. steps. This resulted in 3589 faults being applied for each of the following fixed delays: 0 µs, 4 µs, 10 µs, 100 µs, 500 µs and 1000 µs.

Figure 31 shows four restraint curves with synchronising errors (0 µs, 100 µs, 500 µs and 1000 µs) applied to the LV merging unit (GTNET-SV #2). ‘Ibias’ is the bias current and ‘Idiff’ is the differential current, with both expressed as per unit (p.u.) quantities. The black points are where the relay issued a trip command, and the hollow grey points are where a trip was not issued. The red line is the restraint characteristic of the relay. It can be seen in sub-figures (a) and (b) that the trip/no-trip boundary matches the restraint curve, while in (c) and (d) the relay is tripping at lower ID values than desired. It must be noted that 1000 µs is a deliberately extreme case and correct operation of the relay was not expected.

0 1 2 3 4 5

01

23

45

Fault on HV Terminal

Ibias (p.u.)

Idiff

(p.

u.)

Trip Time (ms)

No Trip(17.3,18.1](18.1,19](19,19.9](19.9,20.7](20.7,21.6](21.6,22.4](22.4,23.3](23.3,24.2](24.2,25](25,25.9](25.9,26.7](26.7,27.6](27.6,28.5](28.5,29.3](29.3,30.2](30.2,31](31,31.9](31.9,32.8]

0 1 2 3 4 5

01

23

45

Fault on LV Terminal

Ibias (p.u.)Id

iff (

p.u

.)

Trip Time (ms)

No Trip(21.8,22.6](22.6,23.4](23.4,24.2](24.2,25](25,25.8](25.8,26.6](26.6,27.4](27.4,28.2](28.2,29](29,29.7](29.7,30.5](30.5,31.3](31.3,32.1](32.1,32.9](32.9,33.7](33.7,34.5](34.5,35.3](35.3,36.1]

Page 28: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 27 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

Figure 31: Differential protection restraint curves with (a) 0 µs, (b) 100 µs, (c) 500 µs and (d) 1000 µs synchronising error.

5.6 Effect of background network traffic on protection response

Artificial sampled value traffic was generated to load the process bus to near capacity. Three sets of synthetic sampled value traffic were created:

1. A unique source address and a unique multicast destination address for each “stream”. 2. A unique source address and the same multicast destination address used by the RTDS. 3. The same source address and multicast destination address used by the RTDS.

The three addresses were used to determine how the RET670 filtered sampled value traffic. The DAG card transmitted the background traffic continuously while the RTDS applied the faults to the protection relay. This gave a maximum network load of 20 merging units (two GTNET-SV streams and 18 background streams).

Figure 32(a) shows that the relay operated with no adverse effects with 20 merging units sharing the network when different multicast destination addresses were used. Figure 32(b) shows when the same multicast destination address was used the response time started to degrade at 14 merging units (two desired and 12 background), and is unacceptable at 19 merging units in total. The response with the source address set to that of the RTDS was the same as Figure 32(b).

Figure 32: Protection performance with varying background sampled value traffic levels with (a) different multicast addresses and (b) the same multicast address as the RTDS.

0 1 2 3 4 5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

No Fixed Delay

Ibias (p.u.)

Idiff

(p.

u.)

No TripTripRestraint Curve

(a)

0 1 2 3 4 5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

100 µs Fixed Delay

Ibias (p.u.)

Idiff

(p.

u.)

No TripTripRestraint Curve

(b)

0 1 2 3 4 5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

500 µs Fixed Delay

Ibias (p.u.)

Idiff

(p.

u.)

No TripTripRestraint Curve

(c)

0 1 2 3 4 5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

1000 µs Fixed Delay

Ibias (p.u.)

Idiff

(p.

u.)

No TripTripRestraint Curve

(d)

0 3 6 12 18

1012

1416

1820

Different Multicast Destination

SV Load (Background Merging Units)

Re

spo

nse

Tim

e (

ms)

0 3 6 12 16 17 18

1012

1416

1820

Same Multicast Destination

SV Load (Background Merging Units)

Re

spo

nse

Tim

e (

ms)

Page 29: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 28 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

6 Discussion The unique characteristics of sampled value networks, which have hard real-time requirements, have been presented in this paper. Measurements from a live process bus substation (the only one outside of China) confirmed that transmissions from merging units can occur at the same time and at the same rate. The term coherent transmission has been introduced to describe this type of data. Coherent transmission from merging units affects the switching performance of Ethernet switches, with additional latency introduced due to output queuing delays. Once the frames are queued subsequent Ethernet switches introduce minimal delay, which is the time required to transmit the frame. This permits the use of Ethernet switches in the field to reduce cabling from the switchyard to the control room of a substation without significantly impacting network performance. The results presented in this paper demonstrate that a multi-function process bus can coexist on a shared Ethernet network. A fully switched Ethernet network with full duplex connections does not experience collisions, however queuing does introduce latency. Provided the data rate is less than the maximum capacity of any link, no frames will be lost. Process bus networks are “mission critical” and simply cannot be permitted to fail.

The best performing grandmaster was PTPC, with the least jitter when used with any of the slave clocks. Similarly, the best performing slave clock was PTPF regardless of the grandmaster used. These clocks had a range of local oscillator types, ranging in quality (from low to high) from a crystal oscillator (XO) in PTPA to an oven controlled crystal oscillator (OCXO) in PTPC. The slave-only clock PTPB had an XO and PTPF had a temperature compensated crystal oscillator (TCXO), and the difference in performance is apparent. The transparent clock “correction” test procedure used a standard grandmaster and general purpose network test tools (Ethernet tap and precision capture card), rather than task-specific test equipment at more than ten times the cost. This test could become part of a standard validation process for evaluating transparent clocks for substation timing, and can be applied throughout a substation. The results shown here demonstrate that the PTP system performance meets the ±1 µs requirements of 9-2LE when a shared process bus network is used for sampled values and for time synchronisation. Increased prioritisation of PTP is of marginal benefit as the capability of transparent clocks to accurately estimate residence time compensates for queuing delays experienced by PTP frames. It has been suggested that PTP messages should be switched with high priority to ensure PTP accuracy, but the results presented in this paper show that this is not necessary when peer-delay transparent clocks are used.

The protection performance with sampled values over Ethernet was very similar to that with traditional 1 A secondary connections when tested with the OMICRON test set. The sampled value response was, on average, 0.4 ms slower than the CT response, and is very close to the 1 ms difference other researchers have measured with a different make and model of protection relay [22]. There was however a significant improvement in tripping performance with GOOSE instead of relay contacts. GOOSE allows for a richer set of data to be transmitted, including time-stamps and quality attributes, yields higher performance and is self-monitoring. The effect of background traffic on protection response was shown in the previous section. This reinforces the need to design a system where multicast destination addresses are allocated to minimise the traffic transmitted to a protection relay. The RET670 was robust, accepting high levels of network traffic before performance degraded, however this cannot be assumed of other protection relays. The tests presented in this paper are a means of verifying this capability.

The results in the previous section demonstrate that the protection response of a transformer protection relay subscribing to sampled values and publishing trip indications over GOOSE meets the requirements of Australian and UK grid codes. The slowest LV restrained trip of 31.7 ms, is still less than the 40 ms time required at 400 kV with two cycle circuit breakers, or at 275 kV with three cycle circuit breakers.

Page 30: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 29 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

7 Conclusion The research presented in this paper has shown that multifunction process buses based on IEC 61850 and IEEE Std 1588 are surprisingly resilient, provided network traffic is maintained under capacity limits. Utilities should be mapping out how they will implement process buses in their substations. The move to digital networks has clear advantages, but the skill set required is quite different to conventional substation automation. Commissioning and maintenance work practices will require significant review and revision.

The successful operation of this particular test bed does not guarantee that all process bus products will meet performance requirements. Independent certification is needed for PTP, and the IEEE Conformity Assessment Program has extended its Precision Time Protocol program to include the C37.238 Power System Profile. Manufacturers need submit their product to this testing to provide confidence that their products meet the minimum requirements of IEEE Std 1588 and IEEE Std C37.237. A limited number of sampled value products (merging units and protection test sets) have passed conformance testing; however sampled value protection relays and a number of merging units on the market have not been assessed for standards conformance [23].

Significant improvements to performance, efficiency and cost have been achieved inside substation control rooms through the adoption of digital technology over the past 30 years. It is now time for similar improvements to be achieved in high voltage switchyards. Once NCITs and interoperable digital process buses become the norm, substations of the future will be safer and reduce environmental harm. New work practices will be enabled by the digital transfer of information, and this will help lower the cost of designing, constructing and operating high voltage substations while improving performance and capability.

This research has shown that substation automation using process bus networks can meet the needs of electricity transmission utilities. The tests presented in this thesis provide evidence that a process bus protection system meets performance requirements—in many cases the performance exceeds that of conventional systems.

References

[1] W. Rogowski and W. Steinhaus, “Die Messung der magnetischen Spannung (Messung des Linienintegrals der magnetischen Feldstärke) [The measurement of the magnetic voltage (Measurement of the line integral of the magnetic field strength)],” Archiv für Elektrotechnik, vol. 1, no. 4, pp. 141-150, 1912.

[2] S. Saito, Y. Fujii, K. Yokoyama, J. Hamasaki and Y. Ohno, “8C1—The Laser Current Transformer for EHV Power Transmission Lines,” IEEE J. Quant. Electron., vol. 2, no. 8, pp. 255-259, 1966.

[3] L. Hossenlopp, D. Chatrefou, D. Tholomier and D. P. Bui, “Process Bus: Experience and Impact on Future System Architectures,” in CIGRÉ 2008, Paris, France, 2008.

[4] IEEE Power Engineering Society, IEEE Std C37.92 — IEEE Standard for Analog Inputs to Protective Relays from Electronic Voltage and Current Transducers, New York, USA, 2005.

[5] IEC TC38, IEC 60044-8 — Instrument transformers – Part 8: Electronic current transformers, Geneva, Switzerland, 2002.

[6] IEC TC57, IEC 61850-9-2 — Communication networks and systems for power utility automation – Part 9-2: Specific communication service mapping (SCSM) – Sampled values over ISO/IEC 8802-3, Geneva, Switzerland, 2011.

[7] UCAIug, “Implementation Guideline for Digital Interface to Instrument Transformers Using IEC 61850-9-2 (R2-1),” 7 June 2004. [Online]. Available: http://iec61850.ucaiug.org/Implementation%20Guidelines/DigIF_spec_9-2LE_R2-1_040707-CB.pdf.

[8] IEC TC57, IEC 61850-8-1 — Communication networks and systems for power utility automation – Part 8-1: Specific communication service mapping (SCSM) – Mappings to MMS (ISO 9506-1 and ISO 9506-2) and to ISO/IEC 8802-3,

Page 31: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 30 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

Geneva, Switzerland, 2011.

[9] F. Baldinger, T. Jansen, M. van Riet, F. Volberda, F. van Erp, M. Dorgelo and W. van Buijtenen, “Advanced Secondary Technology to Evoke the Power of Simplicity,” in 9th IET Int. Conf. Dev. Power Sys. Prot. (DPSP), Glasgow, UK, 2008.

[10] IEEE Instrumentation & Measurement Society, IEEE Std 1588 — IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems, New York, USA, 2008.

[11] IEEE Power & Energy Society, IEEE Std C37.238 — IEEE Standard Profile for Use of IEEE 1588 Precision Time Protocol in Power System Applications, New York, USA, 2011.

[12] IEEE PES PSRC WG H7/Sub C7 Members and Guests, “Standard profile for use of IEEE Std 1588-2008 Precision Time Protocol (PTP) in power system applications,” in Int. IEEE Symp. Precision Clock Synchron. Meas. Control Commun. (ISPCS), San Francisco, CA, USA, 2012.

[13] Endace, “Endace DAG 100 percent packet capture,” [Online]. Available: http://www.endace.com/endace-dag-high-speed-packet-capture-cards.html. [Accessed 28 December 2012].

[14] NetOptics, “10/100/1000BaseT Tap,” [Online]. Available: http://www.netoptics.com/products/10100/101001000baset-tap. [Accessed 18 February 2013].

[15] R. L. Scheiterer, C. Na, D. Obradovic and G. Steindl, “Synchronization Performance of the Precision Time Protocol in Industrial Automation Networks,” IEEE Trans. Instrum. Meas., vol. 58, no. 6, pp. 1849-1857, June 2009.

[16] M. Bloch, O. Mancini and T. McClelland, “Mass-Produced Quartz Oscillators as Low-Cost Replacement of Passive Rubidium Vapor Frequency Standards,” in 2007 IEEE Int. Freq. Control Symp. Jnt 21st Europ. Freq. Time Forum, Geneva, Switzerland, 2007.

[17] D. M. E. Ingram, P. Schaub and D. Campbell, “Multicast Traffic Filtering for Sampled Value Process Bus Networks,” in 37th Ann. Conf. IEEE Indust. Electron. Soc. (IECON), Melbourne, Australia, 2011.

[18] J. B. Burch, K. Green, J. Nakulski and D. Vook, “Verifying the Performance of Transparent Clocks in PTP Systems,” in 2009 IEEE Int. Symp. Precis. Clock Synchr. Meas. Control Commun. (ISPCS)", Brescia, Italy, 2009.

[19] D. M. E. Ingram, P. Schaub, R. R. Taylor and D. A. Campbell, “Performance Analysis of IEC 61850 Sampled Value Process Bus Networks,” IEEE Trans. Indust. Informat., vol. PP, no. 9, pp. 1-9, 2013.

[20] Australian Energy Market Commission, “National Electricity Rules (Version 54),” 1 January 2013. [Online]. Available: http://www.aemc.gov.au/Electricity/National-Electricity-Rules/Current-Rules.html.

[21] National Grid Electricity Transmission plc, “The Grid Code (Issue 5 Revision 2),” 31 January 2013. [Online]. Available: http://www.nationalgrid.com/uk/Electricity/Codes/gridcode/gridcodedocs/.

[22] L. Yang, P. A. Crossley, A. Wen, R. Chatfield and J. Wright, “Performance assessment of a IEC 61850-9-2 based protection scheme for a transmission substation,” in Innov. Smart Grid Tech. Conf. Europe 2010 (ISGTE), Manchester, UK, 2011.

[23] DNV KEMA, “IEC 61850 Test Register,” 6 February 2013. [Online]. Available: http://www.dnvkema.com/Images/COM118%20Conformance%20test%20Register%2061850.pdf.

Page 32: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 31 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

List of Figures

Figure 1:  Multicore cables used for conventional secondary connections between the switchyard and control room. 3 

Figure 2:  The simplicity of Ethernet relay connections. 4 Figure 3:  Schematic of the process bus test and evaluation system. 5 Figure 4:  RSCAD ‘draft’ schematic of the power system model used for transformer protection assessment. 6 Figure 5:  The RTDS ‘run-time’ interface for transformer protection testing, with a fault applied to the high

voltage transformer terminal. 6 Figure 6:  Latency measurement using externally synchronised Ethernet capture card. 8 Figure 7:  Distributions of (a) variation in frame arrival time from the average arrival time and (b) the time

spacing of sampled value frames. 9 Figure 8:  Schematic showing the use of the Ethernet tap and multi-port capture card to measure Ethernet

switch latency. 9 Figure 9:  Six sampled value streams, showing effect of frame spacing and number of switches. 10 Figure 10:  The test network used to investigate interactions between GOOSE and sampled value messages. 11 Figure 11:  Latency of outbound (SF2) and inbound (F2S) GOOSE messages. 11 Figure 12:  Latency of sampled value (SV) messages from the first and last merging units of 20, with no

traffic, “outbound” GOOSE, and “inbound” GOOSE traffic. 12 Figure 13:  Experimental arrangement to assess performance of PTP with directly connected grandmaster

and slave. 12 Figure 14:  Thirty minute time series of the delay between 1-PPS outputs for each combination of directly

connected grandmaster and slave clock. 14 Figure 15:  GPS antenna ‘farm’ located one third the way up a 25 m high steel building. 14 Figure 16:  PTP slave clock response to grandmaster corrections. 15 Figure 17:  Slave clock 1-PPS response to a grandmaster (GM) losing GPS synchronisation, with a backup

grandmaster available. 16 Figure 18:  Experimental equipment used to measure PTP Sync message residence time, based on an

Ethernet tap and an Endace DAG7.5G4 capture card. 16 Figure 19:  Transparent clock Correction field accuracy with 25 Mb/s random length frames background

traffic. 18 Figure 20:  Sample distributions showing the effect of transparent clocks on the best performing

grandmaster/slave clock combination. 18 Figure 21:  Connection of three transparent clocks with sampled value traffic injection to simulate a loaded

process bus network. 19 Figure 22:  Sample distributions for three transparent clocks in series, with six different 9-2LE sampled value

(SV) traffic levels. 20 Figure 23:  Sample distributions for three transparent clocks in series, with two different sampled value traffic

levels and three PTP 802.1Q priorities . 20 Figure 24:  RET670 three-section restraint curve with factory settings. 22 Figure 25:  Connection arrangements to assess performance of RET670. 22 Figure 26:  Sources for protection testing. (a) RTDS and (b) OMICRON test set. 23 Figure 27:  Comparison of protection response times for the RTDS and OMICRON CMC256-6 with (a) HV

three phase faults and (b) LV phase to ground faults. 24 Figure 28:  T3WPDIF protection response time based on disturbance records. 24 Figure 29:  RTDS script that walks the relay restraint characteristic. 25 Figure 30:  Protection response time for (a) HV and (b) LV faults of differing magnitudes. 26 Figure 31:  Differential protection restraint curves with (a) 0 µs, (b) 100 µs, (c) 500 µs and (d) 1000 µs

synchronising error. 27 Figure 32:  Protection performance with varying background sampled value traffic levels with (a) different

multicast addresses and (b) the same multicast address as the RTDS. 27 

Page 33: c Copyright 2013 CIGRE Australia Notice Changes introduced ...(similar to the way 9-2LE limits the options available to IEC 61850-9-2). The IEEE Power and Energy Society developed

CIGRE Australia APB5 SEAPAC 2013

Brisbane 12-13 March Assessment of Real-Time Networks and Timing for Process Bus Applications

IEC 61850 and IEEE Std 1588

Page 32 of 32

CIGREAUB5SEAPAC2013DOC_147_AU_Ingram.docx

List of Tables

Table 1:  Sampled value time accuracy classes from IEC 61850-5. 13 Table 2:  Latency experienced by PTP Sync frames passing through three transparent clocks. 20 Table 3:  Protection clearance times for power system faults. 21 Table 4:  List of measurement configurations for protection relay testing. 23 

Biography: David Ingram

David Ingram received the B.E. (with honours) and M.E. degrees in electrical and electronic engineering from the University of Canterbury, Christchurch, New Zealand, in 1996 and 1998, respectively. He is a Ph.D. candidate at the Queensland University of Technology, Brisbane, Australia, with research interests in substation automation and control. He has previous experience in the Queensland electricity supply industry in transmission, distribution, and generation. Mr Ingram is a Chartered Member of Engineers Australia and is a Registered Professional Engineer of Queensland. He is a corresponding member of IEEE PSRC Working Group H7/SubC7 and the P1588 PTPv3 study group.

Biography: Pascal Schaub

Pascal Schaub received the B.Sc. degree in computer science from the Technical University Brugg-Windisch, Windisch, Switzerland, (now the University of Applied Sciences and Arts Northwestern Switzerland) in 1995. He was with Powerlink Queensland as Principal Consultant Power System Automation, developing IEC61850 based substation automation systems. He is currently the Principal Process Control Engineer at QGC, a member of the BG Group. He is a member of Standards Australia working group EL-050 “Power System Control and Communications” and a member of the international working group IEC TC57 WG10 “Power System IED Communication and Associated Data Models.”

Biography: Duncan Campbell

Duncan Campbell received the B.Sc. degree (with honours) in electronics, physics, and mathematics and the Ph.D. degree from La Trobe University, Melbourne, Australia. He has collaborated with a number of universities around the world, including Massachusetts Institute of Technology, and Telecom-Bretagne, Brest, France. He is currently a Professor with the School of Electrical Engineering and Computer Science, Queensland University of Technology, Brisbane, Australia, where he is also the Director of the Australian Research Centre for Aerospace Automation (ARCAA). His research areas of interest are robotics and automation, embedded systems, computational intelligence, intelligent control, and decision support. Professor Campbell is the Immediate Past President of the Australasian Association for Engineering Education and was the IEEE Queensland Section Chapter Chair of the Control Systems/Robotics and Automation Society Joint Chapter (2008/2009).

Biography: Richard Taylor

Richard Taylor received the B.E. (with honours) and M.E. degrees in electrical and electronic engineering from the University of Canterbury, Christchurch, New Zealand, in 1977 and 1979, respectively, and the Ph.D. degree from the University of Queensland, Brisbane, Australia in 2007. He is the former Chief Technical Officer of Mesaplexx Pty Ltd and currently holds a fractional appointment as an Adjunct Professor in the School of Electrical Engineering and Computer Science at the Queensland University of Technology. He started his career as a telecommunications engineer in the power industry in New Zealand. In the past 25 years Dr Taylor has established two engineering businesses in Queensland, developing innovative telecommunications products.