61850 communication networks

Communication networks for IEC

61850 systems

Dr. Alexander Apostolov

Questions

• What are we doing?

• Why are we doing it?

• How are we doing it?

Technical Report IEC 61850-90-4

• IEC 61850-90-4 Technical report:

Network Engineering Guidelines

• Provides definitions, guidelines, and

specifications for the network

architecture of IEC 61850 based

systems

• Intended for an audience familiar with

network communication and/or

IEC 61850 based systems .


• Not a tutorial on networking or

IEC 61850

• It references and summarizes standards

and papers from other working groups

that may assist the engineers.

• Still provides significant amount of

useful information for people with basic

understanding of communications

technology


• Focuses on engineering a LAN/WAN

that is limited to the requirements of an

IEC 61850 based automation system.

• It outlines the advantages and

disadvantages of different approaches

to network topology, redundancy, clock

synchronization, etc. so that the network

designer can make educated decisions.

• Outlines possible improvements to both

IEDs and networking equipment.


• Addresses the most critical aspects of

IEC 61850 such as protection related

tripping via GOOSE and SVs.

• Addresses in particular the multicast

data transfer of large volumes of

sampled values (SV) from merging units

(MUs).

• Considers the high precision clock

synchronization and “seamless”

guaranteed transport of data across the

network under failure conditions that is

central to the Process Bus concept.


• Addresses the most critical aspects of

IEC 61850 such as protection related

tripping via GOOSE and SVs.

• Addresses in particular the multicast

data transfer of large volumes of

sampled values (SV) from merging units

(MUs).


• Considers the high precision clock

synchronization and “seamless”

guaranteed transport of data across the

network under failure conditions that is

central to the Process Bus concept.

• This report intentionally omits the

subject of network security, since it is

within the scope of IEC TC57 WG15.

Ethernet Technology for Substations

• Ethernet subset for substation

automation

• Topology

• Physical layer

• Link layer

• Network layer

Network Design Checklist

• Substation topology and physical

locations of IEDs

• Protection and control application

• Resiliency and redundancy

• Reliability, availability, maintainability

• Logical data flows and traffic patterns

• Performance

• Latency for different types of traffic

• Time synchronization and accuracy

• Network management

Network Design Checklist

• Environmental issues

• EMI immunity

• Form factor

• Physical media

• Cyber security

• Scalability, upgradeability and future-

proof

• Testing the design

• Cost

Network Topologies

• Single bridge

• Star

• Simple ring

• Multiple ring

• Process Bus

• Station and Process Bus connection

• Duplicate

Other Topics

• Dependability issues – requirements for

availability and reliability,

maintainability, dependability, risk

analysis

• Network configuration - assignment of

IP addresses

• Performance issues

• Quality of service

Other Topics

• Latency requirements

• Traffic control

• Clock synchronization

• Network security

• Network management

• Network testing

Network Symbols

IEC 61850 Engineering Process

Allocate Logical Nodes (Protection & Control Functions) to the Single Line Diagram

Application

requirements &

design criteria

Select and assign IEDs to SLD covering all Logical Nodes,

perform proof of concept, interoperability testing before engineering

Network design including switch

configuration (RSTP, PRP, HSR,..)

Select data communications schemes and protocols (Client-Server services to

SCADA, GOOSE & SV messages, network-based time synchronization) and

other Ethernet-based applications such as video surveillance (SNMP, etc.)

Final communication network design

Select the network topology and redundancy technique used for SA

Perform convergence tests

Are reliability requirement

met ?

Review design network architecture

Yes

No

Decide on primary technology (AIS, GIS, Hybrid GIS, etc.).

Decide to use electronic transformers, intelligent switchgear and

Process Bus

Select physical locations of IEDs inside main building, in close proximity to

primary equipment e.g. mounted outdoor or in kiosk or containers.

Select switches, perform proof of concept, interoperability testing before

engineering, more than one vendor switches are used. Conduct network contingency

analysis to ensure the network topology will meet the SAS reliability requirement

Partially Redundant Network

switched local

area network

(LAN)

node

node node

node

edge link

trunk link

switching nodes

node

nodetrunk link

trunk link

node

edge linkedge port

trunk port

switch

edge port

802.1Q tag

Destination(6) Source(6)

XXXX

ET (2)

tag (4)Destination (6) Source ET (2) LPDU 46..1500 octest

0x8100 0xXXXX

VLAN Tag (2)

12-bit 802.1Q VLAN Identifier

Canonical - 1 bit

3-bit Priority Field (802.1p)

LPDU 46..1500 octets

Ethernet layer MAC header (layer 2) without 802.1Q tag

Ethernet MAC header (layer 2) with 802.1Q tag

FCS

FCS

Ethertype

Network Traffic

Supervisory

Level

clock

SCADAHMI

gateway

firewall

Grid Control

bay

Station Bus

SCADA

IED

IED

IED

Process

Level

8-1 MMS

client-server traffic

SV

9-2 Sampled Valued

hard real-time traffic

pro

cess b

us

SCADAHMI

Engineering

8-1 GOOSE

soft real-time traffic

bay

IEDIED

IEDIED

IEDIED

bay

IEDIED

IEDIED

IEDIED

bay

IEDIED

IEDIED

IEDIED

bay

IEDIED

IEDIED

IEDIED

bay

IEDIED

IEDIED

IEDIED

horizontal communication

Vert

ical com

munic

ation

Electrical and Bay Topology

bay 1 bay 1 bay i bay j bay k bay w

engineering

gateway

networkcontrol centre

loggerprinter

station bus

GPStime

operator workplace(SCADA)

bay 3 bay N

IED

switch

IED

IED

IED

switch

IED

IED

IED

switch

IED

IED

IED

switch

IED

IED

IED

switch

IED

IED

IED

switch

IED

IED

G

switch switch

switch

RSTP

• IEC 62439-1

• RSTP is a widely used redundancy

protocol standardized as

IEC/ISO 8802.1D.

• RSTP is flexible and can convert an

arbitrarily meshed network topology

to a logical tree, by eliminating loops

that redundant links would introduce

on the physical level.

RSTP

• RSTP operates by peer-to-peer

messages, so-called BPDUs (Bridge

Protocol Data Units), between bridges.

• In normal operation, each node

indicates to its neighbors the costs to

reach the root node, which is at the top

of the hierarchy (independent of its

physical location).

RSTP

• When a bridge can reach the root

through multiple ports, it considers only

the port with the cheapest costs and

blocks the other ports.

• If path costs are equal, the port number

is used to break the tie.

• The path costs considers the number of

intermediate bridges, communication

speed of the link between the bridges

and the identity of the bridges.

RSTP Principle

inter-switch

link

edge portleaf

link

edge

links

alternate port

{blocked}

end

node

root bridge

end

node

end

node

end

node

end

node

end

node

end

node

end

node

end

node

end

node

end

nodeend

node

end

node

end

node

end

node

end

nodeend

node

end

node

end

node

end

nodeend

node

end

node

PORT STATE

learning

blocking

forwarding

PORT ROLE

Root (goes to root bridge)

Designated (goes away from root)

{ BackUp / Alternate}

Edge

root port

{forwarding}

designated port

{forwarding}

edge

ports

alternate port

{blocked}

inter-switch

link

edge portleaf

link

edge

links

alternate port

{blocked}

end

node

root bridge

end

node

end

node

end

node

end

node

end

node

end

node

end

node

end

node

end

node

end

nodeend

node

end

node

end

node

end

node

end

nodeend

node

end

node

end

node

end

nodeend

node

end

node

PORT STATE

learning

inter-switch

link

edge portleaf

link

edge

links

alternate port

{blocked}

end

node

root bridge

end

node

end

node

end

node

end

node

end

node

end

node

end

node

end

node

end

node

end

nodeend

node

end

node

end

node

end

node

end

nodeend

node

end

node

end

node

end

nodeend

node

end

node

PORT STATE

learning

blocking

forwarding

PORT ROLE

Root (goes to root bridge)

Designated (goes away from root)

{ BackUp / Alternate}

Edge

root port

{forwarding}

designated port

{forwarding}

edge

ports

alternate port

{blocked}

(Rapid Spanning Tree Protocol)

RPR

• PRP is specified in IEC 62439–3,

Clause 4 as a protocol that offers

seamless failover.

• RSTP achieves redundancy through

redundant network paths and a failover

protocol.

• PRP relies on complete duplication of

the LAN.

RPR

• Both LANs operate in parallel with a

source node duplicating a frames to

send and the destination nodes

discarding the duplicates on the base of

their source and of a sequence number

appended to the frame’s payload.

• To achieve this, a PRP node is a doubly

attached node (DANP) with two ports,

one for each redundant LAN.

RPR Principle

switched local

area network

(tree) LAN_B

RB

switched local

area network

(ring) LAN_A

DANP

DANP DANP DANP

SAN

A2 SAN

B1

SAN

B2

SAN

A1

SAN

R1

SAN

R2

DANP

HSR

• HSR is specified in IEC 62439-3 Clause

5 and provides seamless failover.

• HSR uses the principle of frame

duplication of PRP, but achieves

redundancy through only a single

additional link.

• Nodes in HSR have (at least) two ports,

the nodes are daisy-chained, with each

one node connected to two neighbor

nodes

HSR

• The last node being connected to the

first node and closing the line to a

physical ring structure

• HSR uses nodes similar to PRP nodes

with two network interfaces (DANH).

• A node must be able to forward frames

from port to port at wire speed, which

requires a cut-through bridge in each

node and therefore a hardware

implementation.

HSR

destinations

node node nodenodenode

nodenode

source

„A“-frame

(HSR tagged)

„B“-frame

(HSR tagged)

„C“-frame „D“-frame

AB

node

destinations

Single Switch Topology

NC

P

C

MU

P

C

MU

P

C

MU

P

C

MU

bay bay bay bay

Star Topology

NC

P

C

MU

P

C

MU

P

C

MU

P

C

MU

bay bay bay bay

Redundant Star Topology

PA PBC

AB

NC

Station Bus (2 x RSTP)

PA PBC PA PBC

LAN A

LAN B

Simple Ring

P

C

MU

NC

bay

P

C

MU

bay

P

C

MU

bay

Ring of Bridging IEDs

PCM PCM PCM PCM PCM PCM PCM PCM

NTP

voltage level 1 voltage level 2

Redbox HRS-RSTP

DANH - IEDs

Separate Switches for Main 1 and 2

P1

C

MU

bay

P2

MU

P1

C

MU

bay

P2

MU

P1

C

MU

NC

bay

P2

MU

Multiple Station Bus Ring

PCM

PCM

PCM

PCM

PCM

PCM

PCM

PCM

PCM

PCM

PCM

PCM

NC

Ring-Ring Topology with RSTP

PCM

PCM

PCM

PCM

PCM

PCM

PCM

PCM

PCM

PCM

PCM

PCM

PCM

PCM

PCM

PCM

PCM

PCM

PCM

PCM

PCM

PCM

PCM

PCM

NC

Rings and Sub-rings

P

C

MU

P

C

MU

P

C

MU

Primary Ring

Secondary Ring (voltage level 1) Secondary Ring (voltage level 2)

bay

P

C

MU

P

C

MU

P

C

MU

BP IEDIEDs

NC

bay bay bay bay bay

bussbar

protection

Rings of Rings with HSR

remove these switches

P

C

MU

P

MU

P

C

Primary Ring

Secondary Ring (voltage level 1) Secondary Ring (voltage level 2)

P

C

MU

P

MU

P

C

BP IEDIEDs

NC

bay bay bay bay bay

bussbar

protection

bay

Hierarchical Redundant Ring

Topology with HSR/PRP

P1 C MU1 MU2

RB A RB B

P2

Process Bus

AB

P1 C P2 MU

RB A RB B

Process Bus

P1 C P2 MU

RB A RB B

NC

Station Bus with PRP (2 x RSTP)

Process Bus

Process Bus Point-to-Point

IA1

IA2

UAL

IAL

UAS

IC2

IC1

UCL

ICL

UCS

IB1

IB2

IED

PMC1

U/I sensors

I sensors

switch control

actor

I sensors

U/I sensors

switch control

I sensors

I sensors

station bus with

8-1 and 9-2 traffic

PIPI

PIPI

PIPI

PIPI

PIPI

PIPI

PIPI

PIPI

PIPI

PIPI

PIPI

PMC2

Process Bus Ring

IA1

IA2

UAL

IAL

UAS

IC2

IC1

UCL

ICL

UCS

IB1

IB2

U/I sensors

I sensors

switch control

actor

I sensors

U/I sensors

switch control

I sensors

I sensors

8-1 traffic

(HSR)

9-2 traffic

PMCB

PI

PI

PI

PI

PI

PI

PI

PI

PI

PI

PI

PMCA

RBdiagnostics

gateway

Station and Process Bus Rings

HSR

RedBox A

A B

A

A B

B

PA C PB PM

Process Bus

A B

PA C PB PM

A B

PA C PB MUMU

B

NTP

PRP

Station Bus

HSR

RedBox B

Process Bus Process Bus

Separate Station and Process Bus

Station Bus

Process Bus

MU MU MU

NTP

P C P C

MU MU MU MU MU

BC

Separate Station and Process Bus

P C

Station Bus

Process Bus

MU MU MU

NTP

MU MU

P C P

Multicast Domain

Station Bus (e.g. RSTP ring)

PI = Process Interface

MU = Merging Unit

PIP

PIPI

PIPI

PIMU

PIMU

PA

PI

PI

PI

MU

MU

HSR process bus

PBPA

PI

PI

PI

MU

MU

HSR process bus

PBPA

PI

PI

PI

MU

MU

HSR process bus

PB

PIP

PIPI

PIPI

PIMU

PIMU

NTP

multicast domains

that allow to limit

traffic on the trunk

Hierarchy of Clocks

Test Setup

Annexes

• Annex A: IEC 61850 bridge object

model

• Annex B: IEEE 1588 Clock model

• Annex C: Case study - Process Bus

configuration for busbar protection

system

• Annex D: Case study - An IEC 61850

Station Bus (Powerlink, Australia)

Annexes

• Annex E: Case study - Simple

Topologies (Transener/Transba,

Argentina)

• Annex F: Case study - Station Bus

configuration in a sophisticated

application with VLANs (Trans-Africa,

South Africa)

Annex F: Trans-Africa, South Africa

61850 communication networks

Documents