network emulations with onos controller for communication ... · faculty of engineering,...

Faculty of Engineering, Chulalongkorn Universitywww.eng.chula.ac.th

20/2/2019

1

Network Emulations with ONOS controller for

communication network designing of IEC61850

digital substations interconnection

Mr. Saran JitnukulsiriChulalongkorn University

(IoTcloudServe@TEIN)


2[1] www.egat.co.th

Power Plant

Communication

Division

(WAN)Control Center




Enterprise task

IT system IT system

Substation

IT system

IT Division

IT system

IT Division

Operation task

Power system

OperatorOperation task

OperatorOperator

Power system Power system

Comm. systemComm. system

NOC (Comm.)Substation & Power Plant

Link Rental

Substation

Link Rental

Comm. system Comm. system

Operational Service

Enterprise Service

Enterprise Service

Telecom Network Management

20/2/201920/2/2019


3




[2] http://www.gridunlocked.com/2014/07/06/the-changing-architecture-of-electricity-developing-smart-grids/

Smart grid

Electronic & embedded

systems

Information &

communica-tion

System control&

automation

• Reduce green house gas emission

• Improve grid security and resilience

• Optimize utility asset

• Improve the energy network reliability

[2]

IEC 61850

20/2/201920/2/2019


20/2/2019

4




[3] http://seclab.illinois.edu/wp-content/uploads/2011/03/iec61850-intro.pdf

IEC 61850- It is a part of the International Electrotechnical Commission’s (IEC) reference

architecture for electric power systems.[3]

- It is an international standard defining communication protocols for intelligentelectronic devices (IED) at electrical substation.[3]

- The goal of the standard is to define digital communication within a substation butexpand to wide area connection recently.

IEC 61850-90-1 — Use of IEC 61850 for the communication between substations [Published]IEC 61850-90-2 — Use of IEC 61850 for the communication between control centres and substations IEC 61850-90-3 — Using IEC 61850 for Condition Monitoring IEC 61850-90-4 — IEC 61850 - Network Engineering Guidelines IEC 61850-90-5 — Use of IEC 61850 to transmit synchrophasorinformation according to IEEE C37.118IEC 61850-90-12 – Communication networks and systems for power utility automation: Wide area network engineering guidelines

IEC 61850 Standard Structure [3]

20/2/201920/2/2019


20/2/2019

5




[4] IEC TR 61850-90-12 Communication networks and systems for power utility automation – Part 90-12: Wide area network engineering guidelines, Edition 1.0 2015-07

Scope of interest for digital substations interconnection- Substation Level bus functions are base on L3- Bay level bus Functions are base on L2

IF7: data exchange between substation (level) and a remote engineer’s workplaceIF10: vertical data exchange between the substation and remote control centre(s) includes remote monitoring and telecontrolIF13: Synchrophasors to PDC and control centre

Control & Protection


Substation Bus

Technical services

Transmission

center (TCC)

Transmission control

center (TCC)

Phasor data

(PDC)

Phasor data concentrator

(PDC)

Process Bus Process Bus (Sensor)

Process Bus Process Bus (Sensor)

Bay Bay

Substation A

IF7 IF10 IF13

IF11

IF2

IF1,IF6IF1,IF6


IF2: Protection-data exchange between bay level

and remote protection

horizontal time-constrained exchange involving high traffic analog data (e.g. line differential protection) and low traffic binary data (e.g. protection blocking)IF11: Control data exchange between substations

horizontal exchange involving mainly low-traffic binary data (e.g. for interlocking functions or other inter-substation automatics)

Substation B

IF7, IF10, IF13 are interfaces that base on L3 traffic

20/2/201920/2/201920/2/2019

IF2 and IF11 are time critical requirement traffic


20/2/2019

6




Ref. IEC 61850-90-1 — Use of IEC 61850 for the communication between substations

[5] Real Time Emulation of IEC61850 SV,GOOSE and MMS using NS-3, Journal of Engineering and Applied Sciences 13(3):634-638, 2018, ISSN: 1816-949X, Sung-Ho Hwang, Yong-Soon Im

Scope of interest for digital substations interconnection in EGAT- Message performance classesThere are two basic groups of performance classes1) Control and protection application (focus on transfer times) e.g.

- Type 1A “trip” : - TR1 ≤ 4 ms : “Trip” within one bay- TR3 ≤ 15 ms : “ Trip” to neighboring substation (analog)

- Type 1B “automation”2) Metering and power quality applications (focus on accuracy

requirement) e.g.- Additional requirements for time synchronization for line

differential protection- TT1 : acceptable change of transfer time < 0.2 ms- TT2 : acceptable change of transfer time < 10 ms

- Type 5 – File transfer functions , Type 6 – Time synchronization messages

Since these messages were developed from communication within substationby using GOOSE and SVM (Sampled Value messages) over “Ethernet”, thestandard recommend “Ethernet” for communication between substations also.

20/2/201920/2/201920/2/2019

Overview of functionality and profiles[5]

mms: Manufacturing Messaging Specification (ISO9506)


20/2/2019

7



digital substations interconnection in

[4] IEC TR 61850-90-12 Communication networks and systems for power utility

automation – Part 90-12: Wide area network engineering guidelines, Edition 1.0 2015-07

Scope of interest for digital substations interconnection in EGAT- Communication technologies (1)Technology SDH/SONET Carrier Ethernet IP IP/MPLS MPLS-TP

Layer 1 & 2 2 3 2.5 2.5

Topology Mesh, ring, linear, P2P

Mesh, ring (logical tree)

Mesh Mesh, rings Mesh, rings

Bandwidth Up to 40 Gbit/s Ethernet speed Depends on Layer 2

Depends on Layer 2

Depends on Layer 2

Recovery delay 50 ms RSTP: 50 msITU-T G.8032:< 50 msPRP & HSR:Zero

50 ms with FRR, otherwise no upper bound

50 ms (ring redundancy)

50 ms (ring redundancy)

Path congruency Same circuit back and forth

Yes (broadcast domain)

Path in both directions can vary

Can be enforced by engineer

Can be enforced by engineer

Routing Circuit-switched Broadcast with MAC address filtering and VLANs

Automatic OSPF, IS-IS

Automatic LSP Static

RSTP: Rapid Spanning Tree ProtocolPRP: Parallel Redundancy ProtocolHSR: High-availability Seamless Redundancy

FRR: Fast Re Route [MPLS]OSPF: Open Shortest Path FirstIS-IS: Intermediate System to Intermediate System

20/2/201920/2/201920/2/201920/2/2019


20/2/2019

8






Scope of interest for digital substations interconnection in EGAT- Communication technologies (2)Technology SDH/SONET Carrier Ethernet IP IP/MPLS MPLS-TP

Configuration Control plane Automatic (RSTP, IEEE 802.1D)

Automatic (OSPF, IS-IS, SNMP)

Automatic (LDP) Management Plane

Virtual network None VLAN VRF VPWS, VPLS, L3VPN VPWS, VPLS

Suitability for differential protection

Yes Only with precise time distribution

Only with precise time distribution



Suitable for Large networks Small networks Very large networks Large networks Medium networks

Observations Widely used in Utility networks

Widely used in Metropolitan and access networks

Widely used in WANs

Widely used in carrier networks, increasing use in utilities

Possible migration path from SDH/SONET to PSN

VRF: Virtual Routing and ForwardingLDP: Label distribution ProtocolSNMP : Simple Network Management Protocol [RFC 3410]VPWS: Virtual Private Wire Service (pseudo-wire)VPLS: Virtual Private LAN Service [RFC 4761 and RFC 4762]L3VPN: Layer 3 VPN

SDH: Synchronous Digital Hierarchy [ITU-T]PSN: Packet Switched NetworkMPLS: Multi-Protocol Label Switching [RFC 3031]MPLS-TP: MPLS with Traffic Profiling

20/2/201920/2/201920/2/201920/2/201920/2/2019


20/2/2019

9




Network Emulation- Main purpose of emulation is to calculate the latency and jitter of GOOSE/SV

messages over L2VPN (VPWS/VPLS) between substations.

Simulator/Calculator Model

End to End ServiceSimulation Sample traffic- Performance measurement will be capture by

Wireshark with messages simulation of GOOSE&SV by packet simulator such as Scapy

- OVS emulation by Mininet with python code - Flows/Interfaces/Links ruled by ONOS API

- MPLS should be conduct by ONOS

Emulation by Linc-OE extension which will create ROADM as optical transport devices or

Calculate by algorithms that include compensation from real network

Topology & Link length information will manually setup according to EGAT database

20/2/201920/2/201920/2/201920/2/201920/2/2019


20/2/2019

10




ONOS Controller- Preconfigured VM for tutorial

3 clusters for SDN controllers

Will be store in ubuntu OS as

$OC1 = 172.17.0.4

$OC2 = 172.17.0.2

$OC3 = 172.17.0.3

Network

Preconfigured for Example

Easy icon to start ONOS CLI and GUI

Clean installation

in case something wrong with ONOS

20/2/201920/2/201920/2/201920/2/201920/2/201920/2/2019


20/2/2019

11




Network Emulation- Create virtual device with Mininet/Linc-OE via python code

20/2/201920/2/201920/2/201920/2/201920/2/201920/2/2019

TCLink in mininet can be use to setup

link latency, bandwidth and BER of the link between OVS ports


20/2/2019

12




Ref. https://wiki.onosproject.org/display/ONOS/System+Components

ONOS system tiersProviders in the project are Mininet and Linc-oe

ONOS core setup Flows / Ports configuration from Apps or by ONOS Cli

command then push to Mininet

There are many application create base on ONOS such as “VPLS”, “Carrier

Ethernet Application” or “CORD” with capability of control real SDN devices.

Southbound protocols- OpenFlow - REST- NETCONF - SNMP- TL1(NETCONF/OpenFlow for Mininet)

Developer can create their own project that connect to ONOS via Rest API to push/get information from providers base on

java code or develop by other compiler such as YANG

ONOS Controller & Application

REST API(.JSON)

20/2/201920/2/201920/2/201920/2/201920/2/201920/2/201920/2/2019

Mininet(Devices/LinkInformation)

Linc-oe(Devices/Link

Information)

Emulated Devices


20/2/2019

13





ONOS Controller – Example Usage : VPLS(1)

Group h1 with h3 in the same vlan by

using VPLS CLI that add to ONOS by

VPLS app

(2) Ping from h1 to h3

h1

h3s1 s2

(3) VLAN tag (802.1Q) add at s1 and pop out at s2

20/2/201920/2/201920/2/201920/2/201920/2/201920/2/201920/2/2019


20/2/2019

14





ONOS Controller – Example Usage : VPLS

(1)

VLAN tag

(1)Group h2 with h4 within the same domain without

VLAN tag

(2) Ping from h2 to h4

h2

h4

s1 s2

(3) No VLAN tagged in packets between h2 and h4

20/2/201920/2/201920/2/201920/2/201920/2/201920/2/201920/2/2019


20/2/2019

15




Thank you

20/2/201920/2/201920/2/201920/2/201920/2/201920/2/201920/2/2019


20/2/2019

16

EGAT is Thailand’s leading state-owned power utility under the Ministry of Energy, responsible

for electric power generation and transmission for the whole country as well as bulk electric energy

sales. EGAT is the largest power producer in Thailand, owning and operating power plants of

different types and sizes which are located in 45 sites across the country with a total installed

capacity of 15,757.13 MW (as of March 2018) [1]

Transmission Line Length: 33,393.186 circuit-kilometers

Number of Substations 223

Employees 21,837




[1] www.egat.co.th

20/2/2019


20/2/2019

17



digital substations interconnection in Electricity

Generating Authority of Thailand



Scope of interest for digital substations interconnection in EGAT- definition according to IEC TR 61850-90-1 and IEC TR 61850-90-12

High Voltage up to Extra High Voltage Level(EGAT domain)

Base on Transmission level

(L) Operation backhaul Network(H) Backbone network(G) Inter-center network(F) Inter-substation network(E) Intra-substation Network (partial)(EGAT communication division zone)

defined by levels of hierarchy

by SGAM (Smart Grid Coordination

Group, Smart Grid Architecture

Model) in wide area network

20/2/201920/2/201920/2/2019


20/2/2019

18





Optical Network Emulation

- There are 2 pre-config example program that I use to for optical simulation- 1st is OpticalTest.py which provide devices creation and links by using Mininet for packet switch

and LINC-oe for optical switch- 2nd is OpticalUtils.py which use as translator between OpticalTest.py and ONOS system- Flows ,vlan configuration and MPLS configuration is not included

- TCLink in mininet will be setup to create as link latency between each OVS ports

Mininet(Devices/portsInformation)

Linc-oe(Devices/ports

Information)

ONOS

ONOS/network/configuration

ONOS/devices

Devices information

Topology information

Host informationController information

Emulated Devices

ONOS/hosts

ONOS Controller

20/2/201920/2/201920/2/201920/2/201920/2/201920/2/201920/2/2019


20/2/2019

19





Future works

- Develop own application in ONOS to create ethernet tunnel between substation in emulation testbed focus on MPLS-TP (will take time for java learning!).

- Or Design network-configuration via other tools to create .json for posting into ONOS by REST API.

- Implement link latency calculation algorithm for TCLINK configuration on OVS ports to emulated link delay ( this will include compensation from another traffic in the link also)

20/2/201920/2/201920/2/201920/2/201920/2/201920/2/201920/2/201920/2/2019

network emulations with onos controller for communication ... · faculty of engineering,...

Documents