future-ready communication architectures for integrated ... · future-ready communication...

Smart Connected PipelinesDidi Ismawan

Serhii Konovalov

FUTURE-READY COMMUNICATION ARCHITECTURES FOR INTEGRATED PIPELINE OPERATIONS

Lack of expertise in modern designs and cyber security by EPCs

Increasing Cyber Security Threats

Legacy Infrastructures / Asset Management

Product Theft / Third Party Interference

Minimize downtime in control center software upgrades

Pipeline Operations: Reality of Today

Converged architectures for today and tomorrow

Smart Connected Pipelines

Introduction

Validated Architectures

Benefits

Summary / Resources

Agenda

Converged ArchitecturesAddressing the pipeline requirements of today and tomorrow

Schneider Electric - Cisco: Industry

Partnership

Pipeline SCADA, Process &

Energy Automation

Pipeline Simulation, LDS &

Operational Applications

Domain Expertise

IP Networking, Wireless and Optical

Communications

Industrial Cyber Security, ISA SP99

High Availability Designs

Virtualization and Convergence

Architectures

Smart Upstream

Smart Pipelines

Smart Distribution

Forward-looking functional

architecture for end-to-end

pipeline infrastructure.

• A flexible, modular

approach

that supports a phased Oil

and Gas Pipeline

operational excellence

• End to End Integrated

Solution for Process ,

Safety , Power & Security

• Control Room

Virtualization

• Converged Wide Area

Operational Telecoms

• Pipeline Station Wired and

Wireless Networks

• Integrated Multi-Service

use cases

• IEC 62443 / ISA99 Security

model

SIEM

ProcessControl PowerManagement SafetySystems

Compressor/PumpSta on

Mul serviceDomain

Sta onWAN&Security

ProcessDomain

Metering/PIGSta on

Metering

PIGSystems

GasQuality

Mul serviceDomain

Sta onWAN&Security

ProcessDomain

SCADA&Opera onalBusinessSystemsSIEM

EngineerWorksta ons

Applica onServers

DomainController

Instrumenta on Instrumenta on Instrumenta on Instrumenta on

Quantum Quantum MiComc264

SIL3Controller SIL3Controller

GTW RI/O GTW RI/O

Historian OperatorSta on

Historian PACISOperator

Historian OperatorSta on

HMI

EthernetNetwork EthernetNetwork SafeEthernetNetwork

EthernetNetworkSafeEthernetNetwork

WirelessAP

MobileWorker

IPVoice

AccessControl

CCTV

RFID

EthernetNetwork

WirelessAP

MobileWorker

IPVoice

AccessControl

CCTV

RFID

EthernetNetwork

WirelessAP

Controller Controller Controller

EthernetNetwork

Historian Historian Historian

HMI HMI

Router Firewall Switch Router Firewall Switch

ConvergedOT&ITOpera onalFieldTelecoms

SCADAPrimary

RAS

LeakDetec on

PhysicalSecurity

OperatorWorksta ons

SCADABackup

TrainingServer

Historian

Repor ng

MeteringSystems

MainControlCenter

VideoOpera ons

AccessOpera ons

VideoStorage

IncidentResponse

IP/Ethernet

DWDM

IP/MPLS

(virtualized/non-virtualized)

(virtualized/non-virtualized)

BackupControlCenter

MCCW

AN&Security

BCCW

AN&Security

Mul serviceDomain

MobileWorker

IPVoice

AccessControl

CCTV

RFID

EthernetNetwork

WirelessAP

ProcessDomain

RouterFirewall

Switch

Sta onWAN&Security

BlockValveSta on

Quantum

Instrumenta on

CentralizedOpera ons Offic

e

/ BusinessDomain InternetEdge

Internet3rdPartySupport

Voice

Wireless

WLANController

CallManager

Voicemail

EngineerWorksta ons

Applica onServers

DomainController

SCADAPrimary

LeakDetec on

OperatorWorksta ons

SCADABackup

Historian

Repor ng

MeteringSystems

IncidentResponse

(virtualized/non-virtualized

)

(virtualized/non-virtualized

)

Wireless

WLANController

CallManager

Voicemail

SCADA&Opera onalBusinessSystems PhysicalSecurity Voice

Magelis

IONMetering

SEPAMProtec on

TeSysTMotorMgt

Al varDrive

MiCOMFeeder

Protec on

Magelis

VideoOpera ons

AccessOpera ons

VideoStorage

(Red

undant

Op

ons)

(Red

undant

Op

ons)

(Red

undant

Op

ons)

SIEM SIEM

SIEM SIEM SIEM

Switch

SIEM SIEM SIEM SIEM

SIEM SIEM SIEM SIEM SIEM

SIEM SIEM

SIEM SIEM

SIEM SIEM SIEM SIEM SIEM SIEM SIEM

SIEM SIEM SIEM SIEM SIEM SIEM

RI/O

ScadaPack

SIL3Op onNoSILOp on

Wirelessop on

3G/LTE,WiMax900MhzRFMesh

Satellite,Microwave

ROADM ROADM ROADM

CrewWelfare/Infotainment

SIEM

IDMZ

TImingServer

SIEM

AAA

TImingServerRAS

SIEM

SIEM

AAA

WANNetworks

IDMZ

Cisco Schneider Electric - Pipeline Functional Reference Model

Smart Connected Pipelines

Solution Overview

Pipeline Management Systems

Pipeline management systems consist of multiple services and systems applications to facilitate safe and efficient operations.This combines operational SCADA with real-time applications specific to the oil and gas industry, leak detection and historical flow measurement.

Operations

Control

GMASEnterprise

Asset Operation

ProcessExpert

Resource Advisor

SCADA Expert / Energy

VideoExpert

MeasurementAdvisor

Seconds

Minutes to Hour

Day/ Month/ Year

PipelineOperation Trainer

Offline

Engineering/Designing

PlanningAdvisor

IntegrityOperation

PipelineOperation

SCADAExpert

Note: To explore Invensys offers for Pipeline Designer Tool and to support Scheduling//Planning Advisor

Avantis(Cond Monitoring & Pro)

PipelineDesigner

PACIS

WW Sys Platf/ EVO

Endura

OASyS

Real Time Platform /

Pooling Engine

Operations

Asset

Energy

Business WFAutomation

SKELTA

Manual /Mobile Data Capture

InteltracGas and LiquidsApplications

Leak Detection& Simulation

Measurement & Validation

Web BasedDashboards

Scheduling, Logistics & Revenue Accounting

ePLMS - Software Suite & Supporting Applications

Applications can be categorized as

Operational (those directly involved with supporting pipeline

operations such as the SCADA or leak

detection systems), and Multiservice (those

which either support pipeline operations

such as video surveillance, or those more concerned with business applications

such as voice and corporate data)

Enterprise Pipeline Management System Requirements

Component Function

Supervision & control

Real-time control and supervision of operations along the pipeline through a SCADA system based in one or more control centers

Measurement Accurate measurement of flow, volume and levels to ensure correct product accounting

Leak detection Detect and locate pipeline leakage including time, volumes and location distances

Physical safety & security

Integrated security systems for personnel, the environment, and infrastructure using video surveillance, access control, and intrusion detection systems

Process safety systems

Ensure safe operations through instrumentation and safety systems

Energy management

visualize, manage and optimize energy consumption

Block valve station Isolate a segment of the line for leaks or maintenance

Local Control,Safety System ; RTU, Security

SupervisoryControl

Historian

Equipment (MV,LV,MCC, MV Drives,Valves) & Process

Instrum & Meters

Simulation (Leak Detection)

Energy Management

Asset Management

Supply ChainMngt

(scheduling, nominations,

& revenue accounting)

Real Time Applications(Gas ; Liquids)

IED’s / Power

Data Management

Measurement

Network / Communications

Co

ntr

ol

Op

era

tio

ns

Ente

rpri

se

Operational Telecoms & Communications RequirementsRequirement Description

High availability & reliability

• Backup WAN services to ensure operational services continuation• Provide several primary and failover infrastructure connectivity options (Ethernet, MPLS, DWDM, OTN, Cellular, Wireless etc) depending

on project requirements

Transport multiple traffic types across common infrastructure

• Differentiated quality of service (QoS) between traffic types, ensuring performance requirements of all operational traffic, and multiservice traffic

• Segregation capabilities (physical or logical) between services, ensuring one traffic type does not impact another where designed

Open standards • Multi-level security to protect against cyber attacks and non-intentional security threats• Centralized configurable policy-based services

Multiservice applications Optional services to support pipeline operations including VoIP, local WiFi access, mobility, collaboration tools, and internet access

Management End to end communications network management, security management, and administration management - from the instrumentation/sensor to the control centre application

Ruggedization Locations may require ruggedized equipment due to harsh conditions, local legislation or industry certifications

Network Requirements:

• High availability and reliability mechanisms to ensure operations

• QoS for differentiated services and bandwidth reservation,

• Monitorable SLAs must be delivered to ensure the right applications operate in the right way at the right time.

• Higher bandwidth requirements from newer applications and multiservice applications

• Sub 50ms network re-convergence redundancy and reliability mechanisms

• Traffic engineering for path selection and path redundancy,

• Device redundancy.

Ethernet/IP Based Protocols Serial Based Protocols

Modbus/TCP Modbus

IEC 61850 GOOSE (Station), SV (Station), MMSIEC 60870-104 IEC 60870-101

DNP3/IP DNP3

EtherNet/IP (Industrial Protocol)

OPC

TCP/IP

FTP

Proprietary protocols Proprietary protocols

Pipeline Security

System Patches

NetworkSegmentation

Anti-virus

IncidentResponse

Proactive Monitoring

Security MonitoringIPS /

Signatures

ThreatDefense

DisasterRecovery

Backupand Restore

ContinuousImprovement

Organize Harden Detect RespondDefend

White & Blacklisting

Security Log Collection and Management

AnomalyDetection

MalwareDetection

Intrusion Detection

Security Policy

Virtualization

Encryption

KPI’s and Analytics

Location Awareness

Process Inventory

Assessments

Change Management

Education & Awareness

Dashboards & Reporting

PCN Access & Control

Physical Security

Industrial Wireless

Portable MediaSecurity

PLAN BUILD RUN MONITOR MANAGE

Secure Storage

Asset Inventory & Management

• Architectural approach required (for example: Purdue Model of Control Hierarchy and ISA95/ISA99 security levels. • Security services should include common security policies, management and context, transport security, network policy

enforcement, research intelligence and threat defense, and physical security leading to the key network architecture principles to deliver:

- Access Control : Authentication and authorization of all personnel and devices - Data Confidentiality and Privacy : Data privacy and data integrity for all operational and control data must be ensured - Threat Detection and Mitigation : Protection of all critical assets - Integrity of Platforms and Devices : Secure devices over the entire life-cycle

Pipeline Safety System Architecture

HMI

XPMF Distributed Safety Hot Standby SIL3

Hot Standby

Unity XLS

Ethernet Modbus / TCP

Ethernet Modbus / TCP

Safe Ethernet

(CAT4 equivalent to SIL3) (SIL3)

Machine SafetyProcess Safety

Smart Connected Pipelines -Validated Architectures

Schneider ElectricLab Documentation

Joint Validated

Architecture

Schneider Electric Validated SCADA System

Architecture

SCADA / Leak Detection

Deployment Servers

PRIMDCDES01/02 (Virtual)

Primary Log Server

PRIMDCLOG01 (Virtual)

3

CISCO NEXUS N5548UP

STAT

ID

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

CO

NS

OL

E

PO

WER

ALA

RM

BO

OT

ACTI

VE

VPN

HD

MG

MT

7

6

5

4

3

2

1

0

SPD LNK LNK SPD LNK SPD LNK SPD LNK SPD

100-240V~, 4.85A MAX, 50/60Hz

Firewall/Router

PRIMDCRTR01/02

Cisco ASA 5525

3

CISCO NEXUS N5548UP

STAT

ID

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

Cisco Nexus C5548

Production

Environment

Data Center

Primary

Training LAN (Virtual)

To/From

Backup Data Center

Main Control Center

File: 011-002XXX-11

Visio 2010

Scale

NTS

Dwg Size

B

Drawing Number Sheet

1 of 8

Rev

1

Company Name:

Project Name:

Project Number:

Title:

2014 Enterprise OASyS

System ArchitectureAny information contained in this

document which was not known

by the recipient and in which there

is a proprietary right, must not be

disclosed to any other, nor used

for manufacturing purposes,

without the permission of

Schneider Electric. Acceptance of

this document will be construed as

acceptance of the foregoing

condition.

TNA-ZH-002XXX-DWG-011

2014/01/02

2014/07/16

002XXX

Enterprise OASyS SCADA

System

Updated for Customer

Name

Approved by Date

BM 2014/07/16

Customer Name

SCADA LAN B (Physical)

SCADA LAN A (Physical)

Development

Environment

# Training

Environment A

Test

Environment

! ResetConsole

UCS B200 M3

Hot / Standby

Real-time Servers

RealTime Service

Remote Client Service

PRIMDCRT01/02 (Physical)

2 X E5-2670 v2 10C 2.5GHz

4 X 16GB RDIMM 1866 MT/s

2 X 300GB 15K RPM HD Raid 1

Windows Server 2012 R2 Std.

2 X Cisco UCS B200 M3

! ResetConsole

UCS B200 M3

! ResetConsole

UCS B200 M3

Hot / Standby

Remote Access Servers

Remote Client Service

PRIMDCRCS01/02 (Physical)

2 X E5-2670 v2 10C 2.5GHz





! ResetConsole

UCS B200 M3

! ResetConsole

UCS B200 M3

Hot / Standby

Leak Detection Servers

Leak Detection Service

PRIMDCLD01/02 (Physical)

2 X E5-2670 v2 10C 2.5GHz





! ResetConsole

UCS B200 M3

A B

FAS8020FAS8020

! ResetConsole

UCS B200 M3

! ResetConsole

UCS B200 M3

1/10 GIGABIT ETHERNET 1/2/4/8G FIBRE CHANNEL

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16N55-M16UP

1/10 GIGABIT ETHERNET 1/2/4/8G FIBRE CHANNEL

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16N55-M16UP

CO

NS

OL

E

PO

WER

ALA

RM

BO

OT

ACTI

VE

VPN

HD

MG

MT

7

6

5

4

3

2

1

0

SPD LNK LNK SPD LNK SPD LNK SPD LNK SPD

100-240V~, 4.85A MAX, 50/60Hz

A B

FAS8020FAS8020

Clustered SAN

PRIMDCSAN01

24 X 600GB 10K RPM HD

Raid 6

2 X NetApp 8020

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

Cisco UCS 6248

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

# Leak Detection

Development Server


PRIMDCENGLD01 (Virtual)


! ResetConsole

UCS B200 M3

Hot / Standby

Historical Servers

Historical Service

PRIMDCHS01/02 (Physical)

2 X E5-2637 v2 4C 3.5GHz




MSSQL Server 2012 Std. 4C


NetApp Storage

! ResetConsole

UCS B200 M3

! ResetConsole

UCS B200 M3

Primary / Alternate

Domain Controllers

PRIMDCDC01/02 (Physical)

2 X E5-2637 v2 4C 3.5GHz





! ResetConsole

UCS B200 M3

# Training

Environment B

Training SCADA Server

RealTime/Historical Services

PRIMDCTRNARTH01 (Virtual)



Training Simulation Server

PRIMDCTRNALD01 (Virtual)

Simulation Software

Training SCADA Server


PRIMDCTRNBRTH01 (Virtual)



Training Simulation Server

PRIMDCTRNBLD01 (Virtual)

Simulation Software

Field LAN A (Virtual)

Field LAN B (Virtual)

Field Equipment

Primary

Communication

Path

Secondary

Communication

Path

* Communication

Environment

To/From

Decision

Support

Environment

(No Active Directory Trust)

Engineering Server

PRIMDCENG01 (Virtual)

RTU Simulator



Display / Database Server

XE / ADE

PRIMDCENGDES01/02 (Virtual)


# Leak Detection

Development SCADA Server


PRIMDCENGRTH01 (Virtual)



# Test Leak Detection Server


PRIMDCTSTLD01 (Virtual)


Test SCADA Servers

Domain Controller

PRIMDCTSTDC01 (Virtual 1)

4GB / 2 Core

RealTime Services

PRIMDCTSTRT01 (Virtual 2)

32GB / 10 Core

Remote Client Services

PRIMDCTSTRCS01 (Virtual 3)

32GB / 4C

Historical Services

PRIMDCTSTHS01 (Virtual 4)

64GB / 4C

1 X Cisco UCS B230 M1/M2


2 X E5-2670 v2 10C 2.5GHz

4 X 300GB 15K RPM HD 2 X Raid 1



NetApp Storage

! ResetConsole

UCS B200 M3

Virtual Host Servers

PRIMDCVM01/02/03

2 X E5-2690 v2 10C 2.5GHz



Test & Development LAN A

Test & Development LAN B

City, State

! ResetConsole

UCS B200 M3

Primary Log Server

PRIMDCLOG01 (Virtual)

2 X Cisco UCS B230 M1/M2


1 X E5-2670 v2 10C 2.5GHz


4 X 1.2TB 10K RPM Raid 10


Full in-site and inter-site

redundant Pipeline

Management System

Architecture

• Production, Test, Development

and Training Domains

• Real-time SCADA

• Long Term Historian

• Full suite of pipeline

applications

• Casual user access

• Secure Domain base security

Control Centre

Converged Pipeline Control Centre Architecture

Space, power, and

performance optimized

control center infrastructure

for Pipeline Management

Systems and Multiservice

applications

• Main & Back-up site ensure

reliability & update

• Unified Compute System

• Nexus Data Centre

switching fabric

• Fibre channel storage

• Operational WAN

aggregation

• Enterprise WAN access

• Secure remote access


model

Nexus 5500 Layer 2/3 Ethernet

And SAN Fabric

Nexus 2200 Series

Fabric Extenders

Cisco UCS

C-Series Servers

Cisco UCS Blade Server,

Chassis and Fabric Interconnects

Fiber Channel

Storage Array

WANDMZ

WANEnterprise

ASA5525 DC Core ASR 903

Fiber Channel

Twinax

Ethernet

Cisco Pipeline Station Architecture Overview

Station wired architecture –

greenfield considerations

• DWDM, L2 ethernet, Wireless

failover

• Station edge operational

field telecoms

• Level 2.5 protection zone

• Process, safety & energy

domains

• Multiservice domain

• Industrial and wireless

mobility

• Legacy serial transport

• Distributed acoustic optical

sensing (DAS)


model

Main Station

ASR 903

(Dual RP)

Industrial Ethernet Switches

Tree or Ring Topology

Serial

Controller

Ethernet

Controller

Ethernet

Controller

PhoneAccess

ControlCCTV

Mobile

Worker

Asset

Tracking

Stateful Failover

Pair

819H

Industrial

Switches

ONS

Industrial

Switch

Industrial

Wireless AP

Sta

tio

n E

dg

e

Pro

ce

ss D

om

ain

Mu

ltis

erv

ice

Do

ma

in

L2.5

Industrial

Wireless AP

Instruments

DWDM Main station to Main station

Industrial

SwitchL2 Ethernet Main station to Block Valve

DAS

Distributed

Optical Sensing

MPLS to Control Centres

HMI

Cisco Pipeline Block Valve Station Architecture

Block Valve station wired

architecture – greenfield

considerations

• L2 Ethernet, Wireless failover

• Station edge access

• Process, safety & energy

domains

• Multiservice domain

• Industrial and wireless

mobility


model

Block Valve Station

Serial

Controller

Ethernet

Controller

Ethernet

ControllerPhone

Access

ControlCCTV

Mobile

Worker

819H

Industrial

Switch

Sta

tio

n E

dg

eP

roc

ess

Do

ma

in

Mu

ltis

erv

ice

Do

ma

in

Industrial

Wireless AP

Instruments

Industrial

Switch

L2 Ethernet Block Valve to Block Valve or Main station

L2.5

Smart Connected PipelineTangible benefits

Customer pain points•Safe, Reliable and Efficient

Operations•Secure Communications •Mitigate Risks (leaks, accidents,

hazards)

Customer benefits•Enhanced performance and safety•Reduce Operational Risk & rapid

response to events •Increased Availability & Uptime

•Predictable performance, Standarization, Network Mngt & Diagnostics

Creating Greater Value for our

Customers

Interconnecting Gas Pipeline into Gas Storage

Customer Challenge:

● Tight Project Schedule

● Strict environmental conditions

● Harsh climate environment (weather)

Solution Value Proposition:

● Complete End to End Pipeline Management Solution delivered through MAC-MEC approach

Scope:

● Electrical: UPS, Electrical Systems, Shelters

● Control: SCADA, Bup Control Center, Simulation, PLC’s (redundant), ESD

● Security: Video System, Intrusion Detection

● Telco: Fiber Optic Ring & Switches. VOIP

● Instrumentation: P,T,L, Gas Quality, Motorized Valves (w profibus)

Benefits:

● Single Source Solution Provider

● Reduce Project Risk via integrated offer

BVS1 BVS2 BVS3

Bunde Etzel

FRI

Crystal

EKB

BEP Pipeline

Block Valves

GQS

Pipeline Management Solution

http://www.flags.net/GERM.htm

http://www.flags.net/GERM.htm

http://www.flags.net/NETH.htm

http://www.flags.net/NETH.htm

Creating Greater Value for our Customers

Export Pipelines & Gathering System

Customer Challenge:

● Safe Operation (including Leak Detection)

● Major Upgrade of Trunk Line w/out Interrupting Operation

Solution Value Proposition:

● Complete Solution comprised of Modern Pipeline SCADA, Leak Detection & Telecommunication System

Scope:

● Network & Telcom: Fiber Optic Ring & Switches. VOIP

● Control Center: SCADA ; Leak Detection

● Leak Detection

Benefits:

• Complete Modern Solution from Control Room , Operational Software & Communications Infrastructure

• Upgrade w/out disruption to operation

Smart Connected Pipeline

● CPC (Caspian Pipeline Consortium) operates a 1,600 km crude oil

pipeline originating from the Tengiz Fields in Kazakhstan transporting oil to a marine terminal in Novorossiysk on the Black Sea. The original project was delivered between1999-2001 and subsequently maintained, extended and expanded several times over the next ten years.

● In 2011, a contract was awarded to completely upgrade and

expand the SCADA and Pipeline Simulation System, as part of a major expansion CPC capacity expansion project.

● In parallel, Schneider Electric is executing a project to completely upgrade the entire telecommunication network for the CPC pipeline.

Summary

• Predictable Performance

• Differentiated Services

• Standardization

• Network Management and Diagnostics

• Network Cyber Security

• Reliability

• Real-Time Traffic Performance

• Power Availability and Equipment Footprint

Schneider Electric Approach to Realize an

enterprise PipeLine Management Solution

Joint Innovations:

Distributed Leak Detection enabled by Cisco Fog Computing Platform

Available Today:

White Paper - Recommended Practices

for O&G Pipeline Designs

White Paper - Recommended Practices for Pipeline

Control Centre Virtualization and Designs

Schneider & Cisco: Smart Connected

PipelinesAvailable Today

THANK YOU

future-ready communication architectures for integrated ... · future-ready communication...

Documents