anatomy of modern process control networking …anatomy of modern process control networking...

Anatomy of Modern Process Control Networking InfrastructureMichael Boland, Distinguished Systems Engineer

BRKIOT-2130

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 3BRKIOT-2130

“This session will provide an overview of the design requirements and technologies

involved in communications infrastructure for process control environments,

focusing on discrete and process manufacturing. The art of balancing business

and control systems requirements with technology capabilities and limitations will

provide the main theme throughout the discussion.

A special focus on mapping cyber security and converged infrastructure

management into the process control environment will be covered, along with a

summary of what is new in the latest joint Cisco/Rockwell Converged Plant-wide

Ethernet (CPwE) design guides. While design discussions will be generic in

nature, Cisco technologies, products and solutions will be the focus of the solution

design examples presented.”

Abstract

anatomynoun, plural anatomies.

“… the study of the structure and part of organisms …”

Source: Merriam Webster Dictionary

4

• DNA - Industry Models and Design Patterns

• Organisms - Organisational Structure

• Skeleton - Architecture

• Physiology - Design

• Systems - Timing, QoS, Virtualisation

• Immune System - Security

• Evolution - Future Industrial Networks

• Anatomy Guides - Design Guides

Agenda


Characteristics of Industrial Networks

The characteristics of industrial networks differ from commercial or residential networks in two ways:

1. They provide the additional real-time performance capabilities needed for a majority of manufacturing applications.

2. They meet the requirements for survival in various types of industrial environments, normally characterised by higher levels of electrical noise, shock, vibrations, ambient temperature, humidity, etc.

Source: ODVA https://www.odva.org/Portals/0/Library/Publications_Numbered/PUB00035R0_Infrastructure_Guide.pdf


Key Design Attributes, Metrics and Objectives

Attributes Metrics Objectives

Performant Latency

↑ OEE 1

• Quality

• Performance

• Availability

Available Resilient Reliable Jitter

Deterministic Predictable VisibleConvergence

Time

Secure Conformant Manageable MTTR

EnvironmentalTime

Synchronisation

1 http://www.oee.com/

DNAIndustry Models & Design Patterns

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public

Purdue Enterprise Reference Architecture/ISA-95Scheduling and Control Hierarchy Levels in Industrial Companies

9BRKIOT-2130

Business Planning and Logistics

Plant production scheduling,

Operational management, etc.

Level 4

Manufacturing Operations and Control

Dispatching production, detailed production

Scheduling, reliability assurance, etc.

Level 3

Area Control

Cell/Line supervision, operations and process

control functions

Levels 2, 1 & 0

Batch

Control

Continuous

ControlDiscrete

Control

Defines the interfaces between

Enterprise activities and Control

activities.

Provides standard models and

terminology for describing the

interfaces between the business

systems of an Enterprise and its

manufacturing-control systems.

ISA-88 Standard


ISA-95 Computer-Integrated Manufacturing LevelsLevel 4 - The business-related activities needed to manage a manufacturing organisation

that are executed by enterprise-level software and systems to include:

• Plant scheduling - material use, delivery, and shipping

• Determining inventory levels

• Delivery of materials to the right place on time for production

• Timeframe – Months, weeks, days, shifts

Level 3 - The activities of work flow to produce the end products that are executed by the MES and MES-related systems. Timeframe – shifts, hours, minutes, seconds.

Level 2 - The activities of monitoring and controlling the physical processes that are executed by the PLC, the HMI, and the Area and Unit Operations portion of the Supervisory Control and Data Acquisition (SCADA) system.

Level 1 - Activities involved in sensing and manipulating the physical processes executed by valves, sensors, motors, etc.

Level 0 - The actual physical processes

10BRKIOT-2130

Source: http://www.pharmpro.com/article/2012/07/manufacturing-execution-systems

Ma

nu

factu

rin

g


Level 1

Infrastructure Logical Mapping to ISA-95 CIM Levels

11

Source: http:// http://www.et.tu-dresden.de/ifa/uploads/media/PLT1_010-ISA95_05.pdf

Business Process Information Network

Operations Information Network

Automation Network

Discrete & Process Device Communication Networks

Level 4

Level 3

Level 2

ERP, APO,

Logistics Systems

MES, LIMS, WMS

CMM Systems

HMI, SCADA

Batch Systems

PLC, DCS,

Packaged Systems

I/O, Devices,

Sensors

BRKIOT-2130 11


IEC-62443 (formerly ISA-99)

Security

Not addressed in

IEC-62443

BRKIOT-2130 12


NIST Special Publication 800-82 revision 2

Guide to Industrial Control Systems (ICS) Security

• Overview of Industrial Control Systems

• ICS Risk Management and Assessment

• ICS Security Program Development and Deployment

• ICS Security Architecture

• Applying Security Controls to ICS

BRKIOT-2130 13


NERC CIP Cyber Security Standards and Requirements

BRKIOT-2130

CRITICAL CYBER

ASSETS

SECURITY

MANAGEMENT

CONTROLS

PERSONNEL

AND TRAINING

ELECTRONIC

SECURITY

PHYSICAL

SECURITY

SYSTEMS

SECURITY

MANAGEMENT

INCIDENT

REPORTING &

RESPONSE

PLANNING

RECOVERY

PLANS FOR CCA

CIP-002 CIP-003 CIP-004 CIP-005 CIP-006 CIP-007 CIP-008 CIP-009

1. PLAN

2. PHYSICAL

ACCESS

CONTROLS

3. MONITORING

PHYSICAL

ACCESS

4. LOGGING

PHYSICAL

ACCESS

5. ACCESS LOG

RETENTION

6. MAINTE-NANCE

& TESTING

1. TEST

PROCEDURES

2. PORTS & SERVICES

3. SECURITY PATCH

MANAGEMENT

4. MALICIOUS

SOFTWARE

PREVENTION

5. ACCOUNT

MANAGEMENT

6. SECURITY STATUS

MONITORING

7. DISPOSAL OR

REDEPLOY-MENT

8. CYBER

VULNERABILITY

ASSESSMENT

9. DOCUMEN-TATION

1. CYBER

SECURITY

INCIDENT

RESPONSE

PLAN

2. DOCUMEN-

TATION

1. RECOVERY

PLANS

2. EXERCISES

3. CHANGE

CONTROL

4. BACKUP &

RESTORE

5. TESTING

BACKUP MEDIA

1. CRITICAL

ASSETS

2. CRITICAL

CYBER

ASSETS

3. ANNUAL

REVIEW

4. ANNUAL

APPROVAL

1. ELECTRONIC

SECURITY

PERIMETER

2. ELECTRONIC

ACCESS

CONTROLS

3. MONITORING

ELECTRONIC

ACCESS

4. CYBER

VULNER-

ABILITY

ASSESSMENT

5. DOCUMEN-

TATION

1. AWARENESS

2. TRAINING

3. PERSONNEL

RISK

ASSESSMENT

4. ACCESS

1. CYBER SECURITY

POLICY

2. LEADERSHIP

3. EXCEPTIONS

4. INFORMATION

PROTECTION

5. ACCESS

CONTROL

6. CHANGE

CONTROL

14

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKIOT-2130

Industry Norms

• Oil & Gas

• Pharmaceutical

• Nuclear

• Military

Some Industry Baseline Design Patterns Mandate Separate IT and OT Networks

Perfectly Valid Design Pattern.

However …

Should NOT obfuscate proper design

and operations methodologies within

both network infrastructures!

15


Communication Network Model – ISO/IEC 10731

Open Systems Interconnection

Model 7 Layers

Session Inter-host Communication

Application Network Process to Application Layer 7

Presentation Data Representation and Encryption Layer 6

Transport End-to-End Connections & Reliability Layer 4

Network Path Determination Layer 3

Data-Link MAC and Logical Link Control Layer 2

Physical Media, Signal and Binary Transmission Layer 1

Layer 5

Missing Important

Cynical Layers!

Politics Distribution of Power and Resources Layer 8

Religion Cultural System of Behaviours & Practices Layer 9

Finance Science of Money Management Layer 10

BRKIOT-2130 16

OrganismsOrganisation Structure


Don’t Mention the War!

18BRKIOT-2130

OT comes from Mars IT comes from Venus

Images Source: http://www.nasa.gov/sites/default/files/images


• Digital Transformation → Operations and IT Collaboration / Integration

• Increased integration of manufacturing into the digital supply chain

• New Hardware and Software systems skills and technologies are required

• Insource and/or Outsource service functions

Organisational Changes

Operations IT

+

Lines of Business

BRKIOT-2130 19

SkeletonIndustrial NetworkArchitecture

20


Presentation Legend

21

Reference Material

Standalone Multilayer Switch

Virtual Switching System

Key Points

Layer 2 Link

Layer 3 Link

BRKIOT-2130

WiFi Access Point

WiFi Workgroup Bridge

WiFi Wireless LAN Controller

Next Generation FirewallRouter

Fabric Interconnect


Does Your Network Have Good Bones?

Core

Distribution

Access

Control

BRKIOT-2130 22



• Provides non-stop connectivity between distribution layers for large sites – all about scale

• Site-wide Redundancy

• Non-disrupting In-Service Upgrades

Core

Distribution

Access

Control

BRKIOT-2130

Spatial Diversity*

* The level of diversity should be driven by the

business risk assessment process

23



Core

Distribution

Access

Control

• Aggregates access layers and provides connectivity services

• Connectivity and Policy Services within the Access-Distribution network

• Distribution, Policy Control and Isolation/Demarcation points between Cell/Area Zones and rest of network

BRKIOT-2130 24



Core

Distribution

Access

Control

• Provides endpoints (PCs, controllers, I/O devices, drives, cameras, etc.) and users access to the network

• Security, QoS and policy trust enforcement

• Labels packets to enforce segmentation

• Network Address Translation

BRKIOT-2130 25



Core

Distribution

Access

Control

• Connects time-critical function components

• Time-critical function components segmented from non-time-critical components

• Rapid convergent ring topologies or parallel access network topologies

• Multicast-rich local traffic flows

BRKIOT-2130 26

PhysiologyIndustrial Network Design

27


Industrial Networks live in the Real World

• Must take into consideration the physical challenges of the facility environment

• Location, routing and equipment choices should be based on a complete understanding of cause and effect conditions

• And M.I.C.E.

BRKIOT-2130 28

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 29

Environmental Focus – M.I.C.E.

Office Industrial

Increased Environmental Severity

TIA/EIA 1005

Electro

magnetic

Climatic

Chemical

Ingress

• Water

• Dust

Mechanical

• Shock

• Vibration

E1

C1

I1

M1

E2

C2

I2

M2

E3

C3

I3

M3

BRKIOT-2130


MICE Table pg1


MICE Table pg2

BRKIOT-2130


Key Infrastructure Requirements

• Determinism – predictability of performanceDelay

Jitter

Packet Loss

Path Symmetry

• Availability and ResiliencyConvergence time

Mean Time To Repair

• Security

• Visibility and ManageabilityOperations

32BRKIOT-2130


Convergence Times

Requirement Class Target Cycle Time Target RPITarget Network

Convergence

Information/Process

(e.g. HMI)< 1 s 100 - 250 ms < 1 sec

Time critical processes

(e.g. I/O)30 - 50 ms 20 ms < 40 ms

Safety 10 - 30 ms 10 ms < 15 ms

Motion 500 µs - 5ms 50 µs - 1 ms < 1ms

33BRKIOT-2130

MMachine/process cycle times — The frequency with which the control application makes decisions

Request Packet Interval (RPI) or I/O update time — The frequency which input/outputs are sent/received

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 34

Plant Floor Control Networks

Control

• Rapid convergent ring topologies dominate e.g. DLR, MRP (IEC 62439-2)

< 50 nodes per ring

< 3ms ring convergence time*

• Single fault auto-recovery

• Dedicated rings for time-critical traffic

• Separate Ethernet links for non-time-critical traffic

• Star and Bus topologies also common

* Cisco IE switch MRP default max recovery time is 200msDrive

PLCsRemote

I/O

DeviceNet

ControlNet

Local I/O

Access

BRKIOT-2130

HMI


Plant Cell/Zone Control Networks

Drive

PLC

Remote

I/O

• Dedicated rings for safety and time-critical functions/components

Issue: extending control rings across plant

• Separate Ethernet interfaces for non-time-critical traffic

• Design constraints with PLCsNetwork sub-system performance

May not support IEEE 802.1Q (VLANs)

Typically do not support IEEE 802.1X

Switch

HMI

Remote

I/O

PLC

Access

Control

BRKIOT-2130


Plant Floor Control NetworksNetwork Redundancy (Power Networks Example)

Control

• Dual access LANs e.g. PRP - Parallel Redundancy Protocol (IEC 62439-3.4)

• Source dual-homed to two networks

• Source sends a frame simultaneously on both LANs

• Destination receives both frames and discards one (normal operation)

• 0ms recovery time from single point network faults

Red

Box

IED

IED

Network

A

Network

B

Access

BRKIOT-2130


Plant Floor Wireless Networks

Wireless

• Sensor networking

• Mobile worker

• Mobile fleet

ISA 100.11a

WiHART

6LowPAN

Access

RAP

MAP

5GHz/2.4GHz

WiFi

RAP

BRKIOT-2130 37


Network Access & Distribution

• Rapid convergent technologies < 250ms

• Primary with hot standby secondary trunk path (determinism)

• Topologies:

Dual Point-to-Point Ethernet links

Ethernet Ring

• Spanning Tree Protocol enabled on south-bound Ethernet ports for loop prevention*

Access

Control Native

VLAN

Native

VLAN

VLANVLAN

HMI

BRKIOT-2130

* N.B. Enable Layer 2 control plane security features: VTP authentication,

Root guard, Loop guard, BPDU guard, BPDU filter

38


Resiliency ProtocolRing Topology

(Switch or Device Level)

Redundant Star or Mesh

Topology

Typical Network Convergence

Time

Max Number of

Switch Nodes

Remark

Standardised

STP (802.1D) S X 30s 7 Limited network diameter

RSTP (802.1w) S X 2s 7 Superseded by 802.1D-2004

MRP (IEC 62439-2) D 10-500ms 50 Recovery increases with number of nodes

MSTP (802.1s) S X 250ms 255 Number of VLANs and node increases convergence time significantly

RSTP (802.1D-2004) S X 50-200ms 255 Recommend limit of 40 nodes. Needs optimising for rapid convergence

EtherChannel (LACP 802.3ad) X 100ms 2 Switch to switch redundancy only

G.8032v2 (ITU-T) S X 50ms 255 Recommend limit of 16 nodes

DLR (IEC & ODVA) D 3ms 50 Worst case 3ms for 50 nodes

HSR (IEC 62439-3.5 2012) D X 10ms per hop HSR is a device ring, requires FPGA

PRP-1 (IEC 62439-3.4 2012) D N/A 0ms N/A PRP requires duplicate L2 networks, no special hardware

Proprietary

S-Ring (GarettCom) S 200ms-700ms Unlimited No upper limit to number of nodes but recommend 50

HiperRing (Hirschmann) S 200-500ms Unlimited Recovery depends on number of nodes

TurboRing (Moxa) S 200-300ms Unlimited Recover depends on number of nodes

FlexLinks (Cisco) X 100ms 2 Switch to switch redundancy only

REP (Cisco) S 50-250ms Unlimited Recovery tested up to 130 nodes

eRSTP (RuggedCom) S X 5ms per hop 80 Recover depends on number of nodes

StackWise (Cisco) S X 5ms 9 Offers L2 and L3 redundancy

L2 Industrial Network Redundancy Protocols


Network Access/Distribution – Goal State Issues

L3

L2

L3

L2

Site-Wide

Fibre Cabling

Opt. Plant Ring

Distribution

Access

Control

OSPF,

EIGRP

REP

• Fibre minimisation

Point-to-Point vs Ring

• Legacy protocols

• Multicast implementation

• IP Multicast TTL = 0 in the Control Layer

• Inter-control zone/cell communications segmentation

• Cost of Layer 3 protocol support to the Access Layer

OSPF,

EIGRP

BRKIOT-2130 40


Network Access & Distribution

IEEE 802.3ad

or PAgP

REP

MC-LAG,

VSS

FlexLinks IEEE 802.3ad

or PAgP

Satellite

L3

L2

L3

L2Site-Wide

Ring Fibre

Cabling

Distribution

Access

Control

HSRP,

VRRP

OSPF,

EIGRP

VSS

Horizontal Switch Stacking may not be suitable as convergence times typically > 500ms

BRKIOT-2130 41


Network Core

M-LAG,

VSS

Satellite

L3

L2

L3

L2

L3L3

L2

Distribution

Access

Control

Core

MC-LAG,

VSS,

vPC

MC-LAG,

nV

Edge

OSPF,

EIGRP

OSPF, EIGRP,

IS-IS, BGP

Native Routing,

Multi-VRF (VRF-Lite, EVN)

VRFL3

VRF

MPLS, VPLS, PBB-EVPN

VXLAN EVPNMPLS @core and @distribution,

VPLS, PBB-EVPN, FabricPath

OSPF,

EIGRP

BRKIOT-2130 42


“Industrial” Data Centre

Distribution

Access

Control

Core

Data Centre

Collapsed

Core and

Distribution

Fabric

Interconnect

Hyperconverged

Compute and Storage• Conventional Computing

DAS

• Converged Infrastructure

NAS, SAN

• Hyperconverged Systems

Software Defined Storage

Tight integration of compute, storage, networking and virtualisation

BRKIOT-2130 43


Data Centre – Service Mapping to IEC-62443 Levels

Demilitarised Zone* — Shared

Access, “Jump Zone”

Enterprise Network Level 5

Site Business Planning and

Logistics Network Level 4

Site Manufacturing

Operations and ControlLevel 3

Area Control Level 2

Basic Control Level 1

Process Level 0

VM

CPU

Zone / CellIndustrial Data Centre

PLC

HMI

SCADA

SITE HISTORIAN

“Level 3.5”

* DMZ Level not formally part of IEC-62443

Patch Management, Terminal

Services, Application Mirrors, AV

Servers

44BRKIOT-2130


Data Centre – Service Mapping to IEC-62443 Levels

Level 2

CPU

Zone / Cell

Level 2

VM

IEC-62443 Level

HMI

PLC Level 3SITE HISTORIANLevel 3

Level 2

SCADA

SITE HISTORIAN

SCADA

SCADA

HISTORIAN

BRKIOT-2130

OPC DA

OPC DA

SQL

Historian implemented as “single point of production truth model”

“TAGs”

45

SystemsTiming, QoS, Virtualisation

Image: http://cliparts.co/human-body-outline-printable


Synchronisation: the Two Signals That Matter

Frequency Time

router# show clock

*13:38:54.805 UTC Mon Apr 4 2016

BRKIOT-2130 47


Timing Network OrgansAn analogy Clock Servo

Node architecture

Local Reference

Network nodesNetwork links

Primary Reference(s)

Master clock(s)

Network type

Standard specifications

Application and

environment

BRKIOT-2130

Image: http://cliparts.co/human-body-outline-printable

48


Where to Look

Power Profile

Defined by IEEE PSRC (C37.328)

Substation LAN Applications

Telecom Profile

Defined by ITU-T (G.8265.1, G.8265.2)

Telecom WAN Applications

Default Profile

Defined by Annex J of IEEE 1588 specification

LAN/Industrial Automation Applications

• Time and Frequency synchronisationprofiles for different industry use cases defined by industry standards bodies

• Industry design guides

• CiscoLive! (BRKSPG-2170) Synchronisation in Packet-Based Networks

BRKIOT-2130 49


Quality of Service – Output Queue Prioritisation

PTP-Event

Critical Data

Video

Call Signalling

Best Effort

Voice

Bulk Data

Network Control

Scavenger

Critical Data

Video

Call Signalling

Best Effort

Voice

Bulk Data

Network Control

Scavenger

CIP Explicit - Low

Messaging – Class 3

CIP Motion

PTP Management,

Safety I/O and I/O

Typical Enterprise QoS Cell/Area Zone QoSPriorityQueue 1

Output Queue 2

Output Queue 3

Output Queue 4

Output Queue 2

Priority Queue 1

Output Queue 3

Output Queue 4

BRKIOT-2130

DSCP

59

55

47

43

48

46

3127

24

50


Bandwidth Control

HMI

Drive

PLCs

Remote

I/O

Control

Access

Why control bandwidth?

• Prevent data traffic from the contractor occupying the network and affecting process control traffic e.g. delaying PTP messages, CIP, etc.

• Prevent a malicious user taking up the bandwidth and starve critical application traffic

How?

• Rate limiters can limit traffic per VLAN, port or user to mitigate the impact of packet-blasting worms and limit amount of traffic a user can send onto the network

• Can rate limit using either traffic policing (ingress) or shaping (egress) functions

• Rate limiting should be configured on the access switches

PTP & Control

Traffic Given

Highest

Priority

Contractor


Virtualisation

• Network Virtualisation - The capability to share a common infrastructure while supporting multiple virtual networks with isolated data and control planesproviding multi-tenancy and security

• Device Level Virtualisation

Virtual LANs (VLANs)

Virtual Routing and Forwarding (VRFs)

• Path Isolation

VRF-Lite End-to-End

MPLS VPN


Virtualisation Example

• Switched device virtualisation via VLANs

• Service network virtualisation via VRFs

Access

Control Native

VLAN

VL

AN

30

VL

AN

40

VL

AN

10

VL

AN

20IEEE

802.1Q

Trunks

Core or

Distribution

Network

Management

VRF

Production

VRF

Unified

Comms.

VRF

Physical

Safety & Sec.

VRF

BRKIOT-2130 53

Immune SystemSecurity


NIST 800-82 (Revision 2)

Enterprise Zone

DMZ

Manufacturing Zone

Cell/Area Zone

Demilitarised Zone — Shared Access, “Jump Zone”


Site Business Planning and Logistics Network Level 4

Site Manufacturing Operations and Control Level 3



Process Level 0

BRKIOT-2130 55


Industrial Network Models – IEC-62443-3-2







Process Level 0

Zo

ne

A

Zo

ne

B

Controlled Conduit

IEC-62443-3-2

BRKIOT-2130

Enterprise Zone

DMZ

Manufacturing Zone

Cell/Area Zone

56


Industrial Network Models – ISA-62443-3-2 + DMZ







Process Level 0

Zo

ne

A

Zone E

Con

tro

lled C

on

du

its

X

BRKIOT-2130

Enterprise Zone

DMZ

Manufacturing Zone

Cell/Area Zone

57


Demilitarised Zone

Industrial Network Models – DMZEnterprise Network Level 5





Process Level 0

Zone

Remote

CC

CC

HT

TP

S

Zone

Terminal

Services

Zone C

RD

P

HMI

Contractor

BRKIOT-2130

Enterprise Zone

DMZ

Manufacturing Zone

Cell/Area Zone

58


Visibility with NetFlow

Distribution

Access

Control

Core

Data Centre Network Flow Telemetry• Network Baseline

• Analytics

• Security

NetFlow provides• Visibility into conversations in your network

• Collect statistics on infrastructure, who’s

communicating, how long, protocols, etc.

• Network usage measurement

• An ability to find north-south as well as

east-west communication flows

• Light weight visibility compared to SPAN

based traffic analysis

Virtual Switch

DC Fabric

Interconnect

Core Switch

Distribution

Switch

Access Switch

Collection, Correlation, Analysis, Context e.g.

StealthWatch

Wireless LAN

Controller

VM

VM

BRKIOT-2130


Identification

Distribution

Access

Control

Core

Data Centre

IEEE 802.1x

MAB

Port Security

(Static MAC)

Policy

Controller,

e.g. ISE

IEEE 802.1x

VM

VM

Why Identity?

• Allow only known devices

• Limit number of devices that can be connected to the port

• Prevent rogue/unknown devices connecting to the network

How?

• Configure IEEE 802.1x authentication on access switch ports

• Use MAB for endpoints that do not have support for IEEE 802.1x supplicants

• Port security with static MAC address for all static end points


Segmentation

Distribution

Access

Control

Core

Data Centre

Virtual Switch

Access Switch

Policy Controller, e.g. ISE

Wireless LAN

Controller

VM

VM

Firewall

Routing

Network Segmentation

“The capability to segment a network in order to achieve data plane isolation over physical and virtual networks”

Segmentation

Mechanisms

Group Based Policy*

TrustSec SGT

Dynamic ACL

Statefull ACL

Static ACL

VRF

Routing

DHCP Scope

Address

VLAN

Physical

* E.g. End Point Groups within a data centre


Cloud ServicesAnalyticsService

Distribution

Access

Control

Core

Data Centre

VM

VM

WAN

Demilitarised Zone

Shared Access, “Jump Zone”

Enterprise

Network Level 5

Site Business Planning

and Logistics Network Level 4

Site Manufacturing

Operations and Control Level 3



Process Level 0

CPU

Telemetry Aggregation Server

AnalyticsService

VLA

N 1

4

Packaging

Zone P

Analytics Vendor X VRF

Telemetry Aggregation

Server

VLA

N 2

4

Level 3.5

BRKIOT-2130

VLA

N 2

4

VLA

N 5

4

Packaging VRF

Example of

Fog

Computing


More and More Porous Boundaries …

LTE

Backdoor

• Machine builders want access to machine telemetry

• Requesting direct machine-to-cloud connectivity

• In some cases machines come with embedded LTE networking which is outside operations control

• Presents a security “backdoor” problem


Security Functions in Action

Level 2

CPU

Zone / Cell

“M”

Level 2

VM

HMI

PLC Level 3SITE HISTORIANLevel 3

Level 2

SCADA

SITE HISTORIAN

SCADA

SCADA

HISTORIAN

BRKIOT-2130

VM

VM

MAB Authentication of PLC

“Zone M” VLAN for L2 broadcast containment

“Zone M PLCs” SGT for Policy-based

segmentation

SGFW

“Zone M

PLC” to

“Historian” ✔

ACL = OPC DA, Historian : PERMIT

SGFW “SCADA” to

“Historian” ✔

ACL = SQL, Historian : PERMIT

SGFW “Historian” to

“SCADA” ✔

ACL = SCADA, SQL : PERMIT

NetFlow for flow VisibilityNetFlow for flow VisibilityNetFlow for flow VisibilityNetFlow reporting for flow

Visibility

NetFlow reporting for flow

Visibility

NetFlow reporting for flow

VisibilityNetFlow for flow VisibilityNetFlow for flow VisibilityNetFlow for flow VisibilityNetFlow reporting for flow

Visibility

ACL HMI, SCADA OPC DA :

PERMIT

SGFW

“Zone M

HMI” to

“SCADA”

MACsec MACsec MACsecMACsec

MACsec MACsec MACsecMACsec

MACsec

MACsec

MAB Authentication of HMI

“Zone M” VLAN for L2 broadcast containment

“Zone M HMIs” SGT for Policy-based

segmentation

Network Visibility and Enforcement,

e.g. StealthWatchPolicy Controller, e.g. ISE

64

EvolutionFuture Industrial Networks


Industrial Network Future → Fabric

BRKIOT-2130

Fabric

Users or Devices

Secure Industrial Fabric

Controller Management

• Policies based on User, Device or Application Group

• Traffic Visibility and Fabric Orchestration

• Single User Interface for Network Management

Programmable Overlay

• Dynamic Path Setup and Client Mobility

• Network Segmentation via Virtual Networks (VNs)

• User/Device Segmentation via Segments (Groups)

Prescriptive Underlay

• Topology and Protocol Independent

• Leverage Standards-based Network Infrastructure

• Optimised Forwarding, Time Distribution & Scale

MES

LIMS

ERP

APO

SCADAHISTORIAN

66

Anatomy GuidesCisco/Rockwell CPwE


• Tested, validated and documented reference architectures

Developed from use cases - customer and application

Tested for performance, availability, repeatability, scalability and security

Validated Designs

• Built on technology and industry standards

“Future-ready” network design

• Content relevant to both OT and IT Engineers

• Deliverables

Recommendations, best practices, design and

implementation guidance, documented test results and configuration settings

Simplified design, quicker deployment, reduced risk in deploying new technology

Reference ArchitecturesConverged Plantwide Ethernet (CPwE)

68BRKIOT-2130 68


Design GuidesConverged Plantwide Ethernet (CPwE) Design and Implementation Guides

BRKIOT-2130

Baseline Sept. 2011

CPwE

REP

Jun. 2014

CPwE

WLAN

Nov. 2014

CPwE

NAT

Jun. 2015

CPwE

Ident. Serv.

Jul. 2015 CPwE

IDMZ

Jul. 2015

CPwE

Resiliency

Dec. 2015

CPwE

Migration

Jan. 2016

CPwE

VPN

Mar. 2016

CPwE

Resiliency

Jul. 2016

CPwE

Ind. Firewall

Aug. 2016

CPwE

Loc. Services

Aug. 2016

CPwE

Ind. Computing

Aug. 2016

69


Sigmund FreudCollected Writings (1924) vol. 5, p. 210

“ Anatomy is destiny.”


Complete Your Online Session Evaluation

72BRKIOT-2130

Learn online with Cisco Live!

Visit us online after the conference

for full access to session videos and

presentations.

www.CiscoLiveAPAC.com

Give us your feedback and receive a

Cisco Live 2017 Cap by completing the

overall event evaluation and 5 session

evaluations.

All evaluations can be completed via the

Cisco Live Mobile App.

Caps can be collected Friday 10 March

at Registration.

http://www.ciscoliveapac.com/

Thank you


Further Reading / Viewing

• CiscoLive! (2012 Melbourne) BRKRST-2063 vPC and VSS Best Practice, Deployment and Operation, Evan Rose - Networking Consulting Engineer

Time

• CiscoLive! (BRKSPG-2170) Synchronisation in Packet-Based Networks -Laurent Montini – Technical Leader

Security

• Cisco Safe CVD

• http://www.cisco.com/c/en/us/solutions/enterprise/design-zone-security/landing_safe.html#~overview

Availability Design

http://www.cisco.com/c/en/us/solutions/enterprise/design-zone-security/landing_safe.html#~overview


Cisco Design Zone for Industry Solutions

Manufacturing• CPwE and Connected Factory

• Connected Factory – PROFINET

• Connected Machine

Oil & Gas• Connected Pipeline – Control Centre

• Connected Pipeline – Operational Telecoms

• Connected Refinery and Processing Facility

http://www.cisco.com/c/en/us/solutions/enterprise/design-zone-industry-solutions/index.html

Transportation• Connected Rail

• Connected Mass Transit

• Connected Roadways

Power Utilities• GridBlock Reference Model

• Field Area Networks

• Distribution Automation

• Substation Automation and Utility WAN

• Substation Security

http://www.cisco.com/c/en/us/solutions/enterprise/design-zone-industry-solutions/index.html


Rockwell Automation Design Guides

Rockwell Published Industrial Network Design Guides

• Converged Plantwide Ethernet (CPwE) Design Guide

• Deploying Network Address Translation (NAT)

• Site-to-Site VPN to a Ethernet Architecture

• Migrating Legacy IACS Networks

• Deploying a Resilient Ethernet Architecture

• Deploying the Resilient Ethernet Protocol

• Securely Traverse Data Across IDMZ

• Deploying Identity Services

• Deploying 802.11 Wireless LAN Technology

http://www.rockwellautomation.com/global/capabilities/industrial-networks/technical-data/overview.page

http://www.rockwellautomation.com/global/capabilities/industrial-networks/technical-data/overview.page


Panduit Design Guides and Whitepapers

• Physical Infrastructure for a Resilient Converged Plantwide Ethernet Architecture

http://www.panduit.com/ccurl/1013/385/physical-infrastructure-cpwe-architecture.pdf

• Fibre Optic Infrastructure Application Guide

http://www.panduit.com/ccurl/150/459/FiberOpAppGuide2011_GU_ENET-TD003A-EN-E_ENG,0.pdf

• A Manufacturing Network Fabric Maturity Model (Whitepaper)

http://www.panduit.com/ccurl/7/943/network-fabric-maturity-model-white-paper-cpat18.pdf

http://www.panduit.com/ccurl/1013/385/physical-infrastructure-cpwe-architecture.pdf

http://www.panduit.com/ccurl/150/459/FiberOpAppGuide2011_GU_ENET-TD003A-EN-E_ENG,0.pdf

http://www.panduit.com/ccurl/7/943/network-fabric-maturity-model-white-paper-cpat18.pdf


Acronyms

ACL – Access Control List

AP – wireless (IEEE 802.11) Access Point

APO – Advanced Planner and Optimiser

BGP – Border Gateway Protocol, IETF RFC 4271 since 2006

CIM – Computer Integrated Manufacturing

CIP – Common Industrial Protocol, ODVA

CMMS – Computerised Maintenance Management System

CPwE – Converged Plantwide Ethernet (Cisco and Rockwell design guides)

CVD – Cisco Validated Design


Acronyms

DAS – Direct-Attached Storage

DCS – Distributed Control System

DMZ – Demilitarised Zone

DLR – Device Level Ring protocol, EtherNet/IP

EIA – Electronic Industries Alliance

EIGRP – Enhanced Interior Gateway Routing Protocol

ERP – Enterprise Resource Planning

EVPN – Ethernet Virtual Private Network


Acronyms

HMI – Human Machine Interface

HSR – High-availability Seamless Redundancy protocol

IDMZ – Industrial Demilitarised Zone

I/O – Input/Output

IEC – International Engineering Consortium

IED – Intelligent Electronic Device

IEEE – Institute of Electrical and Electronics Engineers

IETF – Internet Engineering Task Force


Acronyms

ISA – International Society of Automation

ISO – International Organisation for Standardisation

IS-IS – Intermediate System to Intermediate System protocol, ISO/IEC 10589:2002

IT – Information Technology

ITU-T – International Telecommunication Union - Telecommunication Standardisation Sector


Acronyms

LACP - Link Distribution Control Protocol

LAN – Local Area Network

LIMS – Laboratory Information Management System

LTE - Long Term Evolution (standard for high-speed wireless communications)

MAB – MAC Authentication Bypass

MC-LAG – Multi-Chassis Link Aggregation Group

MES – Manufacturing Execution System

MICE – Mechanical, Ingress, Climatic/Chemical, Electromechanical


Acronyms

MPLS – Multiprotocol Label Switching

MRP – Media Redundancy Protocol

MSTP – Multiple Spanning Tree Protocol – IEEE 802.1s

MTTR – Mean Time To Repair

NAS – Network-Attached Storage

OT – Operations Technology

ODVA – Open DeviceNet Vendor Association

OEE – Overall Equipment Effectiveness

OSPF – Open Shortest Path First protocol


Acronyms

PBB-EVPN – Provider Backbone Bridging Ethernet Virtual Private Network

PLC – Programmable Logic Controller

PTP – Precision Time Protocol, PTP Version 2 IEEE 1588-2008

PAgP - Port Distribution Protocol

PRP – Parallel Redundancy Protocol

REP – Resilient Ethernet Protocol

RSTP – Rapid Spanning Tree Protocol – IEEE 802.1w

QoS – Quality of Service


Acronyms

SAN – Storage Area Network

SCADA – Supervisory Control And Data Acquisition

SGT – Security Group Tag

STP – Spanning Tree Protocol – IEEE 802.1D

TIA – Telecommunications Industry Association

TTL – Time To Live


Acronyms

VLAN – Virtual Local Area Network

VPLS – Virtual Private LAN Service

VRF - Virtual Routing and Forwarding

VSS – Virtual Switching System

VXLAN – Virtual Extensible Local Area Network

WAN – Wide Area Network

WiFi – Trademark of the WiFi Alliance – IEEE 802.11 standards-based network

WiHART – Wireless Highway Addressable Remote Transducer Protocol

WMS – Warehouse Management System

anatomy of modern process control networking …anatomy of modern process control networking...

Documents