ISA—The Instrumentation, Systems, and Automation SocietyPhiladelphia Section

Samuel M. Herb, PE

May 18, 2005

Implications of System Security…the “old days”…

– Reliability, – Diagnostics, – Redundancies– Access by local personnel

vs. remote access to information – Availability vs safety shutdown

– All are still part of the complete story!

System Security Building Blocks

HIGH STRENGTH

DIAGNOSTICS

COMMON CAUSE STRENGTH

ARCHITECTURE

FAULTAVOIDANCE

FAULTTOLERANCE

FAULTTOLERANCE

Designing for NO Component Failures!

Need for Safety Systems

• High availability process control• Burner management systems• Safety instrumented systems • Fire and gas protection systems• Turbine control systems

Safety Lifecycle• S84 & other SIS standards formalized development, design,

operation, testing, & maintenance of safety systems• U.K. Health and Safety Executive (HSE) analyzed a number

of control and safety system failures:

!

Safety Contest

The Top Five WinnersFor

The worst safety practices at work

5th Place

4th Place

3rd Place

?

2nd Place

and the winner is...

System Security• Increased safety• Improved reliability• Higher system availability• Reduced start-up time

and minimized downtime• Increased security• Open communication !

How Open?

BANK

Open banking?

Microprocessor-based Equipment

• Inherent diagnostics• Can notify operator• Can even trigger corrective action

Introduction

FATAL EXCEPTION OCCURRED

AT ADDRESS 00C4:4F51H

Introduction

Resetting costs time and money !!!

Typical Security Protection Model

Secure Systems are Necessary• To prevent problems from happening • From directions you least expect…

Plant Control Network

Machine ControlPackaging

Process

Single Loops

History

Engineering

Central ControlThe Business…

…The WorldCustomersSuppliers

Clients

Which Partsare

Vulnerable??

SCADA Network System

T&D Power SCADAWater, Oil & GasPipeline SCADA

History

Engineering

Central ControlThe Business…

…The WorldCustomersSuppliers

Clients

Which Partsare

Vulnerable??

Control System Must Link to Plant

InformationInformationDomainDomain

Enterprise – Site – Area

Instructions(make a batch)

ControlControlDomainDomain

UnitSupervision

ProcessControl

ProductionInformation

Management

RecipeManagement

ProductionPlanning andScheduling

Controllers

Automation System

Data(Batch Record)

ProcessManagement

Safety Protection

MES

TheoreticalJunction

Disparate Computing Communities:

Enterprise Resource Plan

Data

Control SystemControl System

Information !

B u s in e s sP la n

C o n tro l S y s te m

ProcessFolks

• Control People understand Plant Floor Complexity

• Enterprise is some cloud…..

• Business People understandEnterprise Complexity

• Control system is a little box...

IT Folks

BUT – No time for battles!BUT – No time for battles!

Security Protection Needed Now

• 2001 demonstrated that self-propagating viruses are tangible threat to most process control systems.

• Critical national infrastructure organisations warn that viruses are real threat to Process Control & SCADA systems.

• Analysis of attack activity over 6 months Jan-July 02 reveals that Internet attacks remain significant threat to organisations of all types.*

• High Tech, Financial Services, and Power and Energy companies continue to show highest rates of attack per company*

* = Riptech Internet Security Threat Report – July 02

CERT/CC Statistics 1988-2003Number of incidents reported1988-1989

Established in 1988, the CERT® Coordination Center (CERT/CC) is a center of Internet security expertise operated by Carnegie Mellon University

1326Incidents

19891988Year

1990-1999

9,8593,7342,1342,5732,412 2,3401,334773406252Incidents

1999199819971996199519941993199219911990Year

2000-2003

76,40482,09452,65821,756Incidents

1Q-2Q 2003200220012000Year

Total incidents reported (1988-2Q 2003): 258,867Please note that an incident may involve one site or hundreds (or even thousands) of sites. Also, some incidents may involve ongoing activity for long periods of time.

!

Rapidly Increasing Cyber Threat

• World becoming more interconnected.• Skill level needed to hack is reducing.• Number of vulnerabilities are increasing as

systems are becoming more complex.• Becoming more difficult

to defend from attack.

An Ernst and Young Security Survey Reported:- That Over 90% of Fortune 500 Networks Have Been

Hacked

2003/2004 CSI/FBI ReportWhile Only 50% of the Respondents Were Willing to

Quantify Financial Losses…

-They Reported approximately

$200 Million in Financial Losses Per year

Connecting Your Unprotected PC to Internet

• In 2003 – attack occurred within 15 minutes. • In 2004 – attack occurred within 15 seconds.

• Don't take it personally.– whole process usually automated.– your machine just another number in a range

of targeted IP addresses.

Source: Symantec

Security Incidents

Speed of Virus Propagation

“ Rate change within the past year…”Decreasing Vulnerability to Exploit Time

The Spread of the Sapphire/Slammer Worm

“ Within Just 30 Minutes! ”

Key threats to process control systems

• Network worm/virus (e.g. Nimda, SQL Slammer)• Targeted external hacker or cracker

– with process control knowledge– without process control knowledge

• Targeted internal hacker or cracker– with process control knowledge– without process control knowledge

• General virus attack• Protest hacking or cracking• Amateur hacking or cracking

Sources of Threats

Organised crime

Nation states/Governments

‘Insider’ threats

Competitors, contractors, corporations

Corporate intelligence/Investigation companies

Disaffected staff(Including contractors)

Malicious code attack specifically directed against a Customer

Illegal information brokers andfreelance agents

General malicious code threat

Common criminals

General hacker threatAnimal rights activistsAnti world trade/

Anti globalisationactivists

Environmental groups

Regional political activism

Non state-sponsoredterrorism

In support of these Threats

Business Risks

• Cyber attack could lead to:• Failure of control systems• Loss of integrity or control of systems• Loss of process monitoring and visibility of plant

• Which may lead to:• Risk of injury or loss of life• Loss of production• Environmental damage• Damage to the Company brand and reputation• Company’s licence to operate being jeopardised

Some Definitions• Virus — Unauthorized program that replicates itself, attaches itself

to other programs, and spreads onto various data storage media (floppy disks, magnetic tapes, random access memory, etc.) and/or across a network; - symptoms of infection include much slower computer response time, inexplicable loss of files, changed modification dates for files, increased file sizes, and total computer failure.

• Worms — Absorb memory & slow performance much like viruses, but do not attach themselves to other programs and generally do not destroy data, software, or other system resources.

• Trojan Horses — Apparently innocuous but unauthorized software programs hidden within authorized programs which, when loaded into a system or network, will otherwise function similar to viruses, will allow access to a virus, or can be designed to give system access to a specific “cracker.”

• Larger systems do not generally suffer from viruses, but they dosuffer from worms and Trojan Horses

Some Definitions• Cracking—Breaking into computers for criminal purposes;

- typical way for hackers--both black hat and ethical--to gain access to some organization's network is to use analyzers that can sniff or probe for passwords for networked systems.

• Hacking—Fooling around with computing functions to learn new or different functions and techniques; - due to misuse in the news media, term has come to mean by some, to cause unauthorized access to other computers over public networks.

• Hacktivism—Act of hacking into website or computer system to communicate politically or socially motivated message, as opposed to causing malicious harm

Some Definitions

• DoS — Denial of Service; implemented by crackers who break into business website to disrupt or destroy service to users & customers of that business by denying access to their site.

• IP Spoofing—Technique used to gain unauthorized access to computers, whereby intruder sends messages to computer with some IP address indicating that intruding message is coming from a trusted port; - to engage in IP spoofing, a cracker must first use a variety oftechniques to find an IP address of a trusted port and then modify the packet headers so that it appears that the packets are coming from that port; - newer routers and firewall arrangements can offer protection against IP spoofing.

Nature of Attack

Sources of Attack

Quick Solution: Isolation!!

Differing Operating Systems

BUT in Typical System Architecture……there are other sources too!

SCADA RTU

Vulnerability

Low High

Internet

ESD—Emergency Shut DownEMS—Enterprise Messaging Server

DCSMTU

EMS

Remote Service Center

ESDDCS Offerings

EnterpriseSystems

Noise or Bad Packets

Engineering Network

Repeater

Accounting NetworkCut

• Propagation of noise or bad packets throughout an entire network is a serious risk.

• Pulp mill case history-– Cable damage problem in one area creates bad

packets from reflections.– “Dumb” network equipment spreads problem to other

areas !

IP Address Duplication• TCP/IP protocol demands that every device has

an unique IP address.• Paper Machine Profile Controller Case History:

– Controller & Scanners use TCP/IP to communicate.– Printer in administration gets same address as controller.– Scanners try to talk to printer instead of controller !

Process NetworkAccounting Network

ControllerScanner

Scanner sends data to printer

X

Switch

Broadcast Storms

• Broadcasts are messages addressed to all network nodes.

• A few broadcasts are okay. Many create broadcast storms and will use up a device’s CPU resources.

• Case History- Steam Plant DCS:– DCS uses Ethernet to communicate between screen server and

operator consoles. – Broadcasts from mis-configured Windows 95 machine in

another mill area overloads screen server…Shuts down all DCS operator consoles!

Internal Intranet Intrusion• Eastern plant does major upgrade of DCS.• Several months later, head-office engineer

connects to the mill DCS from head office, using the company's wide area network (WAN)…

East Coast Plant Business NetworkDCS Network

Head OfficeRouter

Router

PLC PLC

Head Office Engineer

DCSPLC Gateway

…Internal Intranet Intrusion• Engineer loads program onto operator station

to send data to head office for expert system. • This new task overloaded DCS/PLC gateways.• Operators lose control of devices

connected to PLCs!Router

DCS Network

Head Office

Router

DCS

PLC PLC

Head Office Engineer

PLC Gateway

Control Highway Intrusion• Disgruntled employee attacks PLC in another

plant area over PLC highway.• Password changed to obscenity, blocking

legitimate maintenance and forcing process shutdown!

Disgruntled Employee

PLC PLC PLCPLCSteam Plant Paper Plant

Plant Highway #@*!%^!#

Unplanned Workstation Activities

• Operator chooses to load TurboTax into workstation to “better use his time.”

• System crashes DCS controllers… …taxes boss! (total loss = 10 man/yrs)

Stupid Employee

Control

Plant Highway

Control Control Control

Taxed Staff

NOTHING gets added!

External Wireless Intrusion

PLC PLCSewage Plant

• Hacker attacks sewage control system using radio link.

• Causes millions of liters of raw sewage to spill out into local parks, rivers

…and the grounds of a Hyatt Regency hotel!

Disgruntled Contractor

Rogue Radio

Control System’s Vulnerabilities

Internetand Extranets

DBFirewalls

COE

Router

ControlApplication

Servers

PI Server WebServer

Operator Consoles

Distributed Control System

RS CS TR RD TD CDTALK / DATA

TALK

Modem

UnattendedPC

in a laboratory

Process ControlNetwork

Plant

RAS

Dial-up

Radio / Wireless Links

Center Of Excellence user accidentally causes virus outbreak

Attacker breaks in thru Remote Access Service or third-party link

Accidental act by COE user or administrator

Attacker connects via poorly protected modem

Maintenance staff insert virus via infected floppy

or CD

Housekeeping gets access to unprotected PC in unlocked lab

Attacker taps into wireless communications

New Internet worm attacks web servers(e.g. CodeRed, Nimda, SQL Slammer)

““Awareness of all Awareness of all possible entries”possible entries”

Information Protection Model

Firewalls, DMZ, “Air Gap”, perimeter defense, Firewalls, DMZ, “Air Gap”, perimeter defense, PKI (Public key infrastructure), policy & PKI (Public key infrastructure), policy & procedures procedures

AvoidanceAvoidance

Vulnerability analysis, log reviews, alarms, Vulnerability analysis, log reviews, alarms, regular schedule reviewsregular schedule reviewsAssuranceAssurance

Intrusion detection systems, port scannersIntrusion detection systems, port scannersDetectionDetection

Incident Response, disaster recovery, offIncident Response, disaster recovery, off--site site backups, response teambackups, response teamRecoveryRecovery

ExamplesExamplesLevelLevel

Secured Environment = Secured Environment = [(Policy and Procedure) + Tools] X Commitment[(Policy and Procedure) + Tools] X Commitment

DMZ

DeMilitarized Zone-- Middle ground between organization's trusted internal network and untrusted, external network such as Internet; subnetwork (subnet) that may sit between firewalls or off one leg of a firewall.

ISPs typically place their Web, mail and authentication servers in the DMZ.

DMZ is a military term that refers to the area between two enemies.

? Question ?• If you had unlimited $$$ & Resources

what would you spend it on for security?• Technology?

• Consultants?

Spend it on a Security Policy!Spend it on a Security Policy!

…then ENFORCE it!

Risk Assessment and ReductionRisk Assessment and Reduction

•• Develop a Internal Security TeamDevelop a Internal Security Team–– Establish a permanent core Security Team to Establish a permanent core Security Team to

establish and direct internal practicesestablish and direct internal practices–– Have corporate buyHave corporate buy--inin–– Have local site responsibilitiesHave local site responsibilities–– Execute the initial Site Security Auditing Execute the initial Site Security Auditing

processesprocesses–– Establish key third party partnersEstablish key third party partners

Risk Assessment and ReductionRisk Assessment and Reduction

• Provide pragmatic advise on process control security measures in 6 areas:

• Dial-up modems • Network connectivity• Virus protection• Remote workstations/servers• Wireless Ethernet• Cyber incident and crisis response

Implement Physical Security

• Before any electronic measures are considered, physical access to sensitive equipment must be controlled

• Access to a local keyboard creates more damage then a virus or ‘hacking’

• Access to system equipment where any changes can be made must be controlled

Secured Environment = [(Policy and Procedure) + Tools] X Commitment

SecurID card – RSA Security Inc. Authentication token from RSA Security, Inc., Bedford, MA (www.rsasecurity.com) that uses a smart card that authorized users keep in their possession. The card's microprocessor and the host computer are synchronized by a unique number and the time of day. When users log onto a SecurID-enabled host, they type in the number displayed on their cards at that moment as an additional passcode. If the number matches the number that the host computes, the user is presumed to be the valid holder of the card.

Isolate Network Segments

• Avoid use of corporate LAN for DCS and other control systems

• Use API/ODBC services to exchange data between systems

• Define and pass only necessary information

• Use secure routers (firewall) to segregate systems:• ERP• MIS• Metering

• EMS• DCS• SCADA• DMS

Control Network Connectivity

I/O & Field Communications

DCS Network Security Considerations:•Access Control•Intrusion Detection•What resources are you trying to protect?•Define the host-specific

security measures needed.•Decide who will grant access to services•Teach users about password protection• Keep up with

latest security-related technologies

Ethernet PortWorkstations & Servers

Balance of PlantNetworkSecure Router

(Firewall)

Controllers

Use Operating System Basic Security Services

• Administrative Access Tools• Activity and Users Tracking logs• Remote Access Tracking logs• Restrict or Disable

Direct and Remote File Transfer and Access Services (e.g, NFS, FTP, RFTP, HTTP)

• Investigate the new OpenSSH tcp wrappers• Password Enforcement

- e.g., Aging, minimum 8 characters…• Backups

ALL Operating Systems are Vulnerable

• Not just Microsoft– Certainly most common– Large number of systems are better target

• UNIX• LINUX• MACINTOSH

Security is an Industry ProblemSecurity Advisories 2002Security Advisories 2002

Redhat, Redhat, SuSESuSE, Mandrake, , Mandrake, TrustixTrustix, Debian, and Sun OS have , Debian, and Sun OS have released more security advisories than Microsoft in this periodreleased more security advisories than Microsoft in this period

Trustix1.5

DebianEnGarde Sun(OS)

Mandrake 8.x

00

2020

4040

6060

8080

100100

120120

RedHat7.2

Windows2000

WindowsXP

33 34 3751

67

86 86 87

124

SuSE

Source: Company web sitesSource: Company web sites

Security is Still an Industry Problem

Mandrake DebianTrustix SuSE Sun(OS)

Security Advisories 2003Security Advisories 2003Redhat, Redhat, SuSESuSE, Mandrake, , Mandrake, EnGardeEnGarde, Debian, and Sun OS have , Debian, and Sun OS have

released more security advisories than Microsoft in this periodreleased more security advisories than Microsoft in this period

00

3030

6060

9090

120120

150150

180180

RedHatWin 2000

WinXP

24 30 32 3351

68

119 120

184

EnGarde

Source: Company web sitesSource: Company web sites

Risk Reduction – Starting Guidelines

• Baseline of authorized and proper use of DCS network should be established.

All other activity should be denied.

• All network connections to and from DCS network should have single point of entry.

• Physical and Layer 2 access to DCS network should be controlled through change management.

• All DCS network interaction with other networks should be monitored and logged 24x7.

Risk Reduction – Starting Guidelines

• All traffic leaving DCS carrying sensitive information should be encrypted.

• Intrusion attempts and policy violations should be reported and discussed on an ON-GOING basis.

• All passwords should be change periodically; Enabled NO default users or vendor backdoors.

• Repeat - Ownership of security policies and process should be local to each site.

Key Take Aways From All This• Security is very important and should not be taken lightly.• Connecting any computer to a non-private network

creates a risk that must be evaluated.• Ease of moving information and security

are mutually exclusive.• Protect passwords, enforce user accounts. • Beware of connections to corporate WAN.• Trust NO ONE !!• Consider hiring a security consultant

to evaluate your policy, location, and corporate network.

Where to begin?• “The easiest thing to do is nothing,

…which is exactly the wrong thing to do”

• Assemble team to help in planning, assessment, implementation, and response

• Review the asset your trying to protect • Break down the system into sections• Apply security in phases• Monitor and maintain the security “systems”

It’s not a one shot deal & there are no silver bullets

Material Stolen from:

• ISA instructor & author Eric Byers, manager, British Columbia Institute of Technology Internet Engineering Lab

• Charlie Piper, Alex Johnson, & others – Invensys Foxboro

• Intelligent Systems Div. of NIST• Probably more…

There are Never Simple Answers!!There are Never Simple Answers!!If there were...• all of this stuff would be sold mail order,

and... • talks like this would be unnecessary!!

…Sam

Samuel M. Herb, PEJAOMAD Consultancy117 Pawnee RoadNew Britain, PA 18901-51142Voice:(215) 345-1464 Fax:(215) 345-4387SamHerb@JAOMAD.comhttp://www.JAOMAD.com