reinventing the role of the scada historian · reinventing the role of the scada historian...

Standards

Certification

Education & Training

Publishing

Conferences & Exhibits

Reinventing the Role of

the SCADA Historian

Distributed Redundancy, Centralized Access

2013 ISA Water / Wastewater and Automatic Controls Symposium

August 6-8, 2013 – Orlando, Florida, USA

Speakers:

Blair Sooley, M. B. A., P. Eng. – Trihedral Engineering

2013 ISA WWAC Symposium

Aug 6-8, 2013 – Orlando, Florida, USA 2

Presenter

Blair Sooley is a Pre-Sales Account Manager

with Trihedral Engineering

• Electrical Eng. - Dalhousie Univ., Canada

• M.B.A. - St. Mary’s University, Canada

• 18 years in SCADA and process monitoring

• Specialized in high-level solution design



Presentation Outline

• Analysis of the historian

• Limitations and threats

• Currently employed protection schemes

• Improving efficiency, redundancy and synchronization

• Redefining the role of the historian

• Examples of advanced historian deployments

• Selecting a database format

• Servicing data requests

• Cost control



Historians Don’t Take Vacations

• Ever-increasing demands

– Reporting

– Upset analysis

– Process improvement

• Storage has become cheap

• Focus – meet demands rather than high availability

Why not focus on high-availability?

The answer is Simple. Because it’s Hard.



Analysis of the Typical Historian

Separate Working (real-time)/Historical (long-term) databases

Key steps to a Historian’s life

STEP 1: Periodic sampling of real-time DB

STEP 2: Comparison to set of rules

STEP 3: Copy to database or disregard

STEP 4: Service data requests



Limitations of Current Paradigm

• Don’t forget to have 1-2 summary slides at the end

Process EFFECT

Periodic sampling of real-time

database whether or not data

changed

Significant loading of CPU. Worsens

as system scales

Storage to business DB’s, such as

SQL and Oracle.

Platform mismatch. Such DB’s are

designed for many concurrent

connections and complex data

analysis. SCADA requires fast access,

simple format, few concurrent

connections and time-based datasets.

Data logged to single DB. DB is

queried from various sources.

No redundancy. Potential for data loss

due to single point of failure.



Understanding the Threats

• Natural disasters

• Equipment malfunctions

• Open networks make software more accessible.

Technology makes it more difficult to protect.

1US Department of Homeland Security (DHS), US Department of Energy (DOE) (2006). Introduction SCADA Security for Managers and Operators [Presentation].

Retrieved from http://energy.gov/sites/prod/files/oeprod/DocumentsandMedia/Introduction_to_SCADA_Security_for_Managers_and_Operators.pdf

DHS & DOE - Cyber threat sources1

1. Strategic Information Warfare (e.g. terrorism)

2. Direct Cyber Attacks (e.g. disgruntled employees)

3. General Cyber Attacks (for notoriety/fame)



Current Protection Schemes

RAID (Redundant Array of Inexpensive Disks)

Basic – RAID 1 (mirroring) and RAID 2 (striping)

Complex – RAID 10 (mirroring and striping)

• Concerns

– Single controller = Single point of failure

– SATA (Serial Advanced Technology Attachment) randomly

oriented disks cause performance degradation.

Number of Disks in Array Average Degradation vs. Single Disk

2 33% longer to process data read request

5 67% longer to process data read request




Offline Backup

Removable storage survives catastrophic failure.

• Concerns

– No synchronization - Current data only at the moment of creation

– Backup and restore take time

– Systems may not support online backup/restore

– Keeping up with media evolution (floppy, tape, ZIP, CD, USB)




Redundant Independent Historians

Data storage in two locations for fault tolerance

• Concerns

– No synchronization – differing data if sampling at different times

– No data backfilling after server outage

– Trends and reports differ based on which Historian is queried



Evolution of the Historian

11

Must become Faster, More scalable, Fault tolerant

New Historian RoleGuide the process of storage and

retrieval by dedicated sub processes

Old Historian RoleStore to and retrieve from a

single DB



Identify Wasted Effort

Historian servers are typically overloaded with unnecessary

tasks.

Play note

Is this the

correct note?

Disregard

No

Record

Yes

Switch to

different note

Record song• Periodic evaluation of

specific set of real-time

variables

• Unchanged values are

sampled and disregarded

• Repeat

Analogy – An Overworked Musician



Improve Efficiency by

Reducing Wasted Actions

Event-driven execution registers specific events of interest

with the SCADA System.

• SCADA notifies when a

specific event occurs

• Quicker response

• Able to handle a greater

number of events.

= Lightly loaded resources

Analogy Revisited

A Well-rested Musician

Play C note

‘Play ‘C’ Note’

event

Wait for

next event



Identify Synchronization Problems

Unsynchronized Historians may record at different times.

• Historian data may differ

• Summary data differs

based on DB queried;

• Trends

• Reports

• Calculations

Analogy

2 musicians play the same song with

slightly different arrangements



Improve Synchronization through

Assigned Leadership

Assigning Leader (primary) and follower (backup) positions

guarantees synchronization.

• Primary defines data to

be recorded

• Backup copies actions

explicitly and in sync

The leader selects the arrangement and

provides direction for the follower

Leader Follower



Multi-Level Redundancy

Allows Greater Data Security

Any number of redundant data storage locations can be

synchronized in this way.

Imagine this horn section had no leader!

In an orchestra, the 1st Chair is the section leader. 2nd and maybe 3rd Chairs are

appointed to ensure section members have a hierarchy of leaders to follow.

1st Chair Etc.2nd Chair 3rd Chair

1st Chair absent?

2nd Chair assumes

leadership position



Data Backfill

Mitigates Data Loss

Synchronization ensures data can be backfilled to a data

storage location that has been out of service.

A recording of one musician can be copied to fill in where

another musician was unavailable to record.

1st Chair Etc.2nd Chair 3rd Chair

Data is

backfilled

from primary

recording



Synchronization and Data Backfill

Georgetown, KY

Geographically separated data storage. Each plant

provides local storage and backup remote storage.

Real-time synchronization

Bi-directional backfill

Secure VPN IPSec tunnel

Minimal equipment

Fault tolerant



Traditional Historian

With Scalability comes Complexity

The Historian role includes recording, prioritization, data

longevity management and servicing data requests.

• Powerful servers required

• Limited scalability

Traditional Historians

Each server performs all functions

Server 1 Server 2

http://1.bp.blogspot.com/-qEF6pQy0fWI/Ta-Z48nxrQI/AAAAAAAAAIA/WcOOAGf888I/s1600/Bill+OneManBand.jpg






Distributed Data Storage Management

A Scalable Alternative

Lightly loaded SCADA workstations offer opportunity for

distributed task assignment.

• Storage and retrieval

activities reassigned to

computers with excess

CPU and storage

• Distribute actions by

functional area

• Dedicated Historian

computers eliminated

Distributed Task Assignment

Conducting the Orchestra

Conductor

directs

activities

Each orchestra section

assumes responsibility

for its own part



Redefining RAID for Historian Use

Drives on SCADA computer network are clustered to form a

data storage network of subparts with central read access.

Clustered data striping and mirroring (RAID 10)

Greater fault tolerance than standard RAID due to dedicated controllers

Plant 1

primary

storage

Plant 1

backup

storage

Mirroring

(RAID 1)

Plant 2

primary

storage

Plant 2

backup

storage

Striping

(RAID 5)



Distributed Historian Storage

Ocala, FL

6 distributed storage locations

Small local storage, large central

Backup local computers eliminated

If the centralized DB is unavailable, data requests are broken into sub-requests for servicing

by each of the remote sites. The responses are combined and delivered to the requestor.



Eliminating a new

Single Point of Failure

Identify hierarchy to eliminate the Historian as a SPOF.

Help, the conductor is sick! – Alas, the show must go on.

The leadership position is assumed based on an established hierarchy such

that the orchestra is never without leadership.

Conductor Etc.Backup

Conductor

2nd Backup

Conductor

Conductor

unavailable?

Backup conductor

assumes leadership

position



Enhanced Fault Tolerance

Data Buffering at the Source

Distributed historian storage and autonomy allows greater

fault tolerance by buffering local data at remote assets.

Buffered data storage at

remote asset

Centralized

data storage

Data

Backfill

• No data loss after network

or central computer outage

• Replace local HMI with

integrated SCADA node

• Lightly loaded local node

easily handles logging.



Data Buffering at the Source

Ontario Power Generation, ON – (plan)

Existing central DBs provide collocated redundancy

Expansion plan

• Add local buffering at 26 remote generation facilities

• Data compression on backfill limits network traffic



Database Format Selection

The Historian controls the process. As long as it can R/W

data to the format, the storage format should not matter.

Database considerations

• Native SCADA formats may

be faster

• 3rd party software access

(reports, CMMS, etc.)

• Some formats provide better

security against hacking

• Cost



Combining Database Formats

Gainesville Regional Utilities, FL

Single SCADA application with mixed historical

database formats.

• Data sync and backfill across

differing DB formats

• Native DB’s high speed

required by operations team

• Existing SQL DB leveraged

by engineering team



Servicing Data Requests

Historian = single point of contact for servicing data requests.

• Storage mechanism is

irrelevant to requestor

• Historian knows where all

data is stored

• Complex requests (min,

max, ave., totals, counts)

can be derived from raw

data on demand

Taking Requests

From the Audience

Conductor

determines if his

orchestra has the

capabilities to

service the

request.

Only specific orchestra

sections are required to

meet request



Cost Control

Expectation

Faster/more robust/scalable historian = Increased cost

Actual

Reduced infrastructure = Reduced cost

$ SAVINGS OPPORTUNITIES $

Eliminate dedicated historian servers

Exchange server hardware for workstation hardware

Eliminate collocated server redundancy

Use native DB formats over expensive business formats



Summary

Problem

• Scalability and efficiency concerns plague existing systems

• Demands are increasing. Historical data protection has become critical.

• Current protection schemes are inadequate

Solution

• Synchronization and data backfill are key to data quality and service

availability

• Role of the new Historian is to manage synchronization, to guarantee

data backfill to all replicated storage notes, and to protect the integrity

of the data storage

• Cost control requires realization of underutilized infrastructure. Highly

organized management of distributed architecture increases reliability.



Thank you!