Cybernetics Symposium Phoenix, AZ

April 2012

Alarm Management

Tactics and Analysis

Maurice Jones

Alarm Management Coordinator

Plains All American Pipeline

• Getting Started

• Define the Goal

• Tactics

• Analysis

• Execution

• Measuring Results

• Going Forward

Overview

2 Alarm Management – Tactics and Analysis

Getting Started

Alarm Management

Team

Alarm Management

Plan & Philosophy

Define Goals

3 Alarm Management – Tactics and Analysis

Goals

Clarity of Alarms

(Meaningful Alarms)

Manage Safety-Related Alarms

Reduce Alarm

Volume

4 Alarm Management – Tactics and Analysis

Tactics

Establish Baselines

Identify Alarms / Events

Received

Promote Involvement

5 Alarm Management – Tactics and Analysis

Baselines

Snapshots Reports

6 Alarm Management – Tactics and Analysis

Benchmarking…6 Weeks of Data

* Baseline – 1/2/11 to 2/6/11 (Console were run from different SCADA system)

7 Alarm Management – Tactics and Analysis

Baseline Summary - 7/4/10 to 8/8/10

Console Alarms/Hour

A 46

B 152

C 79

D 30

E 141

F 70

G 13 *

H 33

I 41

J 40

K 26 *

L 14 *

M 117

N 41

O 70

Maximum Manageable 12

Acceptable (ISA 18.2) 6

Plains Goal (?) 24

8 Alarm Management – Tactics and Analysis

46

152

79

30

141

70

13

33

41

40

26

14

117

41

70

12

6

24

- 20 40 60 80 100 120 140 160

A

B

C

D

E

F

G

H

I

J

K

L

M

N

O

Maximum Manageable

Acceptable (ISA18.2)

Plains Goal (?)

Alarms/Hour Baseline – Console A thru O

2 out of 15 meet Plains‟ Goal

0 out of 15 meet ISA 18.2 “Acceptable” Goal

Analysis

Create/Distribute Reports

Interpret Results

Dig Deeper

Team Review

Understand Operational Influence /

Impact

Unexpected Finds

Alarm / Event Types

Instrument Failures

9 Alarm Management – Tactics and Analysis

Determine Strategies

Execution

Implement

Strategies

Single Tag

Multiple Tags

Entire Pipeline

Entire Console

Control Center

Measure Effectiveness

Weekly

Monthly

Are Improvements Sustained?

Share with Team

Standardize

Apply to Other Consoles

(If Appropriate)

Establish New Baselines

Acknowledge Teams and Individual

Contributions to Success

10 Alarm Management – Tactics and Analysis

Console “A” Fluctuates between nearly 0% reduction to 40% reductions when compared to

baseline. Definitely a „Work in Progress‟.

11 Alarm Management – Tactics and Analysis

Baseline Summary - 2/5/12 to 3/11/12

Alarms/Hour

Console Percent Change Baseline New

A -9% 46 42

B -78% 152 33

C -56% 79 35

D -84% 30 5

E -62% 141 53

F -52% 70 34

G -73% 13 4

H -57% 33 14

I -56% 41 18

J -52% 40 19

K -72% 26 7

L -62% 14 5

M -71% 117 34

N -49% 41 21

O -68% 70 23

This Console has the Greatest Number of Configured Alarms

This Console has been split into (2) Consoles – bringing total # to (16).

12 Alarm Management – Tactics and Analysis

5 out of 15 meet Plains‟ Goal

3 out of 15 meet ISA 18.2 “Acceptable” Goal

Console „E‟ has been split into (2) consoles with an average of (21) and (32) Alarms/Hour.

42

33

35

5

53

34

4

14

18

19

7

5

34

21

23

12

6

24

- 20 40 60 80 100 120 140 160

A

B

C

D

E

F

G

H

I

J

K

L

M

N

O

Maximum Manageable

Acceptable (ISA18.2)

Plains Goal (?)

Alarms/Hour Current Performance – Console A thru O

13 Alarm Management – Tactics and Analysis

22929

7176

1583 0

5,000

10,000

15,000

20,000

25,000

7/4

/10 0

:00

9/2

6/1

0 0

:00

12/1

9/1

0 0

:00

3/1

3/1

1 0

:00

6/5

/11 0

:00

8/2

8/1

1 0

:00

11/2

0/1

1 0

:00

2/1

2/1

2 0

:00

CommunicationAlarms

Early decrease from changes made to field equipment, SCADA system corrections,

disabling decommissioned sites.

Decrease between 1/9/2011 and 2/20/11 from changes made to SCADA parameters.

Recent decreases from specific focus on improving individual locations.

About 13% of all alarms.

About 3% of all alarms.

14 Alarm Management – Tactics and Analysis

152,588

82,822

52,345

0

20,000

40,000

60,000

80,000

100,000

120,000

140,000

160,000

180,000

7/4

/2010

8/1

/2010

8/2

9/2

010

9/2

6/2

010

10/2

4/2

010

11/2

1/2

010

12/1

9/2

010

1/1

6/2

011

2/1

3/2

011

3/1

3/2

011

4/1

0/2

011

5/8

/2011

6/5

/2011

7/3

/2011

7/3

1/2

011

8/2

8/2

011

9/2

5/2

011

10/2

3/2

011

11/2

0/2

011

12/1

8/2

011

1/1

5/2

012

2/1

2/2

012

3/1

1/2

012

Combined Control Center Alarms Per Week (12 Consoles)

152,588 – 52,345 = 100,243 Reduction in Alarms/Week or 66% in Combined Control

Center Alarms.

Data was gathered on just 12 Consoles when AM activities began.

15 Alarm Management – Tactics and Analysis

Alarm Type by Console – January 2012 (Monthly Report)

Alarm Type A B C D E F G H I J K L M N O P

\Total 34,295 24,683 16,759 24,601 3,817 25,654 23,422 2,747 12,890 12,002 12,187 5,650 3,997 23,994 15,159 16,473

\PerWeek 7,744 5,574 3,784 5,555 862 5,793 5,289 620 2,911 2,710 2,752 1,276 903 5,418 3,423 3,720

\PerWkChange 324 (68) 51 (518) (180) 303 343 (237) (317) (380) 326 (9) (129) 191 (118) 147

\PerHour 46 33 23 33 5 34 31 4 17 16 16 8 5 32 20 22

\PerHrChange 2 (0) 0 (3) (1) 2 2 (1) (2) (2) 2 (0) (1) 1 (1) 1

HILO 25,390 18,543 14,529 19,455 995 19,629 18,660 798 6,297 6,456 8,515 798 1,455 18,538 8,631 13,362

COS 3,570 2,455 651 2,442 1,148 4,254 3,070 564 4,649 1,967 1,760 1,434 684 2,615 2,949 860

RTUCOMM 359 1,313 134 682 226 385 484 228 136 262 239 181 215 1,010 28 731

OPWARNTI 53 563 235 276 104 73 73 626 108 182 1 18 1,019 188

OPWARNVO 553 113 237 61 33 13 240 66 161 34 3 460 106

OPTRKCOM 21 487 204 161 52 43 66 135 66 149 24 7 281 127

COMMAND 1,820 248 48 64 16 104 236 38 78 87 132 124 166 119 525 142

RANGE 234 8 2 4 18 2

* Restricted * 19 108 101 215 18 210 117 9 78 55 232 75 204 6 85

WARNTIME 604 100 665 2 350 124 1,410 2 561 708 688 4 224

TARGET 976 47 454 627 13 627 568 134 2 67 619 337 130 573 105 301

STALE 13 34 172

XXXXX_MB 6 10 65 112 1 58 42 8 2 5 71 9 12 2

ACCRESET 6 9 65 112 1 58 42 4 2 5 71 9 12 1

NODE 496 2 19 19

BATMTR 13 4 5

ROC 24

COMMPATH 19 14

PPL1PIDL 12

ABORTED_ 4 1 1 4 2 3 1 11 1 1 10 1 6

IF_STATU 6 5 3

TIME_TO_ 9 318 190 193

TRKCOMPL 258 1 270 1 1 193 1 410 1 192 234 2 127

ABORT20_ 1

BAD_LEVE 1 1

DEV 495 134 151 213 414 589 494 1,630 11 182 958

WARNVOLU 1 312 241

NULL 1

POLL_CMD 28 18 27 39 38 3 18

STERROR 3

* Restricted * 152 139

OPTRK 2

16 Alarm Management - Tactics and Analysis

Alarm Type by Console – January 2012 (Monthly Report)

Alarm Type A B C D E F G H

\ Total 34,295 24,683 16,759 24,601 3,817 25,654 23,422 2,747

\ Per Week 7,744 5,574 3,784 5,555 862 5,793 5,289 620

\ Change 324 (68) 51 (518) (180) 303 343 (237)

\ Per Hour 46 33 23 33 5 34 31 4

\ Per Hour Change 2 (0) 0 (3) (1) 2 2 (1)

HILO 25,390 18,543 14,529 19,455 995 19,629 18,660 798

COS 3,570 2,455 651 2,442 1,148 4,254 3,070 564

RTUCOMM 359 1,313 134 682 226 385 484 228

OPWARNTI 53 563 235 276 104 73 73

OPWARNVO 553 113 237 61 33 13

OPTRKCOM 21 487 204 161 52 43 66

COMMAND 1,820 248 48 64 16 104 236 38

RANGE 234 8 2 4

* Restricted * 19 108 101 215 18 210 117 9

WARNTIME 604 100 665 2 350

TARGET 976 47 454 627 13 627 568 134

STALE 13 34 172

XXXXX_MB 6 10 65 112 1 58 42 8

ACCRESET 6 9 65 112 1 58 42 4

NODE 496

BATMTR 13 4 5

Going Forward

Documentation

And

Rationalization

Auditing

Compliance

And

Maintenance

17 Alarm Management - Tactics and Analysis

• Establish Team (Dedicate AM Champion)

• Get Others Involved (I&E Personnel, Controllers)

• Establish Baselines / Review Reports

• Analyze / Interpret Alarm Behavior (Dig Deeper)

• Implement Strategies (Uses Pilot Tests)

• Measure Results (Know What to Measure / Standardize Strategies When Appropriate)

• Report / Share Successes with Team

• Focus on the Operation, not just the Alarm!

Summary

18 Alarm Management – Tactics and Analysis

Thank You!

19