tacoma power’s compaq server and ethernet rtu performance testing presented by joe orth...
TRANSCRIPT
Tacoma Power’s Compaq Server and Ethernet RTU Performance Testing
Presented By Joe Orth253/502-8785
EMS Group MeetingPresented April 3, 2001
Tacoma Overview• Municipal utility serving multiple jurisdictions
in Pierce County governed by Utility Board that is approximately double the size of the Tacoma City Limits.
• The City operates Tacoma Public Utilities of which Tacoma Power is an operating division (along with Tacoma Water and Tacoma Rail).
• Within Tacoma Power there are 5 business units: T&D, Power Management, Energy Services, Click!Network and Generation
Tacoma Power Overview• The City of Tacoma, Washington is
about 30 miles south of Seattle in the South Puget Sound region
• Approximately 150,000 customers
• System hour peak 2/3/89 of 1,212 MW
• Currently Struggling to Overcome West Coast Power Market, Government Coup and 50 Year Drought Events
Tacoma Power Electrical System
• In Tacoma Power’s control area there are 4 major substations, 2 of which have 230 kV inter-ties with the BPA.
• Major substations are interconnected by a combination of 110 kV and 230 kV transmission lines.
• A 110 kV sub-transmission system distributes power from the major substations to Tacoma Power’s unit substations.
Tacoma Power Electrical System
• The typical unit substation has one 25 MVA 110 kV – 12kV transformer with four 12 kV, 800 Amp feeders which are often loaded to 600 Amps during peak hours.
• 50+ unit substations.• 2040+ miles transmission and distribution lines . • Average feeder length 2 to 3 miles.• Click!Network has 600+ Miles of Fiber and Coax
Installed in City Limits
ENERGY CONTROL CENTER and CLICK NETWORK OPERATION CENTER
SCADA System Overview• New RTUs in 80+ locations. • LAN and WAN (Nortel J-Mux) installation
at about half of these sites. Provides 10 MB dedicated SCADA LAN acting as flat LAN. Other sites use various communications options.
• PC Based system communicating with RTUs using DNP over UDP/IP during system transition.
• Substation and Distribution Automation.
exist ing J MUXplanned 1999 order
proposed Loveland nodes
highlight indicates fi ber installed and available
Legend:
O C- 1A dmin Ring
A utochem* in operat ion
N EO C- 1
* in operat ion
H ylebos
I ndian H ill Browns Point
Cedar
A dmin* in operat ion
East F* in operat ion
S impson* in operat ion
Pioneer* in operat ion
T idefl ats* in operat ion
N isqually* in operat ion
H illtop
A lexander* in operat ion
O C- 1S E Ring
Port land
Crandel
Polk
Crof tCollins
Lidf ord
CowlitzO C- 1
Roosevelt
Cresent
Graham
Elk Plain
Loveland
Fredr ickson
W apato
O C- 3Backbone
CowlitzO C- 3
Pear lO C- 3
N EO C- 3
ECCO C- 3
O C- 1S W RingCuster
Menlo
Mcneil
Plaza
S tone
Clover Park
Clement
S WO C- 1
A tlas
O rchard
Br idgeport
Flett
O C- 1N W Ring
Gove
Defi anceH ighland
Pear lO C- 1
Union
H awthorne
H uson
S tadium
Cushman
O ld T own
S WO C- 3
SCADA SYSTEM ARCHITECTUREPASS Network
Design
SCADA System Overview• New EMS.
• Desire to integrate RTUs on LAN directly with new EMS – signed contract with ABB September 2000.
• Preliminary testing of DNP Serial and Ethernet Data Acquisition Sub-system completed Feb. 2001 using Master Blaster RTU.
"Test 1" High Activity State conditions: - Exception Analogs - 15,000 changes per 6 sec - AGC Analogs - 10,000 changes per 2 sec - Analog alarms - 300 alarms per 6 sec - Status Alarms - 340 Alarms per 10 sec, plus alarm burst 8 DI per sec - Automatic acknowledging of alarms via scripts - Alarms saved to disk-based event file (Oracle)
Master Blaster Overview
Pass/Fail Criteria
•No CPU Utilization 5 Second Average Over 60%•LAN Utilization Less Than 20% Excluding One 3
Second Spike•.5 Second Max Delay Responding to Operator Request•Open One New Graphic In 2 Secs 95% of Time•Verify Screens Update Every 2 Seconds•Alarm Summary Page Shown on Each Console•Verify Vendor DNP Profile Correct•Verify Large Data Packets Processed Properly
Laptop withNet XRay
RDAS 1Force CPUSimpac PCI
RDAS 2Force CPUSimpac PCI
RAS 2ES 40 Historian
DS 20EXP1000
Workstations
RAS 1GS 60E
100MB LAN
RTU1-64
GEH D200Two, 7 Node CCUs
RTU65-98
Single, Doubleand
Triple Headed
Note: Test 2 used ES40as RAS1
9600 BaudSerial DNP
Test 1 and 2 May 2000
RDAS 1Force CPUSimpac PCI
RDAS 2Force CPUSimpac PCI
DS 20EWorkstation
XP1000Workstations
RAS 1GS 60E
100MB LAN
RTU1-8
GEH D20 + D200RTUs
RTU9-64
Each 56KBitSerial DNP
Test 3 J une 2000
Each 9600Baud
Serial DNP RTU65-98
RDAS 1Force CPUSimpac PCI
DNP/ UDP/ I P
RDAS 2Force CPUSimpac PCI
DS 20EWorkstations
RAS 1GS 60E
100MB LAN
GEH D20 + D200RTUs
Test 4 J anuary 2001
10MB LAN
38 Diff erentDNP RTUs
on this LAN
Each of 58 DNPSerial Channel
56KBits
Tests 1 and 2 – May 2000
RTU Communications for Test 1 and 2•98 Serial RTUs•DNP 3.0 Protocol•All Operating at 9600 Baud•Each RTU has 408 Status and 408 Analogs•Approximate Total 40,000 DIs and AIs•Could Not Achieve 2 second Updates @ 9600
ES-40 versus GS-60E
0
10
20
30
40
50
60
70
80
90
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 701 SEC DURATIONS
CP
U ID
LE T
IME
(Per
cent
)
ES-40
GS-60E
RDAS-1 vs RDAS-2 (ES-40 RAS)
0
10
20
30
40
50
60
70
80
90
100
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 651 Sec. Durations
CP
U Id
le T
ime
(Per
cent
)
RDAS-1 (ES-40) RDAS-2 (ES-40)
RDAS-1 vs RDAS-2 (GS-60E RAS)
0
10
20
30
40
50
60
70
80
90
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 631 Sec. Durations
CP
U Id
le T
ime
(Per
cent
)
RDAS-1 (GS-60E) RDAS-2 (GS-60E)
ES-40 CPU % Idle Time
0
10
20
30
40
50
60
70
80
90
0 10 20 30 40 50 60 70Seconds
CP
U Id
le T
ime
(Per
cent
)
1 Sec Avg 5 Sec Avg
RDAS1 CPU % Idle w/ ES40 RAS
0
10
20
30
40
50
60
70
80
90
0 10 20 30 40 50 60Seconds
CP
U Id
le T
ime
(Per
cent
)
1 Sec Avg 5 Sec Avg
RDAS 2 % CPU Idle Time w/ ES40 RAS
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70Seconds
CP
U Id
le T
ime
(Per
cent
)
1 Sec Avg 5 Sec Avg
GS60E RAS % CPU Idle Time
0
10
20
30
40
50
60
70
80
90
0 10 20 30 40 50 60 70 80Seconds
CP
U Id
le T
ime
(Per
cent
)
1 Sec Avg 5 Sec Avg
RDAS1 CPU % Idle Time w/ GS60 RAS
0
10
20
30
40
50
60
70
80
90
0 10 20 30 40 50 60 70Seconds
CP
U Id
le T
ime
(Per
cent
)
1 Sec Avg 5 Sec Avg
RDAS 2 % CPU Idle Time w/ GS60 RAS
0
10
20
30
40
50
60
70
80
90
0 10 20 30 40 50 60 70Seconds
CP
U Id
le T
ime
(Per
cent
)
1 Sec Avg 5 Sec Avg
Test 1 XP1000 Workstations Performance Data
25
35
45
0 0 00
10
20
30
40
50
60
70
80
90
100
XP1000 - 3 Headed XP1000 - 2 Headed XP1000 - 1 Headed
CP
U Id
le T
ime
(Per
cent
)
Avg. CPU idle time Min CPU idle time
Test 1 ABB Hardware Performance (1 Sec. Averages)
0
10
20
30
40
50
60
70
80
90
100
ES-40 RDAS-1 (&ES-40) RDAS-2 (&ES-40) GS-60E RDAS-1 (&GS-60E)
RDAS-2 (&GS-60E)
CP
U Id
le T
ime
(Per
cent
)
Avg. CPU idle time
Median CPU idle time
Max CPU idle time
Min CPU idle time
Test 1 ABB Hardware Performance (5 Sec. Averages)
0
10
20
30
40
50
60
70
80
90
ES-40 RDAS-1 (&ES-40) RDAS-2 (&ES-40) GS-60E RDAS-1 (&GS-60E)
RDAS-2 (&GS-60E)
CP
U Id
le T
ime
(Per
cent
)
Avg. CPU idle time
Median CPU idle time
Max CPU idle time
Min CPU idle time
No. Description1 Test Scenario and Full System2 ES 40 as Primary RAS w/o Backup RAS3 GS 60 as Primary RAS w/o Backup RAS
Test 1 & 2 LAN Data Scenarios
Average LAN Statistics Values
3 2 2
7
4 4
21
7 6
0
5
10
15
20
25
1 2 3
LAN Scenarios
Per
cent
100
Mbp
s LA
N U
tiliz
atio
n
Minimum
Average
Maximum
Scenario 1: Backup, Primary, and Test ScenarioScenario 2: ES-40, No Backup, and Test Scenario
Scenario 3: GS-60, No Backup, and Test Scenario
Test 3 – June 2000
RTU Communications for Test 3•98 Serial RTUs•DNP 3.0 Protocol•8 Operating at 56 kbit (fills one ICP Card)•Others Operating at 9600 Baud•Each RTU has 408 Status and 408 Analogs•Approximate Total 40,000 DIs and AIs
Single CPU DS20E 3 Head Workstation % CPU Idle Time w/ GS60 RAS
0
10
20
30
40
50
60
70
80
90
0 5 10 15 20 25 30Seconds
CP
U Id
le T
ime
(P
erc
en
t)
1 Sec Avg 5 Sec Avg
RDAS w/ 8 56 kbps on 1 ICP % CPU Idle Time w/ GS60 RAS
0
10
20
30
40
50
60
70
80
90
100
0 5 10 15 20 25 30Seconds
CP
U Id
le T
ime
(P
erc
en
t)
1 Sec Avg 5 Sec Avg
AI 2 Second Data Updates (30 Samples)
30
27
23
29
25
22
0
5
10
15
20
25
30
Free Wheel Scan by RDAS 1 Sec. Scan by RDAS 2 Sec. Scan by RDAS
# o
f U
pd
ate
s O
ut
of
30
Sa
mp
les
AI Update at RDAS
AI Update at DS20E
Test 3 Hardware Performance
0
10
20
30
40
50
60
70
80
90
100
DS20E 3 Headed (1 Sec) DS20E 3 Headed (5 Sec) RDAS (1 Sec) RDAS (5 Sec)
CP
U Id
le T
ime
(Per
cent
)
Avg. CPU idle time
Median CPU idle time
Max CPU idle time
Min CPU idle time
Avg of 10 Second % CPU Idle Time Avgs by Machine Type
68
94
95
95
95
45
87
80
45
0 10 20 30 40 50 60 70 80 90 100
Single CPU DS20E, 3 Headed
Processor 1 GS60E RAS
Processor 2 GS60E RAS
Processor 3 GS60E RAS
Processor 4 GS60E RAS
RDAS 2 - 2 rlogit 46 RTUs @9600
RDAS 2 - 0 rlogit 46 RTUs @9600
RDAS 1 - 42@9600 8 @56000
XP1000, 1 Headed
Machine % CPU Idle Time
Test 4 – JANUARY 2001
RTU Communications for Test 4•85 RTUs•DNP 3.0 Protocol•55 Operating at 56 kbps•30 On 10 MB Ethernet LAN Using UDP/IP•These 30 RTUs Spread Over 2 IPs•Each RTU has 408 Status and 408 Analogs•Approximate Total 34,000 DIs and AIs•Test in January 2001
Integration Story• Due to dedicated, flat Ethernet LAN,
confidence with DNP and other factors, UDP/IP was selected.
• Required TCP/IP for Telnet, etc.
• Require hot-stand-by on EMS FEP
• We had thought that this would be fairly simple and not require too much thought…
Other Issues
• #1 Issue – Vendor Specific!!!
• DNP Confusion
• LAN Document Interpretation
• Unique UDP Port Numbers?
• Unique DNP Addresses?
• How to Hot-Stand-By on FEP?
• Unsolicited messages?
Other Issues
• SOE Management?• Understand your LAN environment.
Goals
•Up to 60 RTUs on LAN•2 Second Updates Most of the Time•Take Advantage of Click!Network Fiber•System Will Not Crash Due to Data Overload•No Hardware Upgrades Needed for 6 Years•No Proprietary Modifications to DNP•Minimize Risk - Test to See If The System Could Perform Before Signing a Contract
What We Learned
•Most XP1000 Workstations replaced with single processor DS20Es•GB LAN for Real-Time LAN•Back-Up Systems Add About 3% of steady state LAN Traffic and 14% peak•GS60E and ES40 Performance Close but GS60E Didn’t Have Deep Performance Valleys of ES40•Need Firmware Changes in RTUs to Have Ethernet Communication & Resolve RTU Latency•Pre-Contract Award Performance Test Paid Off Big Time and Made Lawyer and Board Happy
DNP Serial and Ethernet CPU Data
0
10
20
30
40
50
60
70
80
90
100
DS20E -1
DS20E -3
GS60 RDAS1 RDAS2
%C
PU
Uti
liza
tio
n
Min
Max
Avg.
Add'l Activity Gen. 0% 25% 50%10 Minute Avg. 5% 29% 55%Observed Updates 2 Sec 5 Sec 8 SecMax 10 Sec Avg. 12% N/A N/ACollisions Over 30 min 0.02% N/A N/A
LAN Activity and Robustness
Ethernet Ethernet %BW %BW SerialTest mSec Field mSec Test Field in mSec
RTU Rx Delay 133 54 77.6 62.6 349Rx Length 1 1 0.3 1.2 140Rx to Ack Delay 18 11 10.5 12.9 19Total Transaction Time 172 86 11.6 23.4 507 - 20 mSec delay assumed from Ack to complete next Tx
Ethernet RTU Response Data
Ethernet Bandwidth Utilization Breakdown @ Test
Delay Between Polls11.6%
RTU Reply0.3%
Master to ACK Message
10.5%
RTU Response Time77.6%
Ethernet Bandwidth Utilization Breakdown Based on Field Data
Delay Between Polls23.4%
RTU Reply1.2%
Master to ACK Message
12.9% RTU Response Time62.6%