schneider electric scada global support provides troubleshooting and technical tips

SCADA & MES Global Support presents troubleshooting and technical tips

Schneider Electric - Industry - O&O Software Activity - SCADA Troubleshooting Tips Copyright © 2012 2

Introduction

Presenters:

L. Duguid – Senior Global Support Engineer● Top Five Issues raised with Support (SCADA)

T. Montmayeur – Senior Global Support Engineer● Best Practices for Upgrading Vijeo Citect / CitectSCADA

SCADA & MES Global Support Portal:

www.citect.schneider-electric.com


Top Five Issues Raised with SupportPresented by L. Duguid

1. Driver Debugging

2. Web Client / Server Issues

3. Cicode Debugging

4. Clustering

5. Runtime Crashes

Schneider Electric SCADA & MES

http://linkedin.com/groups?gid=4412366


Driver Debugging

Symptoms:

Intermittent Communications, #COM, # Graphics, Errors in log files

Diagnosis:

• Check Boards, Ports, I/O Device forms as per documentationNote: Each I/O Device should use its own port

• Upgrade Driver (DriverWeb for documentation and driver releases)

• Check Syslog.dat for errors check against the Knowledge Base

2012-09-27 09:39:21.874 +10:00 Error: Channel not yet initialized2012-09-27 09:39:21.874 +10:00 CINIT 0017 PORT1_BOARD4 Melter_CLX Error_in_CMD-2 162012-09-27 09:39:21.874 +10:00 Generic 000023 Driver 536870937 (0x20000019)2012-09-27 09:39:21.877 +10:00 Error: Channel not yet initialized

• Check Syslog.dat for errors check against the Knowledge Base


Driver Debugging

•Enable Driver debugging – best to delete logs first

Older drivers:[MELSEC, MODNET]DebugStr = * ALL

New Drivers:[OPC, DNPr, ABCLX]DebugCategory=ALLDebugLevel=ALLLogFileSize=50000

Search syslog for ‘Error’ and again check Hex number in Knowledge Base

• Test with a simple one tag, one I/O Device project

• Send log files to support with SOE

• Check variable tag addresses against documentation


Web Client / Server Issues

Symptoms:

#COM, # Graphics, software protection failure on Webclient, changes not propagating

Diagnosis:

• Check configuration as per documentation

Q6268 – V7.20 (Windows 7, Server 2008) – has lots of debugging tipsQ4700 – prior to V7.2 (earlier OS)Note: There are specific settings for Server 2003

• Ensure Service Pack matches between WebServer and Citect Server

• Delete the %tmp% directory

• Check Internet Explorer settings as per Q6268 and Q4464

• Test with Example project – is the issue specific to your project or Citect in general?

• Check the ports are open (Q4946)

•Test Web Client on Server machine first

http://www.citect-kb.schneider-electric.com/ScadaKB/KnowledgebaseArticle6268.aspx?






Cicode Debugging

Symptoms:

Compile Errors, Function help, Cicode Debugging

Diagnosis:

• Read the help

• Compiler will pick up deprecated functions or invalid function calls

• Debug Cicode using break points, stack window, F11

• Ensure Cicode Editor is using correct process [Debug]CodeDebug = Client

• Does running in Single process help? What process should the code be called from?


Implementing Clustering

Symptoms:

#Com, # Graphics

Diagnosis:

Introducing a second cluster will require the use of cluster context in your projects.

• Assign cluster context by either adding prefix to tags (cluster.tag), assign to page in page properties, or via PageDisplay Cicode function on page call.

• Check Cluster Connections Setup in Computer Setup Wizard to ensure client can see all required clusters.

• Accumulators should be assigned to Cluster and use Cluster prefix on tags.

• Connecting via OPC and CtAPI to runtime tags will require prefix.

• V7.30 to resolve tag where possible


Debugging CrashesSymptoms:

Runtime Manager reports process has stopped responding and exception report is generated.

Diagnosis:

• Exception’s stored in Logs directory, should be prompted to save.

• Exception contains many files, Debug.log is most useful to support.

Exception 0xc0000005 occurred at location 0x74149084 at time Thu Jul 26 21:33:08 2012

(741c0000) - (7425afff) v8.0.50727.4940 C:\Windows\WinSxS\x86_microsoft.........8cc06a442b34fc\MSVCR80.dll

• Debug.log can contain multiple crashes.

• Crash locations can be random, harder to fault find.

• Reproduction steps are very important.

• Check for Events or Cicode being executed, change of shift, operators login, any patterns.


Debugging Crashes• Debug Diagnostics should be used if no crash log is generated (Q5387).

• Ensure latest Service Pack and Drivers are installed

• Running in multi-process a great indication as to where the issues is.

• I/O Server CrashUpgrade Drivers, Switch devices to memory mode, Disable devices that are not operational

• Alarm Server CrashCheck alarm formatting in Alarm Categories, Delete Alarm save files

• Report Server CrashDebug Cicode used in Report Format file by copying to Cicode Editor, Verify Output device configuration and file

• Trend Server CrashRun Trend File Clash Utility from website (Toolbox) to verify trend configuration ‘clashes’, Read trend files using Trend Reader (Toolbox), Delete trend files.

• Support use Bugzilla to search for problems or raise issue with developers for more involved crash analysis (user.dmp analysis).

http://www.citect-kb.schneider-electric.com/ScadaKB/KnowledgebaseArticle5387.aspx


Get your cases resolved faster…

• Provide a simple test project that demonstrates the issue

• Report the issue with a clear and concise definition of the issue

• Unless you are sure, faster to let us decide where the problem is

• Tell us what you know, and what you think you know

• Provide concise information where possible (rather than “it doesn’t work”)

• Divide and Conquer – what do you know, try to refine it through trial and error

• Try and use standard terminology (Web Client, IO Server, IO Device, etc.)

• What is the history of the issue (What changed? What have you tried?)

• Keep issues separate with multiple CSR’s

• What are you trying to achieve, let us know from the outset

• Shift times are 7am-3pm, 3pm-11pm, 11pm-7am (EST), there is a staff change over between shifts. While we endeavour to provide continuous coverage for open cases if you prefer to speak to your specific case Engineer be sure to call within the corresponding shift times


Best Practices for UpgradingPresented by T. Montmayeur

1. Standard process for standalone systems

2. Upgrade on a redundant production system

3. Useful Tips

4. Upgrade tools

5. Potential issues and how to troubleshoot them


Best Practices for Upgrading●Why upgrade?

● New features and enhancements● Performance and product capabilities● Operating System compliance

Old trend style template against Process Analyst view..


Best Practices for UpgradingStandard Process

*This is the minimal upgrade process

Paramount

> Backup Project

> License Upgrade

> Test environment

> Rollout Process

In this presentation, the initial version is 7.0 spk4 and it is meant to be upgraded to version 7.30.

1.Backup V7.0 project2.Uninstall SCADA V7.03.Install SCADA V7.304.Restore your V7.0 project into SCADA V7.30


I want to upgrade to get the latest features. But I don’t want any downtime and want to keep all my historical data.


Best Practices for UpgradingRedundant system on Production

Primary

Client Client Client Client

StandbyExample Configuration

• Primary Server• Standby Server• 4 Control Clients


Best Practices for UpgradingRedundant system on Production

1. Upgrade the keys2. Set up a development environment and proceed to testing

• Test your project in memory mode• If possible test on actual device

3. Production environment:• On both servers: [LAN]AllowLegacyConnections = 1• Shutdown CitectSCADA on the standby server.• Upgrade CitectSCADA on this server according to the standard upgrade

procedure. • Restart you standby server • Perform testing• Shutdown the primary• make sure the standby takes over and perform testing again• Once standby is running fine Upgrade Citect on the primary server

according to the standard upgrade procedure. • Restart the primary server and check your hardware alarms to make sure

you don’t have any connection failure. • Upgrade your clients one by one.


Best Practices for UpgradingUseful Tips

1. History Data• Back up trend files • Backup alarm files ALMSAV.DAT and

ALMINDEXSAVE.DAT

If upgrading hardware as well:

Trends:• Create the same file hierarchy on the new system• Place the file in the same folders

Alarms:• Run the computer setup wizard and edit Alarm paths

2. Additional files to backup, when upgrading hardware

• Data folder• Report files (User defined folder)• [Ctedit]<Pathname> check in citect.ini


Best Practices for UpgradingUpgrade Tools

1. Environment manager

Development EnvironmentMultiple versionsMultiple projects backup and restore

2. Help• What’s new in CitectSCADA• Upgrading to CitectSCADA

3. Knowledgebasehttp://www.citect-kb.schneider-electric.com/ScadaKB/KnowledgebaseCategory32.aspx

http://www.citect-kb.schneider-electric.com/ScadaKB/KnowledgebaseCategory32.aspx


Best Practices for UpgradingPotential issues and how to troubleshoot - 1

Obsolete functionsHelp: Getting Started > What's New in v7.x > What's New in CitectSCADA 7.20 >

Cicode Functions

Operating Systems (OS) compatibility

• SCADAhttp://www.citect.schneider-electric.com/documents/frk/SCADA/SCADA-Compatibility-Matrix.pdf

• DriversNot all of 150+ drivers have been fully tested on latest OS. If not compatible, own

testing required.Once tested OS check can be disabled:[<DriverName>]OverrideOSProtection

http://www.citect.schneider-electric.com/documents/frk/SCADA/SCADA-Compatibility-Matrix.pdf



Slow system after upgrade

Hardware & Software. Refer to installation guide

Ex: 7.20 Installation Guide, System Requirements



DowntimeRedundant system is necessary to achieve a “No Downtime” Upgrade.

Upgrading keys

• Make sure authcode is correct• Make sure the key is showing correct version• Upgrade CiuSafe• Upgrade Sentinel Driver

Further issues?




Best Practices for UpgradingUseful Links

Knowledge base – Upgrade related articleshttp://www.citect-kb.schneider-electric.com/ScadaKB/KnowledgebaseCategory32.aspx

Installation Guide - Installation Files (DVD / Zip file)

Help Getting Started > What's New in CitectSCADAV7.x Getting Started > Upgrading to CitectSCADA V7.20

Website - Upgrading SCADA guidelineshttp://www.citect.schneider-electric.com/index.php?option=com_content&view=article&id=1553&Itemid=1540

- Authcode Generatorhttp://www.citect.schneider-electric.com/index.php?option=com_content&view=article&id=1592&Itemid=942

Driver webhttp://www.citect.schneider-electric.com/forum/forumdisplay.php?f=2

schneider electric scada global support provides troubleshooting and technical tips

Documents

driver releases check

check boards

check configuration

initialized check

citect server

support scadat

software protection

cicode editor