© 2012 Invensys. All Rights Reserved. The names, logos, and taglines identifying the products and services of Invensys are proprietary marks of Invensys or its subsidiaries. All third party trademarks and service marks are the proprietary marks of their respective owners.

WW TSS-05InBatch Advanced Troubleshooting

Matt Mummert Shri Mariyala

October, 2012

Slide 4

Project Start.Best Practices.

Batch systems are often considered mission critical in the production process; therefore, batch systems need to be designed so that the process integrity is ensured even during unexpected external events.A batch management solution, facilitating the batch execution on physical process equipment, expands across multiple layers of the system, including (but not limited to):

Control system infrastructure Network infrastructure Computer system infrastructure Batch Management software User interaction

Each one of these layers must be thoroughly analyzed and considered as a critical part of the batch management solution. It is vital that, after the critical variables surrounding the batch execution are identified, the system is designed to address their needs at the right level.(From InBatch R&D WHITE PAPER - Best Practices and Considerations for Designing Reliable InBatch Solutions)

Slide 5

One of the most important things you can do on a project is to make sure that the system functional specification completely describes the desired operation of the system and no more.(From InBatch deployment guide)

Project Start.Best Practices.

Slide 6

Best Pratices – What Do We Take Away?

- InBatch Deployment Guide

- InBatch R&D WHITE PAPER - Best Practices and Considerations for Designing Reliable InBatch Solutions

Slide 7

Advanced Troubleshooting - Topics

• InBatch Architecture

• Batch Equipment

• Batch Phase Logic

• IBCli/DAServer

• Batch IBMX

• Batch Warm Restart Files

• BatchMngr Parameters

Slide 8

Advanced Troubleshooting - Topics

• BatchGuiConfig Ctrl

• Batch Redundancy

• Batch History

• Reports

• System Monitoring

• Log Flags

Slide 9

InBatch Architecture 1

Batch History Information Server

InBatch Dev Client

Batch Server + DAServer

PLCs

Access Reports Remotely

InBatch Runtime Clients/Touch

Slide 10

InBatch Integrate with System Platform

Batch History Information Server

InBatch Dev Client

Batch Server + DAServer

PLCs

Access Reports Remotely

InBatch Runtime Clients/Touch

• AOr

GRAOS

Slide 11

PHASE 4

PHASE 5

PHASE 6

Phases modeled within System Platform

Extremely flexible and powerful batch management capabilities

Extremely flexible and powerful platform and plant model

InBatch Server

PHASE 1

PHASE 2

PHASE 3

Phases modeled within Control System

MX communication

SystemPlatform

Device Integration Device

System Platform Integration – How ?

Slide 12

Batch Equipment Analysis

● Batch started but no activity, why?

● What is equipment status?

● How to track and identify which equipment in use?

● What is Unit, Connection & Segment Status?

● How to update equipment status (TagView)

Slide 13

Equipment Definition - ModelEditor

Slide 14

Units/Connections Status - Batch Display

Slide 15

Segments Overview

Slide 16

Segments Linked to Connections

Slide 17

Equipment Status – TagView

Slide 18

Where is Tagview.exe?

Slide 19

Equipment Status – What Do We Take Away?

• Unit Status values must be consistent - Process Model Vs USTAT tags

• USTAT tags are linked to the correct addresses in PLC

• Blank or incorrect values for USTAT leads to “waiting for available equipment” state

• Allocating a connection allocates all segments of that connection

• All connections that contain segment allocated by another connection are ‘busy’

• Technote 25 – Using Segments to define Model Limitations in InBatch.

Slide 20

Monitor Phase Status – Batch Display

Slide 21

Monitor Phase Status – Phase Logic Tool

Slide 22

Where is InBatch Phase Logic Tool?

Slide 23

Phase Logic Tool – Add to Env. Display

Slide 24

Phase Logic Tool-Add to Env. Display

Slide 25

Monitor Phase Status – TagView

Slide 26

Monitor Phase Status – WWClient

Note: Phase status values using WWClient

Slide 27

Phase Monitor – What Do We Take Away?

• Phase Control/Status bits linked to the control system must be unique and valid addresses in PLC. Duplicate items cause issues

• Phase must always be in ready state prior to the beginning of the phase execution

• At any point in time, only status is present.

• Technotes as follows:

Slide 28

Phase Monitor – What Do We Take Away?

• TN 555 – Active batch behavior when IBCli looses communications to IOServer

• TN 504 - Phase control bit behavior

• TN 312 – InControl phase logic for InBatch

• TN 23 – Developing Allen Bradley Phase logic for InBatch

Slide 29

InBatch IBCli Parameters

● Response Time - Time after which an IBCli request to the IOServer will timeout. Default 15 seconds

● Ping Time - Interval at which IBCli will ping the IOServer to detect a connection loss. Default 2 seconds. To enable debug for IBCli, set the Ping Time to “-999”

● Advise All - Performs an Advise All on initialization

● Disable Timestamp – Disable the end of phase timestamp feature for this topic.

Slide 30

IBCli debug – Log Messages with PingTime = -999

Reading a value from DAServer/IADAS:

2011/07/20 16:20:13.399/-03B4:14A4/IBCli /...(ProcessUpdate) Update

for [G2_V9072-V9074-CS-DONE] of [1] 2011/07/20 16:20:13.399/-

03B4:14A4/IBCli /...(ProcessUpdate) Time Stamp for [G2_V9072-V9074-

CS-DONE] of [2011/07/20 20:20:11] Quality [1c0]

Slide 31

IBCli debug – Log Messages with PingTime = -999

InBatch writing value to DAServer/IADAS

2011/07/20 16:20:13.899/-03B4:14A4/IBCli /(tt) wwCliPoke - request:

[22869912]  tag: <G2_V9072-V9074-CS-RESET> Value:[1]

2011/07/20 16:20:13.899/-03B4:14A4/IBCli  /...(tt) Adding tag to request -

request: [22869912]  tag: <G2_V9072-V9074-CS-RESET> req type: [1]

2011/07/20 16:20:13.899/-03B4:14A4/IBCli  /...(ProcessUpdate) Update for

[G2_V9072-V9074-CS-RESET] of [1] 2011/07/20 16:20:13.899/-

Slide 32

Where are IBCli Parameters?

Slide 33

InBatch 9.5 – Trace IBCli via Logger

Slide 34

InBatch - Environment Editor

• Any updates to Environment Editor requires Update Environment to affect changes, which means, stop runtime services, which means, it costs production down time.

Slide 35

DAServer/IADAS Parameters

Slide 36

DAServer/IADAS – Log Flags

Slide 37

InBatch IBMX - Add Application in Env. Editor

Slide 38

InBatch IBMX – Configure in TagLinker

Slide 39

InBatch IBMX – Launch Galaxy Browser from TagLinker

Slide 40

InBatch IBMX – Galaxy Browser

Slide 41

IBMX with Verbose Mode-Log Messages

Slide 42

InBatch IBMX - ObjectViewer

Slide 43

InBatch IBMX – What Do We Take Away?

● Platform must be deployed to enable IBMX

● IDE must be installed in order to browse Galaxy browser inside of TagLinker

● Requires IDE and Runtime license for Platform

● Use TagView, Phase Logic, Batch Display and ObjectViewer to track Phase Control/Status tags

● InBatch and ArchestrA AppServer Communication Recommendations using IBMX – White Paper (Tech Article)

Slide 44

Batch Warm Restart

● Warm Restart files - If there is no warm restart folder, batchmngr doesn’t startup

● .batchwrdebug – Creating a .batchwrdebug folder from command line enables the debug for warm restart information. This folder created under x:\>program files\wonderware\InBatch\Cfg\Config_a\.batchwrdebug

Slide 45

Batch Parameters

Slide 46

BatchMngr Parameters

● Param Timeout - Time in seconds, of all attempts to read a parameter tag value before timeout failure. Default value set to 30 seconds. If you see any parameter read timeouts in the logger tune this setting

● Phase Batch Status – Puts all phases on hold when batch is on hold – Add this setting if all phases needs to be on hold when batch is on hold

● Disconnect Clients – Disconnect all deadlock (misbehaving) clients – Add this setting if any clients misbehaving, otherwise it will cause negative impact to the BatchMngr

Slide 47

InBatch ActiveX Ctrl - BatchGuiConfig

Slide 48

BatchGuiConfig – Properties

● Note: MessageBoxes has to be disabled from popup, otherwise it will cause the view to not respond until message is acked.

Slide 49

BatchGuiConfig – Events

● Terminate/Initialize the control using Startup/Shutdown events of Batch Server accordingly.

Slide 50

SFCCtrl and OcxBatchCtrl

● Terminate/Initialize these controls inside of view using Shutdown/Startup events of BatchServer

Slide 51

InBatch ActiveX Controls – What Do We Take Away?

● View locks up due to InBatch ActiveX Controls not terminated gracefully when connection lost to BatchServer (Server shutdown/loss network connection)

● Technote 364 – How to configure BatchGuiConfig ActiveX Ctrl for Redundancy Client Switch-Over. This technote is good for Standalone Server also.

● Starting with InBatch 9.5, use Technote 797 “Configuring InBatch Runtime Client Communication Timeouts.” These timeouts are configured in registry and technote covers details on configuration.

Slide 52

InBatch Redundancy● Configuration Utility

Slide 53

InBatch Redundancy (cont.)

● RedCfg File

● LMHost File

● Messages in the logger

Slide 54

InBatch Redundancy (cont.)● Contents of RedCfg file

Line 1: Remote machine name. Line 2: (blank, not used). Line 3: The path to the remote machine’s config_A directory. Line 4: (blank, not used). Line 5: The local hostname for the primary network (must not be a machine name). Line 6: The local hostname for the secondary/backup network, if one exists. Line 7: The remote hostname for the primary network (must not be a machine name). Line 8: The remote hostname for the secondary/backup network, if one exists.

Note: The file content must end with a carriage return. Just press the Enter key after the last entry and then save the file. The contents of the RedCfg file must exactly match the entries in the Hosts or LMHosts files.

Slide 55

InBatch Redundancy (cont.)● Contents of LMHost file

100.100.100.1 BATCHMASTER #PRE100.100.100.2 BATCHBACKUP #PRE100.100.100.10 INTOUCH1 #PRE100.100.100.11 INTOUCH2 #PRE100.100.100.12 INTOUCH3 #PRE100.100.100.1 BATCHMAST #PRE100.100.100.2 BATCHBACK #PRE200.200.200.1 BACKUP1 #PRE200.200.200.2 BACKUP2 #PRE

Slide 56

InBatch Redundancy (cont.)● Technotes:

● 139 – Installing and configuring Redundancy Option for InBatch

● 302 – How to create host file and redcfg file for InBatch 8.0 redundancy setup which is still valid for later versions also

● 881 – Configuring InBatch Redundant Servers to Sync in a WorkGroup Environment

Slide 57

Batch History & Reports (cont.) ● ReportQReader

● Custom Reports – Tech Note 767

Slide 58

Log Flags for 9.5 ● XES Diags – all service apps

● App Load: %

● Max Delay: s

● Trace – HistQReader

● E.g. transactions and admin thread activity

● Trace – IBMX / IBCLI

● E.g. read, write, data updates, quality, ack/nak, etc

● Replaces –d (debug) and –p-999 (PingTime) options

Slide 59

Log Flags for 9.5 (cont.)● Trace – OcxBatch ActiveX / Batch Display / Batch Scheduler

● Init, term, heartbeat, focus

● Trace – Recipe Editor

● XML Import and Export

Slide 60

Log Flags for 9.5 (cont.)Security and Active Directory Trace Log Options

● [HKLM\SOFTWARE\Wonderware\InBatch\ArchestraSec\Debug]

● [HKLM\SOFTWARE\Wonderware\InBatch\NativeSec\Debug]

Slide 61

Log Flags for 9.5 (cont.)Batch WR File Log Option

● Create folder before startup. Logs to folder. Send to Tech Support.

Slide 62

Diagnostic Tools

● Operating System Log Files - Application/System/Security Event Log files

● ArchestrA Log files

● Windows Debugging tool

● Phase Logic Tool

● WWClient

Slide 63

Diagnostic Tools (cont.)Microsoft WinDbg - used by Tech Support to examine DMP files.

Slide 64

Diagnostic Tools (cont.)Technotes:

Slide 65

Questions?

THANK YOU