copyright © 2005, nortel networks nortel networks...

Nortel Networks ConfidentialCopyright © 2005, Nortel Networks

CBM Fault Management for Wireless Networks

Fault management strategyThe Core Billing Manager (CBM) fault management strategy includes the dual functions of Fault Delivery and Test and Diagnostic capabilities.

The core manager component handles many of the fault delivery features.

Tools and utilitiesThe primary fault management tools and utilities are alarms and logs.

AlarmsFor a list and descriptions of all SSPFS alarms, refer to NN10275-909, Succession Fault Management Alarms Reference.

LogsThe Log Delivery application, included as part of the base software platform on the core manager, collects logs generated by the core manager, the computing module on the call server, and other network elements, and delivers them to operational support systems (OSS). For more information on the Log Delivery application and tools, refer to NN-20000-244, CBM Basics for Wireless Networks.

CAUTION Do not attempt to RTS failed hardware. If you experience any core manager hardware failure, do not attempt to return this hardware to service (RTS). Replace the failed hardware with an available spare as soon as possible. Contact your next level of technical support for further analysis and instructions as necessary.

CBM850 Fault Management for Wireless Networks

Nortel Networks Confidential2Copyright © 2005, Nortel Networks

Table 1, SDM/CBM logs matrix for SDM logs and Table 2, SDM/CBM logs matrix for SBA logs provide a matrix between logs applicable to SDM and if they are applicable to CBM.

Table 1 SDM/CBM logs matrix for SDM logs

Log SDM CBM Comments

SDM300 X

SDM301 X

SDM302 X

SDM303 X X

SDM304 X X

SDM306 X X

SDM308 X

SDM309 X

SDM314 X

SDM315 X X

SDM317 X

SDM318 X X

SDM325 X X Specific to P-MSC.

SDM330 X

SDM331 X X

SDM333 X X

SDM336 X

SDM375 X X

SDM500 X

SDM501 X

SDM502 X

NN-20000-250 Standard 01.08 September 2005


SDM503 X

SDM504 X

SDM505 X

SDM550 X X Generated on MTX.

SDM600 X

SDM601 X

SDM602 X

SDM603 X X

SDM604 X X

SDM608 X

SDM609 X

SDM614 X

SDM615 X

SDM616 X

SDM617 X

SDM618 X

SDM619 X X

SDM620 X

SDM621 X

SDM622 X X

SDM625 X Specific to P-MSC.

SDM630 X

SDM632 X

Table 1 SDM/CBM logs matrix for SDM logs (Continued)




SDM636 X Introduced in SN07.

SDM650 X X Generated in MTX.

SDM700 X X

SDM739 X New in SDM20 (MTX13)

SDMO375 X X

SPFS310 X

SPFS320 X

SPFS330 X

SPFS350 X

SPFS400 X

Table 2 SDM/CBM logs matrix for SBA logs

SBA Log SDM CBM Comments

SDMB300 X X

SDMB310 X X

SDMB315 X X

SDMB316 X X

SDMB320 X X

SDMB321 X X

SDMB330 X X

SDMB350 X X

SDMB355 X X

SDMB360 X X

SDMB365 X X

Table 1 SDM/CBM logs matrix for SDM logs (Continued)




SDMB366 X X

SDMB367 X X

SDMB370 X X Not applicable to CDMA.

SDMB375 X X

SDMB380 X X

SDMB390 X X

SDMB400 X X

SDMB530 X X

SDMB531 X X

SDMB550 X X

SDMB600 X X

SDMB610 X X

SDMB615 X X

SDMB620 X X

SDMB621 X X

SDMB625 X X

SDMB650 X X

SDMB655 X X

SDMB660 X X



SDMB675 X X

SDMB680 X X





Log Delivery proceduresThe following table lists tasks and procedures associated with the Log Delivery system and tools. Use this table to determine which procedure to use to complete a specific log-related task.

SDMB820 X X

SDMB690 X X Introduced in SN07.

SDMB691 X X Introduced in SN07.



Table 3 Log Delivery procedures

If you want to Use procedure

access log devices from a remote location

“Accessing TCP and TCP-IN log devices from a remote location” in the Fault Management section

add a TCP, TCP-IN, or file device “Configuring a CBM for log delivery” in the Configuration Management document

modify parameters for an existing device “Modifying a log device using logroute” in the Configuration Management document

specify logs to be delivered to a specific device

• for a new device, use “Configuring a CBM for log delivery” in the Configuration Management document

• for an existing device, use “Modifying a log device using logroute” in the Configuration Management document

delete a log device “Deleting a device using logroute” in the Configuration Management document

define the set of logs sent from the CM “Specifying the logs delivered from the CM to the CBM” in the Configuration Management document



SDM logsCore manager events are recorded internally to the core manager in a series of log reports.

Core manager log reports fall into two categories: trouble (TBL) logs, and information (INFO) logs.

• Trouble logs provide an indication of some type of fault for which corrective action can be taken. These logs are generated for connectivity failures, system resource problems, and application software and hardware failures. Each of these trouble conditions corresponds to an alarm on the alarm banner of the core manager maintenance interface.

• Information logs provide information about events that do not normally require corrective action. These logs are generated for system restarts, non-service-affecting state changes, and for events that clear TBL logs.

change the log delivery global parameters (applicable to all devices)

“Configuring the Log Delivery global parameters” in the Configuration Management document

configure the Generic Data Delivery (GDD) parameter

“Configuring GDD parameter using logroute” in the Configuration Management document

display log records “Retrieving and viewing log records on page 57”

install log delivery service “Installing the Log Delivery application” in the Configuration Management document

install the logreceiver tool “Installing the logreceiver tool on a client workstation” in the Configuration Management document

view logs “Retrieving and viewing log records on page 57”

store logs in a file “Retrieving and viewing log records on page 57”

troubleshoot log delivery problems “Troubleshooting Log Delivery problems on a CBM on page 67”

Table 3 Log Delivery procedures

If you want to Use procedure



SDM logs describe events general events related to the operations of the core manager. The following table lists SDM logs.

Table 4 Core manager logs

Log Trigger Action

SDM303 A core manager application or process has failed more than three times in a day, or has declared itself to be in trouble.

Users with root permissions can examine the log files in /usr/adm to determine the cause of the process failure. If required, contact your system administrator or Nortel Networks for assistance.

SDM304 The Log Delivery application cannot deliver logs to the specified UNIX file.

Use the Log Delivery online commissioning tool (logroute) to verify the existence and validity of the device name. Refer to the following procedures in for more information:

• “Configuring a CBM for log delivery” in the Configuration Management document

• “Deleting a device using logroute” in the Configuration Management document

If required, contact your system administrator or Nortel Networks for assistance.

SDM306 The Table Access Service application on the core manager has detected that the software load on the Core is incompatible with the software load on the core manager.

Upgrade the CM software to a version that is compatible with the SDM software.

Note: The software on the core manager must not be at a lower release level than the software on the Core.

SDM315 The Table Access Service application on the core manager has detected corruption in the Data Dictionary on the Core.

Contact your next level of support with the information provided in the log. The log information contains essential information for identifying the Data Dictionary type that is corrupt.



SDM318 An operational measurements (OM) report was not generated. (The OM report failed to complete within one report interval.)

Contact Nortel Networks.

SDM325 Indicates a lost connection to a Preside network management component.

None

SDM330 Indicates a communication problem between two mated nodes on a CBM850 HA cluster

Use the description field to determine necessary action.

SDM331 OMD audit deleted files from the OMD storage volume to free up space.

None

SDM333 OMD audit discovers that the OMD storage usage has gone above 60%

Delete the old OM reports from the volume reported by the log. Otherwise, older files will get deleted in the next audit if the usage has gone up to above 70%.

SDM336 No heartbeat response received Use the logs command from the hw level of the cbmmtc display to check for Ethernet link faults on the CBM. Check on core mapci;mtc;xac level for Ethernet connectivity faults.

SDM375 OMD discovered a problem while performing outbound file transfer and could not ensure that the OM report got transferred downstream.

Contact your next level of support.

SDM603 A fault on a core manager application or process has cleared.

None


Log Trigger Action



SDM604 The Log Delivery Application generates this log when the Core generates logs at a higher rate than can be transferred to the Log Delivery Service and the device buffer on the core is too full to accept more logs.

Increase office parameter PER_OPC_LOGDEV_BUFFER_SIZE to its maximum size of 32,000. (For more information about this parameter, refer to the SuperNode Data Manager Log Report Reference Manual, 297-5051-840.) If you still continue to receive SDM604 logs after you have increased the size of the parameter, or if large numbers of logs are lost, contact Nortel Networks for assistance.

SDM622 The SDM log delivery application generates this log when the file device reaches its maximum size.

Check if you have configured enough space for the file device. If there is a software error causing the increase of logs, contact Nortel Networks for help.

SDM625 Indicates a re-established connection to a Preside network management component.

None

SDM636 Heartbeat alarm cleared None

SDM700 Log report SDM700 reports a Warm, Cold, or Reload restart or a norestartswact on the core

None

SDMO 375

Indicates that OMD discovered a problem while performing an outbound file transfer and could not ensure that the OM report was transferred downstream.



Log Trigger Action



SDMB logsSDMB logs describe events related to the operations of the SuperNode Billing Application (SBA) and the SDM Billing System that resides on the SDMCS 2000 Core Manager. The following table lists SDMB logs.

Table 5 SDM Billing Application (SBA) logs

Log Trigger Action

SDMB300 Memory allocation has failed. Contact your next level of support.

SDMB310 A communication-related problem has occurred.

Determine the reason that the core manager is not communicating with the Core. Determine whether the core manager, the Message switch (MS) and the Frame Transport bus (FBus) are in service (InSv) or in-service trouble (ISTb). If the core manager is InSv or ISTb, return the billing stream to service.

SDMB315 A general software-related problem has occurred.


SDMB316 A billing-related process has been manually “killed”.

Restart the process.

SDMB320 A billing backup-related problem occurred, which affects more than one file.

Ensure that the backup volumes configured for the stream have enough available space.

SDMB321 A billing backup-related problem occurred, which affects one file.

Ensure that the backup volume is not busy or full.

SDMB330 The configuration of a billing stream failed.

Configure the billing stream using the procedure “Configuring a billing stream” in the Accounting document.

SDMB350 An SBA process has reached a death threshold and made a request to restart. A death threshold occurs after a process has died more than 3 times less than 1 minute apart.

SBA will automatically restart. What for logs that indicate that SBA is in normal operation. If the system generates this log more than once, contact your next level of support.



SDMB355 A problem with a billing disk has occurred, which can consist of any one of the following problems:

• Records cannot be written to file (by stream). When this occurs, alarm DSKWR is raised.

• The Record Client/File Manager is unable to write to the disk.

• The disk use is above the critical threshold specified in the MIB in parameter. When this occurs, alarm LODSK is raised.

• The disk use is above the major threshold specified in the MIB in parameter. When this occurs, alarm LODSK is raised.

• The disk use is above the minor threshold specified in the MIB in parameter. When this occurs, alarm LODSK is raised.

• Reached limit for disk space or for the number of files that can reside on the system for a particular stream.

• The SBA cannot close or open a file.

• Flush file failed

• Check the disk space on the core manager. You may need to FTP files or may need to clean up the disk.

• Check the disk space on the core manager. You may need to FTP files or may need to clean up the disk.

• Check to see if files are being sent FTP. If not, set the system up to FTP files or back up files to the DAT tape.




• Check to see if files are being sent FTP. If not, set the system up to FTP files or back up files to the DAT tape. Also check file permission for the destination directories.

• Contact your next level of support.

SDMB360 SBA has lost the connection to the Persistent Store System (PSS) and cannot restore it. When this occurs alarm SBAIF is raised.



Log Trigger Action



SDMB365 A serious problem is preventing the creation of a particular stream. Generated when a new version of SBA does not support a stream format on an active stream that was present in a previous load.

Revert to the previous running version of the SBA. If you removed the support for the stream format in the new release, turn off the stream before installing the new version. If the new version is supposed to support all existing streams, contact Nortel Networks for the latest appropriate software.

SDMB367 A trapable Management Information Base (MIB) object was set. The modification of some MIB objects provides notification of failures to the System Manager by way of a trap. Because there is no System Manager, the system logs messages. While most SDM logs report the stream, the logs associated with the MIB do not. Consideration for separate streams is not built into the Automatic Accounting Data Networking System (AMADNS) MIB specification.


SDMB370 The CDR-to-BAF conversion encountered a problem that prevents it from converting CDR to BAF. When this occurs, alarm NOSC is raised because the BAF record was not generated.

Clear the alarm.


Log Trigger Action



SDMB375 A problem occurred during the transfer of a file to the Data Processing Management System (DPMS). When this occurs, alarm FTP is raised. The error text can be any of the following:

Note: The system may escalate these logs and minor alarms to critical status when the DPMS transmitter exhausts all possible retries. The MIB parameter SessionFtpMaxConsecRetries specifies the condition.

Contact your next level of support if log indicates any one of the following errors:

• insufficient storage space in system

• exceeded storage allocation on downstream DPMS

• unable to fork child process

• unable to open pseudo terminal master

• unable to setsid in child process

• unable to open pseudo terminal slave in child process

• unable to set stdout of child process to pseudo terminal slave

• unable to set stderr of child process to pseudo terminal slave

• unable to set stdin of child process to pseudo terminal slave

• local error in processing

• DPMS FTP service not available

• DPMS FTP connection closed

• requested file action not taken: <command>. File unavailable

Verify FTP if the log indicates any one of the following errors:

• not logged in while executing command: <command>

• unable to exec FTP process

SDMB380 The file transfer mode for the specified stream has an invalid value

Set the file transfer mode to either Inbound or Outbound.


Log Trigger Action



SDMB390 A schedule-related problem has occurred. When this occurs, alarm SBAIF is raised.

Clear the alarm and any alarms related to failure.

SDMB400 This log is generated for every active stream every hour and lists all of the current active alarms.

Clear alarms immediately using the corresponding procedure in the Fault section.

SDMB530 A change in the configuration or status of a stream has occurred.

None

SDMB531 The configuration for backup volumes has been corrected.

None

SDMB550 The SBA has shut down either because the core manager was busied or the SBA was turned off.

Determine the reason SBA shut down.

SDMB610 A communication-related problem with the SBA has been resolved.

None

SDMB615 A software-related condition has been resolved.

None

SDMB620 A backup-related problem with the SBA has been resolved.

None

SDMB621 A new backup file has been started. None

SDMB625 Recovery has started on a backup file.

None

SDMB650 The SBA is restarting one or more of its processes.

None

SDMB655 • The state of a billing file has changed.

• Disk utilization for a particular stream has dropped below a threshold.

• A billing file could not be moved to closedSent.



Log Trigger Action



SDMB660 A problem related to communications with other SBA features was resolved.

None

SDMB665 A software problem on the Core that prevents the synchronization (downloading) of FLEXCDR data at the core manager.

Restart the Core with a load that supports the SBA enhancements for CDR on the core manager.

SDMB670 Either a CDR-to-BAF conversion process used default values to create a BAF field because a CDR field was missing, or the problem was corrected.

For the missing CDR field(s), determine which are needed to generated the BAF field. Use the BAF field displayed in the log report and refer to the applicable Billing Records Application Guide for a list of the CDR fields associated with each BAF field. Update the CDR to include the missing field.

SDMB675 A problem related to file transfer was resolved.

None

SDMB680 The file transfer mode has changed value.

None

SDMB820 Minimal backup space is available. Increase the size of backup volumes.


Log Trigger Action



Clearing a minor or major or critical CBM alarm

ApplicationUse this procedure to clear a minor or major or critical CBM alarm.

IndicationAn alarm indication is displayed on the Office Alarm Unit or the INMS Alarm Management System. These alarms generate logs which can be monitored at the client output device. These alarms are also displayed on the APPL;SDM level of the MAPCI.

MeaningThis indicates that there are one or more alarms reported by the CBM.

ImpactIf the CBM status at the MTC level of the CBMMTC display does not show InSv, then one or more of the following conditions exist:

• one or more CBM applications have failed.

• CBM application is reporting an in-service trouble condition.

• a system software resource has exceeded its alarm threshold.

• a hardware device failure has been reported.

• communication with the core has failed.

Note: If all CBM applications fail, the CBM appl state is system busy (SysB). The system generates a minor alarm.

ActionThe following flowchart provides a summary of the procedure. Use the instructions in the procedure that follows the flowchart to clear the alarm.



Summary of clearing a minor or major or critical CBM alarm

Clearing a minor or major or critical CBM alarm

At the local or remote VT100 terminal

1 Log into the core manager as a maint class user, or root user, and obtain fault status information from the core manager:

> querycbm flt

This flowchart summarizes theprocedure.

Use the instructions in theprocedure that follows thisflowchart to perform theprocedure.Determine the

type of fault at theCBMMTC display

Isolate andclear the fault

ISTb or SysB

End

ManB

Hardwarefaults?

Y Isolate andclear thehardware fault

N

Return to service the affected com-ponents



2 Use the table below to determine the type of fault indicated by the response. Note the log type and the reason for use in later steps.

3 Proceed according to the type of fault.

Fault typelog number Description

Application SDM303 Exceeded failure threshold Package: <package> Process: <process>

Trouble condition asserted Package: <package> Process: <process> <reason>

Connection to the Core

SDM314 Major Crossed Link: link 0 (domain 0 port 0) crossed to Core with link 1 (domain 0 port 1)

SDM334 OC3 Card Fault: transmit fault on link 0 (domain 0 port 0)

SDM335 Minor Link Fault: Bad Incoming CRCs on link 0 (domain 0 port 0)

Communication SDM336 Heartbeat alarm. No heartbeat response received.

Network Time Protocol

SDM327 NTP alarm. Synchronization started, can take up to 30 minutes.

Platform related SPFSxx Specific to the platform, such as a hardware fault or resource exceeded threshold

If the fault is Do

Platform related (SPFSxxx) problem

step 4

Communication problem with the Core (SDM314, SDM 334, SDM335)

Refer to "Clearing a major or minor or critical APPL;SDM alarm"

Communication problem with the Core (SDM336)

Refer to " Clearing a major Heartbeat alarm"



4 If the fault indicates that the logical volume is exceeded, continue with step 5; otherwise, refer to the appropriate SSPFS procedure to clear the alarm.

5

If the GDD logical volume is exceeded, continue with step 6; otherwise, refer to the appropriate SSPFS procedure to increase the size of a logical volume.

6 There are two choices when the GDD logical volume is exceeded:

• increase the size of the logical volume, or

• decrease the number of days to keep the logs

Network Time Protocol problem

have your system administrator isolate and clear the problem.

Application problem (SDM 303)

step 14

CAUTION Potential service interruptionA logical volume on the CBM must never reach 100% disk full. The system operation is unpredictable when a logical volume reaches 100% disk full. If a logical volume exceeds its alarm threshold, contact your system administrator. The system administrator must assess the current condition of the logicalvolume and take appropriate action immediately. If required, contact Nortelfor assistance.

If you decide to Do

Increase the size of the GDD logical volume

proceed to the SSPFS procedure Increasing the size of a file system on an SSPFS-based server.

Decrease the number of days to retain logs

step 7

If the fault is Do



7 Access the Logroute commissioning tool:

# logroute

Example response:

Logroute Main Menu 1 - Device List 2 - Global Parameters 3 - CM Configuration File 4 - Gdd Configuration 5 - Help 6 - Quit Logroute Enter Option ==>

8 Access the GDD configuration menu:

> 4

Example response:

GDD Menu 1 - Number of days to keep log files in /gdd :30 2 - Help 3 - Return to Main Menu

Enter Option ==>

9 Enter the option number for the number of days to keep log files in /gdd:

Enter Option ==> 1

10 Enter the number of days to retain the log files:

Enter number of days(range - 1 To 30) ==>

11 Confirm to save the changes by entering "y":

Save GDD Value [Y/N][N] :- Y

Example response:

Warning: This would change the number of days to store logsin/gdd. Logfiles older than the day specified would be deleted.

Press the Enter key to acknowledge that the data was saved.

Example response

Save data completed -- press return to continue

12 Press the Enter key to acknowledge that the data was saved.

13 Go to step 23.



14 Log into the CBM as a maint class user, or root user, and access the maintenance interface:

# cbmmtc

15 Access the application (Appl) menu level of CBMMTC:

> appl

Example response:

Group: CBM State: ISTb# Application State1 Generic Data Delivery .2 OSS Comms Svcs ManB3 Log Delivery Service . 4 Table Access Service . 5 OM Access Service . 6 OM Delivery . 7 GR740 Pass Through . 8 Passport Log Streamer ISTb9 Base Maintenance Utility .

10 FTP Proxy . Applications showing: 1 to 10 of 10

16 Determine the affected application from the display and note its key number, shown under the header "#".

17 Proceed depending on the state of the application.

18 Determine from office records or other personnel why the application was manually removed from service. When permissible, return the application software package to service:

> rts <key>

where

<key>is the key number of the application, shown under the header “#”

If the state is Do

ManB step 18

ISTb step 19

SysB step 20

Fail step 21



Example response:

RTS Application - Command initiated.Please wait...

Note: When the RTS command is finished, the "Please wait..." message disappears. The word "initiated" also changes to "complete" as follows:

RTS Application Command complete.

19 This state can result from a recent change of state, or if this application is dependent on another application that has not completed initialization.

• if you suspect either situation to be true, wait 10 minutes for the applications to complete initializing.

• if you do not suspect either situation to be true, use the value in the reason field to resolve the problem.

20 Use the reason given to resolve this problem.

21 The specified application software package was set to Fail state because it failed for one of the following reasons:

• the system cannot restart the package

• the application has restarted and failed three times within 10 minutes

If Do

the application returns to service

step 24

the application does not return to service

step 17

If you Do

can resolve this problem step 24

cannot resolve this problem Contact your next level of support.

If you Do

can resolve this problem step 24

cannot resolve this problem Contact your next level of support.



At the application menu level of the RMI, manually busy the affected application software package:

> bsy <key>

where


Example response:

Bsy Application - Command initiated. Please wait...

Note: When the Bsy command is finished, the “Please wait...” message disappears. The word “initiated” also changes to “complete” as follows:

Bsy Application - Command complete.

22 Return the application to service:

> rts <key>

where


Example response:

RTS Application - Command initiated. Please wait...

Note: When the RTS command is finished, the “Please wait...” message disappears. The word “initiated” also changes to “complete” as follows:

RTS Application - Command complete.

23 Proceed depending on the state of the application.

If the application Do

remains in a Fail state refer to the configuration or installation information modules in the Configuration or Upgrades documents, specific to that application

changes to InSv state go to step 24



24 Obtain the fault status information from the CBM:

> querycbm flt

If Do

more faults are reported step 2

all faults are cleared you have completed this procedure.



Clearing a major Heartbeat alarm

ApplicationUse this procedure to clear a major Heartbeat alarm on the CBM.

IndicationAt the net level of the cbm mtc display, the Core Heartbeat State indicates a SysB condition.

MeaningThe CBM is not receiving responses from the Core.

ImpactIf the CBM is unable to communicate with the Core, the applications will also be unable to communicate with the Core.

Action

Clearing a major Heartbeat alarm

At the CBM

1 Verify that the CBM Ethernet interface is in service.

Clear the fault if it is not in service.

2 Verify that the Core Ethernet interface is in service.

Clear the fault if it is not in service.

3 Verify that Ethernet packets are routed properly between the CBM and Core interfaces.




Replacing a failed power supply

ApplicationUse the following procedure to replace a power supply on a CBM server.

ActionThe power supply is a field replaceable unit (FRU). It can be replaced while the server is powered up and in-service.

Replacing a power supply on a CBM server

1 Refer to the manufacturer documentation for the procedure on how to replace the power supply.

2 You have completed this procedure.


Replacing a failed CBM

PurposeUse the following procedures to replace a failed CBM.

ApplicationThe CBM is not a field replaceable unit. The server must be powered down before hardware can be removed from the shelf.

Action

Replacing failed CBM

At the shelf

1 Record the stream_name for the stream you wish to busy as determined in the procedure "Preparing for SBA installation and configuration" in NN-20000-247, CBM Accounting for Wireless Networks.

2 If the server is still powered up, perform the procedure Shutting down an SSPFS-based server on page 162; otherwise, go to step 3.

3 Remove and replace the CBM server by following instructions provided by the hardware manufacturer.

Note: Remove both disk drives from the server being replaced and place them in the replacement server.

4 To bring the server back up, turn on the power to the server at the circuit breaker panel of the frame.



At the CBM

5 Go to the appl level of the cbmmtc tool by typing:

#cbmmtc appl

Example response:

6 Proceed depending on the state of the application. If the applications you want to RTS are in the Offline state, go to step 7; otherwise, go to step 9.

7 Manually busy all the applications by entering:

> bsy group

8 Confirm the BUSY operation:

> y


10 Manually busy all the applications which are in the Offl state:

> bsy <application number 1><application number 2><....>

Note 1: The Bsy command can take multiple application numbers, each separated by a space, to manually busy multiple applications at the same time.

Note 2: Do not apply the Bsy command to the applications you do not want to RTS.

11 If the CBM group state is in ManB state, go to step 12; otherwise, go to step 13.

12 RTS all the applications which are in the ManB state by typing:

> rts group

Go to step 14.



13 RTS each application by typing:

> rts <application number 1><application number 2><....>

14 Ensure that the stream is in Recovery mode by verifying the state is indicated as Rcvy by typing:

> mapci;mtc;appl;sdmbil;post <stream_name>

where

<stream_name>is the stream name value determined in step 1.

Note 1: Rcvy indicates that the stream is in-service and also sending previously created backup files to the CS2000 Core Manager.

Note 2: The state may also be InSv, which indicates that the stream is in a normal working state if recovery has already completed.

15 Clear any application and system alarms if they are present.




Replacing failed Ethernet interfaces

PurposeUse the following procedures to replace a failed Ethernet interface.

ApplicationThe Ethernet interface is not a field replaceable unit. The server must be put out of service and powered down before hardware can be removed from the shelf.

Action

Replacing failed Ethernet interfaces

At the CBM

1 Record the stream_name for the stream you wish to busy as determined in the procedure "Preparing for SBA installation and configuration" in NN-20000-247, CBM Accounting for Wireless Networks.

2 Access the SDMBIL level:


where


3 Busy the stream at the SDMBIL level by typing:

> bsy

4 Proceed with busying the stream by typing:

> y

5 Ensure that the stream is in Backup mode by verifying the state is indicated as ManB by typing:


where


6 Follow the procedure "Sending billing files from disk" in NN-20000-247, CBM Accounting for Wireless Networks.




#cbmmtc appl

Example response:


> bsy group


> y

Example response:

10 Offline each application by entering:

> offl <application number 1><application number 2><....>

Note: Application numbers are separated by spaces if multiple applications are expected to be offlined.

11 Offline the CBM group by entering:

> offl group

At the shelf

12 Follow the procedure Shutting down an SSPFS-based server on page 162.

13 Remove and replace the CBM server by following instructions provided by the hardware manufacturer.



Note: Remove both disk drives from the server being replaced and place them in the replacement server.

14 To bring the server back up, turn on the power to the server at the circuit breaker panel of the frame.

At the CBM


#cbmmtc appl

Example response:

16 Proceed depending on the state of the application. If the CBM group state is Offl go to step 17; otherwise, go to step 19.


> bsy group


> y


20 Manually busy all the applications which are in the Offl state:

> bsy <application number 1><application number 2><....>

Note 1: The Bsy command can take multiple application numbers, each separated by a space, to manually busy multiple applications at the same time.

Note 2: Do not apply the Bsy command to the applications you do not want to RTS.

21 If the CBM group state is in ManB state, go to step 22; otherwise, go to step 23.



22 RTS all the applications which are in the ManB state by typing:

> rts group

Go to step 24.

23 RTS each application by typing:

> rts <application number 1><application number 2><....>

24 Ensure that the stream is in Recovery mode by verifying the state is indicated as Rcvy by typing:


where


Note 1: Rcvy indicates that the stream is in-service and also sending previously created backup files to the CS2000 Core Manager.

Note 2: The state may also be InSv, which indicates that the stream is in a normal working state if recovery has already completed.

25 Clear any application and system alarms if they are present.




Accessing TCP and TCP-IN log devices from a remote location

PurposeUse this procedure to access TCP and TCP-IN devices, from a remote location.

ApplicationThe TCP and TCP-In log devices can be accessed from either a local, or a remote location (console). The following procedures describe how to access these log devices from a remote location. These procedures can be used when you are performing the related procedures listed in the table Remote access to log devices procedures.

Procedure

Accessing a TCP device from a remote location

At the remote workstation1 Start the logreceiver tool:

> logreceiver <port_number>

where:<port_number>

is the port number used for the TCP device on the core manager

2 Continue with the desired procedure listed in the table Remote access to log devices procedures on page 36.

Remote access to log devices procedures

Log device Procedure Applies to

TCP Accessing a TCP device from a remote location

“Configuring a CBM for log delivery” in the Configuration Management document Displaying or storing log records using logreceiver on page 55

TCP-IN Accessing a TCP-IN device from a remote location

“Configuring CBM for log delivery” in the Configuration Management document “Deleting a device using logroute” in the Configuration Management document




Accessing a TCP-IN device from a remote location

At the remote workstation1 Use telnet to access the core manager:

> telnet <ip_address> <port_number>

where:<ip_address>

is the address of the core manager<port_number>

is the number of the port of the device on the core manager2 Log into the core manager either as maint or admin. 3 Start the logroute tool:

# logroute

4 Continue with the desired procedure from the table Remote access to log devices procedures on page 36.




SBA alarm troubleshooting

PurposeIn the SBA environment, there are many conditions that can cause an alarm to be raised. While there is a log message associated with each alarm, the information that is supplied is not always enough to determine what raised the alarm.

Note: When alarms related to a filtered stream are sent to the CM, they are sent under the name of the associated CM billing stream. When this occurs, the name of the filtered stream is prepended to the text of the alarm.

ApplicationThe majority of the alarms raised on the SBA system that you can resolve can be traced back to one of two problem areas:• a problem in the FTP process• an insufficient amount of storage

A problem in the FTP processIf you receive numerous FTP and LODSK alarms, this can indicate a problem with either the SBA or the general FTP process on the core manager. LODSK generally indicates that your primary files (closedNotSent) are not being moved from the core manager to the downstream processor. Review any accompanying logs.

The downstream processor can be full with no space to write files to, which can cause an FTP error. When this happens, you see core SDMB logs, which indicate that the file is not sent. In addition, if you do not receive an FTP alarm, it is possible that scheduling is turned off, which prevents FTP alarms from being sent.

Insufficient amount of storageIf you receive numerous alarms for the backup system without receiving an FTP or LODSK alarm, this indicates a communication problem. The core is not communicating with the core manager.

Use the following procedures to clear alarms based on the FTP process:• Verifying the file transfer protocol on page 143

• Verifying the FTP Schedule on page 149



Use the following procedures to clear alarms based on communication problems between the core and the core manager:• Clearing a DSKWR alarm on a CBM on page 91

• Clearing a NOCOM alarm on page 110

• Clearing a major SBACP alarm on page 134

• Clearing a minor SBACP alarm on page 138

APPL Menu level alarmsBecause SBA processing takes place in both the CM and the core manager environment, the SBA program displays core manager-generated alarms in the MAPCI;MTC window at the CM. The figure Alarms layout shows the SBA alarms that are displayed under the APPL Menu level at the MAPCI;MTC level on the CM side.

Alarms layout

POST

CI

MAPCI

SASelect NWM CPSys IBNMEAS FPE TESTTOOL

MTCNA BERP CPSTATUS DMS MS IOD Net PM CCS Lns Trks ExtActivity

DCAP OAM&P

MTC

Appl

SWMTCSDMBILDIRP SDM



Maintenance for SBAMaintenance for SBA on the CM side centers around the following entities:• table SDMBILL• MAP level SDMBIL• logs• states • alarms

Maintenance for SBA on the core manager side is performed using the interface on the SBA RMI. For example, you perform maintenance on the core manager side of SBA by using commands in the billing level (billmtc) of the core manager RMI display.

You can also display the alarms raised by the core manager side for the SBA by using the DispAl command from the billmtc level. The DispAl command displays the alarm criticality, stream, and text of the alarms.

Alarm severityThere are three levels of severity for SBA alarms:• Critical:

a severe problem with the system that requires intervention• Major:

a serious situation that can require intervention• Minor:

a minor problem that deserves investigation to prevent it from evolving to a major problem

When multiple alarms are raised, the alarm with the highest severity is the one displayed under the SDM header of the MAP banner. If multiple alarms of the same severity (for example, critical) are raised, the first alarm that is raised is the one displayed under the SDM header of the MAP banner. For example, if a NOBAK critical alarm is raised before a NOSTOR critical alarm, the NOBAK alarm is the one that is displayed. Use the DispAl command to view all outstanding alarms, and use the associated procedure to clear each outstanding alarm.



CM MAP statesIn the SBA environment, an SBA stream can have different state values due to some action or condition on the SBA system. You can view the state of a stream from the CM by entering:

>mapci;mtc;appl;sdmbil;post <stream_name>

where

<stream_name> is the name of the stream

The possible state values and their definition are as follows:• Offline pending (OffP):

the stream has been turned off and is waiting for the core manager to complete processing its data

• Offline (OffL): the stream is offline

• Manual busy (ManB): the stream has been manually busied by a user from the CM; data is being written to backup files

• System busy (SysB): the stream has been busied by the SBA system due to a communications or internal software error; data is being written to backup files

• Remote busy (RBsy): the stream has been busied by the SBA system due to a communications or internal software error; data is being written to backup files

• Backup (Bkup): the stream is writing data to backup files due to a performance problem

• Recovery (Rcvy): the stream is in service and is also sending backup files previously created to the core manager

• In-service (InSv): the stream is in a normal working state

•



Common proceduresThere are a few procedures that are common to all of the alarm clearing procedures. These common procedures include the following:• Verifying the file transfer protocol on page 143 helps you determine

that the FTP process is configured correctly and is able to transfer files

• Verifying the FTP Schedule on page 149 helps you determine that the system is able to send FTP files on a regular basis

• “Configuring SBA backup volumes on the core” in the core manager Accounting document is used to create and activate alternative backup volumes for a stream

Use the following procedures to clear alarms based on insufficient storage capacity:• Clearing a BAK50 alarm on page 73

• Clearing a BAK70 alarm on page 77

• Clearing a BAK90 alarm on page 81

• Clearing a BAKUP alarm on page 85

• Clearing a NOBAK alarm on page 105

• Clearing a NOREC alarm on page 117

• Clearing a NOSTOR alarm on page 119

• Clearing a NOVOL alarm on page 125



Accessing the MATE

PurposeUse this procedure to access the MATE.

ProcedureUse the following procedure to access the MATE.

Accessing the MATE

At the workstation UNIX prompt or VT-100 terminal prompt:

1 Log onto the CBM.

2 Get the current hostname by entering:

# GetCurrentHostNAme

Example response:

3 Access the Report Registration Menu:

4 Determine if the MATE is running by entering:

> ping <mate hostname>

Note: The <mate hostname> is the one determined in step 3.

5

6 Access the MATE using SSH by typing:

> ssh root@ <mate hostname>

If Then

the hostname returned in step 2 contains "unit0"

the MATE hostname is "unit1"

the hostname returned in step 2 contains "unit1"

the MATE hostname is "unit0"

If Do

step 4 indicates the MATE is Active

step 6

otherwise step 8

<CBM hostname>-<unit0 / unit1>



where

Note: You can log into the MATE without a password. To exit the MATE, type > exit to return to the local system.


8 You need to access the MATE through a local VT100 terminal.



Clearing the MATE alarm

PurposeUse this procedure to clear the MATE alarm.

ProcedureUse the following procedure to clear the MATE alarm.

At the workstation UNIX prompt or VT-100 terminal prompt:

1 Log onto the CBM.

2 Start the cbmmtc tool by typing:

# cbmmtc

Note: Check the MATE column on the banner. If the state is not "." (dot), this indicates the presence of an alarm.

3 Access the MATE by performing the procedure Accessing the MATE on page 43.

4 Clear the alarms by performing the procedure Clearing a minor or major or critical CBM alarm on page 17.

5 Log out of the MATE.





Displaying SBA log reports

PurposeUse this procedure to display the current logs raised by the core manager for the SuperNode Billing application (SBA) that have not been acknowledged by the Core.

ApplicationThe MIB parameter “sendBillingLogsToCM” affects the displogs command.

The displogs command does not display logs generated by the Core.

PrerequisitesNone

Procedure

Displaying SBA logs

At any workstation or console1 Log into the core manager using the root user ID and password.2 Access the billing maintenance interface:

# billmtc

3 Display the logs:> displogs

The logs are displayed in the format of name, number, event type, alarm status, label, and body. If there are no logs to display, the message No unsent logs is displayed.




Displaying SBA alarms

PurposeUse this procedure to display the current alarms raised by the core manager for the SuperNode Billing application (SBA).

ApplicationThe MAP CI displays the status (critical, major, minor), the stream, and the text of the alarm.

This command displays alarms that have not been sent to the computing module (CM). However, the dispal command does not display Core-side alarms, such as the BAK50, BAK70, BAK90, NOBAK, and BAKUP alarms.

PrerequisitesNone

Procedure

Displaying SBA alarms

At any workstation or console1 Log into the core manager using the root user ID and password.2 Access the billing maintenance interface:

# billmtc

3 Display the alarms:> dispal

The alarms are displayed in the format of alarm status (critical, major, minor), stream, alarm short text, and alarm long text. If there are no alarms to display, the message, “No alarms” is displayed.



Collecting DEBUG information using the CBMGATHER command

PurposeUse this procedure to collect DEBUG information from the core manager.

ApplicationUse either of these procedures to collect the following DEBUG information from the core manager:

• the output of cbmgather

• the content of /var/adm directory

It is important to collect DEBUG information from the system in case of a failure (before recovery). The information assists in discovering the root cause of the problem and in preventing similar problems in the future.

Note: Instructions for entering commands in the following procedure do not show the prompting symbol, such as #, >, or $, displayed by the system through a GUI or on a command line.

Procedure

At the core manager command line (UNIX prompt) of the active node

1 On the active node, run the utility to collect the output:

cbmgather

The output file from this command is located under /var/adm and has a name in the format: cbmgather_<machine>_<date_and_time>.tar.Z

Example /var/adm/cbmgather_hadry2_20050221141300.tar.Z

2 Tar and compress the content of directory /var/adm:

cd /var/adm

tar cvf varadm_active.tar *.day* *.log

compress varadm_active.tar

The output of the compressed tar file in the example is called varadm_active.tar.Z.



3 Move the files generated by commands executed in steps 1 and 2 out the system to a secure location using FTP (in BINary mode).

4 Remove the gathered output/files from the system:

rm -f /var/adm/cbmgather_<machine>_<date_and_time> .tar.Z

Note: The command shown above is entered on a single line. When entering the command, ensure that there is a single space between -f and /var, and that there is no space between time> and .tar.

rm -f /varadm_active.tar.Z

At the core manager command line (UNIX prompt) of the inactive node

5 On the inactive node, run the utility to collect the output:

cbmgather

6 Tar and compress the content of directory /var/adm:

cd /var/adm

tar cvf varadm_inactive.tar *.day* *.log

compress varadm_inactive.tar

Example response:

The output of the compressed tar file in the example is called varadm_inactive.tar.Z.

7 Move the files generated by commands executed in steps 5 and 6 out the system to a secure location using FTP (in BINary mode).

8 Remove the gathered output/files from the system:

rm -f /var/adm/cbmgather_<machine>_<date_and_time> .tar.Z

Note: The command shown above is entered on a single line. When entering the command, ensure that there is a single

If Do

your system is a CBM 850 cluster configuration step 5

your system is not a CBM 850 cluster configuration step 9



space between -f and /var, and that there is no space between time> and .tar.

rm -f /varadm_inactive.tar.Z




Controlling the SDM Billing Application

PurposeUse the following procedure to busy the SDM Billing Application (SBA) or return the SBA to service.

PrerequisitesYou must establish communications between the core manager and the core for SBA to run successfully.

Procedure

At any workstation or console1 Log in to the CBM.2 Access the Application level:

# cbmmtc appl

The system displays a list of applications.Note: Use the up and down commands to scroll through the list of applications.

3

Busy the SDM Billing Application:

> bsy <x>

where:<x>

is the number next to the CBM Billing Application

If you want to Do

busy the SBA step 3

return the SBA to service step 5

CAUTION

Busying the SBA causes SBA to go into backup mode, and triggers an SBACP (major) alarm under the SDMBIL banner at the MAP terminal.



Example response:The application is in service. This command will cause a service interruption. Do you wish to proceed? Please confirm (“YES”, “Y”, “NO”, or “N”):

4 Confirm the busy command:> y

5 Return the CBM Billing Application to service:> rts <x>

where:<x>

is the number next to the CBM Billing ApplicationNote 1: This command causes SBA streams to go into a recovery mode.Note 2: Any streams configured for real-time billing (RTB) are also returned to service. Log report SDMB375 is generated when a stream configured for RTB fails to return to service.

6 Determine if log SDMB375 was generated.

If the SBA Do

busied successfully and you want to return the SBA to service

step 5

busied successfully but you do not want to return the SBA to service at this time

step 13

did not busy successfully contact your next level of support

If the SBA Do

returned to service successfully

step 6

did not return to service successfully

contact your next level of support

If the system Do

generates log SDMB375 step 7



7 Return the RTB streams to service. Exit the Application level:> quit all

8 Access the billing maintenance level:# billmtc

9 Access the schedule level:> schedule

10 Access the real-time billing level:> rtb

11 Busy the stream:> bsy <stream name>

where:<stream name>

is the name of the billing stream configured for RTB (for example OCC)

12 Return the stream to service:> rts <stream name>

where:<stream name>


13 Quit the billing maintenance level:> quit all

14 You have completed this procedure

does not generate log SDM375

you have completed this procedure

If the billing stream configured for RTB Do

returns to service successfully

step 13

does not return to service successfully


If the system Do



Displaying or storing log records using logreceiver

PurposeUse this procedure to display or store log records on a workstation using the logreceiver tool.

ApplicationThe commands that you enter to display or store log records on a workstation must include a port number. The port number must be the same as the port number used to configure the TCP device on the core manager. The port number must not be used for any other purpose on the workstation, otherwise the following error message appears:

Failed to listen for connection request on port <port_number>, exiting

You must change the port number used to configure the TCP device on the core manager.

Storage fileIf the storage file does not exist, it is created automatically. The logs from the core manager are stored in this file.

If the file exists, the logs from the core manager are added to it provided its UNIX access permissions allow writing to the file. In either case, a message ‘Accepted connection request from host <hostname>’ is displayed on the screen just before the first log received is written to the file. Press ctrl -c and press the Enter key to terminate execution of the logreceiver tool.

If the file exists, but its permissions do not allow writing to it, an error message ‘Failed to open <filename>’ displays on the screen. Press ctrl -c, and press the Enter key to terminate execution of the logreceiver tool.

The file continues to fill up until either the logreceiver execution terminates or all free storage in the file system is exhausted. In the latter case, the logreceiver execution terminates automatically. The error message ‘Failed to open <filename>’ displays on the screen and you must remove the file or free up some storage.



Procedure

Checking the port numbers in use on a workstation

At the client workstation1 Check the port numbers in use:

> more/etc/services

The list of port numbers in use is displayed. Scroll through the display by pressing the Enter key again.

Storing logs in a file

At the client workstation1 Start the logreceiver tool to store logs in a file:

> logreceiver <port> -f <filename>

where<port> is the port number used when configuring the TCP device on the core manager<filename> is the name of the file

Displaying log records on a workstation

At the client workstation1 Start the logreceiver tool to display the log records on the

screen:> logreceiver <port>

where<port> is the port number used when configuring the TCP device on the core manager




Retrieving and viewing log records

PurposeUse this procedure to retrieve and view CM and core manager log records using the core manager log query tool.

ApplicationWhen you enter the log query tool, the system automatically displays the log records using the following default settings:• log type: all• format: std• date: current date• time: midnight of current date• display of log records: page by page• arrangement of logs displayed: show latest log first

Procedure

Retrieving and viewing logs

At a terminal or terminal session connected to the core manager1 Log into the core manager.



2 Start the log query tool using the default settings:# logquery

Example response:

3 Access a list of available parameters and variables to view logs:> logquery -help

4 Enter the applicable command.5 When you are finished, exit the log query tool:

> quit




Troubleshooting AFT alarms

PurposeUse this procedure to clear alarms generated by the Automatic File Transfer (AFT) application.

ApplicationUse the following procedures to resolve AFT alarms that are specific to the SuperNode Billing Application (SBA).

IndicationAt the SDMBIL level of the MAP, "AFT" and the alarm level indicators for critical (*C*) and major (M) alarms appear in the alarm banner under the SDMBIL header.

MeaningAn AFT alarm is generated under the conditions listed in the table AFT alarms.

ImpactWhen conditions exist for a critical or major AFT alarm, billing records are not being transferred to the downstream collector.

ProcedureThis section describes the methods for clearing critical and major AFT alarms.

AFT alarms

Alarm Occurs when:

Critical (*C*) • an AFT session network connection has been disrupted during file transfer

• the retry count has been exceeded on a file• the message transfer protocol (MTP) timer

has expired

Major (M) an AFT session has been stopped using the AFT level Stop command



Clearing critical alarmsTo clear a critical alarm, use one of the following methods:• Deleting a tuple from automaticFileTransferTable on page 61

• manually clear the alarm through the Alarm command at the AFT level of the BILLMTC remote maintenance interface (RMI)

Critical alarms also are cleared when the network connection disruption is corrected.

Clearing major alarmsTo clear a major alarm, use one of the following methods:• restart the session using the Start the command available at the

AFT level of the BILLMTC RMI• delete the tuple from the automaticFileTransferTable table• manually clear the alarm through the Alarm command available at

the AT level of the BILLMTC RMI

ProcedureUse the following procedure to clear an AFT alarm manually.

Clearing an AFT alarm manually

At the core manager1 Access the BILLMTC level:

> billmtc

2 Access the Application (APPL) level:> appl

3 Access the Automatic File Transfer (AFT) level:> aft

4 Clear the alarm:> alarm cancel <session_name>

where:<session_name> is the unique name of the network connection for which you want to clear the alarm

Example response:*** WARNING: Alarm(s) will be cancelled for AFT session <session_name> Do you want to continue? (Yes or No)



5 To cancel the alarms, enter:> yes

Example response:Cancelled alarms for AFT session: <session_name>


Deleting a tuple from automaticFileTransferTable

At the core manager 1 Access the BILLMTC level:

> billmtc

2 Access the APPL level:> appl

3 Access the AFT level:> aft

4 Access the AFTCONFIG level:> aftconfg

5 Delete the tuple from the automaticFileTransferTable:> delete <session_name>

where:<session_name> is the unique name of the network connection that generated the alarm

Example response:*** WARNING: Alarm(s) will be cancelled for AFT session <session_name> Do you want to continue? (Yes or No)

6 To delete the table entry (tuple), enter:> yes

CAUTION

An AFT tuple must be stopped before it can be deleted. When an AFT tuple is deleted, billing files are no longer being transferred downstream.



Example response:Deleted table entry for AFT session: <session_name>


Restarting an AFT session

At the core manager1 Access the BILLMTC level:

> billmtc

2 Access the APPL level:> appl

3 Access the AFT level:> aft

4 Restart the AFT session that generated the alarm:> start <session_name>

where:<session_name> is the unique name of the network connection that generated the alarm

Example response:*** WARNING: Started AFT session: <session_name>




Troubleshooting RTB problems

Use the following flowchart, and the procedures in your documentation for this product, to troubleshoot problems related to real time billing (RTB).

Is auto-recovery on?

Y

N

Query RTBauto-recovery.

Check for logSDMB675 thatshows problemresolved.

Is the

resolved?

Y

Nproblem

Use the error text in the log

Start

Check for log

Use the error text in the logto clear the

Finish

SDMB375 thatdescribes the problem.

problem.



Troubleshooting problems with scheduled billing file transfers

Use the following flowchart, and the procedures in your product documentation, to troubleshoot problems related to the scheduled transfer of billing files from the core manager to a downstream destination.

Note: The length of time for the SuperNode Billing Application (SBA) to resume transferring billing files depends on the following configured parameters:

• the number of active scheduled tuples

• the time interval to transfer files

Clear theSBAIF alarm

SBAIF alarm Log SDMB390 Log SDMB691

Make sure the system is freeof faults

Confirm alarmclears with log SDMB690

Confirm transfersresume with log SDMB691

Finish



Troubleshooting Log Delivery problems on a CBM

PurposeUse the procedure to• troubleshoot the ISTb state of the log delivery application• isolate and clear faults• change the state of the log delivery application from ISTb to InSv

Fault conditions affecting Log DeliveryLost logs

When the system detects that logs are being lost, an internal report indicating the number of logs lost is sent to all client output devices.

To clear the problem:1 Access the Log Delivery commissioning tool2 Select the Global Parameters menu, and3 Increase the buffer size

Refer to procedure “Configuring Log Delivery global parameters” in the CBM Configuration Management document.

No logs being received at a Log Delivery clientIf no logs are being received at a Log Delivery client, do the following at the Device List menu of the Log Delivery commissioning tool: • verify that the client is defined• verify that the log stream for the client is defined

Refer to procedure “Modifying a log device using logroute” in the CBM Configuration Management document.

Logs not formatted properlyIf the log reports at a Log Delivery client device are not formatted correctly, access the Log Delivery commissioning tool and check the following:• at the Device menu, verify that the correct log format has been

commissioned for the device (STD, SCC2, STD_OLD, SCC2_OLD)• at the Global Parameters menu, check that the parameters for start

and end of line, and start and end of log, are set correctly.



For more information, refer to procedure “Modifying a log device using logroute” in the CBM Configuration Management document.

Log devices on the computing module are fullIf a CBM cannot detect computing module (CM) logs, it is possible that there are no free log devices on the CM. In the event that all the log devices on the CM are full, the Log Delivery application generates an alarm. The application state changes to ISTb, and generates an SDM303 log at the RMI.

The log delivery alarm can be cleared when any log device on the CM/Core is freed, and the Log Delivery application is manually busied and returned to service.

IntervalPerform this procedure when the state of the log delivery application in the Apply menu level of the cbmmtc user interface is ISTb.

Procedure

Troubleshooting the log delivery application when its state is ISTb

At the local or remote VT100 console1 Log into the CBM as the root user.2 Access the maintenance interface:

# cbmmtc

3 Access the application level (Appl):> appl

4 Busy the GDD application:> bsy <fileset_number>

where<fileset_number>

is the number next to the GDD application

If GDD is Do

Offl step 4

ManB step 5

InSv step 6



5 Return the GDD application to service:> rts <fileset_number>


is the number next to the GDD application on the screenNote: Wait at least one minute for the ISTb state to change to InSv.

6 Check the CBM for any faults:> querycbm flt

7 Exit the maintenance interface:> quit all

Note: You must be a root user of the CBM to continue with the procedure.

8 Access the /gdd directory:# cd /cbmdata/00/gdd

9 Check all log files:# ls -l

If the Log Delivery application Do

remains ISTb step 6

goes InSv you have completed this procedure

If Do

a fault report indicates “log file is circulating (losing logs)”

step 7

a fault report indicates “Core log device is not Configured”

step 20

no fault report indicates “log file is circulating (losing logs)”




10 Determine if there are any files present that are not log files. Note: Log files start with LOGS.recorddata.

11 Delete files that are not log files:Note: Once you remove the file, there is no way to restore it.

# rm <file>

where<file>

is the file in the /gdd directory that is not a log file.12 Return to the maintenance interface:

# cbmmtc

13 Access the application level (Appl):> appl

14 Determine if the state of the log delivery application is ISTb. Wait at least 1 min. to for the ISTb state to change to InSv.


16 Access the /gdd directory:# cd /cbmdata/00/gdd

17 Check the log files:# ls -l

If Do

there are files present that do not start with LOGS.recorddata

step 11

all files start with LOGS.recorddata

step 17


remains ISTb step 15

goes InSv you have completed this procedure



18 Determine if the current log file (LOGS.recorddata) is much larger than the other log files.

19 Increase the size of the /cbmdata/00/gdd file system:Note: Once you have increased the size of a file system, you cannot decrease it.

# filesys grow -m /cbmdata/00/gdd -s <size>{m,g}

where<size>

is the size in megabytes (Mbytes) or gigabytes (g) by which you want to increase the current size of the file system

Note 1: Configure the size of the /cbmdata/00/gdd file system to be equal to the required capacity for 12 hours of log files, multiplied by 2 (for a 24 hour file size) then multiply the value by 50 days. This provides enough storage space to accommodate the required 30 days of log files, with excess capacity available.

Example 3Mb x 2 x 50 days = 300 Mbwhere300 Mb is the average size of a 12 hour log file in the /gdd file system

Note 2: The default value for GDD is set for seven days. If needed, increase the value, but a corresponding increase in GDD size is required.

If the current log file is Do

larger than the other log files contact your next level of support

the same size as the other log files

step 19



At the MAP20 Verify that a log device on the core is available.

>logutil; listdevs

If all 32 log devices are being used, free up one log device for the Log Delivery Service on the CBM to use.For more information, refer to procedure “Deleting a log device using logroute” in the CBM Configuration Management document.

At the local or remote VT100 console21 Busy the Log Delivery application:

> bsy <fileset_number>


is the number next to the GDD application 22 Return the Log Delivery application to service:

> rts <fileset_number>


is the number next to the GDD application23 Determine if the state of the log delivery application is still ISTb.

Wait at least 1 minute for the ISTb state to change to InSv.



remains ISTb contact your next level of support

goes InSv step 24



Clearing a BAK50 alarm

PurposeUse this procedure to clear a BAK50 alarm.

IndicationBAK50 appears under the APPL header of the alarm banner at the MTC level of the MAP display. The alarm indicates a critical alarm for the backup system.

MeaningThe SBA backup system is using more than 50 percent of the total space on backup volumes on the DMS/CM. If the stream is configured as:• both

the alarm severity level is major• on

the alarm severity level is critical

The core manager generates the SDMB820 log report when this alarm is raised.

ImpactIf the disk usage for the SBA backup system reaches 100 percent of its capacity, data that is configured to go to backup storage is lost.

ProcedureThe following flowchart provides a summary of the procedure. Use the instructions in the procedure to clear the alarm.

ATTENTIONThe option to configure a billing stream to both is only intended to be a temporary path while you are performing maintenance and alarm clearing tasks. The option to set a billing stream to the both mode on a permanent basis is not supported.



BAK50 alarm clearing flowchart


At the MAP1 Post the billing stream:


where <stream_name> is the name of the billing stream.

2 Determine why the system is in backup mode.3 Display all of the alarms that have been raised:

> DispAL

Start

End

Receive BAK50alarm

Access the SDMBILlevel

Find out why system is in backup mode

Run a manual audit



4 Determine the billing stream status.

5 Use Audit to clear the alarm.

6 Ensure that the billing system is in recovery:> post <streamname>

7 In the display, look for the status of the billing stream.

8 Perform the procedure Adjusting disk space in response to SBA backup file system alarms on page 89




If the billing stream is Perform the following steps

SysB perform the procedure for the alarm or the condition, and then return to step 5.

RBsy refer to Clearing a major SBACP alarm on page 134, and then return to step 5.

ManB Go to step 8

Bkup Go to step 8

If the alarm Do

cleared step 6

did not clear step 8

If the billing system Do

is in recovery (Rcvy) step 12

is not in recovery contact your next level of support

If the alarm Do

cleared step 10

did not clear contact your next level of support







IndicationBAK70 appears under the APPL header of the alarm banner at the MTC level of the MAP display, and indicates a critical alarm for the backup system.

MeaningThe SBA backup system is using more than 70 percent of the total space on backup volumes on the DMS/CM. If the stream is set to:• both












> mapci;mtc;appl;sdmbil;post <billing_stream>

where <billing_stream> is the name of the billing stream.

2 Determine why the system is in backup mode.3 Display all of the alarms that have been raised:

> DispAL

Start

End

Receive BAK70alarm



Run a manual audit



4 Determine the state of the billing stream.











ManB Go to step 8

Bkup Go to step 8

If the alarm Do

cleared step 6





If the alarm Do

cleared step 10








IndicationBAK90 appears under the APPL header of the alarm banner at the MTC level of the MAP display and indicates a critical alarm for the backup system.

MeaningThe SBA backup system is using more than 90 percent of the total space on backup volumes on the DMS/CM. If the stream is configured as:• both





ProcedureThe following flowchart is a summary of the procedure. Use the instructions in the procedure to clear the alarm.








where <billing_stream> is the name of the billing stream.

2 Determine why the system is in backup mode.3 Display all alarms that have been raised:

> DispAL

Start

End

Receive BAK90alarm



Run a manual audit














ManB Go to step 8

Bkup Go to step 8

If the alarm Do

cleared step 6





If the alarm Do

cleared step 10






Clearing a BAKUP alarm

PurposeUse this procedure to clear a BAKUP alarm.

IndicationBAKUP appears under the APPL header of the alarm banner at the MTC level of the MAP display, and indicates a critical alarm for the backup system.

MeaningRecords are being stored on the DMS/CM backup volume for more than 10 minutes. If the stream is configured as:• both




ImpactA problem with the SBA disk storage capacity can occur depending on the rate at which new data is sent to backup storage. BAKxx alarms provide storage notification (xx is the percentage of disk storage used).


ATTENTIONThe option to configure a billing stream as both is only intended to be a temporary path while you are performing maintenance and alarm clearing tasks. The option to set a billing stream to the both mode on a permanent basis is not supported.



BAKUP alarm clearing flowchart

Clearing a BAKUP alarm



where <billing_stream> is the name of the billing stream


> DispAL

Start

End

Receive BAKUPalarm



Run a manual audit














ManB Go to step 8

Bkup Go to step 8

If the alarm Do

cleared step 6





If the alarm Do

cleared step 10






Adjusting disk space in response to SBA backup file system alarms

PurposeUse this procedure to adjust disk space when SBA backup file system alarms are raised. The procedure enables you to either add logical volumes to a disk or to remove logical volumes from a disk.

Procedure

Adjusting disk space in response to SBA backup file system alarms




2 Display the names of the backup volumes configured for the stream:> conf view <stream_name>

where<stream_name> is the name of the billing stream.

3 Display and record the size of a volume and its number of free blocks:> dskut;sv <volume name>

where<volume name>

is the name of one of the volumes that you obtained and recorded in step 2

If the backup volumes are located on Do

DDU disks step 3

IOP disks step 5

SLM disks step 5

3PC disks step 5



4 Repeat step 3 for each volume name that you recorded in step 2, and then proceed to step 5.

5 Display and record the size of a volume and its number of free blocks:> diskut;lv <volume name>

where<volume name>

is the name of one of the volumes that you obtained and recorded in step 2.

6 Repeat step 5 for each volume name that you recorded in step 2.


If the volumes Do

have enough disk space step 7

do not have enough disk space

perform procedure “Configuring SBA backup volumes on the core” in the Accounting document for your core manager.



Clearing a DSKWR alarm on a CBM

IndicationAt the MTC level of the MAP display, DSKWR appears under the APPL header of the alarm banner and indicates a critical disk alarm.

MeaningThe system is unable to write records to the CBM disk because the disk is unavailable or the disk is full.

ImpactThe DMS/CM cannot send the billing records to the CBM. As a result, the DMS/CM send the billing records to backup storage. However, this backup storage is limited. As the backup storage becomes filled, alarms notify you as to how much of its capacity is used.

Prerequisites

ProcedureUse the following procedure to clear DSKWR alarm.

Clearing a DSKWR alarm

At the MAP interface on the CM1 Access the SDMBIL level:

> mapci;mtc;appl;sdmbil

ATTENTIONIf the NOBAK or NOSTOR alarm appears in addition to the DSKWR alarm, you must configure and activate alternative backup volumes before you clear the DSKWR alarm.



2 Check to see if the NOBAK or NOSTOR alarm exists in addition to the DSKWR alarm on the alarm banner:> dispal

At your workstation3 Check to see if any logs have been raised that indicate a

problem with the system’s disks, by performing the procedure, “Viewing customer logs on a Sun server” .

4 Determine whether the file system holding the billing files has adequate space by performing the procedure, Verifying disk utilization on an SSPFS-based server on page 197.

5 If you want to back up the billing files, perform the procedure “Copying files to DVD” in the NN10363-811 document.

6 Using the information you obtained in step 4 determine whether the file system is full. The file system can be full if you have not sent the primary files downstream.

7 Access the BILLMTC interface: > billmtc

8 Access the FILESYS level: > filesys

If the NOBAK or NOSTOR alarm Do

appears in the alarm banner perform the procedure “Configuring SBA backup volumes on the core” in NN-20000-247, CBM Accounting for Wireless Networks.

does not appear in the alarm banner

step 3

If Do

you want to send the billing files downstream

step 7

you feel that the capacity of the SBA file system requires adjustment




9 Send the primary billing files to the downstream processor:> sendfile <stream_name>

where:<stream_name> is the name of the stream.

Note: The sendfile command sends the billing file to the billing collector.


11 Quit the BILLMTC interface: > quit all

12 At the prompt, check for orphan files and for files someone else copied to the logical volume of your billing stream:> ls /<stream>/<stream_name>/orphan

where:<stream> is the full pathname of the directory you have configured for the billing stream

If the SENDFILE command Do

is successful step 10

is not successful refer to procedures Verifying the file transfer protocol and Verifying the FTP Schedule, then return to this procedure and repeat step 9

If unsuccessful afterwards, contact your next level of support

If the alarm Do

cleared step 17




<stream_name> is the name of the billing stream.

13 Verify the write permission and ownership for the directories in the billing stream:ls -lrt /<stream>/<stream_name>

<stream> is the full pathname of the directory you have configured for the billing stream<stream_name> is the name of the billing stream

14 Change the permissions for a directory:> chmod 755 <directory>

where:<directory> is the billing file directory in which you are changing permissions.

15 Change the ownership of a directory:> chown maint:maint <directory>

where:<directory> is the billing file directory in which you are changing ownership.

If billing files full because of accumulated orphan files Do

and you are unclear as to how to clean up the billing directory


and you have cleaned up the billing directory and are still incurring a problem

step 13

If the Do

permissions (rwx r-x r-x) and file ownership (maint) are correct


permissions for a directory are not rwx

step 14

ownership for a directory is not maint

step 15

the alarm fails to clear contact your next level of support





If the alarm Do

cleared step 17




Clearing an FTP alarm

PurposeUse this procedure to clear an FTP alarm.

IndicationAt the MTC level of the MAP display, FTP appears under the APPL header of the alarm banner and indicates an alarm for FTP.

MeaningThe FTP process failed. The SDMB logs provide details about the FTP problem. This alarm can be either critical or major.


ImpactThe core manager cannot FTP files to the downstream destination. It is possible that the core manager has reached its storage capacity limit, depending on the amount of storage and the volume of records.

As the core manager storage becomes full, alarms notify you of how much of its capacity is used. When this storage is full, the DMS/CM sends subsequent records to backup storage.




FTP alarm clearing flowchart

Clearing an FTP alarm

At the MAP1 Examine the SDMB logs for details about the FTP problem:

> logutil;open sdmb

Note: This command displays the most recent logs.2 Verify that the FTP is working by performing Verifying the file

transfer protocol on page 143 in this document.3 If the alarm fails to clear, contact your next level of support.4 You have completed this procedure.

Y

Start

Receive the

Alarm cleared?Contact yourN

FTP alarm

End

End

Verify FTP

next level of support



Clearing an FTPW alarm

PurposeUse this procedure to clear an FTPW alarm.

IndicationAt the MTC level of the MAP display, FTPW appears under the APPL header of the alarm banner and indicates an alarm for FTP.

MeaningThe FTP process failed. The SDMB375 log report provides details about the FTP problem. Log report SDMB675 is generated when this alarm is cleared. This alarm can be either critical or major.

Note: The FTPW alarm can be present on the CM for a non-existent schedule. For example, the FTPW alarm is generated if an operator • shuts down the server (making the ftp service unavailable to the

core manager), and• did not delete the associated schedule tuple on the core manager

first

ImpactThe core manager cannot send files to the downstream destinations. The core manager has possibly reached storage capacity, depending on the amount of storage and the volume of records. When this storage is full, the DMS switch/CM sends subsequent records to backup storage. When backup storage reaches capacity, billing records cannot be stored and are lost.

Action

Clearing an FTPW alarm

At the core manager1 Complete procedure Verifying the file transfer protocol on

page 143 in this document.

If Do

alarm fails to clear contact next level of support

schedule does not exist step 2



2 Add a schedule tuple with the same stream name and destination defined by the alarm. Use the procedure “Configuring the outbound file transfer schedule” in the CBM Accounting document, then return to this procedure.

3 Once the alarm is cleared, delete the tuple that you added in step 2.




Clearing an inbound file transfer alarm

PurposeUse this procedure to clear an inbound file transfer (IFT) alarm.

IndicationAt the MTC level of the MAP display, inbound file transfer (IFT) appears under the APPL header of the alarm banner and indicates an alarm for the inbound file transfer connection.

MeaningThe IFT alarm indicates the occurrence of an inbound file transfer. This alarm is raised if the link in the ftpdir directory of a stream cannot be managed or if an ftpdir directory is not accessible. This alarm can be minor, major, or critical.

Detailed information about the alarm condition is documented in log reports:• SDMB375 or SDMB380 when the alarm is raised• SDMB675 or SDMB680 after the alarm is cleared

ImpactInbound file transfer for the billing stream is not possible.

ActionThis alarm occurs only in rare situations. If this alarm occurs, ensure all other SBA alarms are cleared. The root user can check the following IFT alarm conditions:• ftpdir directory has no write access• storage for the billing stream has no space available• <rcLogicalVolumeDirectory>/ftpdir directory does not exist

Determine what alarm is present by reading the log text and associating it to the appropriate alarm.



Clearing an IFT alarm

At the MAP1 Log in to the core manager as maint user.

2 Change the permissions of the /home/maint/ftpdir directory: > chmod 777 /home/maint/ftpdir

3 Remove the <rcLogicalVolumeDirectory>/ftpdir directory: > rm /<rcLogicalVolumeDirectory>/ftpdir

If the Do

/home/maint/ftpdir directory has write permissions

no action is required

/home/maint/ftpdir directory does not have write permissions

step 2 only

<rcLogicalVolumeDirectory>/ftpdir directory has write permissions


<rcLogicalVolumeDirectory>/ftpdir directory does not have write permissions

step 3 only

storage disk has sufficient space


storage disk does not have sufficient space

step 4 only

<rcLogicalVolumeDirectory> path is correct


<rcLogicalVolumeDirectory> path is incorrect

correct the <rcLogicalVolumeDirectory> path into the CONFSTRM

<rcLogicalVolumeDirectory>/ftpdir is a directory


<rcLogicalVolumeDirectory>/ftpdir is not a directory

step 5 only

IFT alarm persists once you have performed the appropriate steps in this procedure




where<rcLogicalVolumeDirectory> is the logical volume that is assigned to the billing stream in the confstrm. The billing files are stored in the specified path.

Note: The next interval recreates the correct permissions and recreates all links.

4 Retrieve some closed not sent files and rename them to closed sent.

Note 1: Closed not sent files for DNS and DIRP have the file extensions of .pri and .unp respectively. When you rename them, change the file extensions to .sec and .pro respectively.Note 2: The closed sent files are removed from the system to make available more disk space. If you continue to receive the IFT alarm, consider increasing the size of the logical volume.

5 Remove the <rcLogicalVolumeDirectory>/ftpdir directory: > rm /<rcLogicalVolumeDirectory>/ftpdir

<rcLogicalVolumeDirectory> is the logical volume that is assigned to the billing stream in the confstrm. The billing files are stored in the specified path.

Note: At the next transfer interval, the correct permissions and all links are re-created.

6 You have completed the procedure.



Clearing an LODSK alarm

PurposeUse this procedure to clear a low disk storage (LODSK) alarm.

Indication

At the mtc level of the mapci, LODSK appears under the APPL header of the alarm banner, and indicates a storage alarm.

MeaningThe closedNotSent directory is reaching its capacity. The core manager generates the SDMB355 log report when this alarm is raised.

ImpactAs the storage becomes full, alarms notify you of how much capacity is used. In addition, there is a possibility that the DMS/CM does not go into backup mode if the disks reach 100 percent capacity.

ActionThe following flowchart is a summary of the procedure. Use the instructions in the procedure to clear the alarm.

CAUTION Possible Loss of ServiceIf you receive a LODSK alarm, transfer (FTP) the billing files in the closedNotSent directory, or write to tape immediately. Refer to Verifying the file transfer protocol on page 143 for more information.



LODSK alarm clearing flowchart

Clearing a LODSK alarm

At the MAP1 Use the procedure Verifying the file transfer protocol on

page 143 to determine if the FTP is working properly.

If the alarm Do

clears you have completed this procedure

does not clear refer to procedure Verifying the FTP Schedule on page 149

if the alarm persists, contact your next level of support

Receive LODSKalarm

Send the primarybilling files to thedestination andcontact your nextlevel of support

Verify that FTPis working

Verify that the FTPschedule is working

Start

End



Clearing a NOBAK alarm

PurposeUse this procedure to clear a no-backup (NOBAK) alarm.

IndicationNOBAK appears under the APPL header of the alarm banner at the MTC level of the MAP display and indicates a critical alarm for the backup system.

MeaningThis alarm only occurs if the volumes that are configured for backup are 100 percent full. If the stream is configured as• both




ATTENTIONThe option to configure a billing stream as “both” is only intended to be a temporary path while you are performing maintenance and alarm clearing tasks. The option to set a billing stream to the both mode on a permanent basis is not supported.



NOBAK alarm clearing flowchart

Clearing a NOBAK alarm



where <billing_stream> is the name of the billing stream


> DispAL

Start

End

Receive NOBAKalarm



Run a manual audit












SysB perform the procedure for the alarm or the condition, and then go to step 5.


ManB Go to step 8

Bkup Go to step 8

If the alarm Do

cleared step 6





If the alarm Do

cleared step 10







Clearing a NOCLNT alarm

PurposeUse this procedure to clear a NOCLNT alarm.

IndicationAt the MTC level of the MAP display, NOCLNT appears under the APPL header of the alarm banner and indicates an alarm.

MeaningThe stream was activated by the SDMBCTRL command before initialization was complete. If the stream is set to • on

the alarm is critical• both

the alarm is major

ImpactNo data is buffered by the SBA system. As a result, no data is backed up or made available for delivery to the core manager.

If the stream is set to both, data is still being routed to DIRP. Therefore, you can send the billing records to the operating company collector through the previously-established network used by DIRP.

ActionThis alarm only occurs in rare cases during installation. If this alarm occurs, contact your next level of support.

ATTENTIONThe option to set a billing stream to both is only intended to be a temporary path while you are performing maintenance and alarm clearing tasks. The option to set a billing stream to the both mode on a permanent basis is not supported.


Clearing a NOCOM alarm

PurposeUse this procedure to clear a no communications (NOCOM) alarm.

IndicationAt the MTC level of the MAP display, NOCOM appears under the APPL header of the alarm banner and indicates a communication alarm.

MeaningEthernet infrastructure has failed between the Core and the core manager.

The most likely causes of this alarm are• OC-3 links are not in-service making the core manager SysB• core manager power is off, or• core manager is rebooting

ImpactNo data is transferred to the core manager. Data is sent to the configured backup disk on the core.

If the stream is set to both, data is still being routed to device independent recording package (DIRP). You can send the billing records to the operating company collector through the previously established network used by DIRP.

ATTENTIONThe option to set a billing stream to both is only intended to be a temporary path while you are performing maintenance and alarm clearing tasks. The option to set a billing stream to the both mode on a permanent basis is not supported.



Procedure

At the MAP1 Access the APPL SDM Menu level:

> mapci;mtc;appl;sdm

2 Busy the core manager:> bsy

3 Return the core manager to service:> rts

Note 1: Returning the core manager to service establishes communication between the core and the core manager. If the first attempt fails to return the core manager to service, the system re-attempts to establish communication until it is successful.Note 2: The SDM Billing Application (SBA) and any streams configured for real-time billing (RTB) are also returned to service when the core manager is returned to service. Log report SDMB375 is generated when a stream configured for RTB fails to return to service.

4 Determine the status of the alarm.

If the core manager is Do

Offl or SysB step 2

ManB step 3

InSv or ISTb step 4

If the core manager Do


step 4



If the alarm Do

cleared step 5




At the core manager5 Check for log SDMB375.




9 Busy the stream:> bsy <stream_name>

where:<stream_name>

is the name of the billing stream configured for RTB (for example, OCC)

10 Return the stream to service:> rts <stream_name>

where:<stream_name>

is the name of the billing stream configured for RTB (for example, OCC)

If the system Do

generates log SDMB375 step 6

does not generate log SDMB375









Clearing a NOFL alarm

PurposeUse this procedure to clear a no file (NOFL) alarm.

IndicationNOFL appears under the APPL header of the alarm banner at the MTC level of the MAP display and indicates a critical alarm for the backup system.

MeaningOn startup, the SBA backup file system is unable to create a file. If the stream is set to:• both



ImpactBecause no file is available for SBA data storage, data intended for storage is lost.


ATTENTIONThe option to configure a billing stream as both is only intended to be a temporary path while you are performing maintenance and alarm clearing tasks. The option to configure a billing stream to the both mode on a permanent basis is not supported.



NOFL alarm clearing flowchart

Clearing a NOFL alarm





> DispAL

Start

End

Receive NOFLalarm



Run a manual audit



4 Determine the status of the billing stream.










ManB Go to step 8

Bkup Go to step 8

If the alarm Do

cleared step 6





If the alarm Do

cleared step 10




Clearing a NOREC alarm

IndicationAt the MTC level of the MAP display, NOREC appears under the APPL header of the alarm banner. It indicates an alarm for the recovery system.

MeaningThe SBA system is unable to create a recovery stream. The most likely reasons for not being able to start a recovery stream include the following:• the system is out of buffers (also causes a NOSTOR alarm).• the disk on the core manager is full (also causes DSKWR and

LODSK alarms)

If the stream is set to if the stream is set to: • on

the alarm is major, or• both

the alarm is minor

ImpactNo backup files are recovered by the SBA system.

If the stream is set to both, data is still being routed to DIRP. Therefore, you can send the billing records to the operating company collector through the previously-established network used by DIRP.

ActionContact your next level of support when you receive this alarm.



Clearing a NOSTOR alarm

PurposeUse this procedure to clear a no storage (NOSTOR) alarm.

IndicationNOSTOR appears under the APPL header of the alarm banner at the MTC level of the MAP display and indicates a critical alarm for the backup system.

MeaningThe SBA buffer pool cannot allocate buffers. This means that all buffers are in use, though it does not necessarily mean that the disk is full.

The NOSTOR alarm is usually seen when the system is in backup mode and the traffic is too high for the disk to process. If the disk stream is configured as:• both







NOSTOR alarm clearing flowchart

Clearing a NOSTOR alarm



where <stream_name> is the name of the billing stream


> DispAL

Start

End

Receive NOSTORalarm



Run a manual audit












RBsy refer to Clearing a major SBACP alarm on page 134, and then go to step 5.

ManB Go to step 8

Bkup Go to step 8

If the alarm Do

cleared step 6





If the alarm Do

cleared step 10




Clearing an RTBCF alarm

IndicationAt the MTC level of the MAP display, RTBCF appears under the APPL header of the alarm banner. It indicates a critical alarm for the Real Time Billing (RTB) application.

The core manager generates the SDMB375 log report when this alarm is raised. When this alarm is cleared, the core manager generates the SDMB675 log report.

Refer to the log reports for more information about the condition causing the alarm.

MeaningThe RTBCF alarm indicates that RTB is unable to transfer an open file after RTBMaxConsecutiveFailures.

ImpactRTB moves to the SysB state and stops transferring open files.

ActionRefer to log report SDMB675 for more information about the RTBCF alarm. If required, contact your next level of support.



Clearing a NOVOL alarm

PurposeUse this procedure to clear a no disk volume (NOVOL) alarm.

IndicationNOVOL appears under the APPL header of the alarm banner at the MTC level of the MAP display and indicates a critical alarm for the backup system.


MeaningOn startup, the SBA backup file system is unable to find a volume in which to create a file. If the stream is configured as: • both



ImpactBecause there is no volume available for SBA storage, data intended for backup storage can be lost.





NOVOL alarm clearing flowchart

Clearing a NOVOL alarm



where <stream_name> is the name of the billing stream


> DispAL

Start

End

Receive NOVOLalarm



Run a manual audit



4 Determine the status of the billing stream.










RBsy refer to Clearing a major SBACP alarm on page 134, and then go to step 5.

ManB Go to step 8

Bkup Go to step 8

If the alarm Do

cleared step 6





If the alarm Do

cleared step 10






Clearing an RTBER alarm

PurposeUse this procedure to clear an RTBER alarm.

IndicationAt the MTC level of the MAP display, RTBER appears under the APPL header of the alarm banner, and indicates a critical alarm for real time billing (RTB).

MeaningThe RTBER alarm indicates that RTB has encountered a severe system error trying to re-establish file transfers with the data processing and management system (DPMS).

ImpactThis alarm has the following impact:• RTB is unable to send billing files to the DPMS• RTB moves to the SysB state• the condition generates an SDMB375 log

Action

At the MAP1 Read the text in log SDMB375 for the cause of error. 2 Use the Logs reference documentation for SDMB375 to

determine the actions to take to clear each type of error.3 After you correct the error, return the RTB destination to service.

The system generates SDMB675 when the error is corrected and the alarm is cleared.




Clearing an RTBPD alarm

PurposeUse this procedure to clear an RTBPD alarm.

IndicationAt the MTC level of the MAP display, RTBPD appears under the APPL header of the alarm banner and indicates a critical alarm for the RTB program.


MeaningThe RTBPD alarm indicates that the RTB controlling process died and that RTB is halted.

ImpactRTB moves to the SysB state.

ActionRefer to log reports SDMB375 and SDMB675 for more information about the condition causing the alarm, and corrective actions. If required, contact your next level of support.



Clearing an RTBST alarm

IndicationAt the MTC level of the MAP display, RTBST appears under the APPL header of the alarm banner and indicates a critical alarm for the RTB program.


MeaningThe RTBST alarm is raised if the schedule tuple is deleted or invalid for RTB.

ImpactRTB moves to the SysB state.

ActionRefer to the log reports for more information about the condition causing the alarm.

Refer to log report SDMB675 for more information about the RTBST alarm. You need to verify that the• protocol is set to RFTPW, and • file format type is set to “DIRP” in the schedule tuple associated with

the alarm

If required, contact your next level of support.


Clearing a major SBACP alarm

PurposeUse this procedure to clear an SBACP alarm.

IndicationAt the MTC level of the MAP display, SBACP appears under the APPL header of the alarm banner and indicates a major alarm for the SDM Billing Application (SBA).

MeaningThe SBA is shutting down because either• a user busied the SBA or the core manager, or• a process is repeatedly dying and the SBA shut down

ImpactThe SBA is out of service.

ActionUse the instructions in the following procedure to clear the alarm.

At the MAP1 Access the APPL SDM Menu level:

> mapci;mtc;appl;sdm

2 Busy the core manager:> bsy

3 Return the core manager to service:> rts

Note 1: Returning the core manager to service establishes communication between the core and the core manager. If the first attempt fails to return the core manager to service, the system attempts to establish communication until it is successful.

If the core manager is Do

Offl or SysB step 2

ManB step 3

InSv or ISTb step 6



Note 2: The SDM Billing Application (SBA) and any streams configured for real-time billing (RTB) are also returned to service when the core manager is returned to service. Log report SDMB375 is generated when a stream configured for RTB fails to return to service.

At the core manager4 Go to the Appl level of the cbmmtc tool by typing:

#cbmmtc appl

5 Busy the SBA application:> bsy <SBA_no>where<SBA_no> is the number next to the SBA application.

6 Return the SBA application to service:> rts <SBA_no>where<SBA_no> is the number of the SBA application.

Note: Any streams configured for real-time billing (RTB) are also returned to service.

If the core manager Do

returned to service successfully

step 4



If the SBA application is Do

ISTB, Offl, or SysB step 5

ManB step 6

InSv, and the alarm is cleared step 13

InSv, but the alarm is still present




Log report SDMB375 is generated when a stream configured for RTB fails to return to service.

7 Return the RTB streams to service. Exit the maintenance interface.> quit all




11 Busy the stream:> bsy <stream_name>

where:<stream_name>


If the SBA Do

returned to service successfully and the alarm is cleared

step 7

returned to service successfully and the alarm is still present






12 Return the stream to service:> rts <stream name>

where:<stream name>










Clearing a minor SBACP alarm

IndicationAt the MTC level of the MAP display, SBACP appears under the APPL header of the alarm banner, and indicates a minor alarm for the SBA program.

MeaningThe SBA program is shutting down because one of the processes has failed three times in one minute.

ImpactThe SBA program ends, but restarts within two minutes.

ActionThe following flowchart is a summary of the procedure. Use the instructions in the following procedure to clear the alarm.



SBACP (minor) alarm clearing flowchart

Clearing a minor SBACP alarm

At the MAP1 Wait 2 minutes for the SBA to restart.2 Contact your next level of support if the

• alarm does not clear, or • SBA application fails three times within one minute


Wait 2 minutes

Contact next levelof support

Receive SBACPminor alarm

Start

End

Y NAlarm cleared?



Clearing an SBAIF alarm

PurposeUse this procedure to clear a SuperNode Billing Manager file transfer (SBAIF) alarm.

IndicationAt the MTC level of the MAP display, SBAIF appears under the APPL header of the alarm banner and indicates a major alarm.

The system also generates an SDMB390 log.

MeaningSuperNode Billing Application (SBA) cannot perform a scheduled transfer of billing files from the core manager to a downstream destination.

ImpactIf the alarm does not clear, SBA is not able to transfer files to the downstream destination: • SBA uses local storage on the core manager to store billing files.

Alarms are generated as SBA uses available capacity. • if local storage becomes full, the Core is unable to send billing

records to the core manager. The Core sends the billing records to backup storage. Alarms are generated as the Core uses available capacity.

ActionThe following flowchart is a summary of the procedure. Use the instructions in the procedure to clear the alarm.



SBAIF alarm clearing flowchart

Clearing an SBAIF alarm

At a workstation or console1 Clear all faults in the system using the appropriate procedures in

this document.The SBAIF alarm clears when the fault is corrected.

2 Access the core manager.

If the SBAIF alarm Do

clears step 2

does not clear Contact your next level of support.

Did the alarmclear?

Y

N

Clear all faultsin the system

N

Start

Finish

Contact yournext level ofsupport.

Did

resume?

Y

file transfers



3 Monitor the billing-related logs and look for log SDMB690, which indicates that the SBAIF alarm has cleared.

4 Make sure SBA successfully performs a scheduled transfer of billing files. Monitor billing-related logs and look for log SDMB691, which indicates the file transfer schedule is now working for the stream.

Note: The length of time for SBA to resume transferring billing files depends on the following configured parameters:• the number of active scheduled tuples• the time interval to transfer files


If log SDMB690 Do

is present step 4

is not present contact your next level of support.

If Do

log SDMB691 indicates the file transfer schedule is now working for the stream.

step 5

log SDMB691 or any other log indicates a new problem with the scheduled transfer of billing files




Verifying the file transfer protocol

PurposeYou can use this procedure on the core manager to verify that the file transfer protocol (FTP) is configured correctly to transfer files.

ActionThe following flowchart summarizes the steps outlined in the procedure.

FTP verification flowchart

Is scheduletuple

correct?

Does closec returna value?

Is sendfilesuccessful?

Does listfilereturn a value?

Is FTP N

Start

Next level of support

End

N Correct the

parameters

Y

Manually

a fileFTP

Correct the downstream

successful?Is problem

on downstreamnode?

N

Is sendfilesuccessful?

YY

YY

N

N N

N

Y



Verify the FTP

At the core manager1 Access the bill maintenance level:

# billmtc

2 Access the file system:> filesys

3 Close active billing files:> closec <stream_name>

where<stream_name> is the name of the stream.

Note: You must close any active billing files prior to the FTP process.

4 Determine the results of the closec command.

5 List the primary file (closedNotSent directory):> listfile <stream_name> where<stream_name> is the name of the stream

6 If the listfile command does not return a filename, contact your next level of support because this can indicate a problem with billing generation.

7 Send the primary file (closedNotSent directory):> sendfile <stream_name>where<stream_name> is the name of the stream.

Note: The sendfile command sends the billing file to the operating company billing collector.

8 Go to the previous level:> quit

If the “closec” command Do

returns a filename step 7

does not return a filename step 5



9 Determine the results of the sendfile command.

Note: Observe the SDMB logs on the CM in logutil to determine why the sendfile command is not successful prior to continuing with step 10.


11 List the parameters of the schedule tuple:> list

12 Reset the schedule tuple parameters:> change

13 Enter the stream name (name of billing file).14 Enter the file format.15 Enter the destination name.

Note: The destination name can be up to 15 alphanumeric characters.

16 Observe the schedule tuple displayed.17 Enter the corrected parameters.

Note: You can change parameters one at a time or you can choose to change the entire schedule tuple.

18 Enter the new values of the parameters you have chosen to change.

If the “sendfile” command is Do

successful you have completed this procedure

not successful step 10

If the parameters are Do

correct, but you are receiving an alarm

step 21

incorrect step 12



19 Save the changed parameters:> save


21 Login as root user.22 Attempt to FTP any billing file to the destination used by the

“sendfile” command. This action verifies that FTP is functioning properly for the node and directory.

Note: You can use any billing file for step 22 because you are only verifying login and write ability on the downstream node.

23 Exit back to the command prompt:> quit all

24 Login as root user.25 Copy a billing file from the closedNotSent directory to a

temporary directory:# cp /<logical_vol>/closedNotSent/<file> /tmp

where<logical_vol> is the logical volume for the stream that is in use <file> is the name of the billing file in the closedNotSent directory

Note: You can obtain the logical volume from the confstrm level of the billmtc by requesting a list on the stream.

If you have Do

corrected the parameters in the schedule tuple

step 7

determined that the parameters are correct

• step 20 (verify login and write permissions are correct for FTP process without testing a billing file), OR

• step 23 (verify login and write permissions are correct for FTP process while testing an actual billing file)



26 Access the /tmp directory:# cd /tmp

27 FTP to the downstream node:> ftp <address> <port>

where<address> is the Primary_Destination IP address of the destination node <port> is the Primary_Port of the destination node

28 Log onto the node when prompted by the FTP (Remote_Login and Remote_Password defined in the schedule tuple):

Note: A successful login is confirmed by a “230 User <address> logged in” message returned by the FTP.If the login attempt is unsuccessful, obtain a valid login ID and password and update the schedule tuple with the valid values.

29 Change the directory to the one the schedule tuple is using:ftp> cd <remote_directory>where<remote_directory> is the Remote_Storage_Directory defined in the schedule tuple.

Note: A successful login is confirmed by a “250 CWD command successful” message returned by the FTP.

30 If the “cd” command is unsuccessful, obtain a valid directory from the downstream node and update the schedule tuple with the valid values.

31 Set the file transfer mode to binary:ftp> binary

Note: A successful command is confirmed by a “200 Type set to l” message returned by the FTP.

32 Attempt to write a file to the destination node directory used for billing:ftp> put <file>

where<file> is the name of a billing file that is copied to the /tmp directory in step 25.



33 Exit from the FTP session:ftp> quit

34 Correct the directory permissions to allow write access.35 Repeat steps 21 through 33.36 Send the primary files in the closedNotSent directory:

> sendfile <billing_stream> dest <dest_name>

where<billing_stream> is the name of the billing stream<dest_name> is the name you choose to name the destination (for example, fraud detection).

Note: The sendfile command with the dest option sends the billing file to the specified destination only.

If the file transfer is Do

successful step 36

unsuccessful because of a permission error

step 34

unsuccessful for a reason other than permission error

step 36

If the “sendfile” command is Do

successful you have completed this procedure

unsuccessful contact your next level of support



Verifying the FTP Schedule

PurposeYou can use this procedure to verify that the schedule is configured correctly and can transfer files using FTP.

ActionThe following flowchart summarizes the steps in the procedure.

Verifying the FTP schedule flowchart

Verifying the FTP schedule

At any workstation or console1 Log in to the core manager.2 Access the bill maintenance level:

# billmtc

3 Verify the schedule tuple:> schedule

4 List the parameters of the schedule tuple:> list


correct contact your next level of support

incorrect step 5

Correct theparameter

Y

N

Start

End

Verify theschedule

tuple?



5 Reset the schedule tuple parameters:> change

6 Enter the stream name (billing file name).7 Enter the file format.8 Enter the destination name.

Note: The destination name can be up to 15 alphanumeric characters.

9 Observe the schedule tuple displayed.10 Enter the parameters that you need to correct.

Note: You can change parameters one at a time or you can choose to change the entire schedule tuple.

11 Enter the new values of the parameters you have chosen to change.

12 Save the changed parameters:> save

13 Wait for the next scheduled transfer to execute after the scheduled transfer interval for the alarm not to appear.



correct, but still receiving an alarm


correct and no longer receiving an alarm

step 13



Replacing one or more failed disk drives on an SSPFS-based server

ApplicationUse this procedure to replace one or more failed disk drives on a Succession Server Platform Foundation Software (SSPFS)-based server (a Netra t1400 or a Netra 240 server). Also use this procedure if a disk drive was pulled out by mistake. Simply re-inserting the disk is not sufficient to recover.

Disk failures will appear as IO errors or SCSI errors from the Solaris kernel. These messages will appear in the system log and on the console terminal. To indicate a disk failure, log SPFS310 is generated, and an alarm light is illuminated on the front panel. After the disk is replaced, the alarm light will go off within a few minutes.

Systems installed with SSPFS use disk mirroring. With mirrored hot-swap disks, a single failed disk can be replaced without interrupting the applications running on the server. Thus, a single disk can be replaced while the system is in-service. Follow one of the links below for a view of the disks on a Netra t1400 and Netra 240:

• Netra t1400 on page 152

• Netra 240 on page 153

The steps to replace a failed drive are to identify the failed drive, replace it physically, and replace it logically.

Follow one of the links below according to your office configuration to replace the failed disk drives:

• Replacing failed disks on a Netra t1400 on page 154

• Replacing failed disks on a Netra 240 simplex on page 157

• Replacing failed disks on a Netra 240 cluster (two-server) on page 159



Netra t1400Each Netra t1400 is equipped with four hot-swap drives: “c0t0d0”, “c0t1d0”, “c0t2d0”, and “c0t3d0”. Each physical drive is divided into slices, which are named based on the physical disk and a slice number. For example, “c0t0d0s0” is the first slice of the physical disk “c0t0d0”.

The following figure identifies the disk drives of the Netra t1400.

Netra t1400 disk drives



Netra 240Each Netra 240 is equipped with two hot-swap drives: “c1t0d0”, and “c1t1d0”.

The following figure identifies the disk drives of the Netra 240.

Netra 240 disk drives

PrerequisitesYou have a replacement disk.

c1t0d0

c1t1d0



ActionPerform the following steps to complete this procedure.

Replacing failed disks on a Netra t1400

At the server

1 Locate the failed disk(s) using figure Netra t1400 disk drives on page 152.

2 Physically replace the disk using the documentation for the Netra t1400. When complete, proceed with step 3 in this procedure to logically replace the disk.

Note: If more than one disk needs to be replaced, physically replace one disk and return to this procedure to logically replace the disk (step 3), before you proceed to physically replace the next failed disk.

At your workstation

3 Logically replace the disk you just physically replaced.

If you physically replaced Do

c0t0d0 step a

c0t1d0 step b

c0t2d0 step c

c0t3d0 step d



a Logically replace disk “c0t0d0” by entering the following sequence of commands:

# metadb -d c0t0d0s7

# prtvtoc -h /dev/rdsk/c0t1d0s2 | fmthard -s - /dev/rdsk/c0t0d0s2

# metadb -a -c 2 c0t0d0s7

# metareplace -e d2 c0t0d0s1





Proceed to step 4

b Logically replace disk “c0t1d0” by entering the following sequence of commands.









Proceed to step 4

c Logically replace disk “c0t2d0” by entering the following sequence of commands.



# metadb -a -c 2 cot2d0s7


Proceed to step 4



d Logically replace disk “c0t3d0” by entering the following sequence of commands.





Proceed to step 4

4 Use the following table to determine your next step.


6 Restore the file systems and oracle data. Refer to procedure “Performing a full system restore on a Sun server - SN06.2 or greater” in the ATM/IP Security and Administration document, NN10402-600, if required.

Note: As long as one disk from each pair is good, the data in the system is intact. When both disks in a pair fail, the data needs to be restored.


If you Do

have another disk to physically replace

step 1

do not have another disk to physically replace

step 5

If you replaced Do

1 disk you have completed this procedure

2 non-mirrored disks (i.e. c0t0d0 and c0t2d0 or c0t3d0, or c0t1d0 and c0t2d0 or c0t3d0)


2 mirrored disks (i.e. c0t0d0 and c0t1d0, or c0t2d0 and c0t3d0)

step 6

3 disks step 6

4 disks step 6



Replacing failed disks on a Netra 240 simplex

At the server

1 Locate the failed disk(s) using figure Netra 240 disk drives on page 153.

2 Physically replace the disk using the documentation for the Netra 240. When complete, proceed with step 3 in this procedure to logically replace the disk.

Note: If both disks need to be replaced, physically replace one disk and return to this procedure to logically replace the disk (step 3), before you proceed to physically replace the other failed disk.

At your workstation











Proceed to step 4


c1t0d0 step a

c1t1d0 step b



b Logically replace disk “c1t1d0” by entering the following sequence of commands:









Proceed to step 4




Note: As long as one disk is good, the data in the system is intact. When both disks fail, the data needs to be restored.


If you Do

have another disk to replace step 1

do not have another disk to replace

step 5

If you replaced Do

1 disk you have completed this procedure

both disks step 6



Replacing failed disks on a Netra 240 cluster (two-server)

At the server

1 Locate the failed disk(s) using figure Netra 240 disk drives on page 153.

2 Use the following guidelines to determine the steps you need to do, and do only those steps in the order they appear:

• to replace one disk on one unit

— physically replace the disk (step 3)

— logically replace this disk (step 4)

• to replace one disk on each unit

— physically replace the disk on the Active unit first (step 3)


— physically replace the disk on the other unit (step 3)


• to replace both disks on a unit

— physically replace one disk on the unit (step 3)


— physically replace the other disk on this unit (step 3)


— clone the image from the Active unit onto this unit (step 6)

• to replace three disks

— physically replace the disk on the Active unit first (step 3)


— physically replace one disk on the other unit (step 3)


— physically replace the other disk on this same unit (step 3)





• to replace four disks

— physically replace one disk on the unit with the most recent backup (step 3)




— restore the file systems and oracle data on this unit (step 5)

— physically replace one disk on the other unit (step 3)





3 Physically replace the disk using the documentation for the Netra 240. When complete, proceed with step 4 in this procedure to logically replace the disk.

At your workstation



c1t0d0 step a

c1t1d0 step b












b Logically replace disk “c1t1d0” by entering the following sequence of commands:










Note: As long as one disk is good, the data in the system is intact. When both disks fail, the data needs to be restored.

6 Clone the data from the Active unit. Refer to procedure “Cloning the image of one node in a cluster to the other node” in the ATM/IP Security and Administration document, NN10402-600, if required.



Shutting down an SSPFS-based server

ApplicationUse this procedure to shut down a Succession Server Platform Foundation Software (SSPFS)-based server, which may be hosting one or more of the following components:

• CS 2000 Management Tools

• Integrated Element Management System (EMS)

• Audio Provisioning Server (APS)

• Media Gateway (MG) 9000 Manager

• CS 2000 SAM21 Manager

• Network Patch Manager

• Core Billing Manager (CBM)

Use one of the following procedures according to your office configuration:

• One-server configuration on page 163

• Two-server (cluster) configuration on page 164

PrerequisitesYou must have root user privileges.

ATTENTIONThe SSPFS-based server may be hosting more than one of the above components, therefore, ensure it is acceptable to shut down the server.



ActionUse one of the following procedures according to your office configuration:

• One-server configuration on page 163

• Two-server (cluster) configuration on page 164

One-server configuration

At your workstation

1 Telnet to the server by typing

> telnet <IP address>

and pressing the Enter key.

where

IP addressis the IP address of the SSPFS-based server you want to power down

2 When prompted, enter your user ID and password.

3 Change to the root user by typing

$ su - root


4 When prompted, enter the root password.

5 Shut down the server by typing

# init 0


The server shuts down gracefully, and the telnet connection is closed.

6 If required, turn off the power to the server at the circuit breaker panel of the frame.


To bring the server back up, turn on the power to the server at the circuit breaker panel of the frame. The server recovers on its own once power is restored.



Two-server (cluster) configuration

At your workstation

1 Telnet to the Inactive server by typing



where

IP addressis the physical IP address of the Inactive SSPFS-based server in the cluster you want to power down



$ su - root



5 Shut down the Inactive server by typing

# init 0



6 If required, turn off the power to the Inactive server at the circuit breaker panel of the frame. You have completed a partial power down (one server).

If you want to perform a full power down (both servers), proceed to step 7, otherwise, you have completed this procedure.



7

Telnet to the Active server by typing



where

IP addressis the physical IP address of the Active SSPFS-based server in the cluster you want to power down



$ su - root



11 Shut down the Active server by typing

# init 0



12 If required, turn off the power to the servers at the circuit breaker panel of the frame. You have completed a full power down (two servers).


To bring the servers back up, turn on the power to the servers at the circuit breaker panel of the frame. The servers recover on their own once power is restored.

ATTENTIONOnly perform the remaining steps if you want to perform a full power down, which involves powering down both servers in the cluster.




Erasing the contents of a CD/DVD on an SSPFS-based server

ApplicationUse this procedure to erase the contents of a CD/DVD on a Succession Server Platform Foundation Software (SSPFS)-based server (Netra 240), when you want to re-use the CD/DVD.

PrerequisitesNone


At the server

1 Insert the CD/DVD you want to erase into the drive.

At your workstation

2 Log in to the server by typing

> telnet <server>


where

serveris the IP address or hostname of the SSPFS-based server


4 Erase the contents of the CD/DVD by typing

$ cdrw -b all

and pressing the Enter key

Note: You can also use the “fast” and “session” arguments. For more details, refer to the man pages by typing man cdrw.

5 Remove the CD/DVD from the drive.

You have completed this procedure.


Increasing the size of a file system on an SSPFS-based server

ApplicationUse one of the following procedures to increase the size of a file system on a Succession Server Platform Foundation Software (SSPFS)-based server:

• Simplex configuration (one server) on page 168

• High-availability configuration (two servers) on page 173

It is recommended you perform this procedure during off-peak hours.

The Succession Server Platform Foundation Software (SSPFS) creates file systems to best fit the needs of applications. However, it may be necessary to increase the size of a file system.

Not all file systems can be increased. The table below lists the file systems that cannot be increased, and lists examples of those that can be increased.

Note: Not all the file systems that can be increased are listed.

While file systems are being increased, writes to the file system are blocked, and the system activity increases. The greater the size increase of a file system, the greater the impact on performance.

SSPFS file systems

Cannot be increased Can be increased (examples)

/ (root) /data

/var /opt/nortel

/opt /data/oradata

/tmp /audio_files

/PROV_data

/user_audio_files

/data/qca

/data/mg9kem/logs



PrerequisitesIt is recommended that you back up your file systems and oracle data (if applicable) prior to performing this procedure. Refer to procedures Performing a data backup on an SSPFS-based server (I)SN06.2 or greater on page 183 and Performing a full backup of file systems on an SSPFS-based server (I)SN06.2 or greater on page 187 if required.


Simplex configuration (one server)

At your workstation


> telnet <server>


where

serveris the IP address or host name of the server



$ su - root





5 Determine the amount of disk utilization by the file systems as follows:

a Access the command line interface by typing

# cli


Example response

Command Line Interface 1 - View

2 - Configuration

3 - Other

X - exit select -

b Enter the number next to the “View” option in the menu.

Example response

View 1 - sspfs_soft (Display Software Installation Level Of SSPFS) 2 - chk_sspfs (Check SSPFS Processes) 3 - sw_conf (The software configuration of the znc0s0jx) 4 - cpu_util (Overall CPU utilization) 5 - cpu_util_proc (CPU utilization by process) 6 - port_util (I/O port utilization) 7 - disk_util (Filesystem utilization) X - exit select -



c Enter the number next to the “disk_util” option in the menu.

Example response

The “capacity” column indicates the percentage of disk utilization by the file system, which is specified in the “Mounted on” column.

6 Note the file system you want to increase, as well as its current size (under column “Kbytes”).

7 Exit each menu level of the command line interface to eventually exit the command line interface, by typing

select - x


8

Determine the size by which to increase the file system, by subtracting the desired size for the file system based on your specific needs, from its current size (noted in 6).

For example, to determine the size by which to increase the “qca” file system, subtract its current size, 122847k from the desired size, for example, 256000k. You would increase the size of the “qca” file system by 133153k, or 133MB.

ATTENTIONBefore you proceed with this procedure, ensure the file system you want to increase is full or nearly full and that its content is valid application data. Remove any unneeded files or files generated in error that are taking up disk space.



9 Determine the amount of free disk space that can be allocated to file systems as follows:

a Determine the amount of free disk space on your system by typing

# echo ‘/opt/nortel/sspfs/fs/meta.pl fs‘ 2048 / 5000 - p | dc


Note: Use the back quote on the same key as the Tilde (~) for /opt/nortel/sspfs/fs/meta.pl fs.

The resulting number is the amount of free disk space in megabytes (MB) that can be allocated to existing file systems.

b Use the following table to determine your next step.

If the value is Do

less than zero (0) contact Nortel Networks for assistance

more than zero (0) step b

If Do

the value you determined in step 8 (size by which to increase the file system) is greater than the value you obtained in step 9a (amount of free disk space you can allocate to file systems)

contact Nortel Networks for assistance

the value you determined in step 8 (size by which to increase the file system) is less than the value you obtained in step 9a (amount of free disk space you can allocate to file systems)

step 10



10

Increase the size of the file system by typing

# filesys grow -m <mount_point> -s <size>m

Where

mount_pointis the name of the file system you want to increase (noted in step 6)

sizeis the size in megabytes (m) by which you want to increase the file system (determined in step 8)

Example # filesys grow -m /data -s 512m

Note: The example above increases the “/data” file system by 512 megabytes (MB).


ATTENTIONOnce you increase the size of a file system, you cannot decrease it. Therefore, it is strongly recommended that you grow a file system in small increments.



High-availability configuration (two servers)

At your workstation

1 Log in to the Inactive node by typing

> telnet <server>


where

serveris the physical IP address of the Inactive node in the cluster

Note: If you use the cluster IP address, you will log in to the Active node. Therefore, ensure you use the physical IP address of the Inactive node to log in.



$ su - root



At the Inactive node

5 Verify the cluster indicator to ensure you are logged in to the Inactive node, by typing

# ubmstat


ATTENTIONDuring this procedure, the cluster will be running without a standby node. The duration is estimated at approximately one hour.

If the system response is Do

ClusterIndicatorSTBY step 6

ClusterIndicatorACT step 1



6 Verify the status of file systems on this server by typing

# udstat


7 Determine the amount of disk utilization by the file systems as follows:

a Access the command line interface by typing

# cli


Example response


2 - Configuration

3 - Other

X - exit select -

b Enter the number next to the “View” option in the menu.

Example response


If the file systems are Do

STANDBY normal UP clean step 7

not STANDBY normal UP clean




c Enter the number next to the “disk_util” option in the menu.

Example response

The capacity column indicates the percentage of disk utilization by the file system, which is specified in the Mounted on column.

8 Note the file system you want to increase, as well as its current size (under column Kbytes).

9 Exit each menu level of the command line interface to eventually exit the command line interface, by typing

select - x


10

Determine the size by which to increase the file system, by subtracting the desired size for the file system based on your specific needs, from its current size (noted in 8).

For example, to determine the size by which to increase the “qca” file system, subtract its current size, 122847k from the desired size, for example, 256000k. You would increase the size of the “qca” file system by 133153k, or 133MB.

ATTENTIONBefore you proceed with this procedure, ensure the file system you want to increase is full or nearly full and that its content is valid application data. Remove any unneeded files or files generated in error that are taking up disk space.



11 Determine the amount of free disk space that can be allocated to file systems as follows:

a Determine the amount of free disk space on your system by typing

# echo ‘/opt/nortel/sspfs/fs/meta.pl fs‘ 2048 / 5000 - p | dc


Note: Use the back quote on the same key as the Tilde (~) for /opt/nortel/sspfs/fs/meta.pl fs.

The resulting number is the amount of free disk space in megabytes (MB) that can be allocated to existing file systems.

b Use the following table to determine your next step.

If the value is Do

less than zero (0) contact Nortel Networks for assistance

more than zero (0) step b

If Do

the value you determined in step 10 (size by which to increase the file system) is greater than the value you obtained in step 11a (amount of free disk space you can allocate to file systems)

contact Nortel Networks for assistance

the value you determined in step 10 (size by which to increase the file system) is less than the value you obtained in step 11a (amount of free disk space you can allocate to file systems)

step 12



12

Increase the size of the desired file system by typing

# GrowClusteredFileSystem.ksh <mount_point> <size>m

Where

mount_pointis the name of the file system you want to increase (noted in step 8)

sizeis the size in megabytes (m) by which you want to increase the file system (determined in step 10)

Example # GrowClusteredFileSystem.ksh /data/qca 10m

Note: The example above increases the “/data/qca” file system by 10 megabytes (MB).

13 Reboot the Inactive node by typing

# init 6


14 Wait for the Inactive node to reboot, then log in again using its physical IP address.

15 Verify the status of file systems on the Inactive node by typing

# udstat


ATTENTIONOnce you increase the size of a file system, you cannot decrease it. Therefore, it is strongly recommended that you grow a file system in small increments.

If the file systems are Do

STANBY normal UP clean step 16

not STANBY normal UP clean




16 Log in to the Active node by typing

> telnet <server>


where

serveris the physical IP address of the active node in the cluster



$ su - root



At the Active node

20 Stop the cluster by typing

# StopCluster

and press the Enter key.

This action causes a cluster failover and makes the active node inactive, and the inactive node active.

At the newly Active node

21 Clone the other node using procedure “Cloning the image of one node in a cluster to the other node” if required.




Starting the SAM21 Manager server application

ApplicationUse this procedure to start the SAM21 Manager server application on the CS 2000 Management Tools server.

PrerequisitesNone


At your workstation


> telnet <server>


where

serveris the IP address or host name of the CS 2000 Management Tools server



$ su - root




6 For the (I)SN06 release, start the SAM21 Manager server application as follows:

a Start the SAM21 Manager server application by typing

# /opt/nortel/sam21em/bin/sam21emCtrl start


If the release you are running is Do

(I)SN06 step 6 only

(I)SN06.2 or greater step 7 only



b Verify the SAM21 Manager server application started by typing

# /opt/nortel/sam21em/bin/sam21emCtrl status



7 For the (I)SN06.2 or greater release, start the SAM21 Manager server application as follows:

Note: In a two-server configuration, perform the steps that follow on the active side.

a Start the SAM21 Manager server application by typing

# servstart SAM21EM


b Verify the SAM21 Manager server application started by typing

# servman query -status -group SAM21EM





Stopping the SAM21 Manager server application

ApplicationUse this procedure to stop the SAM21 Manager server application on the CS 2000 Management Tools server.

PrerequisitesNone

Action

At your workstation


> telnet <server>


where

serveris the IP address or host name of the CS 2000 Management Tools server



$ su - root




6 For the (I)SN06 release, stop the SAM21 Manager server application as follows:

a Stop the SAM21 Manager server application by typing

# /opt/nortel/sam21em/bin/sam21emCtrl stop


If the release you are running is Do

(I)SN06 step 6 only

(I)SN06.2 or greater step 7 only



b Verify the SAM21 Manager server application stopped by typing

# /opt/nortel/sam21em/bin/sam21emCtrl status



7 For the (I)SN06.2 or greater release, stop the SAM21 Manager server application as follows:

Note: In a two-server configuration, perform the steps that follow on the active side.

a Stop the SAM21 Manager server application by typing

# servstop SAM21EM


b Verify the SAM21 Manager server application stopped by typing

# servman query -status -group SAM21EM





Performing a data backup on an SSPFS-based server (I)SN06.2 or greater

ApplicationUse this procedure to perform a data backup on a Succession Server Platform Foundation Software (SSPFS)-based server (Sun Netra t1400 or Sun Netra 240) running the (I)SN06.2 or greater release of the SSPFS.

Note: For systems running the (I)SN05 or (I)SN06 release of the SSPFS, use procedure “Performing a full backup of Oracle data on a Sun server (pre-(I)SN06.2)”.

The server can be hosting one or more of the following components:

• CS 2000 Management Tools

• Integrated Element Management System (EMS)

Note: If the server is hosting the Integrated EMS, it is highly recommended to purge the Integrated EMS event and performance data prior to executing the data backup. This reduces the size of the oracle space used by the Integrated EMS, and therefore, reduces the backup time, and can avoid a backup failure. The purge capability is only available in (I)SN07 onward.

• Audio Provisioning Server (APS)

• Media Gateway (MG) 9000 Manager

• CS 2000 SAM21 Manager

• Network Patch Manager

• Core Billing Manager (CBM)

Note: If the server is hosting the Core Billing Manager (CBM), it is not required to perform a data backup.

ATTENTIONIt is recommended that provisioning activities be put on hold during the time of the data backup.



PrerequisitesThis procedure has the following prerequisites:

• you must be running SSPFS (I)SN06.2 or greater

• you need a blank 4mm Digital Data Storage (DDS-3) tape of 125m and 12 GB to store the data on a Sun Netra t1400

• you need one or more blank DVD-RW of 4.7 GB to store the data on a Sun Netra 240 (the backup utility limits the storage to 2 GB for each DVD-RW)

Note: To reuse a DVD-RW, refer to procedure Erasing the contents of a CD/DVD on an SSPFS-based server on page 166, if required.


At the server

1 Insert the blank tape or DVD-RW into the drive.

At your workstation


> telnet <server>


where

serveris the IP address or hostname of the SSPFS-based server on which you are performing the backup



$ su - root


ATTENTIONThe database must be in sync with the Communication Server 2000 and the MG 9000 Manager (if present). Therefore, ensure you have an image of both before you proceed. Performing a restore from the Oracle database alone can cause data mismatches at the Communication Server 2000 and the MG 9000 Manager (if present).




6 If the server is hosting the Integrated EMS, and you want to purge the event and performance data, do step 7, otherwise proceed to step 8.

7 Purge the Integrated EMS event and performance data as follows:

Note: Purging the Integrated EMS event and performance data prior to executing the data backup, reduces the size of the oracle space used by the Integrated EMS, and therefore, reduces the backup time, and can avoid backup failure. The purge capability is only available in (I)SN07 onward.

a

Stop the Integrated EMS server by typing

# servstop IEMS


b Run the script to purge the data by typing

# /opt/nortel/iems/current/bin/purgeTempData.sh


c Start the Integrated EMS server by typing

# servstart IEMS



9 Rewind the tape by typing

# mt -f /dev/rmt/0 rewind


ATTENTIONThis step stops the Integrated EMS server, therefore, ensure it is acceptable at this time to stop the Integrated EMS server.

If you are using Do

a tape step 9

a DVD-RW step 10



10 Backup the data by typing

$ /opt/nortel/sspfs/bks/bkdata


Example response:

Backup Completes Successfully

11 Verify the backup on tape was successful as follows:

a List the content of the tape by typing

# tar tvf /dev/rmt/0


Example response:

-rw-rw-rw- root/other 1291264 2003-10-01 15:58 oracle.dmp -rw-rw-rw- root/other 8192 2003-10-01 15:58 critdata.cpio

b Remove the tape from the drive, label it, write-protect it, and store it in a safe place.

12 Verify the backup on DVD-RW was successful as follows:

a List the content of the DVD-RW by typing

# tar tvf /cdrom/*bkdata*/*.tar


Example response:

-rw-rw-rw- root/other 1291264 2003-10-01 15:58 oracle.dmp -rw-rw-rw- root/other 8192 2003-10-01 15:58 critdata.cpio

b Remove the DVD-RW from the drive, label it, and store it in a safe place.


If you are using Do

a tape step 11

a DVD-RW step 12



Performing a full backup of file systems on an SSPFS-based server (I)SN06.2 or greater

ApplicationUse this procedure to perform a full backup of the file systems on a Succession Server Platform Foundation Software (SSPFS)-based server (Sun Netra t1400 or Sun Netra 240) running the (I)SN06.2 or greater release of the SSPFS.

Note: For system running the (I)SN05 or (I)SN06 release of the SSPFS, use procedure “Performing a full backup of file systems (pre-(I)SN06.2)”.



• you must perform a data backup prior to performing this procedure (refer to procedure Performing a data backup on an SSPFS-based server (I)SN06.2 or greater on page 183, if required)

Note: The data backup is not required prior to this procedure for the Core Billing Manager (CBM) product family.

• you need a blank 4mm Digital Data Storage (DDS-3) tape of 125m and 12 GB to store the data on a Sun Netra t1400

• you need one or more blank DVD-RW of 4.7 GB to store the data on a Sun Netra 240 (the backup utility limits the storage to 2 GB for each DVD-RW)

Note: To reuse a DVD-RW, refer to procedure Erasing the contents of a CD/DVD on an SSPFS-based server on page 166, if required.



Action

At the server

1 Insert a blank tape or DVD-RW into the drive.

At your workstation


> telnet <server>


where

serveris the IP address or host name of the SSPFS-based server on which you are performing the backup



$ su - root



6 Rewind the tape by typing

# mt -f /dev/rmt/0 rewind


If you are using Do

a tape step 6

a DVD-RW step 7



7 Backup the file systems by typing

# /opt/nortel/sspfs/bks/bkfullsys


Example response:

Backup Completed Successfully

Note: If you are using DVD-RW, you may be prompted to insert another blank DVD.

8 Verify the backup to tape was successful as follows:

a List the content of the tape by typing

# gtar -tvMf /dev/rmt/0


b Eject and remove the tape from the drive, label it, write-protect it, and store it in a safe place.

9 Verify the backup to DVD was successful as follows:

a List the content of the DVD by typing

# gtar -tvMf /cdrom/*bkfullsys*/*.tar


b Remove the DVD from the drive, label it, and store it in a safe place.


If you are using Do

a tape step 8

a DVD-RW step 9



Performing a data restore on an SSPFS-based server (I)SN06.2 or greater

ApplicationUse this procedure to restore data from a backup tape or DVD-RW on a Succession Server Platform Foundation Software (SSPFS)-based server (Sun Netra t1400 or Sun Netra 240) running the (I)SN06.2 or greater release of the SSPFS.

Note 1: For systems running the (I)SN05 or (I)SN06 release of the SSPFS, use procedure “Restoring application data to the Oracle database (pre-SN06.2)”.

Note 2: The data restore is not required for the Core Billing Manager (CBM) product family.



• you need the tape or the DVD-RW on which the data was backed up


At the server

1 Insert the backup tape or DVD-RW into the drive.

At your workstation


> telnet <server>


where

serveris the IP address or host name of the SSPFS-based server on which you are performing the data restore



$ su - root





6 If required, stop the server applications that run on the server.

7 Restore the database by typing

$ /opt/nortel/sspfs/bks/rsdata


8 Remove the backup tape or the DVD-RW from the drive, and store it in a safe place.

9 Verify that the database restored properly.

For Refer to

CS 2000 Management Tools server applications

“Stopping the SESM server application” Stopping the SAM21 Manager server application “Stopping the NPM server application”

MG 9000 Manager and mid-tier server applications

the MG9000 Security and Administration document, NN10162-611, if required

Integrated EMS server application

the Integrated EMS Security and Administration document, NN10336-611, if required



10 Start the server applications that run on the server.


For Refer to

CS 2000 Management Tools server applications

“Starting the SESM server application” Starting the SAM21 Manager server application “Starting the NPM server application”

MG 9000 Manager and mid-tier server applications

the MG9000 Security and Administration document, NN10162-611, if required

Integrated EMS server application

the Integrated EMS Security and Administration document, NN10336-611, if required



Performing a full system restore on an SSPFS-based server (I)SN06.2 or greater

ApplicationUse this procedure to perform a full system restore from a backup tape or DVD-RW on a Succession Server Platform Foundation Software (SSPFS)-based server (Netra t1400 or Netra 240) running the (I)SN06.2 or greater release of the SSPFS.

Note: For systems running the SN05 or SN06 release of the SSPFS, use procedures “Restoring root file systems (pre-SN06.2)” and “Restoring non-root file systems (pre-SN06.2)“.

Use one of the methods below according to your office configuration.

• Simplex configuration (one server) on page 193

• High-availability configuration (two servers) on page 195

Note: Only the Simplex configuration (one server) is applicable to perform a full system restore from tape on a Netra t1400 server.



• you need the backup tape or CD/DVD



At the server console

1 Log in to the server through the console (port A) using the root user ID and password.

2 Bring the system to the OK prompt by typing

# init 0




3 Insert SSPFS CD disk#1 into the CD/DVD drive.

4 Insert the backup tape into the tape drive.

5 At the OK prompt, restore the system by typing

OK boot cdrom - restore


6 When prompted, accept the software license restrictions by typing

ok


The system reboots.

Note: If restoring from CD/DVD, you will be prompted to insert Volume 1 of the backup CD/DVD into the drive. During the restore process, the system will prompt you for additional Volumes if more than one CD/DVD was used during the full system backup.

The restore process can take several hours to complete depending on the number and size of the files that are being restored.

Note: Although it can appear as if the system is hanging at times, please do not interrupt the restore process. If you suspect an issue with the restore process, please contact your next level of support.

7 Restore the data. If required, refer to procedure Performing a full system restore on an SSPFS-based server (I)SN06.2 or greater on page 193.

Note: The data restore is not required for the Core Billing Manager (CBM) product family.

8 Once the data restore is complete, reboot the system by typing

# init 6



If restoring from Do

tape step 4

CD/DVD step 5




At the console connected to the inactive node

1 Log in to the inactive node through the console (port A) using the root user ID and password.


# init 0


At the console connected to the active node

3 Log in to the active node through the console (port A) using the root user ID and password.


# init 0


5 Insert SSPFS CD disk#1 into the CD/DVD drive.

6 At the OK prompt, restore the system by typing

OK boot cdrom - restore


7 When prompted, accept the software license restrictions by typing

ok


The system reboots.

8 When prompted, insert Volume 1 of the backup CD/DVD into the drive.

Note: During the restore process, the system will prompt you for additional Volumes if more than one CD/DVD was used during the full system backup.

The restore process can take several hours to complete depending on the number and size of the files that are being restored.

Note: Although it can appear as if the system is hanging at times, please do not interrupt the restore process. If you suspect an issue with the restore process, please contact your next level of support.



9 Restore the data. If required, refer to procedure Performing a data restore on an SSPFS-based server (I)SN06.2 or greater on page 190.

Note: The data restore is not required for the Core Billing Manager (CBM) product family.

10 Once the data restore is complete, reboot the system by typing

# init 6


11 Reimage the inactive node using the active node’s image. If required, refer to procedure “Cloning the image of one node in a cluster to the other node”.




Verifying disk utilization on an SSPFS-based server

ApplicationUse this procedure to verify disk utilization by the file systems on a Succession Server Platform Foundation Software (SSPFS)-based server.

PrerequisitesYou must have root user privileges.


At your workstation


> telnet <server>


where

serveris the IP address or host name of the SSPFS-based server



$ su - root



5 Access the command line interface by typing

# cli


Example response


2 - Configuration

3 - Other

X - exit select -



6 Display the current disk capacity utilization as follows:

a Enter the number next to the “View” option in the menu.

Example response


b Enter the number next to the “disk_util” option in the menu.

Example response




Replacing a DVD drive on a Sun server

ApplicationUse this procedure to replace a DVD drive on a Netra 240 server. This procedure applies to simplex and high-availability (HA) systems. An HA system refers to a Sun Netra 240 server pair.

The following figure shows the location of the DVD drive on the Netra 240.

ATTENTIONThe DVD drive is not hot-swappable. The server must be powered down. Therefore, ensure the server can be powered down before you proceed with the procedure.



Use one of the methods below according to your office configuration:

• Simplex configuration (one server)

• High-availability configuration (two servers)

PrerequisitesNone.

Action


At your workstation

1 Power down the server. Refer to procedure Shutting down an SSPFS-based server on page 162 if required.

2 Physically replace the DVD drive using the Sun documentation for the Netra 240.

3 Once the new DVD drive is in place, restore power to the server by turning on the power at the circuit breaker panel of the frame. The server recovers on its own once power is restored.



At your workstation

1 Use the following table to determine your first step.

2 Initiate a manual failover. Refer to procedure “Initiating a manual failover on a Sun Netra 240 server pair” if required.

3 Once the active server acquires the status of standby (inactive), power down the server. Refer to procedure Shutting down an SSPFS-based server on page 162 if required.

4 Physically replace the DVD drive using the Sun documentation for the Netra 240.

5 Once the new DVD drive is in place, restore power to the server by turning on the power at the circuit breaker panel of the frame. The server recovers on its own once power is restored.

If you are replacing the DVD drive on the Do

active server step 2

inactive server step 3


copyright © 2005, nortel networks nortel networks...

Documents