cisco collection manager user guide...using custom tablespaces 6-12 using table partitioning 6-12...

Cisco Collection Manager User GuideRelease 3.8.x November 04, 2013

Americas HeadquartersCisco Systems, Inc.170 West Tasman DriveSan Jose, CA 95134-1706 USAhttp://www.cisco.comTel: 408 526-4000

800 553-NETS (6387)Fax: 408 527-0883

Text Part Number: OL-26830-04

http://www.cisco.com

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1110R

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.

Cisco Collection Manager User Guide © 2012 Cisco Systems, Inc. All rights reserved.

http://www.cisco.com/go/trademarks

OL-26830-04

C O N T E N T S

About this Guide ix

Introduction ix

Document Revision History x

Organization xi

Related Publications xii

Conventions xiii

Obtaining Documentation and Submitting a Service Request xiv

C H A P T E R 1 Cisco Service Control and Cisco Service Control Engine Overview 1-1

Cisco Service Control Solution 1-2

Service Control for Broadband Service Providers 1-2

Cisco Service Control and Cisco SCE Capabilities 1-3

SCE Platform Description 1-4

Management and Collection 1-6

Network Management 1-6

Subscriber Management 1-7

Service Configuration Management 1-7

Data Collection 1-7

C H A P T E R 2 Collection Manager Overview 2-1

Data Collection Process 2-2

Raw Data Records 2-2

Collection Manager Software Package 2-5

Raw Data Record Server 2-5

Categorizer 2-5

Priority Queue and Persistent Buffer 2-5

Adapter 2-6

JDBC Adapter 2-6

Comma-Separated Value Adapter 2-6

Custom Adapter 2-6

Topper Aggregator (TA) Adapter 2-7

TA Adapter Cycles 2-7

TA Adapter Memory Requirements 2-8

iiiCisco Collection Manager User Guide

Contents

Real-Time Aggregating Adapter 2-10

RAG Adapter Aggregation Buckets 2-10

Flushing a Bucket 2-10

RAG Adapter Process for HTTP Transaction Usage RDR 2-11

RAG Adapter Process for Video Transaction Usage RDR 2-11

RAG Adapter Mobile Based Reports 2-11

RAG Adapter Process for Subscriber Usage RDR 2-11

Managing OSFP support for Subscriber Usage RDR 2-12

Using Databases 2-13

Using the Bundled Database 2-13

Using an External Database 2-14

C H A P T E R 3 Installing Collection Manager 3-1

System Requirements 3-2

Supported Platforms 3-2

Checking System Prerequisites 3-2

Hardware Requirements 3-3

Solaris Requirements 3-3

Software and Environment 3-4

Setting the Locale and Time Zone 3-5

Red Hat Linux Requirements 3-5

Software and Environment 3-6

Setting the Locale and Time Zone 3-7

CentOS Linux Requirements 3-7

Distribution Content 3-7

Default Configuration Settings 3-7

How to Install Collection Manager 3-8

Getting Collection Manager Software 3-8

Collection Manager Software Ports 3-9

Installing the Bundled Sybase Database 3-9

Actions the installsyb.sh Script Performs 3-10

Prerequisites 3-11

Installing the Collection Manager Software 3-12

Information About the install-cm.sh Script 3-12

Configuring the External MySQL Server 3-16

Uninstalling the Sybase Database and Collection Manager Software 3-17

Uninstalling Sybase 3-17

Uninstalling Collection Manager Software 3-17

Upgrading Collection Manager to Version 3.8.x 3-18

ivCisco Collection Manager User Guide

OL-26830-04

Contents

C H A P T E R 4 Managing Collection Manager 4-1

Using Utility Scripts 4-2

Collection Manager Support Information 4-2

Configuring Collection Manager 4-3

Using Scripts to Configure the Collection Manager 4-3

Activating Servers 4-3

Controlling Adapters 4-4

Enabling an Adapter 4-5

Dropping a Cisco SCE Connection 4-5

Configuring a Database 4-5

Configuring the Timezone Offset 4-6

Using Files to Configure the Collection Manager 4-6

Configuring the Categorizer 4-7

Monitoring System Health 4-8

Installing the Periodic Checker 4-8

Periodic Checker Script 4-8

Periodic Checker Script 4-8

Tests 4-9

Managing Users 4-10

Examples 4-10

Managing Virtual Links 4-11

Managing IMEI or MEID TAC with Device Types 4-12

Update the IMEI_DEVICETYPE table 4-12

Update the MEID_DEVICETYPE Table 4-13

Managing the Peak Hour Configuration 4-14

Monitoring Collection Manager 4-15

Checking the Database Capacity 4-15

Checking the RDR Rate 4-15

Checking the SCE Connection 4-16

Example 4-16

Verifying Server Operation 4-16

Viewing Database Statistics 4-17

Viewing Version Information 4-17

C H A P T E R 5 Managing Databases and the Comma-Separated Value Repository 5-1

Common Database Management Tasks 5-2

Listing Database Tables 5-2

Managing the Periodic Deletion of Old Records 5-3

vCisco Collection Manager User Guide

OL-26830-04

Contents

Configuring Periodic Delete 5-4

Applying the Periodic Delete Configuration File 5-5

Deleting a Table 5-6

Deleting Old Records 5-6

Unicode Support for Sybase Database 5-7

Managing the Bundled Sybase Database 5-8

Backing Up the Database 5-8

Updating Sybase with a Changed IP Address 5-9

Updating Sybase Database Statistics 5-9

Managing the External MySQL Database 5-10

Backing Up the MySQL Database 5-10

Restoring a MySQL Database 5-10

Partitioning the MySQL Database 5-10

Prerequisites to Configuring Partitioning 5-11

Configuring Partitioning 5-11

Managing the Oracle Database 5-13

Oracle Support with Minimum Privileges 5-13

Aggregating Database Data 5-14

Configuring Aggregation 5-15

Enabling Aggregation in Sybase 5-16

Enabling Aggregation in MySQL 5-16

Enabling Aggregation in Oracle 5-16

Managing the CSV Repository 5-18

CSV Repository File Structure 5-18

Configuring the CSV File Repository 5-18

Configuring the Comma Escape 5-20

Configuring Escape of Nonprintable Characters 5-20

C H A P T E R 6 Configuring Databases 6-1

Changing Basic Connection Parameters 6-1

Generating SQL Code Using the Velocity Template Language 6-1

Database Configuration Files 6-3

Context Objects 6-4

The tables Object 6-4

The dbinfo Object 6-4

The tools Object 6-5

Application Configuration 6-5

Working Sample 6-6

viCisco Collection Manager User Guide

OL-26830-04

Contents

Macro Definitions 6-6

dbinfo Configuration 6-7

SQL Definitions 6-7

Code for Dropping a Table 6-7

Code for Creating a Table 6-7

Code for Creating Indexes 6-8

Code for Insert 6-8

Code for Metadata Query 6-8

Testing and Debugging 6-9

Parsing a String 6-9

Obtaining Complete Debug Information 6-9

Using the JDBC Framework in Scripts 6-10

Viewing and Setting a Cisco SCE Time Zone Offset 6-10

Scalability Hints for Oracle 6-12

Using Custom Tablespaces 6-12

Using Table Partitioning 6-12

A P P E N D I X A Code Samples A-1

Application Configuration A-2

The dbtables.xml File A-2

The tables.dtd File A-3

RAG Adapter Configuration A-5

The ragadapter.conf File A-5

ragadapter.xml File A-6

The http_TURs.xml File A-6

The video_TURs.xml File A-7

The vlink_BW_per_pkg.xml File A-8

The vsa_SURs.xml File A-8

The cdma_SURs.xml file A-9

Custom Adapter Configuration A-10

A P P E N D I X B Configuration File Descriptions B-1

General Configuration Files B-2

Collection Manager Configuration B-2

The [adapter] Section B-2

The [adapter_mem] Section B-3

The [adapter_start_timeout] Section B-3

The [buffer] Section B-3

The [queue] Section B-4

viiCisco Collection Manager User Guide

OL-26830-04

Contents

The [categorizer] Section B-4

The [server] Section B-5

The [collector] Section B-5

The [csv_adapter] Section B-5

Queue Configuration B-6

Adapter Configuration Files B-7

Topper/Aggregator (TA) Adapter B-7

[config] Section B-7

The [rdr] Section B-9

The [csv] Section B-10

The [state save] Section B-10

The [db] Section B-10

The [app] Section B-12

The [peak_hours] Section B-12

The CSV Adapter Configuration File B-13

The [csvadapter] Section B-13

[bench] Section B-13

The JDBC Adapter Configuration File B-14

The [bench] Section B-14



RAG Adapter Configuration File B-15

The [config] Section B-15

The [housekeeper] Section B-16



Comma-Separated Value Adapter B-17

The [customadapter] Section B-17

[bench] Section B-17

viiiCisco Collection Manager User Guide

OL-26830-04

About this Guide

IntroductionThis guide describes how to install and configure the Collection Manager as part of the Cisco Service Control Engine (SCE) solution.

This guide is intended for technicians who must install and configure the Collection Manager. The guide is also intended for service provider operators who must manage the daily operations of the Collection Manager. Technicians and operators should be familiar with the Cisco Service Control Solution, Cisco Service Control Engine (SCE) platforms, and related components.

ixCisco Collection Manager User Guide

OL-26830-04

About this Guide

Document Revision HistoryTable 1 lists the changes to this document since it was first released.

Table 1 Document Revision History

RevisionCisco Service Control Release and Date Change Summary

OL-26830-04 Supports all 3.8.x releases November 04, 2013

Updated the TA Adapter Memory Requirements, page 2-8

OL-26830-03 Supports all 3.8.x releases January 16, 2013

Minor updates.

OL-26830-02 Supports all 3.8.x releases December 12, 2012

Updated for Release 3.8.5.

OL-26830-01 Supports all 3.8.x releases September 17, 2012

Updated for Release 3.8.0.

xCisco Collection Manager User Guide

OL-26830-04

About this Guide

OrganizationTable 2 describes the sections in this document.

Table 2 Document Organization

Section Title Description

1 Cisco Service Control and Cisco Service Control Engine Overview

Provides a functional overview of the Cisco Service Control solution.

2 Collection Manager Overview Describes the Collection Manager components.

3 Installing Collection Manager Describes how to install the Collection Manager and its database, and explains how to run the Collection Manager.

4 Chapter 4, “Managing Collection Manager”

Explains how to use utility scripts to view and update Collection Manager parameters and other information.

5 Chapter 5, “Managing Databases and the Comma-Separated Value Repository”

Explains how to use utility scripts to manage the Collection Manager database and the CSV repository.

6 Configuring Databases Explains how to configure the Collection Manager to work with your database.

A Code Samples Provides sample lists of code for configuration files.

B Configuration File Descriptions Describes the available parameters in all of the configuration files.

xiCisco Collection Manager User Guide

OL-26830-04

About this Guide

Related PublicationsUse this guide with the following Cisco documentation:

• Cisco Service Control Application for Broadband User Guide

• Cisco Service Control Application for Broadband Reference Guide

• Cisco SCA BB Service Configuration API Programmer Guide

• Cisco Service Control Application Reporter User Guide

xiiCisco Collection Manager User Guide

OL-26830-04

http://www.cisco.com/en/US/docs/cable/serv_exch/serv_control/broadband_app/rel38x/scabbug/scabbug.html

http://www.cisco.com/en/US/docs/cable/serv_exch/serv_control/broadband_app/rel38x/scabbrg/scabbrg.html

http://www.cisco.com/en/US/docs/cable/serv_exch/serv_control/broadband_app/rel38x/scabbpg355/scabb_pg.html

http://www.cisco.com/en/US/docs/cable/serv_exch/serv_control/broadband_app/rel38x/reporterug/reporter.html

About this Guide

ConventionsTable 3 presents and defines the documentation conventions used in this guide:

Note Means reader take note.

Tip Means that the following information will help you solve a problem.

Caution Means reader be careful. In this situation, you might perform an action that could result in equipment damage or loss of data.

Timesaver Means the described action saves time. You can save time by performing the action described in the paragraph.

Warning Means reader be warned. In this situation, you might perform an action that could result in bodily injury.

Table 3 Conventions

Convention Indication

bold font Commands and keywords and user-entered text appear in bold font.

italic font Document titles, new or emphasized terms, and arguments for which you supply values are in italic font.

[ ] Elements in square brackets are optional.

{x | y | z } Required alternative keywords are grouped in braces and separated by vertical bars.

[ x | y | z ] Optional alternative keywords are grouped in brackets and separated by vertical bars.

string A nonquoted set of characters. Do not use quotation marks around the string or the string will include the quotation marks.

courier font Terminal sessions and information the system displays appear in courier font.

< > Nonprinting characters such as passwords are in angle brackets.

[ ] Default responses to system prompts are in square brackets.

!, # An exclamation point (!) or a pound sign (#) at the beginning of a line of code indicates a comment line.

xiiiCisco Collection Manager User Guide

OL-26830-04

About this Guide

Obtaining Documentation and Submitting a Service RequestFor information on obtaining documentation, submitting a service request, and gathering additional information, see the monthly What’s New in Cisco Product Documentation, which also lists all new and revised Cisco technical documentation, at:

http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html

Subscribe to What’s New in Cisco Product Documentation as a Really Simple Syndication (RSS) feed and set content to be delivered directly to your desktop using a reader application. The RSS feeds are a free service and Cisco currently supports RSS version 2.0.

xivCisco Collection Manager User Guide

OL-26830-04

http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html

OL-26830-04

C H A P T E R 1
Cisco Service Control and Cisco Service Control Engine Overview
This chapter presents an overview of the Cisco Service Control solution operating with the Cisco Service Control Engine (SCE). The Collection Manager uses the hardware capabilities of the Cisco SCE platform. It identifies the combination of Cisco-specific applications that create the Cisco Service Control solution.

• Cisco Service Control Solution, page 1-2

• Cisco Service Control and Cisco SCE Capabilities, page 1-3

• SCE Platform Description, page 1-4

• Management and Collection, page 1-6

1-1Cisco Collection Manager User Guide

Chapter 1 Cisco Service Control and Cisco Service Control Engine Overview Cisco Service Control Solution

Cisco Service Control Solution The Cisco Service Control solution combines hardware and software components. Service providers can use the SCE platform to classify, analyze, and control Internet and IP traffic.

Service Control for Broadband Service ProvidersTheCisco SCA BB adds a layer of service intelligence and control to existing networks that can:

• Report and analyze network traffic at subscriber and aggregate level for capacity planning

• Provide customer-intuitive tiered application services and guarantee application service level agreements (SLAs)

• Implement different service levels for different types of customers, content, or applications

• Identify network abusers who are violating the acceptable use policy (AUP)

• Enforce the AUP

• Identify and manage peer-to-peer traffic, NNTP (news) traffic, and spam abusers

• Integrate Service Control solutions easily with existing network elements and business support systems (BSS) and operational support systems (OSS)


OL-26830-04

Chapter 1 Cisco Service Control and Cisco Service Control Engine Overview Cisco Service Control and Cisco SCE Capabilities

Cisco Service Control and Cisco SCE CapabilitiesThe core capabilities of the SCE platform that supports a wide range of applications for delivering service control solutions, include:

• Subscriber and application awareness—Application-level investigation of IP traffic to understand and control its use down to the subscriber level.

– Subscriber awareness—The ability to map IP flows to a subscriber. The status of each eligible subscriber is maintained. An eligible subscriber transmits traffic through the SCE platform. Subscriber awareness enforces the appropriate policy on subscriber traffic.

The Cisco Service Control solution achieves subscriber awareness through dedicated integrations with subscriber management repositories. (DHCP or RADIUS servers are examples of subscriber management repositories.) The Cisco Service Control solution also achieves subscriber awareness by sniffing RADIUS or DHCP traffic.

– Application awareness—The ability to understand and analyze traffic up to the application protocol layer (Layer 7).

For application protocols implemented using bundled flows, the SCE platform processes and manages the bundling connection between the flows.

• Application-layer, stateful, real-time traffic control—The ability to perform advanced control functions using application-layer, stateful, real-time traffic transaction processing. This processing requires a highly adaptive protocol and an application-level intelligence. Advanced control functions include granular bandwidth metering and shaping, quota management, and redirection.

• Programmability—The ability to add new protocols and to adapt to new services and applications in the service provider environment. Programmability is achieved by using the Cisco Service Modeling Language (SML).

Programmability enables a service provider to deploy new services. Programmability also provides an upgrade path for network, application, or service growth.

• Back-office integration—The ability to integrate with existing third-party systems that are available through the service provider. These systems include provisioning, subscriber repositories, billing systems, and Operations Support Systems (OSS). SCE provides a set of open and documented APIs that support this integration.

• Scalable high-performance service engines—The ability to perform all of the operations described in this section at wire speed.


OL-26830-04

Chapter 1 Cisco Service Control and Cisco Service Control Engine Overview SCE Platform Description

SCE Platform DescriptionThe SCE family of programmable network devices performs application-layer stateful-flow inspection of IP traffic, and controls the traffic based on configurable rules. The SCE platform is a network device that uses ASIC components and reduced instruction set computer (RISC) processors to go beyond packet counting and to go into the contents of network traffic. By providing programmable, stateful inspection of bidirectional traffic flows, and mapping these flows with user ownership, SCE platforms perform real-time classification of network use. This classification is the basis of the SCE platform’s advanced traffic-control and bandwidth-shaping functionality. Where most bandwidth shaper functionality ends, the SCE platform provides further control and shaping options, including:

• Layer 7 stateful wire-speed packet inspection and classification

• Robust support for more than 600 protocols and applications, including:

– General—HTTP, HTTPS, FTP, Telnet, Network News Transfer Protocol (NNTP), Simple Mail Transfer Protocol (SMTP), Post Office Protocol 3 (POP3), Internet Message Access Protocol (IMAP), Wireless Application Protocol (WAP), and others

– Peer-to-Peer (P2P) file sharing—FastTrack-KazaA, Gnutella, BitTorrent, Winny, Hotline, eDonkey, DirectConnect, Piolet, and others

– P2P VoIP—Skype, Skinny, DingoTel, and others

– Streaming and Multimedia—Real Time Streaming Protocol (RTSP), Session Initiation Protocol (SIP), HTTP streaming, Real Time Protocol (RTP), and Real Time Control Protocol (RTCP), and others

• Programmable system core for flexible reporting and bandwidth control

• Transparent network and Billing System Support (BSS) and OSS integration into existing networks

• Subscriber awareness that relates traffic and usage to specific customers


OL-26830-04

Chapter 1 Cisco Service Control and Cisco Service Control Engine Overview SCE Platform Description

Figure 1-1 illustrates a common deployment of an SCE platform in a network.

Figure 1-1 SCE Platform in a Network

LINK RX

Cisco SCE 2000 Series4xGBE

TX

RX MM TX

LINK RX TX

RX MM TX

LINK RX TX

RX MM TX

LINK RX TX

RX MM TX

GBE-1SUB LINE NET

PWR B STATUS

PWR ABYPASS

10/100/1000

LINK/ACTIVE 10/100/

1000

LINK/ACTIVE

GBE-2SUB LINE/CASCADE NET

AUXCONSOLE

MNG 2MNG 1

UsersCorporate

AggregationdeviceDSL

CMTS

SCE platform

Providernetwork Peer network

& Internet

9276

4


OL-26830-04

Chapter 1 Cisco Service Control and Cisco Service Control Engine Overview Management and Collection

Management and CollectionThe Cisco Service Control solution has a management infrastructure, which includes the following components to manage all aspects of the solution:

• Network management

• Subscriber management

• Service Control management

These management interfaces comply with common management standards and integrate with existing OSS infrastructure (Figure 1-2).

Figure 1-2 Cisco Service Control Management Infrastructure

Network ManagementThe Cisco Service Control solution provides complete network Fault, Configuration, Accounting, Performance, Security (FCAPS) Management.

Two interfaces provide network management:

• Command-line interface (CLI)—Accessible through the Console port or through a Telnet connection, use the CLI for configuration and security functions.

• SNMP—Provides fault management (through SNMP traps) and performance-monitoring functionalities.

LINK RX

Cisco SCE 2000 Series4xGBE

TX

RX MM TX

LINK RX TX

RX MM TX

LINK RX TX

RX MM TX

LINK RX TX

RX MM TX

GBE-1SUB LINE NET

PWR B STATUS

PWR ABYPASS

10/100/1000

LINK/ACTIVE 10/100/

1000

LINK/ACTIVE

GBE-2SUB LINE/CASCADE NET

AUXCONSOLE

MNG 2MNG 1

9276

3

Aggregationdevice

SCE platform

RDRs

CLI and SNMPXML/RPC

Subscriber info

Router

DHCPor RADIUS

SubscriberManager

Provisioningsystem

Servicepolicy and quota

managementNetworkmanagement

CollectionManager


OL-26830-04

Chapter 1 Cisco Service Control and Cisco Service Control Engine Overview Subscriber Management

Subscriber Management The Cisco Service Control Application for Broadband (SCA BB) enforces policies on different subscribers and tracks usage on an individual subscriber basis. The Cisco Subscriber Manager is middleware used to bridge between the OSS and the SCE platforms. Subscriber information is stored in the Subscriber Manager database. You can distribute subscriber information between multiple platforms according to the actual subscriber placement.

The Subscriber Manager provides subscriber awareness by mapping network IDs to subscriber IDs. It can obtain subscriber information by using dedicated integration modules that integrate with Authentication, Authorization, and Accounting (AAA) devices, such as RADIUS servers or DHCP servers.

You can obtain subscriber information in the following ways:

• Push Mode—The Subscriber Manager pushes subscriber information to the SCE platform automatically when a subscriber logs in.

• Pull Mode—The Subscriber Manager sends subscriber information to the SCE platform in response to a query from the SCE platform.

Service Configuration ManagementService configuration management enables you to configure the general service definitions of a service control application. You create a service configuration file and apply it to an SCE platform. The configuration file contains settings for traffic classification, accounting and reporting, and control.

The SCA BB application provides tools that enable you to automate the distribution of these configuration files to SCE platforms. This standards-based approach enables you to manage multiple devices in a large network.

Service Control provides a GUI for creating and editing configuration files. Service Control also provides a set of APIs for automating the creation of configuration files.

Data CollectionData collection occurs in the Collection Manager from:

1. All analysis and data processing functions of the SCE platform result in the generation of Raw Data Records (RDRs), which the SCE platform forwards using a simple TCP-based protocol and a RDR protocol.

2. The Collection Manager processes the RDRs.

3. The Collection Manager software is an implementation of a collection system that receives RDRs from one or more SCE platforms. It collects these records and processes them in one of its adapters. Each adapter performs a specific action on the RDR.


OL-26830-04

Chapter 1 Cisco Service Control and Cisco Service Control Engine Overview Data Collection

RDRs contain various information and statistics, depending on the configuration of the system. Three main categories of RDRs include:

• Transaction RDRs—Records generated for each transaction. A transaction is a single event detected in network traffic. The identification of a transaction depends on the particular application and protocol.

• Subscriber Usage RDRs—Records generated per subscriber, which describe the traffic generated by that subscriber for a defined interval.

• Link RDRs—Records generated per link, which describe the traffic carried on the link for a defined interval.


OL-26830-04

OL-26830-04

C H A P T E R 2
Collection Manager Overview
This chapter is an overview of the Cisco Collection Manager operating with the Cisco Service Control Engine (SCE). Collection Manager uses the hardware capabilities of the SCE platform and identifies the combination of Cisco-specific applications that create the Cisco Service Control solution.

This chapter describes the following:

• How Collection Manager works

• The Raw Data Records (RDRs) that the SCE platforms generate and send to Collection Manager

• The components of Collection Manager software package

• The database used to store the RDRs

The chapter includes the following sections:

• Data Collection Process, page 2-2

• Raw Data Records, page 2-2

• Collection Manager Software Package, page 2-5

• Adapter, page 2-6

• Using Databases, page 2-13


Chapter 2 Collection Manager Overview Data Collection Process

Data Collection Process The Cisco SCE platform generates RDRs. An application running on the SCE platform, such as the Cisco Service Control Application for Broadband (SCA BB), specifies and defines the RDRs. The following is the data collection process:

1. The SCE platform streams RDRs by using the simple, reliable RDR-Protocol. Integrating the collection of data records with the Service Control Solution involves implementing RDR-Protocol support in the collection system.

2. Collection Manager receives the RDRs from the SCE platform. The software modules recognize and sort the various types of RDRs. The sorting uses preset categories, including type and priority. Collection Manager queues the RDRs in persistent buffers.

3. Collection Manager adapters process each RDR. Each adapter performs a specific function on an RDR:

– Stores it in a customer adapter formatted file on a local machine

– Sends it to an RDBMS application

– Performs custom operations

You can use preinstalled utility scripts to customize many of the parameters that influence the behavior of Collection Manager.

Raw Data RecordsThe SCE platform generates RDRs. The RDRs, their fields, and their semantics depend on the specific service control protocol (SCP) application. Each RDR type has a unique ID known as an RDR tag.

Table 2-1 presents examples of SCP application RDRs:


OL-26830-04

Chapter 2 Collection Manager Overview Raw Data Records

Table 2-1 Examples of SCP application RDRs

Report Type Description RDRs contain...

Periodic Subscriber usage report SCE platforms are subscriber-aware network devices; they can report usage records per subscriber.

• Subscriber identifiers (such as the OSS subscriber ID)

• Traffic types (such as HTTP, streaming, or peer-to-peer traffic)

• Usage counters (such as total upstream and downstream volume).

Note The Periodic Subscriber usage report is necessary for usage-based billing services and for network analysis and capacity planning.

Note The SCA BB Subscriber Usage RDRs fall under this category.


OL-26830-04

Chapter 2 Collection Manager Overview Raw Data Records

Transaction-level report SCE platforms perform stateful tracking of each network transaction conducted on the links on which they are located. Using this statefulness, the SCP tracks several OSI Layer 7 protocols (such as HTTP, RTSP, SIP, or Gnutella) to report on various application-level attributes.

• Basic Layer 3-4 attributes (such as source IP, destination IP, and port number)

• Protocol-dependant Layer 7 attributes (such as user-agent, hostname for HTTP, or email address of an SMTP mail sender)

• Generic parameters (such as time of day and transaction duration).

Note These RDRs are important for content-based billing schemes and detailed usage statistics.

Note SCA BB application Transaction RDRs fall under this category.

SCP application activity reports The SCP application can program the SCE platform to perform various actions on network traffic. These actions include blocking transactions, shaping traffic to certain rates and limits, and performing application-level redirections. When such an operation is performed, the SCP application can produce an RDR.

• SCA BB application’s Breaching RDRs

Breaching RDRs are generated when the system changes its active enforcement on a subscriber (because usage exceeded a set quota).

• SCA BB application’s Blocking RDRs

Blocking RDRs are generated when an SCE platform blocks a network transaction (according to the rules contained in the current service configuration).

Table 2-1 Examples of SCP application RDRs (continued)

Report Type Description RDRs contain...


OL-26830-04

Chapter 2 Collection Manager Overview Collection Manager Software Package

Collection Manager Software PackageCollection Manager software package contains a group of processing and sorting modules. These modules include the following components:

• Raw Data Record Server, page 2-5

• Categorizer, page 2-5

• Priority Queue and Persistent Buffer, page 2-5

Raw Data Record ServerAn SCE platform sends RDRs to the Collection Manager. The RDR server adds an arrival timestamp and the ID of the source SCE platform. The RDR server then sends the RDR to the categorizer.

CategorizerA categorizer classifies each RDR according to its RDR tag. It determines the destination adapter for the RDR and the priority queue through which the RDR is sent.

An RDR can be mapped to more than one adapter. A qualified technician defines the flow in a configuration file, based on user requirements.

Priority Queue and Persistent BufferEach adapter has one or more priority queues; a persistent buffer is assigned to each priority queue.

A priority queue queues each RDR according to the RDR’s priority level and stores it in a persistent buffer until the adapter processes it.

A persistent buffer is a nonvolatile storage area that ensures that the system processes RDRs even in cases of hardware, software, or power failures.


OL-26830-04

Chapter 2 Collection Manager Overview Adapter

Adapter Adapters are software modules that transform RDRs to match the requirements of the target system. An adapter distributes the RDRs upon requests from a database. Currently, the following adapters are shipped with the Collection Manager:

• JDBC Adapter, page 2-6

• Comma-Separated Value Adapter, page 2-6

• Custom Adapter, page 2-6

• Topper Aggregator (TA) Adapter, page 2-7

• Real-Time Aggregating Adapter, page 2-10

Some of the adapters send data to the database or write data to CSV files. The structures of the database tables, and the location and structures of these CSV files are described in the Cisco Service Control Application for Broadband Reference Guide.

Each adapter has its own configuration file. All the configuration files are similar in structure. For a sample Real-Time Aggregating Adapter (RAG) configuration file, see the “The ragadapter.conf File” section on page A-5.

JDBC AdapterThe Java Database Connectivity (JDBC) adapter receives RDRs, processes them, and stores the records in a database.

The JDBC adapter is compatible with any JDBC-compliant database server, You can configure the JDBC adapter to use a database operating on a remote machine.

The JDBC adapter is preconfigured to support the following databases:

• Sybase Adaptive Server Enterprise (ASE) 12.5 and 15.0

• Oracle 9.2, 10.2, and 11

Disable the recycle bin feature available in Oracle 10 and later versions. You can set the initial value of the recyclebin parameter in the text initialization file init<SID>.ora, for example, recyclebin=off.

• MySQL 4.1, 5.0, and 5.1

Comma-Separated Value AdapterThe comma-separated-value (CSV) adapter receives RDRs, processes them, and writes the records to the files on the disk in the CSV format. Enhanced accounting and network traffic analysis records are generated from the records. The service provider OSS can retrieves these records using standard mechanisms. A third-party billing system can also retrieve these records using standard mechanisms. FTP is an example of a standard retrieval method.

Custom AdapterThe Custom Adapter allows you to configure the RDR format. The Custom Adapter receives records with specified fields from an RDR.


OL-26830-04



Chapter 2 Collection Manager Overview Topper Aggregator (TA) Adapter

You can configure only a few fields of TUR. Some of the configurable fields are Sub-ID, Upstream + Downstream, and Application. The corresponding RDR is processed with those specific fields and stored in a CSV file. The Custom Adapter also supports basic arithmetic operations (add, sub, maul) on a few of the RDR fields.

The Custom Adapter process is disabled by default. Edit the ~scmscm/cm/config/cm.conf file in the Collection Manager and restart Collection Manager to enable the Custom Adapter.

For more information on enabling Custom Adaptor, see the “Enabling an Adapter” section on page 4-5.

Topper Aggregator (TA) Adapter The TA Adapter receives subscriber usage RDRs and aggregates the data. The TA Adapter outputs Top Reports to the database in CSV format. Top Reports are lists of top subscribers for different metrics. The top 500 volume or session consumers in the last hour is an example of a Top Report metric. The TA adapter also sends the aggregated daily statistics of all the subscribers to the database.

The TA Adapter maintains a persistent saved state (saved to disk) to minimize data loss in case of failure.

The TA Adapter uses the JDBC adapter infrastructure. The TA Adapter operates with either local or remote JDBC-compliant databases.

TA Adapter supports IP-based aggregation along with the existing service-based or package-based aggregation.The service and IP type-based aggregation is enabled by default. The TA Adapter aggregates the records and lists the top subscribers for each combination of service and IP type with a different metric.

The package-based aggregation is disabled by default. Update the agg_pkg_level property as true in the ~scmscm/cm/config/taadapter.conf file and restart Collection Manager to enable the package-based aggregation.

Note The TA Adapter information is aggregated locally on a Collection Manager server. We do not reccomend using the Collection Managers with a single database. This may affect the accuracy of TA Adapter information.

Note A TA Adapter file is in the CSV format and populated with the RDR type SUR.

The TA Adapter has the following specifications:

• TA Adapter Cycles, page 2-7

• TA Adapter Memory Requirements, page 2-8

TA Adapter Cycles

The TA Adapter works in three cycles: short, long, and peak hours (specific hour range). Cycles are fixed intervals. This data is temporarily stored in a persistent buffer directory with a default location of ~scmscm/cm/adapters/TAAdapter. At the end of an interval, the adapter outputs its aggregated information to the database and to a CSV file. The default interval for the short cycle is 1 hour. The default interval for the long cycle is 24 hours (every day at midnight). You can configure an interval (defined in minutes) and its start and end time. Configure the interval for the peak hours cycle. At the end of the peak hours, the adapter aggregates and outputs details of the top subscribers to the database.


OL-26830-04


Note The long-cycle interval must be a multiple of the short-cycle interval.

The activities in each cycle differ slightly, as follows:

• Short cycle—At the end of each short cycle, the adapter:

– Adds the aggregated Top Reports of the cycle to the short cycle database table

– Saves the current state file in case of power failure

• Long cycle—At the end of each long cycle, the adapter:

– Adds the aggregated Top Reports of the cycle to the long cycle database table


– Creates a CSV file to contain the aggregated statistics for the long-cycle period

• Peak-hour cycle—At the end of each peak-hour cycle, the adapter:

– Adds the aggregated Top Reports of the cycle to the peak hour cycle database table


Note The long cycle data is saved every hour. The save file has 0 to 23:00 hours of data if a power shutdown occurs between 23:00 hrs and 24:00 hrs. The CSV file contains aggregated data. The current state is saved in a nonreadable format.

TA Adapter Memory Requirements

The TA Adapter requires sufficient dedicated memory. Configure the value of the memory in the cm.conf file in the following location:

[adapter_mem] com.cisco.scmscm.adapters.topper.TAAdapter=<Memory for TA Adapter>

To calculate the recommended amount of memory for the TA Adapter, use the following formula:

To calculate the recommended amount of memory for the TA Adapter for a 32 bit operating system, use the following formula:

Memory (MB) = (3 * TOTAL_SUBSCRIBERS * ((AVG_SUBS_ID_LENGTH + (2 * NUM_OF_PERIODS * NUM_OF_SERVICES * NUM_OF_IPTYPES ) + (32 * NUM_OF_SERVICES) )) /(1024 * 1024)) * (NUM_OF_IPTYPES)

To calculate the recommended amount of memory for the TA Adapter for a 64 bit operating system, use the following formula:

Memory (MB) = (5.25 * TOTAL_SUBSCRIBERS * ((AVG_SUBS_ID_LENGTH + (2 * NUM_OF_PERIODS * NUM_OF_SERVICES * NUM_OF_IPTYPES ) + (32 * NUM_OF_SERVICES) )) /(1024 * 1024)) * (NUM_OF_IPTYPES)

To calculate the recommended amount of memory for the TA Adapter with Package Enabled Aggregation for a 32 bit operating system, use the following formula:

Memory (MB) = (3 * TOTAL_SUBSCRIBERS * ((AVG_SUBS_ID_LENGTH + ((20 * NUM_OF_PKGS) +48) + (2 * NUM_OF_PERIODS * NUM_OF_SERVICES * NUM_OF_IPTYPES)+ (32 * NUM_OF_SERVICES) ))) /(1024 * 1024) * (NUM_OF_PKGS ) * (NUM_OF_IPTYPES)

To calculate the recommended amount of memory for the TA Adapter with Package Enabled Aggregation for a 64 bit operating system, use the following formula:


OL-26830-04


Memory (MB) = (5.25 * TOTAL_SUBSCRIBERS * ((AVG_SUBS_ID_LENGTH + ((20 * NUM_OF_PKGS) +48) + (2 * NUM_OF_PERIODS * NUM_OF_SERVICES * NUM_OF_IPTYPES)+ (32 * NUM_OF_SERVICES) ))) /(1024 * 1024) * (NUM_OF_PKGS ) * (NUM_OF_IPTYPES)

The definitions of the fields in the formula are:

• TOTAL_SUBSCRIBERS is the total number of subscribers across the network for each aggregation period.

• AVG_SUBS_ID_LENGTH is the average character length of a subscriber.

• NUM_OF_PERIODS = Total number of Aggregation cycles. Default value is 2.

• NUM_OF_IPTYPES = Total no of ip types supported +1

• NUM_OF_SERVICES = Total number of active services + 1

• NUM_OF_PKGS = Maximum number of packages for a particular subscriber +1

Note Cisco Service Control Collection Manager requires 75 percentage additional memory on a 64 bit machine when compared to a 32 bit machine. The 64 bit machine consumes 64 bit of memory to store each memory address pointers for the java application process. This is twice the memory required to store the addresses in a 32 bit machine. This doubles up the memory requirement while using the Cisco Service Control Collection Manager on 64 bit machine. The Package based aggregation and IP Type based aggregation functions of Cisco Service Control Collection Manager consumes more memory to store data and uses more object references that uses 64 bit to store each address in the memory.

Note For Linux JRE 32 bit, the configured memory maximum is 2 GB. For Linux JRE 64 bit, you can set higher values for the configured memory.

Note For Solaris JRE 32 bit, the configured maximum memory is 3.5 GB. For Solaris JRE 64 bit, you can set higher values for the configured memory. To configure the TA or RAG adapters to run with the JRE 64 bit, see the “The [adapter_mem] Section” section on page B-3.


OL-26830-04

Chapter 2 Collection Manager Overview Real-Time Aggregating Adapter

Real-Time Aggregating AdapterThe RAG adapter processes RDRs of one or more types and aggregates the data from predesignated field positions into buckets. The contents of the buckets are written to CSV files. The RAG adapter has the following components and processes:

• RAG Adapter Aggregation Buckets, page 2-10

• Flushing a Bucket, page 2-10

• RAG Adapter Process for HTTP Transaction Usage RDR, page 2-11

• RAG Adapter Process for Video Transaction Usage RDR, page 2-11

• RAG Adapter Process for Subscriber Usage RDR, page 2-11

• Managing OSFP support for Subscriber Usage RDR, page 2-12

RAG Adapter Aggregation Buckets

A RAG adapter aggregation bucket is indexed by combining values from fields in the RDR. The indexing relation can be one-to-one or many-to-one.

The values in the bucket-identifying fields are processed using closures (equivalence classes), which are configured per type of RDR.

Bucket-identifying field = field number 3 Closures: 4 = 4,5,6; 10 = 8,10,11 Value in field 3 = 4, 5, or 6; field reported as 4 Value in field 3 = 8, 10, or 11; field reported as 10

You can configure the RAG adapter to monitor the values in certain fields for change relative to the values in the first RDR that entered the bucket. For each monitored field, an action is performed when a value change is detected. The supported actions are:

• Checkpoint the bucket without aggregating this RDR into it, and start a new bucket with this RDR.

• Issue a warning to the user log.

Buckets, closures, triggers, and trigger actions are defined in an XML file. For a sample XML file, see the “ragadapter.xml File” section on page A-6.

Flushing a Bucket

When a bucket is flushed, it is written as one line to a CSV file.

The trigger for flushing a bucket (a checkpoint) is the earliest occurrence of any of the following:

• Time elapsed since the creation of a bucket reaches a configured duration

• Volume in an accumulated field in a bucket exceeds a configured amount

• A RAG Adapter, or the entire Collection Manager, goes down

• An RDR arrives at the bucket with some new value (relative to the bucket contents) in some field

The trigger to close a CSV file is the earliest occurrence of one of the following:

• Time elapsed since the creation of a file has reached a set duration

• Number of lines in a file has reached a set number

• A RAG Adapter, or the entire Collection Manager, goes down


OL-26830-04


RAG Adapter Process for HTTP Transaction Usage RDR

If a corresponding RDR TAG is configured under the RAG adapter section in the queue.conf file before Collection Manager is started, the RAG adapter processes the HTTP_TUR RDRs.

An XML file specific to HTTP_TUR is included in Collection Manager distribution for the RAG adapter in order to manage the HTTP_TUR RDR fields. The aggregation period for processing the RDRs is specified in the http_TURs.xml file.

Aggregation is based on the domain, package, and service for the corresponding HTTP Transaction Usage RDR. At the end of the aggregation period, the adapter adds the aggregated Top Reports for hosts to the RPT_TOP_HTTP_HOSTS database table. The adapter adds the aggregated Top Reports for domains to the RPT_TOP_HTTP_DOMAINS database table.

RAG Adapter Process for Video Transaction Usage RDR

If you configure a corresponding RDR TAG under the RAG adapter section in queue.conf file before you start Collection Manager, the RAG adapter processes VIDEO_TUR RDRs.

Collection Manager distribution includes an XML file specifically for VIDEP_TUR, which enables the RAG adapter to manage VIDEO_TUR RDR fields. The aggregation period for processing the RDRs is specified in the video_TURs.xml file.

Aggregation is based on the domain, package, and service for the corresponding VIDEO Transaction Usage RDR. At the end of the aggregation period, the adapter adds the aggregated Top Reports for hosts to the RPT_TOP_VIDEO_HOSTS database table. Also, the adapter adds the aggregated Top Reports for domains to the RPT_TOP_VIDEO_DOMAINS database tables.

RAG Adapter Mobile Based Reports

Collection Manager supports both GSM and CDMA mobile-based reports. GSM reports are enabled by default.

The ~scmscm/cm/config/ragadapter.conf

file property (vsa_type=gsm or cdma) configures the mobile report type processed.

A user should to place the related XML files in ~scmscm/cm/config/ragadapter for the Collection Manager. By default all the XML files are available in the ~scmscm/cm/config/ragadapter/repository directory.

RAG Adapter Process for Subscriber Usage RDR

If a corresponding RDR TAG is configured under the RAG adapter section in the queue.conf file before Collection Manager is started, the RAG adapter processes the NUR RDRs. The Subscriber Usage RDRs are processed in two ways:

• Aggregation Based on VSA Fields, page 2-11

• Aggregation Based on Package, page 2-12

Aggregation Based on VSA Fields

VSA field aggregation supports either GSM or CDMA mobile-based reports:


OL-26830-04


Collection Manager supports one of the mobile-based reports (either GSM or CDMA). By default, GSM reports are enabled. In the ~scmscm/cm/config/ragadapter.conf file, a new property (VSA_Type = gsm or cdma) is added to configure which mobile report Collection Manager needs to process.

During restart, based on the vsa_type configuration, the Collection Manager will copy the corresponding xml config file to the ~scmscm/cm/config/ragadapter file for processing.

Aggregation Based on GSM-Specific VSA Fields

The vsa_SURs.xml file manages GSM-specific VSA fields in the Subscriber Usage RDRs. The aggregation period for processing the RDRs is specified in this file. Aggregation is based on the GSM-specific VSA fields (APN, SGSN, NETWORK_TYPE, DEVICE_TYPE, and USER_LOCATION). At the end of the aggregation period, the adapter adds the aggregated Top Reports related to the GSM-specific VSA fields to their corresponding database tables:

• RPT_TOP_APN

• RPT_TOP_SGSN

• RPT_TOP_NETWORK_TYPE

• RPT_TOP_DEVICE_TYPE

• RPT_TOP_USER_LOCATION

Aggregation Based on CDMA-Specific VSA Fields

The cdma_SURs.xml file manages the CDMA-specific VSA fields in the Subscriber Usage RDRs. The aggregation period for processing the RDRs is specified in this file. Aggregation is based on the CDMA-specific VSA fields (PCF, HOME AGENT, MEID). At the end of the aggregation period, the adapter adds the aggregated Top Reports related to the CDMA-specific VSA fields to their corresponding database tables:

• RPT_TOP_PCF

• RPT_TOP_HOME_ AGENT

• RPT_TOP_MEID

Aggregation Based on Package

The vlink_BW_per_pkg.xml file manages upstream and downstream VLINKs in the Subscriber Usage RDRs. The aggregation period for processing the RDRs is specified in this file.

Aggregation is based on the up and down VLINK fields. At the end of the aggregation period, the adapter adds the aggregated Top Reports for the up VLINKS to the RPT_UVLINK database table. Also, the adapter adds the aggregated Top Reports for the down VLINKs to the RPT_DVLINK database table.

Rag Adapter package aggregation happens based on PACKAGE_ID, UP_VLINK_ID/DOWN_VLINK_ID, and IP_TYPE.

Managing OSFP support for Subscriber Usage RDR

Operating System Finger Printing (OSFP) is supported in Collection Manager software.

The ~scmscm/script/updateVSAindex.sh script supports backwards compatibilty.

The following two properties are added in the ~scmscm/cm/config/ragadpater.conf file:

• attr_index - index position of attribute indicator field in NUR RDR in current the Cisco SCOS version


OL-26830-04

Chapter 2 Collection Manager Overview Using Databases

• attr_shift_pos - Number of fields to be shifted

Check the VSA Attribute Indicator field index value for the current Cisco SCOS version. This is based on the Cisco SCOS version update attr_index , attr_shift_pos properties in ~scmscm/cm/config/ragadpater.conf file. The “RAG Adapter Configuration File” section on page B-15 details the RAG adapter configuration file ragadapter.conf. Then execute the script ~scmscm/script/updateVSAindex.sh script. The script updates the corresponding XML file (vsa_SURs.xml/cdma_SURs.xml).

Table 2-1 shows the VSA Attribute Indicator field index values.

Using DatabasesCollection Manager can use either a bundled database or an external database to store the RDRs supplied by the SCE platforms. The following sections describe the procedures:

• Using the Bundled Database, page 2-13

• Using an External Database, page 2-14

Using the Bundled DatabaseIn the bundled mode, Collection Manager uses the Sybase Adaptive Server Enterprise database. This database enables you to do the following:

• Support transaction-intensive enterprise applications

• Store and retrieve information online

• Warehouse needed information by putting it in a central depository

The Sybase database is located on the same server as the other Collection Manager components. It uses a simple schema that includes a group of small, simple tables.

1. The JDBC adapter sends the converted RDRs to the database, where the RDRs are stored in the tables of the simple schema.

2. You can access these records by using standard database query and reporting tools. (Cisco provides a template-based reporting tool that can generate reports on subscriber usage, network resource analysis, and traffic analysis. For information about the Service Control reporting tool, see the Cisco Service Control Application Reporter User Guide.)

You can maintain the database by using operating system commands and scripts. Collection Manager supports automatic purging of old records from the bundled database. By default, Collection Manager automatically purges the report table of every record that is older than two weeks and polls the records

Table 2-2 VSA Attribute Indicator Field Index Values

SCOS Version attr_index attr_shift_pos

3.8.5 17 0

3.8.0 17 0

3.7.5 17 0

3.7.0 14 3


OL-26830-04



Chapter 2 Collection Manager Overview Using an External Database

once every hour. You can configure database maintenance by using the dbperiodic.sh utility script. For more information on database maintenance, see the “Managing the Periodic Deletion of Old Records” section on page 5-3.

Using an External DatabaseWith the JDBC adapter, you can use any JDBC-compliant database (for example, Oracle or MySQL). The database can be either local or remote. To use an external database, perform the following tasks:

Step 1 Configure the JDBC adapter to use this database.

Step 2 Configure a database pack to supply Collection Manager with the parameters of the database (such as its IP address and port).

Step 3 Supply a JDBC driver for the database so that the adapter can connect to the database.

For details about configuring Collection Manager to work with an external database, see Chapter 5, “Managing Databases and the Comma-Separated Value Repository”.


OL-26830-04

OL-26830-04

C H A P T E R 3
Installing Collection Manager
This chapter describes how to install Collection Manager and the connected database. The chapter also explains how to run Collection Manager.

Note For a typical installation and configuration, see the Collection Manager Quick Start Guide.

This chapter includes the following sections:

• System Requirements, page 3-2

• How to Install Collection Manager, page 3-8

• Configuring the External MySQL Server, page 3-16

• Uninstalling the Sybase Database and Collection Manager Software, page 3-17

• Upgrading Collection Manager to Version 3.8.x, page 3-18


http://www.cisco.com/en/US/docs/cable/serv_exch/serv_control/broadband_app/rel38x/cm_qsg/cm_qsg.html

Chapter 3 Installing Collection Manager System Requirements

System Requirements The Collection Manager has the following system requirements:

• Supported Platforms, page 3-2

• Checking System Prerequisites, page 3-2

• Hardware Requirements, page 3-3

• Solaris Requirements, page 3-3

• Red Hat Linux Requirements, page 3-5

• CentOS Linux Requirements, page 3-7

• Distribution Content, page 3-7

• Default Configuration Settings, page 3-7

Supported PlatformsYou can install Collection Manager on any of the following platforms:

• Sun SPARC machine (64- bit) running 64-bit versions of Solaris 9 or Solaris 10. (See the “Solaris Requirements” section on page 3-3.)

• Intel machine (32 bit or 64-bit) running 32-bit versions of Red Hat Enterprise Linux 4.x, Red Hat Enterprise Linux 5.x, or 64-bit versions of Red Hat Enterprise Linux 5.x. (See the “Red Hat Linux Requirements” section on page 3-5.)

• Intel machine (32 bit or 64-bit) running 32-bit or 64-bit versions of CentOS 5.x. (See the “CentOS Linux Requirements” section on page 3-7.)

• Intel machine (32 bit or 64-bit) running VMware Server or VMware ESX VMware-VMvisor-Installer-4.0.0-164009.x86_64.iso.

• Any Cisco Unified Computing System

• All the configurations use a 32-bit Java Virtual Machine (JVM).

Caution Collection Manager must run on a dedicated server. The local database for Collection Manager is also located on that server. You cannot run the Cisco Subscriber Manager or other software applications on the same server that contains the database.

Note When using the bundled Sybase database, the server on which you install Collection Manager can have a maximum of 4 CPU cores.

Checking System Prerequisites The Collection Manager distribution contains a script, check_prerequisites.sh, located in the install_scripts directory. The script helps to determine if a system meets the requirements for installing a Collection Manager and the bundled Sybase database.

check_prerequisites.sh [ --sybhome=SYBHOME ] [ --cmhome=CMHOME ] [ --datadir=DATADIR ]


OL-26830-04

Chapter 3 Installing Collection Manager Hardware Requirements

Table 3-1 provides a list of the check_prerequisites.sh script options.

The main prerequisites checked are:

• CPU speed

• Amount of RAM

• Operating System version (Solaris 9 or 10, Red Hat Enterprise Linux 4 or 5)

• Additional required and optional packages

• Free space for the Collection Manager and the Sybase database

• Names for all network interface cards (NICs)

• Sybase kernel parameters

• Locale and time zone formats

Hardware RequirementsThe minimum hardware requirements for Collection Manager are:

• 500-MHz CPU

• One GB RAM per CPU

• Hard disk:

– One hard disk (18 GB minimum)

– For bundled installations, a second hard disk of at least 30 GB is recommended to store Sybase data.

• 100BASE-T network interface

Note The minimum hardware requirements apply only to small deployments. Medium or large deployments have higher hardware requirements. Your network design expert is the best source of information on the required hardware for a particular deployment.

Solaris RequirementsInstall Collection Manager Release 3.1.0 or later on Sun SPARC Machines:

• Software and Environment, page 3-4

• Setting the Locale and Time Zone, page 3-5

Table 3-1 check_prerequisites.sh Script Options

--sybhome=SYBHOME Intended home directory for Sybase installation

--datadir=DATADIR Intended data directory for Sybase data files (for the Datadir installation method)

--cmhome=CMHOME Intended home directory for Collection Manager installation


OL-26830-04

Chapter 3 Installing Collection Manager Solaris Requirements

Software and Environment

Collection Manager requires the following software version when running on Solaris:

• Solaris Version 5.9 64-bit build 04/01 or later (currently, only Solaris Version 5.9 and 5.10 are supported).

– Solaris 9—Patch level 9 is recommended

– Solaris 10—Patch level 10 is recommended

• Solaris Core Installation.

• Install the additional packages, as shown in Table 3-2.

• If you are installing Collection Manager in bundled mode with the Sybase database, install the package shown in Table 3-3.

• (Optional) You can install the packages listed in Table 3-4 (for sysadmin applications such as sys-unconfig).

• You can download these packages from http://www.sunfreeware.com/

The root (/) partition must have at least 104 MB of free space to install these packages.

• At least 8 GB free on the partition for Collection Manager. (This memory is used for CSV storage and persistent buffers.)

• At least 3 GB free on one partition for the Sybase home directory (for installations with bundled Sybase).

• For bundled installations, a second hard disk of at least 30 GB is recommended to store Sybase data.

Table 3-2 Additional Packages

system SUNWbash GNU Bourne-Again shell (bash)

system SUNWgzip The GNU Zip (gzip) compression utility

system SUNWzip The Info-Zip (zip) compression utility

system SUNWlibC Sun Workshop Compilers Bundled libC

system SUNWlibCx Sun WorkShop Bundled 64-bit libC

Table 3-3 SUNWipc Package

system SUNWipc Interprocess Communication

Table 3-4 Optional Packages

system SUNWadmap System administration applications

system SUNWadmc System administration core libraries


OL-26830-04

Chapter 3 Installing Collection Manager Red Hat Linux Requirements

• Free space on one partition to hold the desired size of the Sybase data and logs (for installations with bundled Sybase). You can configure memory size during installation.

• The FTP server requires listening to port 21 so that the SCA Reporter can authenticate the server.

• All the IP addresses configured for the machine NICs require hostnames associated with them in the /etc/hosts or in another active naming service. The requirement is for installations with the bundled Sybase database. Verification is required for an installation due to a limitation of the Sybase Adaptive Server Enterprise.

• Use the set_shmmax.sh script to configure the kernel memory for installations with the bundled Sybase. The script is located under install-scripts/.

• At startup ,you must load the IPC module by inserting the following lines in the file /etc/system:

forceload: sys/shmsys

• If you are using database periodic delete, enable the scmscm user to schedule and run cron jobs.

Setting the Locale and Time Zone

• For correct Collection Manager and Sybase operation, use the locale U.S. English.

Add the following line in the /etc/TIMEZONE configuration file to set the locale:

LANG=en_US

Note Restart Collection Manager to change to this configuration.

To use the U.S. English locale, Collection Manager must be running on the Solaris operating system. Verify that the locale is installed by ensuring that the directory /usr/lib/locale/en_US exists. If the directory does not exist, install the locale files from the Solaris CDs.

• Setting the OS time zone as an offset from GMT in POSIX format is not recommended, and can lead to problems. Set the time zone in the /etc/TIMEZONE configuration file by country (supported) name, as in the following example.

TZ=Japan

Verify that the country name is supported as a time zone setting by ensuring that it is listed in the directory /usr/share/lib/zoneinfo.

If you must use GMT offset, use the zoneinfo format by prepending the prefix :Etc/, as in the following example:

TZ=:Etc/GMT+5

Red Hat Linux RequirementsThe following information applies to running the Collection Manager using Red Hat Linux.

• Software and Environment, page 3-6

• Setting the Locale and Time Zone, page 3-7


OL-26830-04

Chapter 3 Installing Collection Manager Red Hat Linux Requirements

Software and Environment

Collection Manager Version 3.7.5 or later requires the following software when running on Red Hat Linux:

• Red Hat Linux 4.x:

– kernel-2.6.9-5

– glibc-2.3.4-2

– compat-libstdc++-33-3.2.3-47.3

– Minimum patch level required—Update 7

• Red Hat Linux 5.x:

– kernel-2.6.18-8.el5

– glibc-2.5-12

– compat-libstdc++-33-3.2-61

– Minimum patch level required—Update 5.3

• Red Hat Enterprise Linux 6.x

Installing Red Hat Linux

For Red Hat Linux installations, perform the following steps:

• Apply the latest recommended patches from Red Hat.

• Reserve at least 8 GB on the partition where you want to install Collection Manager.

Collection Manager uses this disk space for CSV storage and persistent buffers.

• If you are using database periodic delete, enable the scmscm user to schedule and run cron jobs.

• If you use Red Hat Enterprise Linux 6.x, confirm that the desired database is compatible with Red Hat Enterprise Linux 6.x.

Note Red Hat Enterprise Linux 6.x does not support the Collection Manager bundled Sybase database.

Installing Red Hat Linux and the Sybase Database Bundled with the Collection Manager

Note Red Hat Enterprise Linux 6.x does not support the Sybase database bundled with the Collection Manager.

For installations with the Sybase database bundled with the Collection Manager, perform the following steps:

• Install the compat-libstdc++ package. This package is available on the Red Hat installation CD.

• Install the current Sybase patches.

• Reserve at least 1 GB on one of the partitions for the Sybase home directory.

• Verify that all the configured IP addresses for the machine NICs have hostnames associated with them in the /etc/hosts or in another active naming service. This action is required due to a limitation of Sybase Adaptive Server Enterprise.


OL-26830-04

Chapter 3 Installing Collection Manager CentOS Linux Requirements

• Use the set_shmmax.sh script (located under the install-scripts/) to configure the kernel memory.

Setting the Locale and Time Zone

For correct Collection Manager and Sybase operation, use the locale U.S. English (en_US).

CentOS Linux RequirementsInstall Collection Manager Version 3.6.5 or later on any i386 that runs CentOS Linux. Collection Manager requires the following software version when running on CentOS:

• CentOS Linux 5.x:

– kernel-2.6.18-8.el5

– glibc-2.5-12

– compat-libstdc++-33-3.2.3-61

Note CentOS is supported only on Collection Manager software Release 3.6.5 and later.

Distribution ContentCollection Manager installation kit contains scripts for installing Collection Manager and the Sybase database.

It also contains:

• Scripts to support file gathering

• Scripts for periodic Sybase maintenance

Default Configuration SettingsConfigure the settings for Collection Manager during installation.

These settings include:

• Adapters to enable and their locations

• Priority Queue parameters

• Target adapters for each type of RDR (by RDR tag value)

• Various logging policies

Note Permit only qualified personnel to change these settings.


OL-26830-04

Chapter 3 Installing Collection Manager How to Install Collection Manager

How to Install Collection ManagerThis section describes how to install Collection Manager Version 3.1.0 or later and the Sybase database:

• Getting Collection Manager Software, page 3-8

• Collection Manager Software Ports, page 3-9

• Installing the Bundled Sybase Database, page 3-9

• Installing the Collection Manager Software, page 3-12

Note You can install the Collection Manager on Solaris, Red Hat Linux, or CentOS Linux.

Getting Collection Manager SoftwareTo download Collection Manager software:

Step 1 Goto to the Cisco Product/Technology Support Download Software page for the Cisco Service Control Collection Manager at:

http://www.cisco.com/cisco/software/release.html?mdfid=279573187&flowid=7155&softwareid=282767641&release=3.7.5-P1&relind=AVAILABLE&rellifecycle=&reltype=latest

Note Optionally, you can go to the same Cisco Service Control Collection Manager software download page from Cisco.com by navigating to Cisco Product/Technology Support > Download Software > Service Exchange > Cisco Service Control > Cisco Service Control Management Suite > Cisco Service Control Collection Manager.

Step 2 Enter your Cisco CCO password when prompted.

Step 3 Download the SCMS-CM Installation software package for your specific operating system. Specific software packages may consist of a single download or consecutive multiple downloads. If there is a single file package, go to Step 5.

Step 4 If there are consecutive multiple downloads, move the downloaded files to the target machine and join them into a single .tar file.

For example:

# cat cm_part1 cm_part2 >/usr/tmp/cm_full_package.tar

Step 5 Extract the complete package into a temporary directory.

For example:

# mkdir /usr/tmp/cm_install_temp# cd /usr/tmp/cm_install_temp# tar xvf ../cm_full_package.tar


OL-26830-04

http://www.cisco.com/cisco/software/release.html?mdfid=279573187&flowid=7155&softwareid=282767641&release=3.7.5-P1&relind=AVAILABLE&rellifecycle=&reltype=latest

Chapter 3 Installing Collection Manager Collection Manager Software Ports

Collection Manager Software PortsTable 3-5 describes the TCP/UDP ports on which Collection Manager software and associated components (such as the Sybase database) listen. This table helps a network administrator to understand the behavior of the software and its adherence to security policies.

The device listens constantly on the ports in the table. If you cannot enable access on these port numbers, certain operations can fail. Some operations (such as file transfer) cause a device to temporarily open ports other than the listed ports. However, these ports close automatically when the operation that opens additional ports ends.

Installing the Bundled Sybase DatabaseIf you do not want to install Sybase (for example, when working in unbundled mode), see the “Installing the Collection Manager Software” section on page 3-12.

Note The maximum size for the bundled Sybase database is 50 GB. Use an external database for databases over 50 GB.

Note Installing the Sybase database can take as many as 3 hours.

Table 3-5 Ports on Which Collection Manager Listens Constantly

Port Number Description

33000 The SCE devices use this port to send RDRs for data collection.

33001 Internal collection manager.

Note Access is required only from the local machine; external access can be blocked.

9092 HTTP technician interface.

4100 Sybase database connectivity through ODBC or JDBC. Required for access to the database (for installations with bundled Sybase).

1099—1120 RMI. Used as the management interface between the data collector and the Service Control management server.

22000 FTP server of Collection Manager.

Note FTP transactions can listen on other ports (22001 to 22100) for data transfer. The protocol negotiates the port number.

7787 Internal logging of the management user log.

Note Access is required only from the local machine; external access can be blocked.

14375 The Cisco SCA BB console uses this port to send symbol definitions (values.ini) to Collection Manager.


OL-26830-04

Chapter 3 Installing Collection Manager Installing the Bundled Sybase Database

Note You can install Collection Manager with the bundled Sybase database on a server with a maximum of four CPU cores. This is a limitation of the Sybase license.

During installation, if you want to reverse the Sybase installation actions (for example, if an installation is interrupted because of a power failure), do the following:

1. Log in as the root user.

2. Run the /install-scripts/uninstall.sh --sybase script.

Actions the installsyb.sh Script Performs

The installsyb.sh script installs the Sybase database. The script performs the following actions:

• Verifies the shmem setting for Sybase in the /etc/system (for Solaris) or the /etc/sysctl.conf file (for Red Hat Linux). If the setting is not there, the script inserts it and reboots (after prompting you).

• Adds a user Sybase and group Sybase.

• Runs the Sybase installer for your platform.

• Builds a Sybase server that includes Sybase users and passwords.

• Starts Sybase.

• Runs SQL scripts to create Collection Manager database structure. This process is lengthy and requires restarting Sybase several times.


OL-26830-04

Chapter 3 Installing Collection Manager Installing the Bundled Sybase Database

Prerequisites

Log in as the root user and make the distribution kit contents available on your system or local network.

Step 1 Change directory to Sybase in the distribution kit root.

Step 2 Run the installsyb.sh script.

installsyb.sh --sybhome=SYBHOME {-datadir=DATADIR}{--y|--n}

• SYBHOME is the home directory of the Sybase user (have 4 GB free for Solaris and 3 GB free for Linux)

• Select one of the following data location options:

– Specify --datadir=DATADIR, where DATADIR is a directory in which to store all Sybase data.

Use a location in a partition in which at least 15 GB is free.

• If you specify a DATADIR, all Sybase data is stored as normal files in that directory. The default sizes are 10 GB for data, 3 GB for logs, and 2 GB for the temporary storage of Sybase. During the installation process, ownership of the directory changes to the Sybase user.

• A fresh install requires an empty "DATADIR" directory.

Note Using an external database, such as MySQL or Oracle, requires creating a schema and DB USER with the required privileges. Reconfigure Collection Manager to the database using the ~scmscm/scripts/dbconf.sh script option.


OL-26830-04

Chapter 3 Installing Collection Manager Installing the Collection Manager Software

Installing the Collection Manager SoftwareThis section describes how to install Collection Manager.

Information About the install-cm.sh Script

To install Collection Manager server, use the install-cm.sh script.

install-cm.sh Options

The usage message for the install-cm.sh script is:

Usage: install-cm.sh [-h] (-d CMDIR | -o)

Options: -d CMDIR select directory for ~scmscm (must not exist and must be on 8 GB free partition) -o upgrade the existing installation while preserving the current configuration (can't be used with -d) -h print this help and exit

Description of the options:

-d CMDIR Used to designate the directory of the newly created scmscm user's home. Should be the name of a non-existing directory, whose parent resides on a partition where at least 8 GB is free. As an alternate to this option, you can specify -o :

-o Use this option when you wish to upgrade the existing installation while preserving the current configuration. (can't be used with -d)

Actions install-cm.sh Performs

The install-cm.sh script performs the following actions:

• If needed, creates an scmscm user and a scmscm group.

• Optionally, creates the home for this user

• Populates the home of scmscm with Collection Manager files and scripts

• Installs the extra component private JRE in the ~scmscm/cm/lib library.

• Creates boot script symbolic links for the Sybase and scmscm users in /etc/init.d and /etc/rcX.d

Note All Collection Manager scripts run in the Bash shell environment. No other shells are supported.

Step 1 Change the directory to install-scripts under the distribution kit root.

Step 2 Run the install-cm.sh script.

After running the script, a user-driven configuration manager presents options for the basic configuration of Collection Manager.

Step 3 Choose whether to enable or not enable the RAG adapter. For more information on the RAG adapter, see the “Real-Time Aggregating Adapter” section on page 2-10.

Do you want to enable the RAG Adapter? (yes/no):


OL-26830-04


Step 4 You can choose to set up the Collection Manager database at the time of installation or configure it at a later time.

Would you like to configure the database?: yes

Step 5 If you choose to configure Collection Manager database:

a. Select the number corresponding to the relational database management system of the connecting database:

Enter the DB type:1 - Oracle2 - MySQL3 - Sybase4 - exitEnter your choice:

b. Enter the following server information or press enter to leave at the default shown.

Enter DB server host (default localhost):Enter DB server listening port (default port no) :Enter DB server instance id (default schema) :Enter DB schema user name (default user_id) :Enter DB schema user password (default password):Do you want to test the DB connection? (yes/no): Please answer using "yes" or "no"

If you chose yes, the following text appears:

Testing the DB connection...PASS:db is upDB connection succeeded.Updating the time zone...

NOTE: The user name created for SCMS-CM is scmscm. You may want to set a password for this account now.

Step 6 Restart Collection Manager for the database configuration to take effect.

Step 7 To start Collection Manager, choose yes:

Would you like to start CM now? (yes/no): yes

The most basic installation of Collection Manager software is done.

Step 8 (Optional) Set a password.

Run the following command to set the password for the scmscm user:

passwd scmscm

Note Be sure to record the password that you choose.

Step 9 (Optional) Increase the amount of memory allocated to the TA adapter.

An application that uses the TA adapter may require an increase in the amount of allocated memory. The need to increase memory depends on the number of subscribers Collection Manager manages. To increase the memory allocation:

a. Open the file ~scmscm/cm/config/cm.conf.

b. Locate the setting containing TAAdapter in the [adapter_mem] section.

c. Change the default value (512 MB) to a larger value.


OL-26830-04


For example, to allocate 1024 MB of memory, set the value to -Xmx1024M.

d. Save and close the file.

Note For information about calculating the TA Adapter memory, see the “TA Adapter Memory Requirements” section on page 2-8.

Step 10 (Optional) Increase the amount of memory allocated to the RAG adapter.

An application that uses the RAG adapter can require an increase in the amount of allocated memory. The need to increase memory depends on the number of subscribers Collection Manager manages and on the RAG Adapter configuration. To change the setting:

a. Open the file ~scmscm/cm/config/cm.conf.

b. Locate the setting containing RAGAdapter in the [adapter_mem] section.

c. Change the default value (512 MB) to a larger value.

For example, to allocate 1024 MB of memory, set the value to -Xmx1024M.

d. Save and close the file.

Note To use an external database, configure a dbpack to connect Collection Manager to the database. See Chapter 5, “Managing Databases and the Comma-Separated Value Repository” for instructions.

Note The app_conf_dir values of the JDBC (jdbcadapter.conf) and TA (taadapter.conf) adapter configuration files point to the most recent directory of the desired applications by default.


OL-26830-04


Step 11 (Optional) Install and activate the periodic delete procedures for the database tables.

(For information about configuring the behavior of periodic delete, see the “Managing the Periodic Deletion of Old Records” section on page 5-3.)

Note If reports are sent to the database, and you do not install and activate the periodic delete procedures, the second disk might overflow.

a. Install the periodic delete procedures.

Log in as the scmscm user, start Collection Manager. Wait 1 - 2 minutes for the creation of the database tables. Run the script:

~scmscm/db_maint/create_periodic_del_procs.sh

b. Activate the automatic invocation of the periodic delete procedures by running the following command:

~scmscm/scripts/dbperiodic.sh --load

Step 12 Start Collection Manager by running the ~scmscm/cm/bin/cm start.

The script performs the following actions:

• Starts all the adapters enabled in Collection Manager.

• Updates the CONF_SE_TZ_OFFSET table with the timezone offset.

• Populates the IMEI_DEVICETYPE table with default values. This occurs when the table is not created and no records exist in the table.


OL-26830-04

Chapter 3 Installing Collection Manager Configuring the External MySQL Server

Configuring the External MySQL ServerYou can configure Collection Manager to operate with an external server operating the MySQL Server 4.1 software or the MySQL Server 5.0 software. To do this, log in to the MySQL Server and manually create a schema and a user with necessary privileges before configuring Collection Manager.

Log in to the server using root privileges and execute the following MySQL commands to create pqb_admin with root privileges:

create database apricot;

create user 'pqb_admin' identified by 'pqb_admin';

grant all privileges on *.* to 'pqb_admin'@'localhost' identified by 'pqb_admin' with grant option;

grant all privileges on *.* to 'pqb_admin'@'<CM_server_IP>' identified by 'pqb_admin' with grant option;

grant all privileges on *.* to 'pqb_admin'@'%' identified by 'pqb_admin' with grant option;

This set of privileges will grant root privilege to the pqb_admin user. Redefine user privileges to restrict user access.


OL-26830-04

Chapter 3 Installing Collection Manager Uninstalling the Sybase Database and Collection Manager Software

Uninstalling the Sybase Database and Collection Manager Software

The following sections describe how to uninstall the Sybase database and Collection Manager:

• Uninstalling Sybase, page 3-17

• Uninstalling Collection Manager Software, page 3-17

Uninstalling SybaseTo uninstall the Sybase database, perform the following steps:

Step 1 Log in to the Collection Manager as the root user.

Step 2 Uninstall Sybase.

Change the directory to install-scripts under the distribution kit root directory, and enter:

./uninstall.sh --sybase

Step 3 Edit the /etc/system file for Solaris or the /etc/sysctl.conf file for Red Hat Linux and remove the Sybase shmem setting.

Uninstalling Collection Manager SoftwareTo uninstall Collection Manager software, perform the following steps:

Step 1 Log in as the root user.

Step 2 Uninstall Collection Manager software.

Change the directory to install-scripts under the distribution kit root directory, and enter:

./uninstall.sh --cm


OL-26830-04

Chapter 3 Installing Collection Manager Upgrading Collection Manager to Version 3.8.x

Upgrading Collection Manager to Version 3.8.xTo upgrade the CM:RDR database with 3.8.x compatibility, perform the following steps:

Step 1 Download the Collection Manager software, as described in the Getting Collection Manager Software, page 3-8.

Step 2 Change directory to install-scripts under the distribution kit root.

Step 3 Log in as the scmscm user, stop the Collection Manager server:

$ ~scmscm/cm/bin/cm stop

Step 4 Log in as root user, run the install-cm.sh script:

# ./install-cm.sh -o

Step 5 Respond to the displayed text:

Do you want to upgrade the RDR Tables to latest Schema? (yes/no): yes

Note If the no option is selected, run the ~scmscm/scripts/ upgradeRDRTables.sh before manually starting Collection Manager.

Note Stop Collection Manager before running upgradeRDRTables.sh.

Note The upgrade requires more time for a larger amount of data.

After the upgrade, when Collection Manager comes up for the first time, the database tables that are new are created automatically.

Note You can perform a direct upgrade to Version 3.8.x only from Version 3.7.x. To upgrade from Version 3.6.x or earlier, you must first upgrade to Version 3.7.x


OL-26830-04

OL-26830-04

C H A P T E R 4
Managing Collection Manager
This chapter explains how to use utility scripts to view and update Collection Manager parameters.

Any machine connected to Collection Manager through, for example, Telnet or Secure Shell (SSH), can use utility scripts to monitor and manage Collection Manager. The utility scripts are located in the installation directory of Collection Manager.

For information on managing the database and the CSV repository, see Chapter 5, “Managing Databases and the Comma-Separated Value Repository.”

This chapter contains the following sections:

• Using Utility Scripts, page 4-2

• Collection Manager Support Information, page 4-2

• Configuring Collection Manager, page 4-3

• Configuring the Categorizer, page 4-7

• Monitoring System Health, page 4-8

• Managing Users, page 4-10

• Managing Virtual Links, page 4-11

• Managing IMEI or MEID TAC with Device Types, page 4-12

• Managing the Peak Hour Configuration, page 4-14

• Monitoring Collection Manager, page 4-15


Chapter 4 Managing Collection Manager Using Utility Scripts

Using Utility ScriptsGeneral instructions for using utility scripts include:

• To invoke any script, log in as the scmscm user, except when otherwise specified. If you attempt to run these scripts as the root user, the script generates an error.

• To display a description of the script, with an explanation of all the flags and parameters, invoke the script with the help flag.

Note Specify the help flag differently depending on the script. To manage Collection Manager, specify -help. To manage the database, specify -h.

The following example shows how to display a description of the dbperiodic.sh script:

>~scmscm/scripts/dbperiodic.sh --help Usage: ~scmscm/scripts/dbperiodic.sh --load load configuration from /export/home/scmscm/db_maint/dbperiodic.conf ~scmscm/scripts/dbperiodic.sh --loadfile=FILE load configuration from FILE ~scmscm/scripts/dbperiodic.sh --dump print the current configuration in INI format to standard output ~scmscm/scripts/dbperiodic.sh --help print this help message

Collection Manager Support InformationIf you are experiencing difficulties with Collection Manager installation, you might need to provide the Cisco Technical Assistance Center (TAC) with information about the current system setup. Collection Manager can create the required support information or files for TAC by running one of the following scripts:

• ~scmscm/unsupported/getinfo/get_cm_info.sh—This script generates the Collection Manager support information.

• ~scmscm/unsupported/getinfo/get_support_files.sh—This script creates a support file.

The output is a zip file with the investigation supporting files. The following example shows the output that results from running the script:

bash-2.05$ ~scmscm/unsupported/getinfo/get_support_files.sh Gathering support files into Sun_14_Oct_2007_06-05-03_PM_JST.zip, please wait................done.The generated support file contains the output of ~scmscm/unsupported/getinfo/get_cm_info.sh


OL-26830-04

Chapter 4 Managing Collection Manager Configuring Collection Manager

Configuring Collection ManagerYou can use both scripts and files to configure Collection Manager. The following sections provide details on these options:

• Using Scripts to Configure the Collection Manager, page 4-3

• Using Files to Configure the Collection Manager, page 4-6

Using Scripts to Configure the Collection Manager Use utility scripts to:

• Specify which servers to activate at startup.

• Start or stop the database.

• Start or stop an adapter.

• Drop an SCE connection.

Use the following scripts to configure Collection Manager:

• ~scmscm/setup/on-boot.sh

• ~scmscm/scripts/adapterconf.sh

• ~scmscm/scripts/sceconf.sh

• ~scmscm/scripts/dbconf.sh

Note For information about the scripts used for managing the database and the CSV repository, see Chapter 5, “Managing Databases and the Comma-Separated Value Repository.”

The below sections detail the scripts used to configure the Collection Manager.

• Activating Servers, page 4-3

• Controlling Adapters, page 4-4

• Enabling an Adapter, page 4-5

• Dropping a Cisco SCE Connection, page 4-5

• Configuring a Database, page 4-5

• Configuring the Timezone Offset, page 4-6

Activating ServersTo specify the servers (Collection Manager or Sybase) that are activated at startup, use the on-boot.sh script:

~scmscm/setup/on-boot.sh --cm=flag --sybase=flag

The table Table 4-1defines the on-boot.sh script parameters:


OL-26830-04

Chapter 4 Managing Collection Manager Controlling Adapters

The following example shows how to set Collection Manager and Sybase servers to run at startup. (This setting is the default setting of the script.)

>~scmscm/setup/on-boot.sh –-cm=on –-sybase=on

To view the current startup status of each component, run the on-boot.sh script with no parameters.

Server specification take effect when the system restarts. To restart Collection Manager, run the following script as the scmscm user:

~scmscm/cm/bin/cm restart

Controlling AdaptersTo shut down or activate a configured adapter, use the adapterconf.sh script. You can use the adapterconf.sh script to list Collection Manager adapters that are currently running:

~scmscm/scripts/adapterconf.sh --op=action [ --adapter=adapter name ]

The table Table 4-2 shows the adapterconf.sh script options:

To shut down an adapter, as the scmscm user, run the following script:

~scmscm/scripts/adapterconf.sh --op=stop –-adapter=adapter name

To activate an adapter, as the scmscm user, run the following script:

~scmscm/scripts/adapterconf.sh --op=start –-adapter=adapter name

Table 4-1 on-boot.sh Parameters

--cm={ on | off } Activate or do not activate Collection Manager at startup.

--sybase={ on | off } Activate or do not activate the Sybase server at startup.

Table 4-2 adapterconf.sh Options

--op=start Activate the adapter specified by the adapter parameter.

--op=stop Shut down the adapter specified by the adapter parameter.

--op=list List Collection Manager adapters that are currently running.

adapter=adapter name Identify the adapter you want to control. Use only with the start and stop options.

--help Display these options.


OL-26830-04

Chapter 4 Managing Collection Manager Enabling an Adapter

Enabling an AdapterTo configure an adapter to turn on when Collection Manager starts, remove the remark character at the start of the appropriate line in the cm.conf file.

The following example shows how to define the RAG adapter to turn on when Collection Manager starts.

adapter.4 = com.cisco.scmscm.adapters.rag.RAGAdapter

The following example shows how to configure the CSV adapter to remain off when Collection Manager starts:

#adapter.2 = com.cisco.scmscm.adapters.CSVAdapter

Note The value of the adapter.<number> must match the adapter_id parameter value configured in the queue.conf file for the corresponding adapter.

Dropping a Cisco SCE ConnectionTo drop a connection to a particular Cisco SCE, run the sceconf.sh script:

~scmscm/scripts/sceconf.sh --op=drop --ip=IP address

Run this script only if the HTTP adaptor of Collection Manager is running.

Also use this script to display information about the Cisco SCE connection. (See the “Checking the SCE Connection” section on page 4-16)

Table 4-3 shows the sceconf.sh script options:

To drop a connection, as the scmscm user, run the following command:

~scmscm/scripts/sceconf.sh --op=drop --ip=IP address

Configuring a DatabaseTo configure a database, run the dbconf.sh script:

~scmscm/scripts/dbconf.sh

The script prompts you to choose the database type and the corresponding database configuration parameters.

The following example shows how to run the dbconf.sh script to configure Collection Manager to work with an external Oracle database:

$ ~scmscm/scripts/dbconf.sh Enter the DB type:1 - Oracle2 - MySQL3 - Sybase4 - exit

Table 4-3 sceconf.sh Options

Adapter=IP address Drop the connection at the specified IP address.

--help Display the seconf.sh options.


OL-26830-04

Chapter 4 Managing Collection Manager Configuring the Timezone Offset

Enter your choice: 1Enter Oracle server host (current is localhost) :10.56.216.80Enter Oracle server listening port (current is 1521) :Enter Oracle server instance id (current is apricot) :Enter CM schema user name (current is pqb_admin) :Enter CM schema user password (current is pqb_admin) :Do you want to test the DB connection? (yes/no): yesTesting the DB connection...PASS:db is upDB connection succeeded.Updating the time zone...

Note Restart Collection Manager for the configuration to take effect.

Configuring the Timezone OffsetYou can configure the timezone offset manually or automatically.

To configure the timezone offset (from GMT) manually, issue the following command:

~scmscm/cm/bin/jselect-sce-tz.sh --offset=offset-in-minutes

Note Enter the offset in minutes.

For example, if the SCE is located in GMT+2, issue the following command:

~scmscm/cm/bin/jselect-sce-tz.sh --offset=120

For example, if the SCE is located in GMT-10, issue the following command:

~scmscm/cm/bin/jselect-sce-tz.sh --offset=-600

To configure the timezone offset (from GMT) automatically, issue the following command:

~scmscm/cm/bin/updatetimezone.sh

The updatetimezone.sh script calculates the offset in minutes and updates the CONF_SE_TZ_OFFSET table.

Note Run the updatetimezone.sh script every time the timezone of the Cisco SCE is updated. For example, when updating the timezone to accommodate daylight savings time, remember to run the updatetimezone.sh script. The SCA Reporter does not use the correct time intervals unless the offset stored in Collection Manager is consistent with the timezone of the SCE.

Using Files to Configure the Collection ManagerUse the following files to configure Collection Manager:

• cm.conf—General configuration of Collection Manager, including the adapters to turn on when Collection Manager starts. See the “Enabling an Adapter” section on page 4-5.

• queue.conf—Configuration of the adapter queues, including the RDR tags that are associated with a specific adapter. See the “Configuring the Categorizer” section on page 4-7.


OL-26830-04

Chapter 4 Managing Collection Manager Configuring the Categorizer

Configuring the CategorizerThe Categorizer classifies each RDR according to its RDR tag. You can route an RDR to a specific adapter by adding its RDR tag to the tags parameter (a comma-separated list of RDR tags) of the adapter. This configuration is contained in the queue.conf file.

The following example shows how to configure the RDR tags 4042321920 and 4042321922 to be sent to the topper/aggregator (TA) adapter.

# Topper/Aggregator Adapter [topper-hi] adapter_id=3 priority=3 warning_size=40000 maximum_size=50000 tags=4042321920,4042321922

Note The value of the adapter_id parameter must match the adapter.<number> defined in the cm.conf file for the corresponding adapter.


OL-26830-04

Chapter 4 Managing Collection Manager Monitoring System Health

Monitoring System HealthCollection Manager contains a small, expandable framework that monitors the system and issues alerts for predefined, potentially problematic conditions.

Use the following scripts to monitor Collection Manager:

• ~scmscm/setup/monitor/setup-monitor.sh

• ~scmscm/setup/monitor/monitor.sh

Installing the Periodic CheckerTo insert (or remove) an entry for monitor.sh, the periodic checker script, in the cron (periodic scheduler) subsystem, run the setup-monitor.sh script:

~scmscm/setup/monitor/setup-monitor.sh -a flag [ -I flag ]

Table 4-4 shows the setup-monitor.sh script options.

The following example shows how to install monitor.sh so that it runs every 30 minutes:

$ ./setup-monitor.sh -a install -i 30m

The following example shows how to uninstall monitor.sh:

$ ./setup-monitor.sh -a uninstall

Periodic Checker ScriptThe following sections provide information on the periodic checker script:

• Periodic Checker Script, page 4-8

• Tests, page 4-9

Periodic Checker Script

The periodic checker script, monitor.sh, calls a series of subscripts that monitor different aspects of a running system:

~scmscm/setup/monitor/monitor.sh { -a | TEST NAME } [ -v ] [ -d ]

The script is not intended to be run from the command line, although you can do so. The script sends test results to the syslog subsystem and logs the results in the /var/log/messages file.

Table 4-5 shows the smonitor.sh script options.

Table 4-4 setup-monitor.sh Options

-a { install | uninstall } Insert or remove an entry for monitor.sh in the cron.

-i { 30m | 1h | 12h | 24h } Run monitor.sh every 30 minutes, 1 hour, 12 hours, or 24 hours.


OL-26830-04

Chapter 4 Managing Collection Manager Periodic Checker Script

Any test returns a result in the following format:

-STATUS: Message

• STATUS—PASS or FAIL

• Message—A short informative status message

For example, FAIL: db "apricot" has only 1523 free blocks

The following example shows how to run all available tests and print system output to the window:

$ ./monitor.sh -d -aTest: 01free_db.sh. Status: PASS. Message: db apricot has 1532 free blocks Test: 02cm_is_up.sh. Status: FAIL. Message: cm process is not running

The following example shows how to run one test to check the installed database has sufficient free space:

$ ./monitor.sh -d free_db Test: 01free_db.sh. Status: PASS. Message: db apricot has 1532 free blocks

Tests

You can run the following tests using monitor.sh:

• db_up—Checks whether the Collection Manager database is running.

• cm_up—Checks whether the Collection Manager application is running.

• free_db—Checks whether the bundled Sybase database has at least 10 percent free space.

• free_log—Checks whether the bundled Sybase database transaction log has at least 70 percent free space.

• cm_persistent_buffers—Checks whether the persistent buffer of each Collection Manager adapter contains less than 500 files.

The scripts for all these tests are located in the ~/setup/monitor/tests directory.

When calling a test called test_name, the script expects to find a file called NNtest_name.sh, where NN is a number that denotes script priority. For example, the test free_db is mapped to the file 01free_db.sh.

Table 4-5 monitor.sh Options

-a Run all tests.

TEST NAME The names of one or more tests. A test name is the test filename, without the leading digits and trailing .sh.

-v Output results in verbose mode. (Log successful tests.)

-d Print results to window. (By default, results are sent to syslog.)


OL-26830-04

Chapter 4 Managing Collection Manager Managing Users

Managing UsersCollection Manager uses the p3rpc utility to manage users for authenticated RPC calls.

The command format is: p3rpc OPERATION [OPTIONS]

Table Table 4-6 lists the p3rpc operations and options.

ExamplesThe following example shows how to add and update a username and password:

bash-2.05$ p3rpc --set-user --username=lulu --password=lili Command terminated successfully bash-2.05$

The following example shows how to validate a username and password. In this example, the user is successfully validated:

bash-2.05$ p3rpc --validate-password --username=lala --password=lala Local machine: user lala was authenticated successfully : (auth level root) Command terminated successfully bash-2.05$

The following example shows how to validate a username and password. In this example, the user validation fails:

bash-2.05$ p3rpc --validate-password --username=lala --password=lulu Error - Failed to authenticate user lala bash-2.05$

The following example shows how to delete a user configuration:

bash-2.05$ p3rpc --delete-user --username=lulu Command terminated successfully bash-2.05$

The following example shows how to display all of the configured users. In this example, only one configured user, clu, exists:

bash-2.05$ p3rpc --show-users clu Command terminated successfully bash-2.05$

Table 4-6 p3rpc Operations

Operation Description--set-user --username=username --password=password

Adds and updates the username and password.

--validate-password --username=username --password=password

Validates the username and password.

--delete-user --username=username Deletes a user configuration.

--show-users Displays all the configured users.


OL-26830-04

Chapter 4 Managing Collection Manager Managing Virtual Links

Managing Virtual LinksA script is included in Collection Manager distribution to enable you to manage virtual link names and indexes that are configured for a specific SCE:

To show or set virtual links, run the update_vlinks.sh script:

~scmscm/cm/bin/update_vlinks.sh --sce=SCE IP address [ --file=file | --show ]

Table 4-7 shows the update_vlinks.sh script options.

To set the virtual link details, as the scmscm user, run the following command:

~scmscm/cm/bin/update_vlinks.sh --sce=SCE IP address --file=file

The CSV file format is link ID (positive integer), link direction (0=upstream, 1=downstream), name (string).

The script performs following validations on the file:

• The file exists.

• There are no duplicate virtual links IDs for each direction.

• The virtual link ID is a positive value from 0 to 1024.

• The direction is either 0 (upstream) or 1 (downstream).

• No duplicate virtual link names or empty names exist for each direction.

• Virtual link names can contain up to 256 characters. All printable characters with an ASCII code from 32 through 126 (inclusive) can be used, except for 34 ("), 39 ('), and 96 (`).

After the file is successfully validated, the script performs the following actions:

Step 1 The input CSVfile must contain three data fields (VLINK_ID, VLINK_DIRECTION, VLINK_NAME) or six data fields (VLINK_ID, VLINK_DIRECTION, VLINK_NAME, CHANNEL_ID, CHANNEL_NAME, CMTS_NAME).

Step 2 If the CSV file contains three data fields, the VLINK_INI table automatically fills the data for the rest of the columns.

Step 3 All entries in the table that match the SCE_IP_Address and CHANNEL_ID, VLINK_DIRECTION, and CHANNEL_NAME are deleted from the table.

Step 4 The script adds two entries to the VLINK_INI table in the following format:

• Timestamp, sce ip, 0, 0, "Default Virtual Link Up,0, 0, "Default Virtual Link Up"

• Timestamp, sce ip, 0, 1, "Default Virtual Link Down, 0, 1, "Default Virtual Link Down"

Table 4-7 update_vlinks.sh Options

--sce=SCE IP --file=file Update the VLINK_INI table with the data in the supplied CSV formatted file for the specified SCE.

--sce=SCE IP --show Query the VLINK_INI table for entries for the specified SCE.

--help Display the above options.


OL-26830-04

Chapter 4 Managing Collection Manager Managing IMEI or MEID TAC with Device Types

Step 5 The script parses the CSV file and each line in the CSV file is entered as a record entry in the VLINK_INI table.

Step 6 If the CSV file is empty, the script inserts default records into the VLINK_INI table.

Managing IMEI or MEID TAC with Device TypesCollection Manager supports either GSM or CDMA mobile-based reports. GSM reports are enabled by default. Once Collection Manager is installed, the CSV file ~scmscm/cm/config/imeitac_device_map.csv populates the IMEI_DEVICETYPE table with default values.

The MEID_DEVICETYPE table stores the CDMA Device Type mapping information in the database. You manually execute the ~scmscm/scripts/update_devicetype.sh script to update the table. The default input mapping file is ~scmscm/cm/config/meidtac_device_map.csv:

~scmscm/scripts/update_devicetype.sh

Table 4-8 shows the update_devicetype.sh Options

Update the IMEI_DEVICETYPE tableTo update the IMEI_DEVICETYPE table, add new TAC details to the CSV file ~scmscm/cm/config/imeitac_device_map.csv in following the format:

Manufacturer <string>, Model <string>, TAC <number>.

Next, run the /update_devicetype.sh script using the --u option.

~scmscm/scripts/update_devicetype.sh --imei --u

This deletes all the existing records, then updates the IMEI_DEVICETYPE table with the data given in the default CSV file.

The update_devicetype.sh script performs the following actions:

• The script parses the CSV file ~scmscm/cm/config/imeitac_device_map.csv, and each line in the CSV file is entered as a record entry in the IMEI_DEVICETYPE table.

Table 4-8 update_devicetype.sh Options

update_devicetype.sh --imei --r Removes all the existing records from the IMEI_DEVICETYPE Table.

update_devicetype.sh --meid --r Removes all the existing records from the MEID_DEVICETYPE Table.

update_devicetype.sh --imei --u Deletes all the existing records and then updates the IMEI_DEVICETYPE table with the data given in the default CSV file.

update_devicetype.sh --meid --u Deletes all the existing records and then updates the MEID_DEVICETYPE table with the data given in the default CSV file.

update_devicetype.sh --h Prints the usage messsage.


OL-26830-04

Chapter 4 Managing Collection Manager Update the MEID_DEVICETYPE Table

• If the TAC number is a positive value, records are considered valid. Otherwise, the records are ignored.

• If the script finds duplicate TAC entries, the first TAC entry is considered valid while the other entries are ignored.

• If the CSV file is empty, the script deletes all the existing records in the IMEI_DEVICETYPE table.

Update the MEID_DEVICETYPE TableTo update the MEID_DEVICETYPE table, add new TAC details to the CSV file ~scmscm/cm/config/meid_device_map.csv following the format:

Manufacturer <string>, TAC <number>.

Next, run the update_devicetype.sh script using the --u.

~scmscm/scripts/update_devicetype.sh --meid --u

This deletes all the existing records then updates the MEID_DEVICETYPE table with the data given in the default CSV file.

The update_devicetype.sh script performs the following actions:

• The script parses the CSV file ~scmscm/cm/config/meidtac_device_map.csv and each line in the CSV file is entered as a record entry in the MEID_DEVICETYPE table.

• If the TAC number is a positive value, records are considered valid. Otherwise, the records are ignored.

• If the script finds duplicate TAC entries, the first TAC entry is considered valid while the other entries are ignored.

• If the CSV file is empty, the script deletes all the existing records in the MEID_DEVICETYPE table.


OL-26830-04

Chapter 4 Managing Collection Manager Managing the Peak Hour Configuration

Managing the Peak Hour ConfigurationYou need to execute the ~scmscm/scripts/updatepeakconfig.sh script before generating the peak hour reports. This script populates the peak_hour_config table with the default peak hour details. By default, the peak_hour_config table is not created.

The updatepeakconfig.sh script then prompts you to enter the peak hour range for any days you want to change from the default. If the value is not entered for the corresponding day, the current value shown in the prompt is considered the default value.

Note Execute the ~scmscm/scripts/updatepeakconfig.sh script once with default values before generating any peak hour reports.

The following output is an example of a Peak Hours configuration:

-bash-2.05b$ ~scmscm/scripts/updatepeakconfig.shEnter the Peak Hour Range for Monday (current is 18-21) :Enter the Peak Hour Range for Tuesday (current is 18-21) :15-16Enter the Peak Hour Range for Wednesday (current is 18-21) :15Enter the Peak Hour Range for Thursday (current is 18-21) :Enter the Peak Hour Range for Friday (current is 18-21) :Enter the Peak Hour Range for Saturday (current is 16-19) :Enter the Peak Hour Range for Sunday (current is 16-19) :Peak Configurations successfully updated!!!!!Restart CM for the new config to take effect!!


OL-26830-04

Chapter 4 Managing Collection Manager Monitoring Collection Manager

Monitoring Collection ManagerYou can use scripts to monitor system statistics that are relevant to Collection Manager, such as:

• Percentage of free space in the database

• Rate of RDRs entering Collection Manager

• SCE platform connection data

• Viewing database insertion rate statistics per table

• Viewing version information

Use the following scripts to monitor Collection Manager:

• ~scmscm/scripts/dbfree.sh

• ~scmscm/scripts/rdr-rate.sh

• ~scmscm/scripts/sceconf.sh

• ~scmscm/setup/alive.sh

• ~scmscm/cm/bin/p3stats

• ~scmscm/cm/bin/cm version

• ~scmscm/cm/bin/cm dbversion

Use the following scripts to configure Collection Manager (see the “Configuring Collection Manager” section on page 4-3). You can also run these scripts to display the relevant configuration:

• ~scmscm/setup/on-boot.sh

• ~scmscm/scripts/adapterconf.sh

Checking the Database CapacityTo display the percentage of free space in the database report tables and the associated transaction log, run the dbfree.sh script:

~scmscm/scripts/dbfree.sh

Use the script only with a bundled Sybase and MySql databases.

Step 1 As the scmscm user, run the dbfree.sh script.

Checking the RDR RateTo display the momentary total rate of reports entering Collection Manager, run the rdr-rate.sh script

rdr-rate.sh

The output is a single floating-point number that represents the total rate per second of incoming RDRs (from all sources), which entered Collection Manager in the past 5 seconds.

Use this script only if the HTTP adaptor of Collection Manager is running.


OL-26830-04

Chapter 4 Managing Collection Manager Checking the SCE Connection

Step 1 As the scmscm user, run the rdr-rate.sh script.

Checking the SCE ConnectionTo display information about the SCE connections, run the sceconf.sh script:

~scmscm/scripts/sceconf.sh --op=list

Use this script only if the HTTP Adaptor of Collection Manager is running.

Also use the script to drop a connection from a particular SCE. See the “Dropping a Cisco SCE Connection” section on page 4-5.

Step 1 As the scmscm user, run the sceconf.sh script.

~scmscm/scripts/sceconf.sh --op=list

Example

The following example shows SCE connection output:

>~scmscm/scripts/sceconf.sh --op=list IP Rate Peak ------- ------- ------- 10.1.6.93 0.71798986 0.718 10.1.9.36 0.14420895 0.1442139 10.1.9.35 0.0 0.027929332 10.1.12.11 0.0 0.0

Verifying Server OperationTo verify that the server is functioning correctly, run the alive.sh script:

~scmscm/setup/alive.sh

The script verifies that the following components are operational:

• Collection Manager

• Database (in the bundled database case)

• Report tables (in the bundled database case)

If any component is down, the script issues an error message.

Step 1 As the scmscm user, run the alive.sh script

Note It takes time for the components to initialize after a startup. After a restart, wait 5 minutes before running this script.


OL-26830-04

Chapter 4 Managing Collection Manager Viewing Database Statistics

Viewing Database StatisticsTo view the statistics of Collection Manager database, run the p3stats command line utility (CLU).

~scmscm/cm/bin/p3stats [options]

Table 4-9 lists the p3stats options.

Note Collection Manager must be running for this CLU to operate.

The following example shows how to display the statistics of Collection Manager database:

$ ~scmscm/cm/bin/p3stats --show-statsLUR { rate=0, peak=0 }MALUR { rate=0, peak=0 }TR { rate=0, peak=0 }VLUR { rate=0, peak=0 }NUR { rate=0, peak=0 }SUR { rate=0, peak=0 }MEDIA { rate=0, peak=0 }PUR { rate=0, peak=0 }Command terminated successfully$

Viewing Version InformationCollection Manager includes two scripts to display the current version of Collection Manager and of the database:

• cm version

• cm dbversion

The following example shows how to view the current Collection Manager version:

$ ~scmscm/cm/bin/cm versionCM CD Version 3.8.5 Build 14

The following example shows how to view the current database version:

$ ~scmscm/cm/bin/cm dbversionCM: DatabaseSybase ase150/

Table 4-9 p3stats Options

Option Description

--show-stats Displays statistics information in the format <DB RDR tag> <RDR rate> <peak RDR rate>

--reset-stats Resets the statistics counters.


OL-26830-04

Chapter 4 Managing Collection Manager Viewing Version Information


OL-26830-04

OL-26830-04

C H A P T E R 5
Managing Databases and the Comma-Separated Value Repository
This chapter explains how to use utility scripts to manage Collection Manager database and the comma separated value (CSV) repository.

This chapter contains the following sections:

• Common Database Management Tasks, page 5-2

• Managing the Bundled Sybase Database, page 5-8

• Managing the External MySQL Database, page 5-10

• Partitioning the MySQL Database, page 5-10

• Managing the Oracle Database, page 5-13

• Aggregating Database Data, page 5-14

• Managing the CSV Repository, page 5-18


Chapter 5 Managing Databases and the Comma-Separated Value Repository Common Database Management Tasks

Common Database Management TasksThe following database management tasks apply to all the supported databases:

• Listing Database Tables, page 5-2

• Managing the Periodic Deletion of Old Records, page 5-3

• Deleting a Table, page 5-6

• Deleting Old Records, page 5-6

• Unicode Support for Sybase Database, page 5-7

The Collection Manager adds a timestamp to every raw data record (RDR) that it receives and stores in the database. This timestamp is used when various maintenance operations are performed on the database tables.

Use the following scripts to configure and maintain the database:

• ~scmscm/scripts/dbtables.sh

• ~scmscm/scripts/dbperiodic.sh

• ~scmscm/db_maint/create_periodic_del_procs.sh

• ~scmscm/scripts/droptable.sh

• ~scmscm/scripts/prunetable.sh

• ~scmscm/scripts/sybutfsupport.sh

• ~scmscm/scripts/upgradeDB.sh

• ~scmscm/scripts/upgradeRDRTables.sh

• ~scmscm/scripts/partitions.sh

• ~scmscm/scripts/createaggevents.sh

Listing Database TablesTo display a list of all the tables in the database, run the dbtables.sh script:

~scmscm/scripts/dbtables.sh

Where applicable, the script displays the number of lines in the table and the earliest and latest timestamps.

For more information about using the Cisco Service Control Application Reporter to display the actual content of the tables, see the Cisco Service Control Application Reporter User Guide.

Table 5-1 lists the dbtables.sh script options.

Table 5-1 dbtables.sh Options

Option Description

-l Lists only the existing table names (without statistics).

-c Lists the existing table names with their record counts.

-a Includes the nonreport tables in the listing.

-f Enables fast-line counting, using the client rather than the JDBC.


OL-26830-04


Chapter 5 Managing Databases and the Comma-Separated Value Repository Managing the Periodic Deletion of Old Records

The following is sample output of the dbtables.sh:

>~scmscm/scripts/dbtables.sh -c Executing query ... name| num_lines| min_time| max_time|----------------+--------+------------------------+------------------------+ RPT_SUR| 131000| 2006-10-30 16:46:42.24| 2007-02-15 12:00:32.216| RPT_LUR| 170000| 2007-04-10 15:25:45.31| 2007-04-11 07:06:05.45| RPT_VLUR| 4694| 2007-04-11 13:12:39.683| 2007-04-11 13:18:07.396| RPT_PUR| 116000| 2007-04-09 04:45:55.566| 2007-04-11 07:44:09.186| RPT_TR| 57766| 2007-04-11 13:12:39.683| 2007-04-11 13:18:07.396| RPT_MALUR| 109000| 2007-04-09 04:46:35.903| 2007-04-09 13:32:18.42| RPT_MEDIA| 120000| 2007-04-05 17:14:24.443| 2007-04-11 13:16:29.436|RPT_TOPS_PERIOD0| 194250| 2007-03-18 20:00:00.01| 2007-04-23 06:00:00.16|RPT_TOPS_PERIOD1| 46940| 2007-03-19 00:00:00.05| 2007-04-23 00:00:00.1|

Managing the Periodic Deletion of Old RecordsTo manage the periodic deletion of old records, perform the following steps:

1. If the periodic delete procedures have not already been installed during the Collection Manager installation, install them:

Log in as the scmscm user, start Collection Manager, wait for 1 to 2 minutes for the creation of the database tables, and then run the script:

~scmscm/db_maint/create_periodic_del_procs.sh

2. Edit the periodic delete configuration file. For more information, see the “Configuring Periodic Delete” section on page 5-4.

3. To apply the new configuration, run the dbperiodic.sh utility script. For more information, see the “Applying the Periodic Delete Configuration File” section on page 5-5.

Caution If you enable both the aggregation and the periodic delete feature, ensure that you increase their respective configuration values.

Periodic deletion of a table does not begin while a previous periodic deletion procedure is still running. This prevents excessive load on the database, which would otherwise degrade insertion performance in the adapters.

When two or more tables are scheduled to be reduced at the same time, the tables are processed in the order in which they are listed in the periodic delete configuration file.

For ease of configuration, you can schedule periodic deletion for all the tables consecutively in one schedule. All the periodic delete activities recorded in the system log file. For Solaris, the system log file is located in the /var/adm/messages directory; for Linux, the system log file is located in the /var/log/messages directory.

-t { sec_num } Displays the maximum waiting time, in seconds, for the response. The default is no timeout.

-h Prints this help message and exits.

-s table_nm Displays the statistics of a given table.

Table 5-1 dbtables.sh Options (continued)

Option Description


OL-26830-04


The following is an example of the system log file at the time when the periodic delete action occurs:

Feb 17 05:00:07 app-redhat79 logger: /opt/scmscm/db_maint/periodic_del.sh for hourly all - running single del for RPT_LUR to 14 daysFeb 17 05:00:09 app-redhat79 logger: Starting periodic delete for RPT_LUR keeping 14 daysFeb 17 05:00:09 app-redhat79 logger: Deleted rows: 0 from the table RPT_LUR

Note MySql server supports periodic delete from MySql 5.0 or later versions.

Caution Be sure to disable the MySQL partitioning process in Collection Manager when you enable the periodic delete feature.

• The following section describes how to configure and apply the Periodic Delete configuration file.

• Configuring Periodic Delete, page 5-4

• Applying the Periodic Delete Configuration File, page 5-5

Configuring Periodic Delete

By default, the periodic delete configuration file (dbperiodic.conf) is located in the ~scmscm/db_maint/ directory. The file has a structure that is similar to an INI file, where each section describes a particular data reduction operation (for a specific set of tables), to be performed according to a specified schedule.

Note The name of each section of the file is not used when the file is parsed. You can use any name you choose.

Each section begins with the section name in brackets, and contains the parameters shown in Table 5-2. (Not all parameters are required in each section of the configuration file.) Separate the parameters and their values by an equal sign (=). Table 5-2 contains examples of periodic delete configuration files.

Table 5-2 Parameters in a Periodic Delete Configuration File

Parameter name Explanation Value Default Example

active Specifies whether or not to use this section of the configuration file

true/false true false

tablenames Names of the tables to which this section of the configurtion file applies

Names of tables separated by commas, or * for all tables

* (all) RPT_SUR,RPT_LUR

daystokeep Number of days to maintain records

Positive integers 14 30

minute

hour

day

month

The time at which to perform the deletion in this section of the configuration file

0 … 59, *

0 … 23, *

1 … 31, *

1 … 12, *

0 *

(all) *

(all) *

(all)

0

0,4,8,12,16,20

1

1,3,5,7,9,11


OL-26830-04


Note Values for all the parameters except active and daystokeep can be either a single value, a list of values separated by commas, a range of values (two values separated by a dash), or an asterisk (*), which signifies all possible values. A range is not possible for tablenames.

In the following example, all the fields are set to their default values:

# This dbperiodic.conf file emulates the legacy style for periodic # deletion. All tables are processed every hour on the hour, and # records are kept for 14 days. [hourly all] active = true tablenames = * daystokeep = 14 minute = 0 hour = *

In this example, all the tables are reduced at 4:30 A.M. This leaves 10 days of data in each table. In addition, the real-time tables are reduced every hour, leaving three days of data in each table.

# This dbperiodic.conf file reduces all tables once a day and # real-time tables once an hour. [daily all] active = true tablenames = * daystokeep = 10 minute = 30 hour = 4 [hourly real-time] active = true tablenames = RPT_SUR,RPT_LUR,RPT_PUR daystokeep = 3 minute = 0 hour = *

Applying the Periodic Delete Configuration File

To load and apply a new periodic delete configuration file or to view the current file, run the dbperiodic.sh script:

~scmscm/scripts/dbperiodic.sh [ --dump ] [ --load | --loadfile=path_to_dbperiodic.conf ]

When the script is used to load a new configuration file, it parses the file, verifies its validity, and updates the scmscm user's crontab to reflect the changes. Table 5-3 lists the dbperiodic.sh options.

Table 5-3 dbperiodic.sh Options

Option Description--load Load the periodic delete configuration file from

the /export/home/scmscm/db_maint/ dbperiodic.conf

--loadfile=path to periodic delete configuration file

Load the periodic delete configuration file from the specified directory.

--dump Print the periodic delete configuration.

--h Displays the above options


OL-26830-04

Chapter 5 Managing Databases and the Comma-Separated Value Repository Deleting a Table

The following example shows how to print the current periodic delete configuration:

~scmscm/scripts/dbperiodic.sh --dump

Note The script prints the loaded periodic delete configuration. If the current periodic delete configuration file is not yet loaded, the configuration can vary from the script output.

The following example shows how to load the periodic delete configuration file from the ~scmscm/db_maint/dbperiodic.conf file:

~scmscm/scripts/dbperiodic.sh --load

The following example shows how to load the periodic delete configuration file from a specified location:

~scmscm/scripts/dbperiodic.sh --loadfile=path_to_periodic_delete_configuration_file

Deleting a TableTo delete a table or all the current tables from the database, run the droptable.sh script:

~scmscm/scripts/droptable.sh [-f] tableParameter

Table 5-4 lists the droptable.sh options.

The following example shows how to drop a table named RPT_SUR from the database with no request for confirmation:

~scmscm/scripts/droptable.sh –f RPT_SUR

The following example shows how to drop all the tables from the database:

~scmscm/scripts/droptable.sh ALLTABLES

Deleting Old RecordsTo remove records from a database table based on the timestamps of the records, run the prunetable.sh script:

~scmscm/scripts/prunetable.sh [ -f ] num_days table_name

Table 5-4 droptable.sh Options

Option Description

table_name Drop table_name from the database.

ALLTABLES Drop all the tables from the database.

-f Drop the table by force (no questions asked or errors reported)

-h Displays the above options.


OL-26830-04

Chapter 5 Managing Databases and the Comma-Separated Value Repository Unicode Support for Sybase Database

Table 5-5 lists the prunetable.sh options.

The following example shows how to delete all the records that are more than seven days old from a table named RPT_SUR.

Because the –f flag is not specified, there can be requests for confirmation and errors:

>~scmscm/scripts/prunetable.sh 7 RPT_SUR

Note Implement the Periodic delete procedure {Refer Section : Managing the CM:Periodic Deletion of Old Records } before invoking the prunetable.sh script

Unicode Support for Sybase DatabaseTo add the UCS Transformation Format—8-bit (UTF-8) support for the Sybase database:

Step 1 Login to Cisco Service Control Collection Manager as a root user.

Step 2 Run the ~scmscm/scripts/sybutfsupport.sh script.

The script restarts the Sybase after enabling the UTF-8 support in the database.

Step 3 Restart the Cisco Service Control Collection Manager.

The UTF-8 support is enabled on the Cisco Service Control Collection Manager.

Table 5-5 prunetable.sh Options

Option Description

num_days The maximum age (in days) of records not to deleted.

table_name The table whose records will be deleted

-f Drop by force (no questions asked or errors reported).

-h Displays the above options.


OL-26830-04

Chapter 5 Managing Databases and the Comma-Separated Value Repository Managing the Bundled Sybase Database

Managing the Bundled Sybase DatabaseThe task of managing the bundled database includes:

• Backing up and restoring a database

• Updating Sybase with a changed IP address

• Updating the Sybase database statistics

Collection Manager adds a timestamp to each RDR it receives and stores in the database. This timestamp is used when various maintenance operations are performed on the database tables.

Use the following scripts to maintain the bundled Sybase database:

• ~scmscm/scripts/sybback.sh

• ~scmscm/scripts/sybrestore.sh

• ~scmscm/db_maint/update_statistics.sh

Backing Up the DatabaseIn Collection Manager Release 3.7.0 and later, by default, the backup database files are stored in the Collection Manager's home directory under the Sybase directory. An example of the home directory is /home/scmscm/sybase.

Note Consult the Sybase Backup Server product documentation for a complete Sybase backup. The bundled Collection Manager backup is only designed for backing up and restoring small amounts of data, for example, transferring small tables between servers.


OL-26830-04

Chapter 5 Managing Databases and the Comma-Separated Value Repository Updating Sybase with a Changed IP Address

Updating Sybase with a Changed IP AddressWhen you change the IP address, you must also update the Sybase server:

Step 1 Change the IP address. Run the interface configuration script:

/etc/sysconfig/network-scripts/ifcfg-eth0

Step 2 Run the network restart script:

/etc/rc.d/init.d/network restart

Step 3 Update the Sybase server. As the root user, issue the following command:

~scmscm/setup/syb_interfaces.sh

Step 4 Run the Sybase stop and start script as the root user:

~scmscm/setup/sybase stop/start

Step 5 Restart Collection Manager, run the following script as the scmscm user:

~scmscm/cm/bin/cm restart

Updating Sybase Database StatisticsTo update the Sybase database statistics, run the update_statistics.sh script once a week at a time of low database load, for example, at night.

~scmscm/db_maint/update_statistics.sh


OL-26830-04

Chapter 5 Managing Databases and the Comma-Separated Value Repository Managing the External MySQL Database

Managing the External MySQL DatabaseThe tasks of managing the MySQL database includes:

• Backing up and restoring a database

Collection Manager adds a timestamp to each RDR it receives and stores in the database. This timestamp is used when various maintenance operations are performed on the database tables.

Use the following scripts to maintain the MySQL database:

• ~scmscm/scripts/mysqlback.sh

• ~scmscm/cm/config/dbpacks/mysql/4.0.20/mysql.properties

• ~scmscm/scripts/mysqlrestore.sh

Backing Up the MySQL DatabaseTo create text file backups of all the tables in the database, use the mysqlback.sh script:

~scmscm/scripts/mysqlback.sh

To create the mysql backup directory, use the script:

~scmscm/cm/config/dbpacks/mysql/4.0.20/mysql.properties

The following is an example of a backup file:

backup_dir=/mysql/backup

Note The destination location path is defined in the *.properties file. Modify the path if you want to change the default location.

Restoring a MySQL Database To restore a database from the backup file created by the mysqlback.sh script, run the mysqlrestore.sh script:

~scmscm/scripts/mysqlrestore.sh

Partitioning the MySQL DatabaseYou can store records in an external MySQL database. When you configure MySQL as the database, partitioning is supported.

• Prerequisites to Configuring Partitioning, page 5-11

• Configuring Partitioning, page 5-11


OL-26830-04

Chapter 5 Managing Databases and the Comma-Separated Value Repository Prerequisites to Configuring Partitioning

Prerequisites to Configuring PartitioningThe partitioning feature is supported only when the MySQL server meets the following conditions:

• MySQL Version 5.1.06 or later

• open_files_limit value in my.cnf file should be set to 8000 for default partitioning configuration under the [mysqld] section

• thread_stack value in my.cnf file should be set to at least 256 k under the [mysqld] section

Restart the MySQL server to enable the configuration changes to take effect.

Configuring Partitioning

Caution If a partition is created on the existing data, all the existing data will be deleted . Back up the existing data before running the partitioning script.

The partitioning configuration file (partitions.conf) is located in the ~scmscm/cm/config/dbpacks/mysql/4.0.20/partitions/ directory.

The following is a sample of the configuration file:

[partitions_config1]table_names = RPT_LUR,RPT_TOPS_PERIOD1,RPT_FUR,RPT_GURno_of_partition = 30period = 1scheduler_time = 23:30

Each section begins with a unique section name starting with partitions_config in brackets and contains the parameters shown in Table 5-6. Separate the parameters and their values by an equal sign (=).

Table 5-6 Parameters in the Partitioning Configuration File

Parameter Name Explanation Value Default Example

table_names Name of the table to partition.

Start the table name with the prefix RPT_. Should not be duplicated across sections.

— RPT_LUR

no_of_partition Number of partitions to create in the database.

1 - 65. — 30

period Number of days that the records are retained in each partition.

1 - 5. — 1

scheduler_time Time at which to trigger the daily partitioning event scheduler in the database.

Hour and minutes. Format: HH:MM

— 23:30


OL-26830-04

Chapter 5 Managing Databases and the Comma-Separated Value Repository Configuring Partitioning

To create and view partitions, run the partitions.sh script:

~scmscm/scripts/partitions.sh [ --create ] [ --non_partitioned ] [ --partitioned ] [--partitions [table_name]] [events][ --help ]

Table 5-7 lists the partitions.sh options.

Perform the following steps to enable the partitioning feature:

1. Log in as the scmscm user.

2. Change the directory to ~scmscm/scripts/.

3. Customize the ~scmscm/cm/config/dbpacks/mysql/4.0.20/partitions/partitions.conf file based on your requirements.

4. Run the ~scmscm/scripts/loadsqlprocedures.sh --par script to load the required support files for partitioning.

5. Run the ~scmscm/scripts/partitions.sh --create to create the partitioning.

Note Ensure that the periodic deletion process is disabled in the Collection Manager when you enable the partitioning feature.

Table 5-7 partitions.sh Options

Option Description--create Create partitions for database tables that are not

partitioned.

--non_partitioned Show details for database tables that are not partitioned.

--partitioned Show details for database tables that are partitioned.

--partitions [table_name] Show all the table partitions details. [table_name] specifies the table to be displayed.

--events Show the detail of the configured partitioning events

--help Display the above options


OL-26830-04

Chapter 5 Managing Databases and the Comma-Separated Value Repository Managing the Oracle Database

During the creation of the partitions, the partitions.sh script performs the following actions:

• Enables the event scheduler on the MySQL server.

• If any of the tables (table_names) listed in the configuration file (partitions.conf) are not available in the database, partitioning is ignored for that particular table. The remaining tables are processed.

When the MySQL server is restarted after the partitioning, the event scheduler is disabled. To ensure the event scheduler remains enabled through a server restart, add event_scheduler = 1 to the [mysqld] section of the my.cnf file.

Note If any missing table can be created in the database later, run the partitions.sh script to create new partitioning. You can apply the same event scheduler for the new partitioning in each configuration section. This does not disturb the existing partitioning.

Note If any new tables are created in the database, define new configuration sections for these new tables and then run the partitions.sh script. This does not disturb the existing partitioning.

Although the partitioning event scheduler runs every day, it creates or deletes a partition based on the no_of_partition and period configuration parameters. Even if the server is restarted, the partitioning event scheduler is not disturbed.

Managing the Oracle DatabaseThe tasks of managing the Oracle database include:

• Oracle Support with Minimum Privileges

Oracle Support with Minimum PrivilegesThe following are the mandatory privileges that are to be granted to the CM-DB user: Create Session, Create Table, Create Procedure, Create Table Space, Unlimited Table Space, Create Index, Execute Any Procedure.

The following is the syntax of mandatory privileges:

grant CONNECT ,CREATE TABLE ,CREATE TABLESPACE ,CREATE PROCEDURE ,UNLIMITED TABLESPACE,CREATE ANY INDEX, EXECUTE ANY PROCEDURE to pqb_admin

Note Connect to the Oracle DB as SYSTEM/SYSDBA, log in to the Oracle server with specific schema (apricot/avocado), and grant the privileges listed here.


OL-26830-04

Chapter 5 Managing Databases and the Comma-Separated Value Repository Aggregating Database Data

Aggregating Database DataYou can aggregate database data according to different time periods and store data in the same database tables. Both raw and aggregated data are stored in order to maintain compatibility with the reporting tools that pull this data to generate analysis reports. All the reports that process raw data for display can also process the aggregated data because the data structure is the same.

An input date parameter specifies the data records that should be processed. These records are stored per time period. For xUR (where x is a particular type of Usage RDR) tables, hourly aggregation is performed, starting from the input date value through the following 14 days. Thereafter, aggregation occurs daily. For HTTP, video, and NUR VSA tables, aggregation occurs daily.

During the aggregation process, table rows are aggregated per time period. These values are placed in a temporary table. The source records are then removed to allow insertion of the aggregated records.

The aggregation feature is supported in the following tables:

• RPT_LUR

• RPT_PUR

• RPT_VLUR

• RPT_MALUR

• RPT_SUR

• RPT_SPAM

• RPT_ZUR

• RPT_TOP_HTTP_DOMAINS

• RPT_TOP_VIDEO_DOMAINS

• RPT_TOP_HTTP_HOSTS

• RPT_TOP_VIDEO_HOSTS

• RPT_TOP_APN

• RPT_TOP_DEVICE_TYPE

• RPT_TOP_NETWORK_TYPE

• RPT_TOP_USER_LOCATION

• RPT_TOP_SGSN

• RPT_TOP_PCF

• RPT_TOP_HOME_ AGENT

• RPT_TOP_MEID

Caution The actual table records are deleted from the table based on the aggregation range values.

Note If both periodic deletion and the aggregation process are enabled, you can increase the number of days for which the records are retained.


OL-26830-04

Chapter 5 Managing Databases and the Comma-Separated Value Repository Configuring Aggregation

Configuring AggregationThe ~scmscm/cm/config/aggregation.conf file specifies the values used in the aggregation process.

The following is a sample configuration file:

[agg_event1]procedure_names = P_RPT_MALUR_AGGREGATE, P_RPT_LUR_AGGREGATEno_of_days = 2scheduler_time = 23:30

Each section in the aggregation configuration file begins with a unique section name in brackets ([]) starting with agg_event . The file contains the parameters shown in Table 5-8. Separate the parameters and their values with an equal sign (=).

To create a new aggregation event, run the createaggevents.sh script:

~scmscm/scripts/createaggevents.sh [ --create ] [ --help ]

Table 5-9 lists the createaggevents.sh options.

Table 5-8 aggregation.conf Parameters

Parameter Name Explanation Value Default Example

procedure_names SQL procedure name to aggregate.

Any aggregation procedure name present in the database configuration location.

Sybase:

~scmscm/cm/config/dbpacks/ sybase/ase12.5.1/aggregation/

MySQL:

~scmscm/cm/config/dbpacks/ mysql/4.0.20/aggregation/

Oracle:

~scmscm/cm/config/dbpacks/ oracle/9204e/aggregation/

— P_RPT_MALUR_ AGGREGATE

no_of_days Number of days of raw data to be retained in the table.

1 to 31 — 2

scheduler_time The time at which to trigger the daily aggregation event scheduler.

Hour and minutes.

Format: HH:MM

— 23:30

Table 5-9 createaggevents.sh Options

Option Description

--create Creates the corresponding aggregation events.

--help Displays these options


OL-26830-04

Chapter 5 Managing Databases and the Comma-Separated Value Repository Enabling Aggregation in Sybase

The Sybase, MySQL, and Oracle databases support aggregation. The following sections describe how to enable the aggregation feature:

• Enabling Aggregation in Sybase, page 5-16

• Enabling Aggregation in MySQL, page 5-16

• Enabling Aggregation in Oracle, page 5-16

Note Use the loadsqlprocedure.sh --agg option to update the database configuration changes.

Enabling Aggregation in SybaseSybase Version 12 and Sybase Version 15 support data aggregation.

Complete the following steps to enable the aggregation feature:

1. Login as the scmscm user.

2. Run the ~scmscm/scripts/loadsqlprocedures.sh --agg script to load the required support files for aggregation.

3. Create a cron job to run the ~scmscm/cm/config/dbpacks/sybase/ase12.5.1/aggregation/cron/run_aggregation.sh script to schedule the aggregation process.

Cron job example:

crontab -e

Update the path and expected time to schedule the job:

30 23 * * * /<path>/run_aggregation.sh

4. Refer to the ~scmscm/cm/config/dbpacks/sybase/ase12.5.1/aggregation/cron/ readme_run_aggregation_procedure.txt text for more information.

Enabling Aggregation in MySQLMySQL Version 5.1.06 and later support data aggregation.


1. Login as the scmscm user.


Enabling Aggregation in OracleOracle Version 9, Version 10, and Version 11 support data aggregation.


1. Log in as the scmscm user.



OL-26830-04

Chapter 5 Managing Databases and the Comma-Separated Value Repository Enabling Aggregation in Oracle

Note Make sure that the Oracle user has sufficient privileges to alter and execute the aggregation events. For example: grant select ,insert ,delete ,update on sys.job$ to pqb_admin.


OL-26830-04

Chapter 5 Managing Databases and the Comma-Separated Value Repository Managing the CSV Repository

Managing the CSV RepositoryRun a utility script to manage the repository of CSV files output by Collection Manager. The CSV adapter writes these files to disk for use by a service provider’s operational support system (OSS) or by a third-party billing system. To prevent disk overflow, monitor the size of the CSV repository.

Note If the backup parameter is set to true, failure to delete CSV files may result in disk overflow. No CSV files are ever deleted.

Note A third-party application is responsible for managing the CSV files and deleting them as necessary.

To successfully invoke this script, the HTTP adapter of Collection Manager must be running. If the adapter is down, an error message is printed.

The following sections detail managing the CSV repository:

• CSV Repository File Structure, page 5-18

• Configuring the CSV File Repository, page 5-18

• Configuring the Comma Escape, page 5-20

• Configuring Escape of Nonprintable Characters, page 5-20

CSV Repository File StructureCSV files are stored in several subdirectories. Each subdirectory is given the number of a Raw Data Record (RDR) tag. RDR tags define the type of the corresponding RDR. Each RDR is stored in the subdirectory whose name matches its RDR tag number. For more information on RDR tags, see the Cisco Service Control Application for Broadband Reference Guide.

The CSV files are numbered sequentially (automatically), with separate numbering in each directory. You can change the location of the parent directory by editing the cm.conf file located in the cm/config directory.

Configuring the CSV File RepositoryUse the ~scmscm/scripts/csvconf.sh script to:

• List the number of RDRs that are currently stored in the repository.

• Configure the maximum number of CSV files and the maximum permissible number of reports (lines) in each file.

• Control whether a backup occurs when an old CSV file is about to be overwritten.

• Control whether each line in a CSV file contains an indication of the IP address of the Cisco SCE that sent this RDR. (By default, this option is off.)

Note Instead of using the ~scmscm/scripts/csvconf.sh script, you can edit the ~scmscm/cm/config/csvadapter.conf file. For changes in this file to take effect, restart Collection Manager.


OL-26830-04



Chapter 5 Managing Databases and the Comma-Separated Value Repository Configuring the CSV File Repository

Note The same configuration is applied to all the subdirectories in the CSV repository.

Note Setting these parameters does not change existing the CSV files. Parameter value changes only affect the files that are created after the parameters are set.

Table 5-10 lists the csvconf.sh options.

The following example shows how to set the maximum number of CSV files per subdirectory to 1000:

>~scmscm/scripts/csvconf.sh --maxfiles=1000

The following example shows how to set the maximum number records per CSV files to 10,000:

>~scmscm/scripts/csvconf.sh --maxlines=10000

The following example shows how to delete all the files from the CSV repository:

~scmscm/scripts/csvconf.sh --clear

The following example shows how to disable backing up old CSV files in the repository:

~scmscm/scripts/csvconf.sh –-backups=false

Table 5-10 csvconf.sh Options

Option Description

--list Display the CSV repository contents (the number of RDRs currently stored in the repository).

--clear Delete all the files from the CSV repository. (This option deletes all the CSV files, but not the directories in which they are contained.)

--maxlines=N Set the maximum number of RDRs per CSV file to N (an integer between 1 and 20,000).

--maxfiles=M Set the maximum number of CSV files in each subdirectory to M (an integer between 10 and 10,000.)

--backups={true |false} Enable or disable the backup of old CSV files.

--recordsource={true |false} Enable or disable the inclusion of the record source in the CSV files.


OL-26830-04

Chapter 5 Managing Databases and the Comma-Separated Value Repository Configuring the Comma Escape

Configuring the Comma EscapeWhen a comma is contained within a field in a CSV file, an escape sequence indicates that the comma does not mark the end of the field.

Three escape sequences are supported:

• Single quotation marks—Single quotation marks surround any field that contains one or more commas. There is no special treatment of single quotation marks that currently exist in the RDRs.

• URL—%2C replaces each comma contained within a field. There is no special treatment of such sequences that currently exist in RDRs.

• Backslash—A backslash (\) precedes each comma contained within a field. There is no special treatment of backslashes that currently exist in RDRs.

The first two escape methods are compatible with Microsoft Excel. The backslash method is not compatible with Excel, but is retained for backward compatibility.

Single quotation marks are used by default. You can change the escape method by modifying the value of the escapeMethod attribute. This attribute is located in the ~/scmscm/cm/config/csvadapter.conf file. You must use one of the following values:

• quote

• doublequote

• backslash

• url

Note escapeMethod is set to quote by default.

Configuring Escape of Nonprintable CharactersThe CSV adapter can escape nonprintable characters. Enabling this option incurs a performance penalty for the adapter. By default, the option is disabled.

When the option is enabled, a backslash (\) precedes each character that cannot be printed, such as CR and LF, contained within a field.

Enable this option in the ~/scmscm/cm/config/csvadapter.conf file. For the changes made to this configuration file to take effect, restart Collection Manager.


OL-26830-04

OL-26830-04

C H A P T E R 6
Configuring Databases
This chapter describes how to configure the Collection Manager to work with your database. The chapter also describes how to use the database infrastructure to extend the Collection Manager functionality.

The following sections describe how to configure the Collection Manager to work with your database:

• Changing Basic Connection Parameters, page 6-1

• Generating SQL Code Using the Velocity Template Language, page 6-1

• Database Configuration Files, page 6-3

• Working Sample, page 6-6

• Testing and Debugging, page 6-9

• Using the JDBC Framework in Scripts, page 6-10

• Scalability Hints for Oracle, page 6-12

Changing Basic Connection ParametersTo use an external database with Collection Manager, change the basic connection parameters, such as the IP address and the port on which the database is deployed. To configure these parameters, run the dbconf.sh script. See the “Configuring a Database” section on page 4-5.

Generating SQL Code Using the Velocity Template LanguageThe JDBC Adapter framework uses macros written in the Velocity Template Language (VTL) to generate all SQL code that is passed to the database server. The following sections describe the configuration file that controls the generation process.

For more information regarding VTL (which is a part of the Apache Jakarta Project) go to http://velocity.apache.org.

Table 6-1 describes VTL constructs.


Chapter 6 Configuring Databases Generating SQL Code Using the Velocity Template Language

Table 6-1 Summary of VTL Constructs

Directive Syntax Example Purpose

#foreach #foreach ($item in $collection) item is $item #end

Iterates over a collection, array, or map.

#if ... #else ... #elseif

#if ($order.total == 0) No charge #end

Conditional statement.

#parse #parse("header.vm") Loads, parses, and includes the specified template in the generated output.

#macro #macro(currency $amount) ${formatter.currency($amount)} #end

Defines a new directive and any required parameters, if any. The result is interpreted when used later in the template.

#include #include("disclaimer.txt") Includes the specified file, as is, in the generated output.

#set #set ($customer = ${order.customer}) Assigns a value to a context object. If the context object does not exist, it is added; otherwise, it is overwritten.

#stop #if ($debug) #stop #end Stops template processing.


OL-26830-04

Chapter 6 Configuring Databases Database Configuration Files

Database Configuration FilesWhen you initialize the database access framework, the first file that the database access framework searches for is main.vm, which contains definitions or pointers to all the required database SQL definitions. The location used to search for this file depends on the dbpack used in Collection Manager. A dbpack is a collection of configuration files pertaining to a specific database installation. The adapter (according to its configuration file) selects the dbpack. The following code fragment from the jdbcadapter.conf file configures the Collection Manager to work with an Oracle dbpack:

db_template_dir = dbpacks/oracle/9204e/ db_template_file = main.vm

Note The directory location is interpreted relative to the main Collection Manager configuration directory (~scmscm/cm/config).

To make the configuration more modular, the main.vm file generally points to other files; however, this mechanism is not strictly necessary. The files can contain arbitrary definitions that can be used later, for example, in scripts. Some definitions are mandatory because the JDBC adapter uses them for its operation. These definitions are listed in Table 6-2.

Some objects representing Collection Manager configuration in the VTL parsing context can be used in the templates. These objects are described in the following sections:

• Context Objects, page 6-4

• Application Configuration, page 6-5

Table 6-2 Mandatory VM Definitions

Object Name Mandatory Definition

$table.sql.dropTable

$table.sql.createTable

$table.sql.createIndexes

$table.sql.insert

$table.sql.metaDataQuery

For each table, these settings control how SQL is generated for the indicated operation

$dbinfo.driverjarfile

$dbinfo.driver

Location and class name for the JDBC driver

$dbinfo.cmdSeparator Pattern used to separate multiple SQL statements

$dbinfo.url

$dbinfo.connOptions

URL for connecting to the database Various connection properties


OL-26830-04

Chapter 6 Configuring Databases Context Objects

Context ObjectsBefore any Collection Manager components (for instance, a TA or JDBC adapter, or a script), load and parse the VM templates, Collection Manager initializes the parsing context with the following Java objects:

• tables

• dbinfo

• tools

The tables Object

The tables object describes application-related database configuration, such as the structure of RDRs that are stored in the database, the structure of the database tables and where they are stored, and the structure of any other database tables that Collection Manager uses. The object is an array in which each row represents one of the database tables used by Collection Manager. In each table, the row can contain the following information (note that not all items are relevant to all the tables):

• Logical name

• Physical name

• RDR tag associated with this table

• List of fields/columns in this table, with the following attributes for each:

– Field ID

– Field name

– Field native type

– Free-form field options

• List of indexes for this table, with the following attributes for each:

– Index name

– Names of columns indexed

– Free-form index options

The contents of the tables object can be inspected or manipulated when loading the templates. The tables object is initialized using the application-specific XML configuration file. See the “Application Configuration” section on page 6-5.

The dbinfo Object

The dbinfo object describes configuration that is specific to the database, such as the parameters and the SID or schema that is used when opening a database connection. The object holds database-specific configuration options. The object contains the following information:

• The name of the JDBC class that is used as a driver for this database

• The name of the JAR file that contains the driver

• The location of the database expressed as a JDBC URL

• Free-form JDBC connection options, such as authentication data (user and password)The


OL-26830-04

Chapter 6 Configuring Databases Application Configuration

The tools Object

The tools object is a container for several utility methods that you can find useful when developing templates or manipulating the context data structures.

Invoke the object methods by using the $tools.method(arg1, ..., argN), where method is the name of the method.

The methods of the tools object are listed in Table 6-3.

For an example that demonstrates how to use these tools, see the “Using the JDBC Framework in Scripts” section on page 6-10.

Application ConfigurationAll application-related configuration is contained in one file (dbtables.xml), which includes the following items:

• Name and version of the application

• Name and properties of each database table, and specifically, the structure of application RDRs that are stored in database tables

• In each database table:

– Names and native types of the table and RDR fields

– Names and properties of the table indexes

This information is used primarily to populate the tables object in the template parsing context. See the “The tables Object” section on page 6-4.

Table 6-3 Methods of the tools Object

Method Name and Arguments Function

getTableByName (allTables, name) Locates the database table object whose logical name corresponds to name.

getTableByDbTabName (allTables, name) Locates the database table object whose physical name corresponds to name.

assignParams (sql, list_of_args) Replaces question mark characters in the SQL string with consecutive elements from the list_of_args parameter, represented as a String. This method is useful if you are working with templates that create SQL insert statements using the JDBC PreparedStatement string as a base.

collapseWhitespace() Converts all instances of more than one consecutive white-space character to one space, and trims the beginning and ending white space. This method can be useful for databases that require SQL with a minimum number of newline and other white-space characters. (Sybase and Oracle do not require this conversion.)


OL-26830-04

Chapter 6 Configuring Databases Working Sample

Working SampleThe main.vm file can contain references to other velocity template (VM) files to support modularization (see the “Database Configuration Files” section on page 6-3). The names of these other files are arbitrary, except for the VM_global_library.vm file, which name is predetermined. Place macros that must be defined in this file to ensure that they are loaded at the right time. For details about this special file, see the Velocity User Guide.

The following sample illustrates the contents of main.vm for an Oracle setup:

#parse ('dbinfo.vm')

#foreach ($table in $tables) #set ($table.sql.dropTable = "#parse ('drop_table.vm')") #set ($table.sql.createTable = "#parse ('create_table.vm')") #set ($table.sql.createIndexes = "#parse ('create_indexes.vm')") #set ($table.sql.insert = "#parse ('insert.vm')") #set ($table.sql.metaDataQuery = "#parse ('metadata.vm')")#end

In this sample, the mandatory database and SQL definitions (see Table 6-2) are moved to separate files that are loaded and parsed using the #parse directive.

The following sections list the possible contents for the various files in the Oracle dbpack. Some of the definitions use macros that are defined in the VM_global_library.vm file. This file contains all macro definitions that any template uses.

• Macro Definitions, page 6-6

• dbinfo Configuration, page 6-7

• SQL Definitions, page 6-7

Macro DefinitionsThe following example shows definitions for the mapping between native types and SQL types, and utility macros, such as the optcomma macro, which inserts a comma between successive elements of lists:

#macro (optcomma)#if ($velocityCount >1),#end#end #macro (sqltype $field) #set ($maxStringLen = 2000) #if ($field.type == "INT8") integer #elseif ($field.type == "INT16") integer #elseif ($field.type == "INT32") integer #elseif ($field.type == "UINT8") integer #elseif ($field.type == "UINT16") integer #elseif ($field.type == "UINT32") integer #elseif ($field.type == "REAL") real #elseif ($field.type == "BOOLEAN") char(1) #elseif ($field.type == "STRING") varchar2(#if($field.size <= $maxStringLen)$field.size #else $maxStringLen #end) #elseif ($field.type == "TEXT") long #elseif ($field.type == "TIMESTAMP") date #end #end


OL-26830-04

Chapter 6 Configuring Databases dbinfo Configuration

dbinfo ConfigurationIn the following code sample, note that the only required fields are the URL and connection options (for authentication).

Blank lines in the code separate the code into distinct fields for readability and to ease later configuration changes.

#set ($dbinfo.driver = "oracle.jdbc.OracleDriver") #set ($dbinfo.driverjarfile = "ojdbc14.jar") #set ($dbinfo.options.host = "localhost") #set ($dbinfo.options.port = "1521") #set ($dbinfo.options.user = "pqb_admin") #set ($dbinfo.options.password = "pqb_admin") #set ($dbinfo.options.sid = "apricot") #set ($dbinfo.url = "jdbc:oracle:thin:@$dbinfo.options.host:$dbinfo.options.port:$dbinfo.options.sid") #set ($dbinfo.connOptions.user = $dbinfo.options.user) #set ($dbinfo.connOptions.password = $dbinfo.options.password) ## the vendor-specific piece of SQL that will return the current ## date and time: #set ($dbinfo.options.getdate = "sysdate")

SQL DefinitionsThe following sections provide SQL definitions:

• Code for Dropping a Table, page 6-7

• Code for Creating a Table, page 6-7

• Code for Creating Indexes, page 6-8

• Code for Insert, page 6-8

• Code for Metadata Query, page 6-8

Code for Dropping a Table

The following code sample shows how to drop a table using normal SQL syntax:

drop table $table.dbtabname

Code for Creating a Table

The following code sample shows how to create a table using normal SQL syntax. Use this definition for customized database configurations that require special directives to create a table. For example, you can modify the configuration to create the table in a unique tablespace or use table partitioning.

create table $table.dbtabname (#foreach ($field in $table.fields) #optcomma()$field.name #sqltype($field) #if ("$!field.options.notnull" == "true") not null #end#end)


OL-26830-04

Chapter 6 Configuring Databases SQL Definitions

Code for Creating Indexes

The following code shows how to create an index using normal SQL syntax. Use this definition to implement a customized database configuration that requires special directives to create an index.

#foreach ($index in $table.indexes) create index $index.name on $table.dbtabname ($index.columns)#end

Code for Insert

The following code shows how to create a JDBC PreparedStatement that corresponds to a table’s structure:

insert into ${table.dbtabname} (#foreach ($field in $table.fields) #optcomma()${field.name}#end)values (#foreach ($field in $table.fields) #optcomma()?#end)

Code for Metadata Query

The following code shows how to define a simple query that gets the table metadata (column names and types). You can use any query that returns an empty result set.

select * from ${table.dbtabname} where 1=0


OL-26830-04

Chapter 6 Configuring Databases Testing and Debugging

Testing and DebuggingAs you develop a set of templates for your database, you can see the results of parsing. To invoke such parsing, the JDBC adapter supports direct invocation using ~scmscm/cm/bin/cm, the Collection Manager’s main script.

The syntax for invocation is:

~/cm/bin/cm invoke com.cisco.scmscm.adapters.jdbc.JDBCAdapter argument

Here, the argument is one of the parameters described in the following sections. You can use this mechanism whether Collection Manager is running or not.

Additionally, you can use the query and update execution methods described in the following sections to test the template results with a live database:

• Parsing a String, page 6-9

• Obtaining Complete Debug Information, page 6-9

Parsing a StringYou can parse any string as a Velocity Template Language (VTL) template, with the complete context in place. The result of parsing is displayed in the standard output. To parse a string, call the adapter with the -parse parameter. The following sample code shows a parsing mechanism (responses are shown in bold):

$ ~/cm/bin/cm invoke com.cisco.scmscm.adapters.jdbc.JDBCAdapter -parse 'xxx'xxx $ ~/cm/bin/cm invoke com.cisco.scmscm.adapters.jdbc.JDBCAdapter -parse '$dbinfo.url'jdbc:oracle:thin:@localhost:1521:apricot $ ~/cm/bin/cm invoke com.cisco.scmscm.adapters.jdbc.JDBCAdapter -parse '$tools.getTableByName($tables, "LUR").sql.createTable'create table RPT_LUR (TIME_STAMP date

,RECORD_SOURCE integer,LINK_ID integer,GENERATOR_ID integer,GLBL_USG_CNT_ID integer,CONFIGURED_DURATION integer,DURATION integer,END_TIME integer,UPSTREAM_VOLUME integer,DOWNSTREAM_VOLUME integer,SESSIONS integer,SECONDS integer,CONCURRENT_SESSIONS integer,ACTIVE_SUBSCRIBERS integer,TOTAL_ACTIVE_SUBSCRIBERS integer,IP_TYPE integer

Obtaining Complete Debug InformationTo see a dump of all the contents of the tables and dbinfo structures as created by the templates, use the -debug parameter. The -debug parameter displays, in standard output format, a detailed view of all the fields, properties, and options of these structures.


OL-26830-04

Chapter 6 Configuring Databases Using the JDBC Framework in Scripts

Using the JDBC Framework in ScriptsYou can send arbitrary SQL commands to the database for execution, and view the resulting data. Viewing the results helps you to perform periodic database maintenance, monitor the contents of database tables, and manage extra database tables.

To perform an update operation, call the adapter using the -executeUpdate parameter. To perform a query and view the results, call the adapter using the -executeQuery parameter.

Viewing and Setting a Cisco SCE Time Zone OffsetThe following sample of an update operation demonstrates how to change programmatically, the value in a database table that contains the Cisco SCE time zone offset setting. The name of this table is usually JCONF_SE_TZ_OFFSET. However, because a table can be assigned another name, it is referred to here by its logical name TZ. See the listings in the “The dbtables.xml File” section on page A-2.

To avoid the need to first check if the table exists, and then update it, the table is dropped (ignoring the error status if it does not exist), and then re-created. The proper values should then be inserted. Since the table contains a timestamp column, get the current date from the database. This operation is specific to each database vendor; therefore, the following example uses the preconfigured getdate operation that is defined in the templates.

Note the use of the assignParams and getTableByName tools to generate the SQL:

#! /bin/bash

this=$0tableName=TZ

usage () { cat <<EOFUsage: $this --status - show currently configured TZ offset $this --offset=N - set the offset to N minutes (-1440 <= N <= 1440) $this --help - print this messageEOF}

query () { ~/cm/bin/cm invoke com.cisco.scmscm.adapters.jdbc.JDBCAdapter -executeQuery "$*"}

update () { ~/cm/bin/cm invoke com.cisco.scmscm.adapters.jdbc.JDBCAdapter -executeUpdate "$*"}

get_tz () { query 'select * from $tools.getTableByName($tables, "TZ").dbtabname'}

set_tz () { update '$tools.getTableByName($tables, "TZ").sql.dropTable' update '$tools.getTableByName($tables, "TZ").sql.createTable' update '$tools.assignParams($tools.getTableByName($tables, "TZ").sql.insert, [$dbinfo.options.getdate, '$1'])'}

case $1 in --status)


OL-26830-04

Chapter 6 Configuring Databases Viewing and Setting a Cisco SCE Time Zone Offset

get_tz ;; --help) usage exit 0 ;; --offset=*) n=$(echo $1 | egrep 'offset=[-]?[0-9]+$' | sed 's/.*=//') if [ "$n" ]; then if [ "$n" -ge -1440 -a "$n" -le 1440 ]; then set_tz $n &>/dev/null ok=1 fi fi if [ ! "$ok" ]; then usage exit 2 fi get_tz ;; *) usage exit 3 ;;esac

When an executed query returns a result set, the result set is displayed as standard output in tabular format with appropriate column headers.


OL-26830-04

Chapter 6 Configuring Databases Scalability Hints for Oracle

Scalability Hints for OracleThe following sections demonstrate ways in which to make database handling more scalable for Collection Manager. These methods are specific to Oracle, and are provided to illustrate the possibilities.

• Using Custom Tablespaces, page 6-12

• Using Table Partitioning, page 6-12

Using Custom TablespacesIf you create several tablespaces and want to distribute Collection Manager tables among them, specify the tablespace to be used for each table, in the dbtables.xml file. The following sample code represents the definition for a table (note that the code is in bold):

This sample adds the required tablespaces (tspace1 and tspace2) for the index and the table. There is no preconfigured meaning to the tablespace option in Collection Manager. Any new option name can be used. An option name’s meaning is derived from its subsequent use in the templates.

To create a table in the correct tablespace, modify the create_table.vm code as follows:

create table $table.dbtabname (#foreach ($field in $table.fields) #optcomma()$field.name #sqltype($field) #if ("$!field.options.notnull" == "true") not null #end#end)#if ("$!table.options.tablespace" != "") TABLESPACE $table.options.tablespace#end

To create an index in its own tablespace, modify the create_indexes.vm code as follows:

#foreach ($index in $table.indexes) create index $index.name on $table.dbtabname ($index.columns)#if ("$!index .options.tablespace" != "") TABLESPACE $index.options.tablespace #end#end

Using Table PartitioningTo implement rolling partitioning for a particular table on a weekly basis, create a partitioned option for the table in the dbtables.xml file in a manner similar to the example provided in the previous section (see the “Using Custom Tablespaces” section on page 6-12). After this, augment the create_table.vm code as follows (note that the code is in bold ):

create table $table.dbtabname ( #foreach ($field in $table.fields) #optcomma()$field.name #sqltype($field) #if ("$!field.options.notnull" == "true") not null #end #end)#if ("$!table.options.partitioned" != "") partition by range (timestamp)


OL-26830-04

Chapter 6 Configuring Databases Using Table Partitioning

(partition week_1 values less than (to_date ('01-JAN-2005 00:00:00','DD-MON-YYYY HH24:MI:SS')), partition week_2 values less than (to_date ('08-JAN-2005 00:00:00','DD-MON-YYYY HH24:MI:SS')) partition week_3 values less than (to_date ('15-JAN-2005 00:00:00','DD-MON-YYYY HH24:MI:SS')) partition week_4 values less than (to_date ('22-JAN-2005 00:00:00','DD-MON-YYYY HH24:MI:SS')) ); #end

Because Oracle does not accept nonconstant expression for time boundaries, the values must be hardwired for the time the tables are created.

Create a cron job to roll the partitions (delete an old partition and create a new one) on a weekly basis. This cron job runs a script that calls the command-line interface of the JDBC Adapter (as explained in the “Using the JDBC Framework in Scripts” section on page 6-10) to issue the appropriate alter table drop partition and alter table add partition SQL commands.


OL-26830-04

Chapter 6 Configuring Databases Using Table Partitioning


OL-26830-04

OL-26830-04

A
P P E N D I X A Code Samples
This appendix contains samples of files used to configure the Cisco Collection Manager.

The following sections includes file samples for the database table configuration file, the DTD file, the RAG adapter configuration file, and the custom adapter configuration file:

• Application Configuration, page A-2

• RAG Adapter Configuration, page A-5

• Custom Adapter Configuration, page A-10

A-1Cisco Collection Manager User Guide

Appendix A Code Samples Application Configuration

Application ConfigurationThe following sections shows part of the XML file dbtables.xml used to configure the database tables, and the DTD file used to verify the structure of the XML file.

• The dbtables.xml File, page A-2

• The tables.dtd File, page A-3

The dbtables.xml FileThe following code is a section of the Cisco SCA BB dbtables.xml file:

<?xml version="1.0" encoding="ISO8859_1"?><!DOCTYPE dbtabconf PUBLIC "-//P-Cube//Engage DB RDR Configuration 2.1.0//EN" "dbtables.dtd"><dbtabconf> <fileversion> ... </fileversion> <application name="Engage" version="2.1"/> <dbtables>

<rdr name="SUR" dbtabname="RPT_SUR" tag="4042321922" createtable="true"> <fields> <field id="1" name="TIME_STAMP" type="TIMESTAMP"> <options> <option property="source" value="timestamp"/> </options> </field> <field id="2" name="RECORD_SOURCE" type="INT32"> <options> <option property="source" value="recordsource"/> </options> </field> <field id="3" name="SUBSCRIBER_ID" type="STRING" size="64"/> <field id="4" name="PACKAGE_ID" type="INT16"/> <field id="5" name="SUBS_USG_CNT_ID" type="UINT16"/> <field id="6" name="MONITORED_OBJECT_ID" type="INT16"/> <field id="7" name="BREACH_STATE" type="UINT8"/> <field id="8" name="REASON" type="UINT8"/> <field id="9" name="CONFIGURED_DURATION" type="UINT32"/> <field id="10" name="DURATION" type="UINT32"/> <field id="11" name="END_TIME" type="INT32"/> <field id="12" name="UPSTREAM_VOLUME" type="INT32"/> <field id="13" name="DOWNSTREAM_VOLUME" type="INT32"/> <field id="14" name="SESSIONS" type="UINT16"/> <field id="15" name="SECONDS" type="UINT16"/> <field id="16" name="OS_FINGER_PRINTING" type="STRING" size="64"/>  <field id="17" name="IP_TYPE" type="UINT8"/> </fields> <indexes> <index name="RPT_SUR_I1" columns="END_TIME"> <options> <option property="clustered" value="true"/> <option property="allowduprow" value="true"/> </options> </index> </indexes> </rdr><rdr name="LUR" dbtabname="RPT_LUR" tag="4042321925" createtable="true">


OL-26830-04

Appendix A Code Samples The tables.dtd File

<options> <option property="tablespace" value="tspace1"/> </options> <fields> <field id="1" name="TIME_STAMP" type="TIMESTAMP">  <field id="10" name="DOWNSTREAM_VOLUME" type="UINT32"/> <field id="11" name="SESSIONS" type="UINT32"/> </fields> <indexes> <index name="RPT_LUR_I1" columns="END_TIME"> <options> <option property="clustered" value="true"/> <option property="allowduprow" value="true"/> <option property="tablespace" value="tspace2"/> </options> </index> </indexes></rdr><aggtable name="TOP_HOURLY" dbtabname="RPT_TOPS_PERIOD0" aggperiod="0"> <fields> <field id="1" name="RECORD_SOURCE" type="INT32"/> <field id="2" name="METRIC_ID" type="INT8"/> <field id="3" name="SUBS_USG_CNT_ID" type="INT8"/> <field id="4" name="TIME_STAMP" type="TIMESTAMP"/> <field id="5" name="AGG_PERIOD" type="INT8"/> <field id="6" name="SUBSCRIBER_ID" type="STRING" size="64"/> <field id="7" name="CONSUMPTION" type="UINT32"/> <field id="8" name="PACKAGE_ID" type="INT16"/>  <field id="9" name="IP_TYPE" type="INT8"/> </fields> <indexes> <index name="RPT_TOPS_PERIOD0_I1" columns="TIME_STAMP"> <options> <option property="clustered" value="true"/> <option property="allowduprow" value="true"/> </options> </index> </indexes> </aggtable>

For each table (either an RDR table, an aggregation table, or an extra table), the fields, indexes, and so on, are listed.

Note A table, an index, or fields can have arbitrary free text options that can be accessed in the templates.

The XML file is verified at runtime against a simple DTD, which is reproduced in the following section.

The tables.dtd FileThe following code represents the DTD file used to verify the dbtables.xml definition file:

<?xml version="1.0" encoding="ISO8859_1"?>

<!ELEMENT dbtabconf (fileversion, application, db?, dbtables)><!ELEMENT fileversion (#PCDATA)><!ELEMENT application EMPTY><!ATTLIST application name CDATA #REQUIRED


OL-26830-04

Appendix A Code Samples The tables.dtd File

version CDATA #REQUIRED><!ELEMENT db (options)><!ELEMENT dbtables (rdr*, aggtable*, table*)><!ELEMENT table (options?, fields, indexes?)><!ATTLIST table name CDATA #REQUIRED dbtabname CDATA #REQUIRED createtable (true | false) "true" inserttodb (true | false) "false"><!ELEMENT aggtable (options?, fields, indexes?)><!ATTLIST aggtable name CDATA #REQUIRED dbtabname CDATA #REQUIRED aggperiod CDATA #REQUIRED createtable (true | false) "true"><!ELEMENT rdr (options?, fields, indexes?)><!ATTLIST rdr name CDATA #REQUIRED dbtabname CDATA #REQUIRED tag CDATA #REQUIRED createtable (true | false) "true" inserttodb (true | false) "true"><!ELEMENT fields (field+)><!ELEMENT field (options?)><!ATTLIST field id CDATA #REQUIRED name CDATA #REQUIRED type CDATA #REQUIRED size CDATA #IMPLIED><!ELEMENT indexes (index+)><!ELEMENT index (options?)><!ATTLIST index name CDATA #REQUIRED columns CDATA #REQUIRED create (true | false) "true"><!ELEMENT options (option+)><!ELEMENT option EMPTY><!ATTLIST option property CDATA #REQUIRED value CDATA #REQUIRED>

You can set the location and name of the DTD and XML files separately for each adapter, in the adapter configuration file.


OL-26830-04

Appendix A Code Samples RAG Adapter Configuration

RAG Adapter ConfigurationThe following sections shows the configuration file (ragadapter.conf). It also shows the associated XML files used to configure the RAG adapter.

The configuration files of the other adapters are similar to the RAG adapter configuration file. Note that only the RAG adapter has associated XML files.

• The ragadapter.conf File, page A-5

• ragadapter.xml File, page A-6

• The http_TURs.xml File, page A-6

• The video_TURs.xml File, page A-7

• The vlink_BW_per_pkg.xml File, page A-8

• The vsa_SURs.xml File, page A-8

• The cdma_SURs.xml file, page A-9

The ragadapter.conf FileTo perform general maintenance of the RAG adapter, use the ~scmscm/cm/config/ragadapter.conf file. The following example represents the RAG adapter configuration file:

# # RAGAdapter main configuration file # [config] xml_conf_dir = ~/cm/config/ragadapterdomain_ext_filename = ~/cm/config/tlds-alpha-by-domain.txtnetwork_type_filename = ~/cm/config/rat-networktype.txtimei_lookup = falsevsa_type=gsmmeid_length=8attr_index=17attr_shift_pos=0

[housekeeper]interval_sec = 10

[db]operations_timeout = 60batch_size = 10transaction_size = 15commit_interval = 6blocking_connects = truedb_template_file = main.vmdb_template_dir = dbpacks/sybase/ase12.5.1#db_template_dir = dbpacks/sybase/ase12.5.1#db_template_dir = dbpacks/oracle/9204e#db_template_dir = dbpacks/mysql/4.0.20

[app]app_conf_file = dbtables.xmlapp_dtd_file = dbtables.dtdapp_conf_dir = apps/scasbb/latest

[bench]rate_period_msec = 5000


OL-26830-04

Appendix A Code Samples ragadapter.xml File

ragadapter.xml FileThe following ragadapter.xml file sample defines the sinks to persist the aggregated data. The sink section specifies the format of the aggregated data store and location to which the data is written (a CSV file or the database).

The CSV file definition is located in the <sinks> section under the <csvsink id="csv1"> tag. This definition specifies the output file name and directory. It also specifies the rate at which new files are opened, either every 5 minutes, or when the current file reaches 1000 lines. Each field in the file is enclosed by quotes (").

The database definition is located in the <sinks> section under the <dbsink id="dbsink1"> tag. If the sink is configured as a database, the aggregated data is written to the database configured in Collection Manager.

<?xml version="1.0" standalone="no"?><ragadapterconf> <description>RAG Adapter basic configuration</description> <config> <aggregations/> <sinks> <csvsink id="csv1" classname="com.cisco.scmscm.adapters.rag.sinks.CSVSink" filenameformat="yyyy-MM-dd_HH-mm-ss-SSS'.csv'" dirname="~/cm/adapters/RAGAdapter/csvfiles" maxagesec="300" maxlines="1000" usequotes="true" active="false"/> <dbsink id="dbsink1" classname="com.cisco.scmscm.adapters.rag.sinks.JDBCSink" active="true"/> </sinks> </config></ragadapterconf>

The http_TURs.xml FileThe following sample represents a portion of the http_TURs.xml file, which defines aggregations for the incoming HTTP Transaction Usage RDRs (intag="4042323004").

This portion of the http_TURs.xml file aggregates RDRs with the RDR tag 71072 (outtag="71072") and persists them into the database (sinkid="dbsink1").

The buckets aggregate the data using three HTTP_TUR fields: 7, 18, and 19. These fields correspond to ACCESS_STRING, SERVICE_COUNTER_ID, and PACKAGE_COUNTER_ID. The bucket identifiers are similar to the fields appearing in the SQL “GROUP BY” clause. The database definition is located in the <sinks> section under the <dbsink id="dbsink1"> tag. The outtag table definition is located in the dbtables.xml tables configuration file.

<aggregation id="HTTP TURs by domain, per service, per package" intag="4042323004" outtag="71072" sinkid="dbsink1" period="60" top="500" topid="1" topby="4,5,6,8"> 

<bucketident><field inindex="7" type="string" index="1"

pretransform="com.cisco.scmscm.adapters.rag.transforms.URLToDomain"/> 

<field inindex="18" type="int" index="2"/> <field inindex="19" type="int" index="3"/> 

<totals> <total set="1"/> <total set="2"/>


OL-26830-04

Appendix A Code Samples The video_TURs.xml File

<total set="3"/> <total set="1,2"/> <total set="1,3"/> <total set="2,3"/> <total set="1,2,3"/> </totals>

</bucketident><closures/>

<accumulators><field inindex="0" type="int" kind="agg_period" index="0"/><field inindex="0" type="long" kind="count" index="4"/> <field inindex="15" type="long" index="5"/> <field inindex="16" type="long" index="6"/> <field inindex="13" type="long" index="7" unitsFactor="1000"/> <field inindex="0" type="long" kind="uniq_count" index="8"/> <field inindex="6" type="long" kind="rank" index="9"/><field inindex="4" type="long" kind="rank" index="10"/> <field inindex="8" type="long" kind="rank" index="11"/>

</accumulators><monitors/></aggregation>

The video_TURs.xml FileThe following sample represents a section of the video_TURs.xml file, which defines aggregations for the incoming VIDEO Transaction Usage RDRs (intag="4042323072"):

<aggregation id="Video TURs by domain, per service, per package" intag="4042323072" outtag="71074" sinkid="dbsink1" period="60" top="500" topid="1" topby="4,5,6,8"> 


pretransform="com.cisco.scmscm.adapters.rag.transforms.URLToDomain"/> 

<field inindex="18" type="int" index="2"/> <field inindex="19" type="int" index="3"/> 

<totals> <total set="1"/> <total set="2"/> <total set="3"/> <total set="1,2"/> <total set="1,3"/> <total set="2,3"/> <total set="1,2,3"/> </totals>

</bucketident><closures/>

<accumulators><field inindex="0" type="int" kind="agg_period" index="0"/> <field inindex="0" type="long" kind="count" index="4"/> <field inindex="15" type="long" index="5"/> <field inindex="16" type="long" index="6"/> <field inindex="13" type="long" index="7" unitsFactor="1000"/><field inindex="0" type="long" kind="uniq_count" index="8"/> <field inindex="6" type="long" kind="rank" index="9"/><field inindex="4" type="long" kind="rank" index="10"/> <field inindex="8" type="long" kind="rank" index="11"/>

</accumulators><monitors/>

</aggregation>


OL-26830-04

Appendix A Code Samples The vlink_BW_per_pkg.xml File

The vlink_BW_per_pkg.xml File The following sample represents a section of the vlink_BW_per_pkg.xml file, which defines aggregations for the uplink and downlink fields in the Subscriber Usage RDRs (intag="4042321920"):

<aggregation id="NUR's by VLINK UPSTREAM and subs usage counter" intag="4042321920" outtag="71080" sinkid="dbsink1">

<bucketident><field index="1" type="string"/><field index="12" type="int"/> 

</bucketident><closures/><accumulators>

<field index="6" type="long"/><field index="8" type="long"/><field index="9" type="long"/><field index="10" type="long"/><field index="11" type="long"/>

</accumulators><monitors>

<timeoutmonitor action="checkpoint" maxsec="900" active="true"/></monitors>

</aggregation>

The vsa_SURs.xml File The following sample represents a section of the vsa_SURs.xml file, which defines aggregations for the VSA fields present in the Subscriber Usage RDRs (intag="4042321920"):

<aggregation id="VSA SURs by device type, per service"intag="4042321920" outtag="71076" sinkid="dbsink1" period="60" top="500"topid="1" topby="3,4,5"> 


pretransform="com.cisco.scmscm.adapters.rag.transforms.IMEIToDeviceType" /><field inindex="2" type="int" index="2" /> 

<totals><total set="1" /><total set="2" /><total set="1,2" />

</totals></bucketident><closures /><accumulators>

<field inindex="0" type="int" kind="agg_period" index="0" /> <field inindex="8" type="long" index="3" /><field inindex="9" type="long" index="4" /> <field inindex="0" type="long" kind="uniq_count" index="5" /> <field inindex="4" type="long" kind="rank" index="6" /> <field inindex="5" type="long" kind="rank" index="7" />

</accumulators><monitors />

</aggregation>


OL-26830-04

Appendix A Code Samples The cdma_SURs.xml file

The cdma_SURs.xml file The following sample represents a section of the Cdma_SURs.xml file, which defines aggregations for the VSA fields present in the Subscriber Usage RDRs (intag="4042321920"):

<aggregations><aggregation id="CDMA SURs by device type, per service"

intag="4042321920" outtag="71082" sinkid="dbsink1" period="60" top="-1"topid="1" topby="3,4,5"> <bucketident>

<field inindex="26" type="string" index="1"

pretransform="com.cisco.scmscm.adapters.rag.transforms.MEIDToDeviceType" /> 

<field inindex="2" type="int" index="2" /> 

<totals><total set="1" /><total set="2" /><total set="1,2" />

</totals></bucketident><closures /><accumulators>

<field inindex="0" type="int" kind="agg_period" index="0" /> 

<field inindex="8" type="long" index="3" /> 

<field inindex="9" type="long" index="4" /> 

<field inindex="0" type="long" kind="uniq_count" index="5" /> 

<field inindex="4" type="long" kind="rank" index="6" /> 

<field inindex="5" type="long" kind="rank" index="7" /> 

</accumulators><monitors />

</aggregation>


OL-26830-04

Appendix A Code Samples Custom Adapter Configuration

Custom Adapter ConfigurationThe Custom Adapter allows users to configure specific fields in a CSV file for a particular RDR. The ~scmscm/cm/config/customadapter/custom.xml file contains this configuration:

<customadapter><description>Customized CSV Adapter</description><config> <rdrs> <rdr tag="4042321925" sinkId="csv1"> <fields> <field outindex="0" name="RDR_TAG" type="STRING" constant="4042321925" /> <field outindex="1" name="LINK_ID" type="STRING" rdrindex="0" /> <field outindex="2" name="GEN_ID" type="STRING" rdrindex="1" /> <field outindex="3" name="SVC_ID" type="STRING" rdrindex="2" /> <field outindex="4" name="UPSTREAM_VOL" type="LONG" rdrindex="6" /> <field outindex="5" name="DOWNSTREAM_VOL" type="LONG" rdrindex="7" /> <field outindex="6" name="TOTAL_VOL" type="LONG" expr="(F6+F7)" />  </fields> </rdr></rdrs><sinks> <csvsink id="csv1" classname="com.cisco.scmscm.adapters.custom.sinks.CSVSink" filenameformat="yyyy-MM-dd_HH-mm-ss-SSS'.csv'" dirname="~/cm/adapters/CustomAdapter/csvfiles/4042321925" maxlines="1000" active="true" /> </sinks></config></customadapter>

Note These configured RDR TAG IDs should be added in the ~scmscm/cm/config/queue.conf file under the [bill-custom] section.

The sink information should also be configured in the ~scmscm/cm/config/customadapter/custom.xml file for the Custom Adapter to store the customized RDR records. The sink IDs should be unique for each RDR.

Table A-1 defines the fields in the custom.xml file.

Table A-1 custom.xml File Fields

Fields Definition

outindex Refers to the column index of the csv file.

name Column name of the csv file.

type Refers to the Data Type for the corresponding column

rdrindex Refers to the rdr fields that need to be processed.

expr Performs Arithmetic Expressions between RDR Fields and with constant values.

(ex:- expr "(F6+F7)"à Perform Addition operation for the rdr field 6 and rdr Field 7 and store the result in the corresponding outindex column


OL-26830-04


expr ” (F0+FC)” à Add rdr Field 0 with a constant value, FC refers to the constant value that needs to be added).

constant Stores the constant value in the corresponding outindex column.

Table A-1 custom.xml File Fields

Fields Definition


OL-26830-04



OL-26830-04

OL-26830-04

A
P P E N D I X B Configuration File Descriptions
This appendix describes each of the configuration files used by Collection Manager along with the configuration parameters in each file.

Each configuration file is divided into sections. A heading identifies each section, for example, [config].

You can comment out any parameter that is not required in your configuration by placing a # at the start of the line.

If you modify the configuration file, restart Collection Manager to enable the changes to take effect.

Caution The configuration files supplied with Collection Manager software installation contain tested valid settings. You are not expected to change the settings. Use care when modifying the settings.

B-1Cisco Collection Manager User Guide

Appendix B Configuration File Descriptions General Configuration Files

General Configuration FilesThis section describes the cm.conf and queue.conf configuration files.

• Collection Manager Configuration, page B-2

• Queue Configuration, page B-6

Collection Manager ConfigurationThis section describes the cm.conf Collection Manager configuration file:

• The [adapter] Section, page B-2

• The [adapter_mem] Section, page B-3

• The [adapter_start_timeout] Section, page B-3

• The [buffer] Section, page B-3

• The [queue] Section, page B-4

• The [categorizer] Section, page B-4

• The [server] Section, page B-5

• The [collector] Section, page B-5

• The [csv_adapter] Section, page B-5

The [adapter] Section

The [adapter] section contains the following parameters:

• automatic_start

Defines whether starting or resetting Collection Manager starts the adapters automatically.

Possible values for this parameter are true and false. The default value is true.

• port

Defines the adapter manager server port number. Possible values for this parameter are 1024-65535. The default value is 33001.

• timeout

Defines the timeout value, in seconds, for client adapters. Setting this parameter to 0 disables the timeout.

• connections

Defines the maximum number of client connections per adapter.

Possible values for the parameter are 2 to 100. The default value is 20.

• directory

Defines the location to which you want to save buffered RDR data. You can use the ~/ prefix for this parameter. The default value is ~/cm/adapters.


OL-26830-04

Appendix B Configuration File Descriptions Collection Manager Configuration

• adapter<number>

Defines the full class name of the adapter, where <number> is a unique ID number. The adapter definitions must match the adapter_id parameters of the queues defined in the queue configuration file. See the “Queue Configuration” section on page B-6.

• parestart_start

Defines the time (in 24-hour format) at which to restart the adapters periodically. A sample value for this parameter is 23:30. The default value is 03:45.

• parestart_interval

Defines the interval, in minutes, between periodic adapter restarts. Possible values for this parameter are 0 to 2147483647 (0 to 231 - 1), where 0 disables this functionality. The default value for this parameter is 10,080 minutes (18 hours).

The [adapter_mem] Section

The [adapter_mem] section contains the following parameter:

• <Adapter class name>

Defines the maximum heap size allocated to an adapter. The following is an example shows how to set this parameter:

com.cisco.scmscm.adapters.topper.TAAdapter=-Xmx512M

To run with Solaris JRE 64 bit, set the -d64 parameter.

com.cisco.scmscm.adapters.topper.TAAdapter= -d64 –Xmx4096M

If the necessary heap size is different from the JVM default maximum heap size, include a definition of the memory allocated for an adapter.

Note Use the value obtained from the TA Adapter memory calculation formula to arrive at the memory size. For more details, refer to the “TA Adapter Memory Requirements” section on page 2-8.

Note The maximum value of this parameter is 2 GB for an installation on a Linux platform and 3.5 GB for an installation on a Solaris platform. When running on Solaris JRE 64 bit, you can set higher values for this parameter.

The [adapter_start_timeout] Section

The [adapter_start_timeout] section contains the following parameter:

• <Adapter class name>

Defines the number of seconds that Collection Manager waits for a specific adapter to start. The default value is 200 seconds. The following example shows how to set this parameter:

com.cisco.scmscm.adapters.topper.TAAdapter=600

The [buffer] Section

The [buffer] section contains the following parameters:


OL-26830-04


• size

Defines the minimum number of accumulated bytes in the queue before a buffer dequeue.

Possible values for this parameter are 1 to 262,144. The default value is 128,000.

• time

Defines the minimum number of milliseconds between buffer dequeues.


• file_limit

Defines the maximum number of files to store in each adapter queue.


The [queue] Section

The [queue] section contains the following parameters:

• upper_hysteresis

Defines the upper hysteresis level as a fraction of the total queue size.

Possible values for this parameter are in the range 0 to 1. The default value is 0.11.

• lower_hysteresis

Defines the lower hysteresis level as a fraction of the total queue size.

Possible values for this parameter are in the range 0-1. The default value is 0.12.

• config

Defines the queue configuration file. The file name defined for this parameter is the name of the queue configuration file described in the “Queue Configuration” section on page B-6. The default value is queue.conf.

The [categorizer] Section

The [categorizer] section contains the following parameters:

• use_table

Defines whether to categorize RDRs by using the table or by decoding the tag bits.


• rate_period_msec

Defines the number of milliseconds between updates of incoming RDR rate calculation.

Possible values for this parameter are 5,000 to 30,000. The default value is 5,000.


OL-26830-04


The [server] Section

The [server] section contains the following parameters:

• port

Defines the RDR server port number.


• connections

Defines the maximum number of client connections.

Possible values for this parameter are 1 to 50. The default value is 50.

• timeout

Defines the client timeout, in seconds. The default value is 600.

• allow_multi

Defines whether to allow concurrent connections from the same IP.

Possible values for this parameter are true and false. The default value is false.

The [collector] Section

The [collector] section contains the following parameters:

• notification_level

Defines the length of time, in milliseconds, between successive UM notifications.

Possible values for this parameter are 0 to 60,000. The default value is 10.

• log_directory

Defines the directory in which to store log files. You can use the ~/ prefix for this parameter. The default value is ~/cm/logs.

The [csv_adapter] Section

The [csv_adapter] section contains the following parameter:

• csvdir

Defines the parent directory for CSV files. The default value for this parameter is ~scmscm/cm/adapters/CSVAdapter/csvfiles.

• maxCSVFiles

Defines the maximum number of entries in a CSV file. When the count reaches this number, the counting wraps back to 1. Possible values for this parameter are 10 to 10,000. The default value is 9,999.

• rdrLimit

Defines the maximum number of RDRs that can be written to a single file. Possible values for this parameter are 1 to 20,000. The default value is 1,000.


OL-26830-04

Appendix B Configuration File Descriptions Queue Configuration

Queue ConfigurationThis section describes the queue configuration file. The name of the queue configuration file is provided in the [queue] section of the cm.conf configuration file, as described in the “The [queue] Section” section on page B-4.

Each section of the queue configuration file defines a queue. The section names and queue names are all user defined. Each queue section has the following parameters:

• adapter_id

Defines a unique adapter ID number. This number must match the adapters defined in the adapter.<number> parameter in the [adapter] section of the cm.conf configuration file. See the “The [adapter] Section” section on page B-2.

• priority

Defines the priority level of the queue. Possible values for this parameter are 1 to 3. This parameter is deprecated. Do not change the parameter.

• warning_size

Defines the number of RDRs that can be present in the queue before a warning is sent.

• maximum_size

Defines the maximum size of the queue.

• tags

Defines a list of RDR tag numbers sent to the queue by the categorizer. The same tag can be sent to multiple queues.

The following is an example of a queue definition in the queue configuration file:

# Topper/Aggregator Adapter[topper-hi]adapter_id=3priority=3warning_size=40000maximum_size=50000tags=4042321920


OL-26830-04

Appendix B Configuration File Descriptions Adapter Configuration Files

Adapter Configuration FilesThis section describes the configuration files for all the possible adapters.

• Topper/Aggregator (TA) Adapter, page B-7

• The CSV Adapter Configuration File, page B-13

• The JDBC Adapter Configuration File, page B-14

• RAG Adapter Configuration File, page B-15

• Comma-Separated Value Adapter, page B-17

Topper/Aggregator (TA) AdapterThis section describes the taadapter.conf TA Adapter configuration file .

• [config] Section, page B-7

• The [rdr] Section, page B-9

• The [csv] Section, page B-10

• The [state save] Section, page B-10

• The [db] Section, page B-10

• The [app] Section, page B-12

• The [peak_hours] Section, page B-12

[config] Section

The [config] section contains the following parameters:

• all_ip_types_index=

all_ip_types_index=-1 #index in DB/CSV marking "all subscribers"

• all_services_index

Provides an index that marks the all subscribers entry in the database or CSV files. The default value is –1. Do not change this parameter.

• agg_lengths

Defines the length, in minutes, of each aggregation cycle. Each cycle must be divisible by the preceding one. If the cycle is not divisible by the preceding one, it is rounded on startup.

• agg_pkg_level

Defines whether package-based aggregation is enabled. The default value is false.

• avg_pkgs_per_subsc

Defines the average number of packages per subscriber. The default value is 2.

• all_packages_index

Provides an index that marks the "all packages" entry in the database or CSV files. The default value is –1.

• ip_types


OL-26830-04

Appendix B Configuration File Descriptions Topper/Aggregator (TA) Adapter

Defines the IP version as either IPv4 or IPv6:

ip_types=0,1 #Supported IP types 0 - IPv4, 1 - IPv6

• num_packages

Defines the maximum number of packages per subscriber. The default value is 5000.

• num_services

Defines the maximum number of recognized services (including service 0). The default value is 32.

• num_top_entries

Defines how many entries that correspond to the agg_lengths parameter are reported. The default value is 50,500.

• time_base

Defines the phase (time skew) of the aggregation cycles. The default value is 00:00.

Note Do not change the all_packages_index, all_service_index, and all_ip_type index parameters.


OL-26830-04


The [rdr] Section

The [rdr] section contains the following parameters:

• divide_1024

Defines whether to divide all consumptions by 1024.


• table_name

Specifies the logical name of the NUR table as it appears in the dbtables.xml file. The default value is NUR.

• field_subs

Defines the name of the subscriber ID field as it appears in the dbtables.xml file. The default value is SUBSCRIBER_ID.

• field_svc

Defines the name of the service counter ID field as it appears in the dbtables.xml file. The default value is SERVICE_ID, SUBS_USG_CNT_ID.

• field_up

Defines the name of the upstream volume field as it appears in the dbtables.xml file. The default value is UPSTREAM_VOLUME.

• field_down

Defines the name of the downstream volume field as it appears in the dbtables.xml file. The default value is DOWNSTREAM_VOLUME.

• field_sess

Defines the name of the sessions field as it appears in the dbtables.xml file. The default value is SESSIONS.

• field_sec

Defines the name of the seconds field as it appears in the dbtables.xml file. The default value is SECONDS.

• field_pkg

Defines the name of the package-id field as it appears in the dbtables.xml file. The default value is PACKAGE_ID.

• field_ip_type

Defines the name of the ip type field as it appears in the dbtables.xml file.


OL-26830-04


The [csv] Section

The [csv] section contains the following parameters:

• active

Defines whether to activate CSV file creation.


• file_name_format

Defines the format for CSV file names. The default value is yyyy-mm-dd_hh-mm-ss'.csv'.

• dir_name

Defines the storage location for the CSV files. The default value is ~/cm/adapters/TAAdapter/csvfiles.

• write_headers

Defines whether to write a descriptive header at the top of each CSV file.


• metric_separator

Defines the character to separate the metrics in the CSV file. The default value is , (comma).

• service_separator

Defines the character to separate the services in the CSV file. The default value is , (comma).

The [state save] Section

The [state save] section contains the following parameters:

• active

Defines whether to activate state saving orloading.


• file_name

Defines the name of the state file used for saving or loading. The default value is taadapter.sav.

The [db] Section

The [db] section contains the following parameters:

• name_of_total_subs

Defines the designation of "all subscribers" in the database.

• db_template_dir

Defines the location of the main database configuration template relative to Collection Manager configuration directory. The configuration file contains default values for each of the three possible databases. However, only one database can be configured at a time. The possible values are:

– Oracle: dbpacks/oracle/9204e

– Sybase: dbpacks/sybase/ase12.5.1

– MySQL: dbpacks/mysql/4.0.20

The default value is dbpacks/sybase/ase12.5.1.


OL-26830-04


• db_template_file

Defines the name of the main database configuration template. The default value is main.vm. Do not change this parameter.

• operations_timeout

Defines the maximum time, in seconds, allowed for the completion of database operations, such as connecting, insertion, and so on. The default value is 60.

• blocking_connects

Defines whether to block connection attempts to the database.


• connection_timeout

Defines the number of seconds to allow the connection attempts to the database. The default value is 900.

This parameter is not relevant if blocking_connects is set to true.


OL-26830-04


• connection_wait_period

Defines the number of seconds to wait between connection attempts. The default value is 10.

The [app] Section

The [app] section contains the following parameters:

• app_conf_file

Defines the name of the application configuration XML file. The default value is the dbtables.xml file.

• app_dtd_file

Defines the name of the application configuration DTD file. The default value is dbtables.dtd.

• app_conf_dir

Defines the location of the application configuration XML and DTD files relative to Collection Manager configuration directory. The default value is apps/scasbb/latest.

The [peak_hours] Section

The [peak_hours] section contains the following parameters:

• active

Activates or deactivates peak hours processing. The default value is false (deactivated).

• cumulative

Enables or disables cumulative reports during peak hours. The default value is false.

• num_top_entries

Defines how many top entries are reported during peak hours. The default value is 500.

• Mon

Tue

Wed

Thu

Fri

Defines the peak hour range during weekdays. The default value is 18 to 21.

• Sat

Sun

Defines the peak hour range during weekends. The default value is 16 to 19.

The hour value can be a single hour or range of hours.

Top subscriber records are collected during peak hours and aggregated at the end of the corresponding peak hour. For example, if the value is Mon=18-21, subscriber records are collected from 18:00 to 21:00 (the peak hours range) and aggregated at 22:00.

Note To disable the peak hour functionality for a specific day of the week, comment out the appropriate line.


OL-26830-04

Appendix B Configuration File Descriptions The CSV Adapter Configuration File

Note Execute the ~scmscm/scripts/updatepeakconfig.sh script before generating the peak hour reports. This script populates the peak_hour_config table with the default peak hour details. By default, the peak_hour_config table is not created.

The CSV Adapter Configuration FileThis section describes the csvadapter.conf CSV adapter configuration file.

• The [csvadapter] Section, page B-13

• [bench] Section, page B-13

The [csvadapter] Section

The [csvadapter] section contains the following parameters:

• maxCSVFiles

Defines the maximum number of CSV files that can be created.


• rdrLimit

Defines the maximum number of RDRs that can be written to a single file.


• backupPreviousFiles

Defines whether to back up the existing CSV files before overwriting them.


• includeRecordSource

Defines whether to include the record source (SCE IP) in each RDR line.


• escapeNonPrintables

Defines whether to escape characters that cannot be printed.


Note Setting the escapeNonPrintables to true can negatively affect the adapter’s performance.

[bench] Section

The [bench] section contains the following parameter:


Defines the time period, in milliseconds, between the write rate updates. The default value is 5,000. Set this parameter to 0 to disable the write rate updates.


OL-26830-04

Appendix B Configuration File Descriptions The JDBC Adapter Configuration File

The JDBC Adapter Configuration FileThis section describes the jdbcadapter.conf JDBC adapter configuration file:

• The [bench] Section, page B-14



The [bench] Section



Defines the time period in milliseconds between insert rate updates. The default value is 5,000.

Set this parameter to 0 to disable the insertions.

The [db] Section


• db_template_dir








• batch_size

Defines how many inserts can be made in a single batch. The default value is 10.

• transaction_size

Defines how many batches can be made in a single transaction. The default value is 15.

• commit_interval

Defines the maximum time, in seconds, between database commits. The default value is 6.


Defines the maximum time, in seconds, allowed for the completion of database operations, such as connecting, insertion, and so on. The default value is 60.


Defines whether to block connection attempts to the DB.



OL-26830-04

Appendix B Configuration File Descriptions RAG Adapter Configuration File

The [app] Section


• app_conf_file


• app_dtd_file


• app_conf_dir


RAG Adapter Configuration FileThis section describes the ragadapter.conf RAG adapter configuration file.

• The [config] Section, page B-15

• The [housekeeper] Section, page B-16



The [config] Section

The [config] section contains the following parameters:

• xml_filename

Defines the location of the application configuration XML and DTD files that define the real-time aggregation performed by the RAG adapter. The default value is ~/cm/config/ragadapter.xml.

• domain_ext_filename

Defines the location of the TLDS based on the domain. The default value is ~/cm/config/tlds-alpha-by-domain.txt.

• network_type_filename

Defines the location for the Radio Access Type (RAT) to network type mapping. The default value is ~/cm/config/rat-networktype.txt.

• imei_lookup

Defines whether to look up the Type Allocation Code values present in the database on processing the Subscriber Usage RDR. Possible values for this parameter are true and false. The default value is false.

• vsa_type

Defines the VSA type i.e GSM or CDMA to be enabled. The default value is vsa_type=gsm.

• meid_length

Defines the number of characters in the Mobile Equipment Identifier (MEID) to be stored. The default value is 8.

• attr_index


OL-26830-04

Appendix B Configuration File Descriptions RAG Adapter Configuration File

Defines the index position of the attribute indicator field in NUR RDR in the current Cisco Service Control Operating System version.

• attr_shift_pos

Defines the shift position of the VSA fields in NUR RDR.

The [housekeeper] Section

The [housekeeper] section contains the following parameter:

• interval_sec

Defines the interval for the scheduled task to perform aggregation/sink operations. The default value is 10.

The [db] Section


• db_template_dir








• batch_size

Defines how many inserts can be made in a single batch. The default value is 10.

• transaction_size

Defines how many batches can be made in a single transaction. The default value is 15.

• commit_interval

Defines the maximum time between database commits. The default value is 6.


Defines the maximum time allowed for the completion of database operations such as connecting, insertion, and so on. The default value is 60.


Defines whether to block connection attempts to the DB.


The [app] Section



OL-26830-04

Appendix B Configuration File Descriptions Comma-Separated Value Adapter

• app_conf_file


• app_dtd_file


• app_conf_dir


Comma-Separated Value AdapterThis section describes the Comma-Separated Value Adapter configuration file:

• The [customadapter] Section

• [bench] Section

The [customadapter] Section

The [customadapter] section contains the following parameters:

• includeRecordSource

Defines whether to include the record source (SCE IP) in each RDR line.


• parserClass

Defines the parser class to load for parsing the expressions. By default, JEPParser will be used for parsing the expression.

Note The user should not change the parserClass property value.

[bench] Section



Defines the time period, in milliseconds, between the write rate updates. The default value is 5000. Set this parameter to 0 to disable the write rate updates.


OL-26830-04

Appendix B Configuration File Descriptions Comma-Separated Value Adapter


OL-26830-04

cisco collection manager user guide...using custom tablespaces 6-12 using table partitioning 6-12...

Documents