cisco application visibility and control collection ... · chapter 1 cisco application visibility...

Cisco Application Visibility and Control Collection Manager User GuideRelease 3.7.xMay 10, 2012

Americas HeadquartersCisco Systems, Inc.170 West Tasman DriveSan Jose, CA 95134-1706 USAhttp://www.cisco.comTel: 408 526-4000

800 553-NETS (6387)Fax: 408 527-0883

Text Part Number: OL-24187-04

http://www.cisco.com

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Cisco and the Cisco Logo are trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and other countries. A listing of Cisco's trademarks can be found at www.cisco.com/go/trademarks. Third party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1005R)

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.

Cisco Application Visibility and Control Collection Manager User Guide© 2011 Cisco Systems, Inc. All rights reserved.

http://www.cisco.com/go/trademarks

OL-24187-04

C O N T E N T S

About this Guide 7

Introduction 7

Document Revision History 8

Organization 9

Related Publications 10

Conventions 11

Obtaining Documentation and Submitting a Service Request 12

C H A P T E R 1 Cisco Application Visibility and Control Overview 1-1

Introduction 1-1

Cisco Application Visibility and Control Components 1-2

Cisco ASR 1000 Series Router 1-2

Cisco Network-Based Application Recognition 2 (NBAR 2) 1-3

Cisco Flexible NetFlow 1-3

Cisco NetFlow V9 Export Format 1-3

Cisco Collection Manager 1-3

Cisco Collection Manager Database 1-4

Cisco Insight 1-4

Cisco Application Visibility and Control Process 1-5

Cisco IOS CLI 1-5

NBAR 2 1-5

Quality of Service (QoS) 1-5

Bandwidth Control 1-6

Cisco Flexible NetFlow 1-6

Usage Records 1-8

Transaction Records 1-8

Global Usage Records 1-8

Cisco Collection Manager 1-8

Cisco Insight v3 1-8

C H A P T E R 2 Cisco Collection Manager Overview 2-1

Introduction 2-1

Collection Process 2-2

Adapter Description 2-3

1Cisco Application Visibility and Control Collection Manager User Guide

Contents

JDBC Adapter 2-3

Comma Separated Value Adapter 2-3

Bundled and External Databases 2-4

Using the Bundled Database 2-4

Using an External Database 2-4

C H A P T E R 3 Basic Installation of the Cisco Collection Manager 3-1

Introduction 3-1

Getting the Cisco Collection Manager Software 3-2

Checking System Prerequisites 3-3

System Requirements 3-4

Solaris Requirements 3-4

Hardware 3-5

Software and Environment 3-5

Setting the Locale and Time Zone 3-7

Red Hat Linux Requirements 3-7

Hardware 3-7


Setting the Locale and Time Zone 3-8

CentOS Linux Requirements 3-8

Hardware 3-9


Distribution Content 3-9

Default Configuration Settings 3-9

Installing the Cisco Collection Manager Software 3-10

Information About the install-cm.sh Script 3-10

Cisco Collection Manager Software Ports 3-13

Installing the Sybase Database 3-14

Actions Performed by installsyb.sh 3-14

Prerequisites 3-15

Starting the Cisco Collection Manager 3-16

Setting the Time Zone 3-16

Starting the Bundled Sybase Database 3-17

Configuring the External MySQL Server 3-17

Router Configuration 3-17

Uninstalling the Sybase Database and Cisco Collection Manager Software 3-18

Uninstalling Sybase 3-18

Uninstalling the Cisco Collection Manager Software 3-18


OL-24187-04

Contents

Upgrading the Cisco Collection Manager 3-19

C H A P T E R 4 Managing the Collection Manager 4-1

Introduction 4-1

Using Utility Scripts 4-2

Collection Manager Support Information 4-2

Configuring the Collection Manager 4-3

Activating Servers 4-3

Controlling Adapters 4-4

Enabling Adapters 4-4

Configuring Databases 4-5

Managing INI values for NetFlow 4-6

Monitoring System Health 4-7

Installing the Periodic Checker 4-7

Periodic Checker Script 4-7

Periodic Checker Script 4-7

Tests 4-8

Monitoring the Collection Manager 4-9

Checking the Database Capacity 4-9

Verifying Server Operation 4-9

Viewing Version Information 4-10

C H A P T E R 5 Managing Databases and the Comma Separated Value Repository 5-1

Introduction 5-1

Common Database Management Tasks 5-2

Listing Database Tables 5-2

Managing the CM:NFR Periodic Deletion of Old Records 5-3

Configuring Periodic Delete 5-4

Applying the Periodic Delete Configuration File 5-5

Deleting a Table 5-6

Deleting Old Records 5-6

UNICODE Support for Sybase Database 5-7

Managing the Bundled Sybase Database 5-8

Backing up the Sybase Database 5-8

Restoring a Sybase Database 5-8

Updating Sybase with a Changed IP Address 5-9

Updating Sybase Database Statistics 5-9

Managing the External MySQL Database 5-10


OL-24187-04

Contents

Backing up the MySQL Database 5-10

Restoring a MySQL Database 5-10

Partitioning the MySQL Database 5-11

Prerequisites 5-11

Configuring CM:NF Partitioning 5-11

Managing the Oracle Database 5-13

Privileges 5-13

Aggregating Database Data 5-14

Configuring CM:NetFlow Aggregation 5-14

Enabling CM:NF Aggregation in Sybase 5-16

Enabling CM:NF Aggregation in MySQL 5-17

Enabling CM:NF Aggregation in Oracle 5-17

Managing the CSV Repository 5-18

CM:NF CSV Repository File Structure 5-18

CM:NF Configuring the CSV File Repository 5-18

CM:NF Configuring the Comma Escape 5-20

CM:NF Configuring Escape of Nonprintable Characters 5-20

C H A P T E R 6 Configuring Databases 6-1

Introduction 6-1

Generating SQL Code Using the Velocity Template Language 6-2

Database Configuration Files 6-3

Context Objects 6-4

tables Object 6-4

dbinfo Object 6-4

tools Object 6-5

Application Configuration 6-5

Working Sample 6-6

Macro Definitions 6-6

dbinfo Configuration 6-7

SQL Definitions 6-7

Code for Dropping a Table 6-7

Code for Creating a Table 6-7

Code for Creating Indexes 6-8

Code for Insert 6-8

Code for Metadata Query 6-8

Testing and Debugging 6-9

Parsing a String 6-9

Obtaining Full Debug Information 6-9


OL-24187-04

Contents

Using the JDBC Framework in Scripts 6-10

Viewing and Setting the Time Zone Offset 6-10

Scalability Hints for Oracle 6-12

Using Custom Tablespaces 6-12

Using Table Partitioning 6-14

A P P E N D I X A NetFlow Field Types and Database Formats A-1

Introduction A-1

NetFlow Field Types A-1

NetFlow Field Types for RPT_USAGE_NF Table A-1

NetFlow Field Types for RPT_TRANSACTION_NF A-3

NetFlow Field Types for RPT_GLB_USAGE_NF Table: A-5

NetFlow Field Types for CONF_TZ_OFFSET_NF Table A-6

NetFlow Field Types for NF_INI_VALUES Table A-6

Database Tables: Formats and Field Contents A-8

Table CONF_TZ_OFFSET_NF A-8

NetFlow Field Types for RPT_USAGE_NF Table A-9

NetFlow Field Types for RPT_TRANSACTION_NF Table A-9

Columns of Table RPT_GLB_USAGE_NF A-10

NetFlow Field Types for NF_INI_VALUES Table A-11

A P P E N D I X B Code Samples B-1

Introduction B-1

Application Configuration B-1

CM:NetFlow nf-dbtables.xml File B-2

tables.dtd File B-5

A P P E N D I X C Configuration Files Descriptions C-1

Introduction C-1

General Configuration Files C-2

Collection Manager Configuration C-2

[adapter] Section C-2

[adapter_start_timeout] Section C-3

[buffer] Section C-3

[queue] Section C-3

[server] Section C-4

[collector] Section C-4

[csv_adapter] Section C-4


OL-24187-04

Contents

[nf-type] Section C-4

CM:NetFlow Queue Configuration C-5

CM:NetFlow Adapter Configuration Files C-6

CSV Adapter C-6

[csvadapter] Section C-6

[bench] Section C-6

JDBC Adapter C-7

[bench] Section C-7

[db] Section C-7

[app] Section C-8


OL-24187-04

About this Guide

IntroductionThis guide describes how to the install and configure the Cisco Collection Manager operating as a component of the Cisco Application Visibility and Control solution.

This guide is intended for technicians who must install and configure the Cisco Collection Manager. The guide is also intended for service provider operators who must manage the daily operation of the Collection Manager. Technicians and operators should be familiar with the Cisco ASR 1000 Series router and related components.


OL-24187-04

About this Guide

Document Revision HistoryTable 1 lists the changes to this document since it was first released.

Table 1 Document Revision History

RevisionCisco Service Control Release and Date Change Summary

OL-24186-04 Release 3.7.5May10, 2012

Updated to resolve caveat CSCtz80659.

OL-24186-04 Release 3.7.5March 28, 2012

Minor updates

OL-24186-04 Release 3.7.5March 22, 2012

Update for release 3.7.5

OL-24186-03 Release 3.7.2February 8, 2012

Update for release 3.7.2

OL-24186-02 Release 3.7.1November 18, 2011

Added Appendix A, “NetFlow Field Types and Database Formats”

OL-24186-02 Release 3.7.1October 21, 2011

Update for release 3.7.1.

OL-24186-01 Release 3.7.xMay 10, 2011

Initial release of this document


OL-24187-04

About this Guide

OrganizationTable 2 describes the sections in this document.

Table 2 Document Organization

Title Description

Chapter 1, “Cisco Application Visibility and Control Overview”

Describes the Cisco ASR 1000 Application Visibility and Control solution.

Chapter 2, “Cisco Collection Manager Overview”

Describes the Collection Manager components

Chapter 3, “Basic Installation of the Cisco Collection Manager”

Describes how to install the Collection Manager and its database, and explains how to run the Collection Manager

Chapter 4, “Managing the Collection Manager”

Explains how to use utility scripts to view and update Collection Manager parameters and other information

Chapter 5, “Managing Databases and the Comma Separated Value Repository”

Explains how to use utility scripts to manage the Collection Manager database and the CSV repository

Chapter 6, “Configuring Databases” Explains how to configure the Collection Manager to work with your database

Chapter A, “NetFlow Field Types and Database Formats”

Provides sample lists of code for configuration files

Chapter B, “Code Samples” Describes the available parameters in all of the configuration files

Chapter C, “Configuration Files Descriptions”

Describes the configuration files


OL-24187-04

About this Guide

Related PublicationsUse this guide with the following Cisco documentation:

• Cisco Service Control Application for Broadband User Guide

• Cisco Service Control Application for Broadband Reference Guide

• Cisco SCA BB Service Configuration API Programmer Guide

• Cisco Service Control Application Reporter User Guide


OL-24187-04

http://www.cisco.com/en/US/docs/cable/serv_exch/serv_control/broadband_app/rel37x/scabbug/scabbug.html

http://www.cisco.com/en/US/docs/cable/serv_exch/serv_control/broadband_app/rel37x/scabbrg/scabbrg.html

http://www.cisco.com/en/US/docs/cable/serv_exch/serv_control/broadband_app/rel37x/scabbpg355/scabb_pg.html

http://www.cisco.com/en/US/docs/cable/serv_exch/serv_control/broadband_app/rel37x/reporterug/reporter.html

About this Guide

ConventionsTable 3 presents and defines the documentation conventions used in this guide:

Note Means reader take note.

Tip Means that the following information will help you solve a problem.

Caution Means reader be careful. In this situation, you might perform an action that could result in equipment damage or loss of data.

Timesaver Means the described action saves time. You can save time by performing the action described in the paragraph.

Warning Means reader be warned. In this situation, you might perform an action that could result in bodily injury.

Table 3 Conventions

Convention Indication

bold font Commands and keywords and user-entered text appear in bold font.

italic font Document titles, new or emphasized terms, and arguments for which you supply values are in italic font.

[ ] Elements in square brackets are optional.

{x | y | z } Required alternative keywords are grouped in braces and separated by vertical bars.

[ x | y | z ] Optional alternative keywords are grouped in brackets and separated by vertical bars.

string A nonquoted set of characters. Do not use quotation marks around the string or the string will include the quotation marks.

courier font Terminal sessions and information the system displays appear in courier font.

< > Nonprinting characters such as passwords are in angle brackets.

[ ] Default responses to system prompts are in square brackets.

!, # An exclamation point (!) or a pound sign (#) at the beginning of a line of code indicates a comment line.


OL-24187-04

About this Guide

Obtaining Documentation and Submitting a Service RequestFor information on obtaining documentation, submitting a service request, and gathering additional information, see the monthly What’s New in Cisco Product Documentation, which also lists all new and revised Cisco technical documentation, at:

http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html

Subscribe to What’s New in Cisco Product Documentation as a Really Simple Syndication (RSS) feed and set content to be delivered directly to your desktop using a reader application. The RSS feeds are a free service and Cisco currently supports RSS version 2.0.


OL-24187-04

http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html

Cisco Application ViOL-24187-04

C H A P T E R 1
Cisco Application Visibility and Control Overview
IntroductionThis chapter is an overview of the Cisco Application Visibility and Control solution.

The Cisco Application Visibility and Control solution integrates multiple internal components of the Cisco ASR 1000 Series Aggregation Services Router and external components. This chapter describes both the internal and external components of the solution. It also details the solution process.

• Cisco Application Visibility and Control Components, page 1-2

• Cisco Application Visibility and Control Process, page 1-5

1-1sibility and Control Collection Manager User Guide

Chapter 1 Cisco Application Visibility and Control Overview Cisco Application Visibility and Control Components

Cisco Application Visibility and Control ComponentsThe core components of the Cisco Application Visibility and Control solution, include:

• Cisco ASR 1000 Series Router, page 1-2

• Cisco Network-Based Application Recognition 2 (NBAR 2), page 1-3

• Cisco Flexible NetFlow, page 1-3

• Cisco NetFlow V9 Export Format, page 1-3

• Cisco Collection Manager, page 1-3

• Cisco Collection Manager Database, page 1-4

• Cisco Insight, page 1-4

Figure 1-1 illustrates the internal and external components of the Cisco Application Visibility and Control solution.

Figure 1-1 Cisco Application Visibility and Control Network Components

Cisco ASR 1000 Series RouterThe Cisco ASR 1000 Series router is Cisco’s midrange routers offering convergence of network services on a scalable routing platform. The Cisco ASR 1000 Series uses the Cisco Quantum Flow Processor (QFP), which offers multiprocessing, advanced memory management, customized quality of service (QoS), and silicon-based service delivery and programmability. The flexibility of this processor allows network services such as packet encryption, packet inspection, application recognition, traffic differentiation, and subscriber management to be integrated on a single routing platform. It does not need to use external network appliances or service modules for this platform. The router delivers the resiliency, intelligent services, and modularity for needed for the future.

QFP

Cisco IOSFlexible NetFlow

Cisco Network-BasedApplication Recognition

(NBAR)

QoS over MQC

Cisco CollectionManager

with OptionalInternal Database

HTTP

NetFlow TrafficRecords

DatabaseRecords

OptionalExternalDatabase

CiscoInsight

Cisco ASR 1000with Application Visibility

and Control

2462

04

1-2Cisco Application Visibility and Control Collection Manager User Guide

OL-24187-04

Chapter 1 Cisco Application Visibility and Control Overview Cisco Network-Based Application Recognition 2 (NBAR 2)

Cisco Network-Based Application Recognition 2 (NBAR 2)Cisco NBAR 2 is an important component of the Cisco Content networking architecture. It acts as a classification engine in Cisco IOS Software and can recognize a wide variety of Layer 7 applications. These applications include Web-based applications and client/server applications that dynamically assign TCP or User Datagram Protocol (UDP) port numbers. After the application is recognized, the network can invoke specific services for that particular application. NBAR 2 can work with quality-of-service (QoS) features to help ensure that the network bandwidth is used efficiently.

Cisco Flexible NetFlowThe original NetFlow is a Cisco IOS technology that provides statistics on packets flowing through the router. It is the standard for acquiring IP operational data from IP networks. It provides data to enable network and security monitoring, network planning, traffic analysis, and IP accounting. Flexible NetFlow improves on the original NetFlow by adding the capability to customize the traffic analysis parameters for your specific requirements. Flexible NetFlow facilitates the creation of more complex configurations for traffic analysis and data export by using reusable configuration components.

Cisco NetFlow V9 Export FormatFlexible NetFlow exports information to reporting servers in various formats including NetFlow version 9. NetFlow version 9 is the format used by the Cisco Application Visibility and Control solution. NetFlow version 9 is a flexible and extensible format. It includes a template to describe what is being exported and the export data. The template is periodically sent to the Flexible NetFlow collector telling it what data to expect from the router or switch. The data is then sent for the reporting system to analyze. Any data available in the device can theoretically be sent in NetFlow version 9 format. Flexible NetFlow allows the user to configure and customize what information is exported using NetFlow version 9. NetFlow version 9 is the basis for the IETF standard IPFIX associated with the IP Flow and Information working group in IETF.

Cisco Collection ManagerThe Cisco Collection Manager is a set of software modules that runs on a server. It receives and processes NetFlow Records. The NetFlow Records used by the Cisco Collection Manager are generated by the Cisco ASR 1000 Series router. The process uses Cisco Network-Based Application Recognition (NBAR) and Cisco Flexible NetFlow. The processed records are stored in the Cisco Collection Manager database.

The Cisco Collection Manager also interoperates with the Cisco Service Control Management Suite (SCMS) and the Cisco Service Control Engine (SCE) platform. The Cisco Collection Manager is able to operate with both the Cisco ASR 1000 Series and the Cisco SCE platform at the same time.


OL-24187-04

Chapter 1 Cisco Application Visibility and Control Overview Cisco Collection Manager Database

Cisco Collection Manager DatabaseThe Cisco Collection Manager database is either a bundled database or an external database. In bundled mode, the Cisco Collection Manager uses the Sybase Adaptive Server Enterprise database. This database is located on the same server as the other Cisco Collection Manager components. It uses a simple schema that includes a group of small, simple tables. The Cisco Collection Manager is also able to use a Java Database Connectivity (JDBC) compliant database. This includes standard databases, such as Oracle or MySQL. The JDBC-compliant database can be located on the same server as the Cisco Collection Manager or hosted on an external server.

Cisco Insight Cisco Insight is reporting platform software. It processes the formatted data from the Collection Manager database. Cisco Insight presents customizeable reports, charts, and statistics about the traffic on the Cisco ASR 1000 Series. It is a web-based application accessed with a browser.


OL-24187-04

Chapter 1 Cisco Application Visibility and Control Overview Cisco Application Visibility and Control Process

Cisco Application Visibility and Control ProcessThe Cisco Application Visibility and Control solution provides network visibility and traffic control. This section details the processes and purposes of the various components in the solution.

The Cisco Application Visibility and Control solution processing functions, include:

• Cisco IOS CLI, page 1-5

• NBAR 2, page 1-5

• Quality of Service (QoS), page 1-5

• Bandwidth Control, page 1-6

• Cisco Flexible NetFlow, page 1-6

• Usage Records, page 1-8

• Transaction Records, page 1-8

• Global Usage Records, page 1-8

• Cisco Collection Manager, page 1-8

• Cisco Insight v3, page 1-8

Cisco IOS CLIThe basic interface for the Cisco Application Visibility and Control solution is the standard Cisco IOS CLI. You can configure NBAR 2, Modular Quality of Service (MQC) queues and policers, Flexible NetFlow counters, and the protocols for NetFlow information. In the typical configuration, the user uses the Cisco IOS CLI to configure NetFlow rules. These generate the required records and configure the QoS features on specific interfaces.

NBAR 2NBAR 2 is a Cisco feature that enables protocol detection for a network. Protocol detection is the process by which the system determines that a particular network flow is from a specific application. This process is done using various techniques including payload signature matching, behavioral classification or classification based on L7 parameters (sometimes called protocol sub-classification). Upon detection, a Protocol-ID is assigned to a flow. The Protocol-ID is then used by the solution to determine the appropriate actions on packets belonging to that flow.

Quality of Service (QoS)Standard Cisco Modular Quality of Service (MQC) is used for the Cisco Application Visibility and Control solution. Cisco MQC is used to create the application-aware policy of the solution. There are no changes or enhancements to the Cisco QoS framework for the Cisco Application Visibility and Control solution.


OL-24187-04

Chapter 1 Cisco Application Visibility and Control Overview Bandwidth Control

Bandwidth ControlThe Application Visibility and Control solution provides global bandwidth control by application. This control allows service providers to set acceptable bandwidth consumption policies for different traffic classes. It can ensure better performance for sensitive application flows, for example voice calls, gaming, web browsing, and video. This performance increase is accomplished by lowering the bandwidth priority of less sensitive traffic, for example file downloads or backups. Bandwidth priority is provided by using platform policies.

Cisco Flexible NetFlowFlexible NetFlow uses the Cisco ASR 1000 Series infrastructure to provide application visibility. Flexible NetFlow exports data in the NetFlow version 9 format. The exported data is in the form of Flexible NetFlow Records. The three types of Flexible NetFlow Records are Usage Records, Global Usage Records, and Transaction Records.


OL-24187-04

Chapter 1 Cisco Application Visibility and Control Overview Cisco Flexible NetFlow

Figure 1-2 illustrates the packet fields used by the Flexible NetFlow Transaction Records and Usage Records.

Figure 1-2 Packet Fields of Transaction Records and Usage Records

The following sections describe the two types of Flexible NetFlow records:

• Usage Records, page 1-8

• Transaction Records, page 1-8

• Global Usage Records, page 1-8

246231

Transac�on record Usage Record

Application ID

Attached Interface (input/output)

Direction

Other interface (input/output)

First timestamp

Last timestamp

Packets counter

Bytes counter

New-Flows counter

Total Seconds counter

IP Version

Input VRF ID

Global Usage Record

Attached Interface (input/output)

Direction

Other interface (input/output)

First timestamp

Last timestamp

Packets counter

Bytes counter

New-Flows counter

Total Seconds counter

IP Version

Input VRF ID

Transaction ID

Input Interface

Output Interface

Direction

Source IP

Source Port

Destination IP

Destination Port

IP Protocol

Application ID

Connection Initiator

First timestamp

Last timestamp

Packets counter

Bytes counter

Bundle/Flow ID

Flow close mode

Sampler ID

IPv6 Source Address

IPv6 Destination Address

IP Version

Input VRF ID


OL-24187-04

Chapter 1 Cisco Application Visibility and Control Overview Cisco Collection Manager

Usage Records

Usage Records are records of the different type of applications running over a specific interface. The operator can use Usage records to monitor how much bandwidth the different applications use. The Usage Records can show this application usage over a specific time period, the peak and average usages, and usage for a specific application type.

Usage Records perform periodic aggregation of the category information for the interface. (Example: export information for peer-to-peer or e-mail usage).

Transaction Records

A transaction is a set of logical exchanges between endpoints. There is normally one transaction within a flow.

The Transaction Record monitors the traffic at transaction levels. Transaction Records provide a detailed analysis of the traffic flows, including extracted Layer 7 fields. Due to the high load of transactions these records are sample or filtered.

Transaction Records are bound to the input and output directions of the network side interfaces. These Transaction Records allow the system to capture each unidirectional flow once.

Global Usage Records

Global usage records are the records of many applications running over an interface. These records monitor the total network traffic. The Global usage records can show the collected application usage over a specific time period, the peak usage, and the average usage.

Cisco Collection ManagerThe Cisco Collection Manager is a set of software modules that runs on a server. It receives and processes NetFlow Records and stores the processed records in a database. The Cisco Collection Manager is covered in detail in the Cisco Collection Manager Overview chapter of the Cisco Application Visibility and Control Collection Manager User Guide.

Cisco Insight v3Cisco Insight v3 is a software platform based on a web 2.0 user experience standard. It accesses information from the database generate by the Cisco Collection Manager. Cisco Insight v3 then collects and presents reports, charts, and statistics about the traffic. Some of the reports you can generate show the active applications on the network in a given time frame. This is according to the estimated amount of bandwidth the applications consume. You can also sort the information as upstream traffic, downstream traffic, or number of sessions.


OL-24187-04


C H A P T E R 2
Cisco Collection Manager Overview
IntroductionThis chapter is an overview of the Cisco Collection Manager operating as a component of the Cisco Application Visibility and Control solution. The Cisco Collection Manager is a set of software modules that runs on a server. It receives and processes NetFlow Records and stores the processed records in a database.

The Cisco Collection Manager was designed to support mix of IPv4 and IPv6 traffic from a Cisco ASR 1000 Series router. The router generates the NetFlow Records used by the Cisco Collection Manager. The Cisco ASR 1000 Series Router depends on the Cisco Network-Based Application Recognition 2 (NBAR 2) and Cisco IOS Flexible NetFlow for the operation. Cisco NBAR 2 and Cisco IOS Flexible NetFlow are internal features of the router. Another component of the Cisco Application Visibility and Control solution is the Cisco Insight v3 software. It uses the Cisco Collection Manager database to generate usage reports.

The Cisco Collection Manager also interoperates with the Cisco Service Control Engine (SCE) series platform. Refer to the Cisco Service Control Management Suite Collection Manager User Guide for information on that solution.

The chapter includes the following sections:

• Collection Process, page 2-2

• Adapter Description, page 2-3

• Bundled and External Databases, page 2-4


Chapter 2 Cisco Collection Manager Overview Collection Process

Collection Process The Cisco Collection Manager collects and processes information on the Cisco ASR 1000 Series traffic.

Figure 2-1 illustrates Cisco Collection Manager collection and processing.

Figure 2-1 Collection Manager Collection and Processing

1. The Cisco ASR 1000 Series sends the NetFlow Records to port 2055 of the Cisco Collection Manager NetFlow Record server.

2. A priority queue queues each NetFlow record according to its priority level and stores it in a persistent buffer until the adapter processes it.

3. The persi stent buffer of the priority queue stores the NetFlow Record until the adapter is able to process it. A persistent buffer is a nonvolatile storage area that ensures that the system processes NetFlow Records even in cases of hardware, software, or power failures.

4. The adapter processes the NetFlow Record and sends it to an internal or external database.

RDR CollectorTCP on Port 33000

Collection Manager

JDBC CSV TA RAG CA

RDR Adapters

Persistent Buffers

RDR Database

NetFlow CollectorUDP on Port 2055

NetFlow JDBC CSV

NetFlow Adapter

Persistent Buffer

NetFlow Database

JDBC CSV TA RAG CA JDBC CSV

2462

05

Cisco ASR 1000

QFP

Cisco SCE Series


OL-24187-04

Chapter 2 Cisco Collection Manager Overview Adapter Description

Adapter DescriptionAdapters are software modules that transform NetFlow Records to match the requirements of the target system. The adapter distributes the NetFlow Records upon request. The following adapters are compatible with NetFlow Records:

• JDBC Adapter, page 2-3

• Comma Separated Value Adapter, page 2-3

Note Some adapters send data to the database or write it to CSV files. The structures of the database tables, and the location and structures of these CSV files are described in Cisco Service Control Application for Broadband Reference Guide.

JDBC AdapterThe Java Database Connectivity (JDBC) adapter receives NetFlow Records, processes them, and stores the records in a database. This adapter is compatible with any database server that is JDBC-compliant, and transforms the records into a format acceptable to that database. You can configure the JDBC adapter to use an external database.

The JDBC adapter is pre-configured to support the following databases:

• Sybase Adaptive Server Enterprise (ASE) 12.5 and 15.0

• Oracle 9.2, 10.2, and 11

Note Disable the recycle bin feature available in Oracle 10 and later versions. You can set the initial value of the recyclebin parameter in the text initialization file init<SID>.ora, for example:recyclebin=off

• MySQL 4.1, 5.0, and 5.1

Comma Separated Value AdapterThe comma-separated-value (CSV) adapter receives NetFlow Records, processes them, and writes the records to files on the disk in comma-separated-value format. A service provider operations support systems (OSS) or a third-party billing system can retrieve these records to generate enhanced accounting and network traffic analysis records using standard mechanisms such as FTP.


OL-24187-04

Chapter 2 Cisco Collection Manager Overview Bundled and External Databases

Bundled and External DatabasesThe Cisco Collection Manager uses either a bundled database or an external database to store NetFlow Records supplied by the Cisco ASR 1000 Series.

• Using the Bundled Database, page 2-4

• Using an External Database, page 2-4

Using the Bundled DatabaseIn bundled mode, the Cisco Collection Manager uses the Sybase Adaptive Server Enterprise database. This database enables you to do the following:

• Support transaction-intensive enterprise applications

• Store and retrieve information online

• Warehouse needed information

The Sybase database is located on the same server as the other Cisco Collection Manager components. It uses a simple schema that includes a group of small, simple tables.

1. The JDBC adapter sends converted NetFlow Records to the database, where the NetFlow Records are stored in these tables.

2. You can access records by using standard database query and reporting tools.

You maintain the database by using operating system commands and scripts. The Cisco Collection Manager supports automatic purging of old records from the bundled database. By default, the Cisco Collection Manager automatically purges the report tables of every record that is more than two weeks old. The Cisco Collection Manager polls the records once every hour. You can configure database maintenance by using the dbperiodic.sh utility script. For more information, see the “Managing the CM:NFR Periodic Deletion of Old Records” section on page 5-3.

Using an External DatabaseWith the JDBC adapter, you can use any JDBC database (for example, Oracle or MySQL) with the Collection Manager. In this case, the database can be local or remote. To use an external database, perform the following tasks:

• Configure the JDBC adapter to use this database.

• Configure a database pack to supply the Cisco Collection Manager with the parameters of the database (such as its IP address and port).

• Supply a JDBC driver for the database so that the adapter can connect to the database.

You can configure an external database during the Cisco Collection Manager software installation. For details see the “Installing the Cisco Collection Manager Software” section on page 3-10.

You can also configure an external database after the Cisco Collection Manager software is installed by running the dbconf.sh script:

~scmscm/scripts/dbconf.sh --nf


OL-24187-04


C H A P T E R 3
Basic Installation of the Cisco Collection Manager
Introduction This chapter gives the information required for a basic installation of the Cisco Collection Manager and the bundled Sybase database. It also gives the procedures for uninstalling the Cisco Collection Manager software and bundled Sybase database, and upgrading the Cisco Collection Manager software version. References are given throughout for additional and more complex configurations of the Cisco Collection Manager software and features.

• Getting the Cisco Collection Manager Software, page 3-2

• Checking System Prerequisites, page 3-3

• System Requirements, page 3-4

• Installing the Cisco Collection Manager Software, page 3-10

• Cisco Collection Manager Software Ports, page 3-13

• Installing the Sybase Database, page 3-14

• Starting the Cisco Collection Manager, page 3-16

• Starting the Bundled Sybase Database, page 3-17

• Setting the Time Zone, page 3-16

• Configuring the External MySQL Server, page 3-17

• Router Configuration, page 3-17

• Uninstalling the Sybase Database and Cisco Collection Manager Software, page 3-18

• Upgrading the Cisco Collection Manager, page 3-19

Note The Cisco Collection Manager also interoperates with the Cisco Service Control Engine (SCE) platform. Refer to the Cisco Service Control Management Suite Collection Manager Quick Start Guide for installing the Cisco Collection Manager to work with the SCE platform.


Chapter 3 Basic Installation of the Cisco Collection Manager Getting the Cisco Collection Manager Software

Getting the Cisco Collection Manager SoftwareTo download the Cisco Collection Manager software:

Step 1 Log in to Cisco CCO http://www.cisco.com/cgi-bin/tablebuild.pl/sccm.

Enter your Cisco CCO password when prompted.

Step 2 Download the relevant package.

Note Each package consists of multiple parts.

• scms-cm-v37X-bXYZ-bundle-solaris-tar.partX

• scms-cm-v37X-bXYZ-bundle-linux-tar.partX

• scms-cm-v37X-bXYZ-unbundled-solaris-linux.tar

If there is a single file package, go to Step 4.

Step 3 Ensure that the names of the files reflect their order (for example, cm_part1, cm_part2, and so on).

Step 4 Place the downloaded files on the target machine and join them into a single .tar file.

For example:

# cat cm_part1 cm_part2 >/usr/tmp/cm_full_package.tar

Step 5 Extract the complete package into a temporary directory.

For example:

# mkdir /usr/tmp/cm_install_temp# cd /usr/tmp/cm_install_temp# tar xvf ../cm_full_package.tar

Step 6 To upgrade from a previous version, go to the Upgrading the Cisco Collection Manager, page 3-19.


OL-24187-04

http://www.cisco.com/cgi-bin/tablebuild.pl/sccm

Chapter 3 Basic Installation of the Cisco Collection Manager Checking System Prerequisites

Checking System PrerequisitesThe Cisco Collection Manager distribution contains a script, check_prerequisites.sh, located in the install_scripts directory. The script helps to determine if a system meets the requirements for installing a Cisco Collection Manager or the bundled Sybase database.

The script checks overall readiness of the system for a Cisco Collection Manager or Sybase installation. The main prerequisites checked are:

• CPU speed

• Amount of RAM

• Operating System version (Solaris 9 or 10, Red Hat Enterprise Linux 4 or 5)

• Additional required and optional packages

• Free space for Cisco Collection Manager and Sybase homes

• Names for all network interface cards (NICs)

• Sybase kernel parameters

• Locale and time zone formats

check_prerequisites.sh [ --sybhome=SYBHOME ] [ --cmhome=CMHOME ] [ --datadir=DATADIR ]

Table 3-1 check_prerequisites.sh Script Options

--sybhome=SYBHOME Intended home directory for Sybase installation

--datadir=DATADIR Intended data directory for Sybase data files (for the Datadir installation method)

--cmhome=CMHOME Intended home directory for Cisco Collection Manager installation


OL-24187-04

Chapter 3 Basic Installation of the Cisco Collection Manager System Requirements

System Requirements The Cisco Collection Manager and its database are software components that run on a server platform. You can install the Cisco Collection Manager on any of the following platforms:

• Sun SPARC machine (64 bit) running 64-bit versions of Solaris 9 or Solaris 10. (See the “Solaris Requirements” section on page 3-4.)

• Intel machine (32 or 64 bit) running 32-bit versions of Red Hat Enterprise Linux 4.0 or Red Hat Enterprise Linux 5.0 or 64-bit versions of Red Hat Enterprise Linux 5.0. (See the “Red Hat Linux Requirements” section on page 3-7.)

• Intel machine (32 or 64 bit) running 32-bit or 64-bit versions of CentOS 5.x. (See the “CentOS Linux Requirements” section on page 3-8.)

• Intel machine (32 or 64 bit) running VMware Server or VMware ESX VMware-VMvisor-Installer-4.0.0-164009.x86_64.iso.

• Cisco Unified Computing System (UCS) server model R210-2121605 with a Intel(R) Xeon(R) X5570 2.93-GHz CPU with eight Cores (minimum memory 4 GB).

All configurations use a 32-bit Java Virtual Machine (JVM).

Caution The Cisco Collection Manager must run on a dedicated server. This server can also host the local collection manager database. You cannot run the Subscriber Manager or other applications on the server dedicated to the Cisco Collection Manager.

Note When using the bundled Sybase database, the server on which you install the Cisco Collection Manager can have a maximum of 4 CPU cores.

• Solaris Requirements, page 3-4

• Red Hat Linux Requirements, page 3-7

• CentOS Linux Requirements, page 3-8

• Distribution Content, page 3-9

• Default Configuration Settings, page 3-9

Solaris RequirementsYou can install Cisco Collection Manager Release 3.7.0 or later on any Sun SPARC Machine running Solaris that conforms to the requirements listed in the following sections:

• Hardware, page 3-5

• Software and Environment, page 3-5

• Setting the Locale and Time Zone, page 3-7


OL-24187-04

Chapter 3 Basic Installation of the Cisco Collection Manager Solaris Requirements

Hardware

• Minimum 500-MHz CPU

• Minimum 1-GB RAM per CPU

• Hard disk:

– One hard disk, at least 18 GB

– For bundled installations, a second hard disk of at least 30 GB is recommended to store Sybase data.

• 100BASE-T network interface

Software and Environment

• Solaris Version 5.9 64-bit build 04/01 or later (currently only Solaris Version 5.9 and 5.10 are supported).

– Solaris 9—Patch level 9 is recommended

– Solaris 10—Patch level 10 is recommended

• Solaris Core Installation.

• Install the additional packages as shown in Table 3-2.

• If you are installing the Cisco Collection Manager in bundled mode with the Sybase database, install the package shown in Table 3-3.

Table 3-2 Additional Packages

system SUNWbash GNU Bourne-Again shell (bash)

system SUNWgzip The GNU Zip (gzip) compression utility

system SUNWzip The Info-Zip (zip) compression utility

system SUNWlibC Sun Workshop Compilers Bundled libC

system SUNWlibCx Sun WorkShop Bundled 64-bit libC

Table 3-3 SUN Wipc Package

system SUNWipc Interprocess Communication


OL-24187-04

Chapter 3 Basic Installation of the Cisco Collection Manager Solaris Requirements

• (Optional) You can install the packages listed in Table 3-4 (for sysadmin applications such as sys-unconfig).

• You can download these packages from http://sunfreeware.com/

The root (/) partition must have at least 104 MB of free space to install these packages.

• Apply the latest recommended patches from Sun:

– For Solaris 9, go to http://sunsolve.sun.com/pub-cgi/show.pl?target=patches/xos-9&nav=pub-patches

– For Solaris 10, go to http://sunsolve.sun.com/pub-cgi/show.pl?target=patches/xos-10&nav=pub-patches

– For Java, go to http://sunsolve.sun.com/pub-cgi/show.pl?target=patches/J2SE

• If you are using Sybase, install the current Solaris patches recommended by Sybase.

• At least 8 GB free on the partition where you intend to install the Cisco Collection Manager. (This memory is used for CSV storage and persistent buffers.)

• At least 3 GB free on one partition for the Sybase home directory (for installations with bundled Sybase).

• Free space on one partition to hold the desired size of the Sybase data and logs (for installations with bundled Sybase). You can configure memory size during installation.

• An FTP server must listen on port 21 so that the SCA Reporter can authenticate it (for installations with bundled Sybase that use the legacy (pre-3.0) Cisco Service Control Application Suite (SCAS) Reporter).

• Before installation, verify that all IP addresses that are configured for the machine NICs have hostnames associated with them in /etc/hosts or in another active naming service (for installations with bundled Sybase). (This action is required due to a limitation of Sybase Adaptive Server Enterprise.)

• Use the set_shmmax.sh script (located under install-scripts/) to configure the kernel memory (for installations with bundled Sybase).

• Additionally, at startup you must load the IPC module by inserting the following lines in the file /etc/system:

forceload: sys/shmsys

• If you are using database periodic delete, enable the scmscm user to schedule and run cron jobs.

Table 3-4 Optional Packages

system SUNWadmap System administration applications

system SUNWadmc System administration core libraries


OL-24187-04

Chapter 3 Basic Installation of the Cisco Collection Manager Red Hat Linux Requirements

Setting the Locale and Time Zone

• For correct Cisco Collection Manager and Sybase operation, use the locale U.S. English.

To set the locale, put the following line in the /etc/TIMEZONE configuration file (to enable a change to this configuration to take effect, you must restart the Cisco Collection Manager):

LANG=en_US

To use the U.S. English locale, the Cisco Collection Manager must be running on the Solaris operating system. Verify that the locale is installed by ensuring that the directory /usr/lib/locale/en_US exists. If the directory does not exist, install the locale files from the Solaris CDs.

• Setting the OS time zone as an offset from GMT in POSIX format is not recommended, and can lead to problems. Set the time zone in the /etc/TIMEZONE configuration file by country (supported) name, as in the following example.

TZ=Japan

Verify that the country name is supported as a time zone setting by ensuring that it is listed in the directory /usr/share/lib/zoneinfo.

If you must use GMT offset, use the zoneinfo format by prepending the prefix :Etc/, as in the following example:

TZ=:Etc/GMT+5

Red Hat Linux RequirementsYou can install Cisco Collection Manager Version 3.1.0 or later on any i386 running Red Hat Linux that conforms to the requirements listed in the following sections:



• Setting the Locale and Time Zone, page 3-8

Hardware



• Hard disk:





OL-24187-04

Chapter 3 Basic Installation of the Cisco Collection Manager CentOS Linux Requirements


• Red Hat Linux 4.0:

– kernel-2.6.9-5

– glibc-2.3.4-2

– compat-libstdc++-33-3.2.3-47.3

– Minimum patch level required—Update 7

• Red Hat Linux 5.0:

– kernel-2.6.18-8.el5

– glibc-2.5-12

– compat-libstdc++-33-3.2-61

– Minimum patch level required—Update 5.3

• Red Hat Enterprise “Base” Installation.

• Apply the latest recommended patches from Red Hat.

• Reserve at least 8 GB on the partition where you want to install the Cisco Collection Manager. The Cisco Collection Manager uses this disk space for CSV storage and persistent buffers.

• If you are using database periodic delete, enable the scmscm user to schedule and run cron jobs.

• For installations with bundled Sybase:

– Also install the compat-libstdc++ package. This package is available on the Red Hat installation CD.

– Install the current patches recommended by Sybase.

– Reserve at least 1 GB on some partition for the Sybase home directory.

– If you intend to use the legacy Cisco Service Control Application Suite (SCAS) Reporter (before Version 3.0), an FTP server listens on port 21 so that the SCA Reporter can authenticate it.

– Before you start the installation, verify that all the IP addresses that are configured for the machine NICs have hostnames associated with them in /etc/hosts or in another active naming service. (This action is required due to a limitation of Sybase Adaptive Server Enterprise.)

– Use the set_shmmax.sh script (located under install-scripts/) to configure the kernel memory.

Setting the Locale and Time Zone

For correct Cisco Collection Manager and Sybase operation, use the locale U.S. English (en_US).

CentOS Linux RequirementsYou can install the Cisco Collection Manager Version 3.6.5 or later on an i386 that runs CentOS Linux. It must conform to the following requirements:




OL-24187-04

Chapter 3 Basic Installation of the Cisco Collection Manager Distribution Content

Hardware



• Hard disk:





• CentOS Linux 5.x:

– kernel-2.6.18-8.el5

– glibc-2.5-12

– compat-libstdc++-33-3.2.3-61

Distribution ContentThe Cisco Collection Manager installation kit contains scripts for installing the Cisco Collection Manager and the Sybase database.

It also contains:

• Scripts to support file gathering

• Scripts for periodic Sybase maintenance

Default Configuration SettingsConfigure settings for the Cisco Collection Manager during installation. These settings include the adapters to enable and their locations, Priority Queue parameters, the target adapters, and various logging policies. Permit only qualified personnel to change these settings.


OL-24187-04

Chapter 3 Basic Installation of the Cisco Collection Manager Installing the Cisco Collection Manager Software

Installing the Cisco Collection Manager SoftwareThis section describes how to install the Cisco Collection Manager.

Note Separate databases are recommended if you are configuring the Cisco Collection Manager for both the Cisco SCE and the Cisco ASR 1000 Series router.

Information About the install-cm.sh Script

To install the Cisco Collection Manager server, use the install-cm.sh script.

install-cm.sh Options

The usage message for the install-cm.sh script is:

Usage: install-cm.sh [-h] (-d CMDIR | -o)

Options: -d CMDIR select directory for ~scmscm (must not exist and must be on 8 GB free partition) -o upgrade the existing installation while preserving the current configuration (can't be used with -d) -h print this help and exit

Description of the options:

-d CMDIR Used to designate the directory of the newly created scmscm user's home. Should be the name of a non-existing directory, whose parent resides on a partition where at least 8 GB is free. As an alternate to this option, you can specify -o :

-o Use this option when you wish to upgrade the existing installation while preserving the current configuration. (can't be used with -d)

Actions Performed by install-cm.sh

The install-cm.sh script performs the following actions:

• If needed, creates an scmscm user and an scmscm group

• Optionally, creates the home for this user

• Populates the home of scmscm with Cisco Collection Manager files and scripts

• Installs the extra component private JRE in ~scmscm/cm/lib

• Creates boot script symbolic links for the sybase and scmscm users in /etc/init.d and /etc/rcX.d

Step 1 Change directory to install-scripts under the distribution kit root.

Step 2 Run the install-cm.sh script.

After running the script, a user-driven configuration manager presents the user with options for the basic configuration of the Cisco Collection Manager.


OL-24187-04


Step 3 Choose one of the options provided by the configuration manager:

Please select one of the following options:1 - Install CM:RDR2 - Install CM:Netflow3 - Install CM:RDR-and-CM:Netflow4 - Exit

• Option 1 is chosen when the Cisco Collection Manager will operate with the Cisco SCE Service Control Engine.

• Option 2 is chosen when the Cisco Collection Manager will operate with the Cisco ASR 1000 Series router.

• Option 3 is chosen when the Cisco Collection Manager will operate with both the Cisco SCE Service Control Engine and the Cisco ASR Series Aggregation Services Routers.

• Option 4 is chosen to exit the installation.

Step 4 Choose option 2, the NetFlow installation, when working with the Cisco ASR 1000 Series router:

a. Choose to set up the database.

Would you like to configure the database?: yes

b. Select the number corresponding to the relational database management system of the connecting database.

You will see the following text:

Configuring CM:Netflow Database...Enter the DB type:1 - Oracle2 - MySQL3 - Sybase4 - ExitEnter your choice:

c. Enter the following server information or press enter to leave at the default shown.

Enter DB server host (default localhost):Enter DB server listening port (default port no) :Enter DB server instance id (default schema) :Enter DB schema user name (default user_id) :Enter DB schema user password (default password) :Would you like to configure the database?: yes

You will see the following text:

PASS:db is upDB connection succeeded.

Note After the user configures the requested database options, the nf-dbinfo.vm file updates for the NetFlow database configuration details.

Note Execute the following script before starting the collection manager ~scmscm/cm/bin/updateNetFlowMap.sh=NF IP address --file=~scmscm/cm/config/AttributesTable.csv This loads the default INI values for NetFlow in the database.


OL-24187-04


Step 5 (Optional) Set a password for the user.

Run the following command to set the password for the user:

passwd asr

Note Be sure to record the password that you choose.

Note To use an external database, configure a dbpack to enable the Cisco Collection Manager to connect to the database. See Chapter 5, “Managing Databases and the Comma Separated Value Repository” for instructions.

Step 6 (Optional) Install and activate the periodic delete procedures for the database tables.

(For information about configuring the behavior of periodic delete, see the “Managing the CM:NFR Periodic Deletion of Old Records” section on page 5-3.)

Note If reports are sent to the database and you do not install and activate the periodic delete procedures, the second disk can overflow.

a. Install the periodic delete procedures.

Log on as the scmscm user, start the Cisco Collection Manager, wait 1 - 2 minutes for the creation of the database tables, and then run the script:

~scmscm/db_maint/create_periodic_del_procs.sh --nf

b. Activate the automatic invocation of the periodic delete procedures.

Run the following command:

~scmscm/scripts/dbperiodic.sh --nf --load

Step 7 Start the Cisco Collection Manager by running the ~scmscm/cm/bin/cm start command.

The script performs the following actions:

• Starts all the adapters enabled in the Cisco Collection Manager.

• Updates the CONF_TZ_OFFSET_NF table with the timezone offset.


OL-24187-04

Chapter 3 Basic Installation of the Cisco Collection Manager Cisco Collection Manager Software Ports

Cisco Collection Manager Software PortsTable 3-5 describes the TCP/UDP ports on which the Cisco Collection Manager software and associated components (such as the Sybase database) listen. This table helps a network administrator to understand the behavior of the software and its adherence to the security policy.

Note The ports listed are TCP ports, except for port 2055. Port 2055 is a UDP port.

The device listens constantly on the ports in the table. If you do not enable access on these port numbers, certain operations can fail. Some operations (such as file transfer) cause a device to open temporarily ports other than the listed ports. However, these ports close automatically when the operation ends.

Table 3-5 Ports on Which the Cisco Collection Manager Listens Constantly

Port Number Description

33002 Internal Cisco Collection Manager for Flexible NetFlow

2055 Used by the Cisco ASR 1000 to send Flexible NetFlow records for data collection. (2055 is a UDP port.)

9093 HTTP technician interface for Flexible NetFlow.

14376 PRPC (the Cisco proprietary version of the Remote Procedure Call protocol)

33001 Internal Cisco Collection Manager.

Note Access is required only from the local machine; external access can be blocked.

4100 Sybase database connectivity through ODBC/JDBC. Required for access to the database (for installations with bundled Sybase).

1099—1120 RMI. Used as the management interface between the data collector and the Service Control management server.

22000 FTP server of the Cisco Collection Manager.

Note FTP transactions can listen on other ports (22001 - 22100) for data transfer, as negotiated by the protocol.

7787 Internal logging of the management user log.

Note Access is required only from the local machine; external access can be blocked.


OL-24187-04

Chapter 3 Basic Installation of the Cisco Collection Manager Installing the Sybase Database

Installing the Sybase DatabaseIf you do not want to install Sybase (for example, when working in unbundled mode), see the Uninstalling the Sybase Database and Cisco Collection Manager Software, page 3-18.

Note The maximum database size supported by the bundled Sybase database is 50GB. For database support larger than 50GB, use an external database.

Note Installing the Sybase database can require as many as 3 hours.

Note According to the Sybase license, you can install the Cisco Collection Manager with the bundled Sybase database on a server with a maximum of four CPU cores.

During installation, if you want to reverse the Sybase installation actions (for example, if an installation is interrupted because of a power failure), do the following:

1. Log in as the root user.

2. Run the /install-scripts/uninstall.sh --sybase script.

Actions Performed by installsyb.sh

The installsyb.sh script installs the Sybase database. The script performs the following actions:

• Verifies the shmem setting for Sybase in /etc/system (for Solaris) or /etc/sysctl.conf (for Red Hat Linux). If the setting is not there, the script inserts it and reboots (after prompting you).

• Adds a user sybase and group sybase.

• Runs the Sybase installer for your platform.

• Builds a Sybase server including Sybase users and passwords.

• Starts Sybase.

• Runs SQL scripts to create the Cisco Collection Manager database structure. This process is lengthy and requires restarting Sybase several times.


OL-24187-04

Chapter 3 Basic Installation of the Cisco Collection Manager Installing the Sybase Database

Prerequisites

Log in as the root user and make the distribution kit contents available on your system or local network.

Step 1 Change directory to sybase in the distribution kit root.

Step 2 Run the script installsyb.sh. Enter the script as follows:

installsyb.sh --sybhome=SYBHOME {-datadir=DATADIR}{--y|--n}

• SYBHOME is the home directory of the Sybase user (It requires having 4 GB of space free for Solaris and 3 GB of space free for Linux)

• Select one of the following data location options:

– Specify --datadir=DATADIR, where DATADIR is a directory in which to store all Sybase data. An empty DATADIR directory is required for a clean installation of the Sybase database.

Use a location in a partition in which at least 30 GB is free.

• If you specify a DATADIR, all Sybase data is stored as normal files in that directory, with default sizes of 20 GB for data, 6 GB for logs, and 2 GB for Sybase temporary storage. During installation, ownership of the directory changes to the Sybase user.

• You can customize the Sybase installation during the installation sequence for RDR and NetFlow by selecting yes/no during the Sybase installation.

• Maximum size for the NetFlow will be the remaining size left after the RDR size was specified. Reserve some space in the DATADIR if you are also creating a schema for NetFlow.

• During the Sybase installation process, you need to enter the size of the Sybase database. Following message is displayed to enter the database size:

Please enter SIZE in 2K blocks of file to be used for the "data[maximum is 102657160; minimum is 5242798]" device. NOTE - the actual size required will include a 1.05 overhead on the amount you specify. SIZE in 2K blocks:

Note If the User is opting to use an external database, such as MySQL or Oracle, then it is mandatory to create a Schema and DB USER with the required privileges. The Cisco Collection Manager should be reconfigured to the database using ~scmscm/scripts/dbconf.sh --nf script option.


OL-24187-04

Chapter 3 Basic Installation of the Cisco Collection Manager Starting the Cisco Collection Manager

Starting the Cisco Collection ManagerTo start the Cisco Collection Manager:

Step 1 As the scmscm user, run the cm start command.

$ ~scmscm/cm/bin/cm start

Step 2 Wait for 1 to 2 minutes to ensure that all the database tables have been created.

You can check that all the data tables were created by running the following command:

$ ~scmscm/scripts/dbtables.sh --nf

For further information about data tables, see the Chapter 5, “Managing Databases and the Comma Separated Value Repository.”

Setting the Time ZoneIt is necessary to set the time zone for the Cico Collection Manager to the same the time zone as the Cisco ASR 1000 Series router.

To set the time zone:

Step 1 Use the jselect-sce-tz.sh script to set the Cisco Collection Manager time zone.

For example, if the Cisco ASR 1000 Series router is located in GMT+2, run the following command as the scmscm user:

$ ~scmscm/cm/bin/jselect-sce-tz.sh --nf --offset=120


OL-24187-04

Chapter 3 Basic Installation of the Cisco Collection Manager Starting the Bundled Sybase Database

Starting the Bundled Sybase DatabaseTo start the bundled Sybase database:

Step 1 As the root user, run the sybase start command.

# ~scmscm/setup/sybase start

Step 2 Wait for several minutes and run the alive.sh script.

# ~scmscm/setup/alive.sh

Note Make sure that the output does not contain the phrase Sybase not functioning.

Note For further information about starting the bundled Sybase database, see the Managing Databases and the Comma Separated Value Repository chapter.

Note If you are using an external database, start it according to the instructions supplied by the database vendor.

Configuring the External MySQL Server You can configure the Cisco Collection Manager to operate with an external server operating MySQL Server 4.1, 5.0, or later versions. You must log into MySQL Server and manually create a schema and a user with necessary privileges before configuring the Cisco Collection Manager.

Log into the server using root privileges and execute the following MySQL commands to create pqb_admin with root privileges:

create database avocado; create user 'pqb_admin' identified by 'pqb_admin';grant all privileges on *.* to 'pqb_admin'@'localhost' identified by 'pqb_admin' with grant option;grant all privileges on *.* to 'pqb_admin'@'<CM_server_IP>' identified by 'pqb_admin' with grant option;grant all privileges on *.* to 'pqb_admin'@'%' identified by 'pqb_admin' with grant option;

Note Redefine the user privileges to restrict the user access.

Router ConfigurationTo generate reports from the Cisco Collection Manager database, it is necessary to configure the Cisco ASR 1000 Series router to send NetFlow data records to the Cisco Collection Manager. For information on the router configuration, see the Cisco Application Visibility and Control Solution Guide.


OL-24187-04

Chapter 3 Basic Installation of the Cisco Collection Manager Uninstalling the Sybase Database and Cisco Collection Manager Software

Uninstalling the Sybase Database and Cisco Collection Manager Software

The following sections describe how to uninstall the Sybase database and the Cisco Collection Manager:

• Uninstalling Sybase, page 3-18

• Uninstalling the Cisco Collection Manager Software, page 3-18

Uninstalling SybaseTo uninstall the Sybase database, perform the following steps:

Step 1 Log in as the root user.

Step 2 Uninstall Sybase.

Change the directory to install-scripts under the distribution kit root directory, and enter:

./uninstall.sh --sybase

Step 3 Edit /etc/system (for Solaris) or /etc/sysctl.conf (for Red Hat Linux) and remove the Sybase shmem setting.

Uninstalling the Cisco Collection Manager SoftwareTo uninstall the Cisco Collection Manager software, perform the following steps:

Step 1 Log in as the root user.

Step 2 Uninstall the Cisco Collection Manager software.

Change the directory to install-scripts under the distribution kit root directory, and enter:

./uninstall.sh --cm


OL-24187-04

Chapter 3 Basic Installation of the Cisco Collection Manager Upgrading the Cisco Collection Manager

Upgrading the Cisco Collection Manager

Note Cisco Collection Manager Version 3.7.x is the earliest version that supports the Cisco ASR 1000 Series router.

To upgrade the Cisco Collection Manager version, perform the following steps:

Step 1 Stop the Cisco Collection Manager.

Step 2 Install the new Cisco Collection Manager using the install-cm.sh script.

When you upgrade, use the -o option to preserve the existing configuration.

Use the scmscm user.

After the upgrade, when the Cisco Collection Manager comes up for the first time, the new database tables are automatically created.

Note In the Cisco Collection Manager Release 3.7.5 and later, all NetFlow tables have been altered to store the UTC values. The user has to run the ~scmscm/scripts/upgradeNFtables.sh --utc script option manually when the user upgrades from 3.7.2 or lower version to 3.7.5 or higher version. This script should be invoked only once with proper argument.


OL-24187-04

Chapter 3 Basic Installation of the Cisco Collection Manager Upgrading the Cisco Collection Manager


OL-24187-04


C H A P T E R 4
Managing the Collection Manager
IntroductionThis chapter explains how to use utility scripts to view and update Collection Manager (CM) parameters.

Any machine connected to the Collection Manager through, for example, Telnet or SSH, can use utility scripts to monitor and manage the Collection Manager. The utility scripts are located in the installation directory of the Collection Manager.

For information on managing the database and the CSV repository, see Chapter 5, “Managing Databases and the Comma Separated Value Repository.”

• Using Utility Scripts, page 4-2

• Collection Manager Support Information, page 4-2

• Configuring the Collection Manager, page 4-3

• Monitoring System Health, page 4-7

• Monitoring the Collection Manager, page 4-9


Chapter 4 Managing the Collection Manager Using Utility Scripts

Using Utility ScriptsGeneral instructions for using utility scripts include:

• To invoke any script, log in as the scmscm user, except where otherwise noted. If you attempt to run these scripts as the root user, the script generates an error.

• To display a description of the script, with an explanation of all flags and parameters, invoke the script with the help flag.

Note Specify the help flag differently depending on the script. To manage the Collection manager, specify -help. To manage the database, specify -h. See the definition for each script.

The following example shows how to display a description of the dbperiodic.sh script:

>~scmscm/scripts/dbperiodic.sh --help Usage:~scmscm/scripts/dbperiodic.sh --nf --loadload configuration from/export/home/scmscm/db_maint/nf-dbperiodic.conf~scmscm/scripts/dbperiodic.sh --nf --loadfile=FILEload configuration from FILE~scmscm/scripts/dbperiodic.sh --dumpprint the current configuration in INI format to standard output~scmscm/scripts/dbperiodic.sh --helpprint this help message

Collection Manager Support InformationIf you are experiencing difficulties with the Collection Manager installation, you might need to provide the Cisco Technical Assistance Center (TAC) with information about the current system setup. The Collection Manager can create the required support information or files for TAC by running one of the following scripts:

• ~scmscm/unsupported/getinfo/get_cm_info.sh—This script creates the Collection Manager support information.

• ~scmscm/unsupported/getinfo/get_support_files.sh—This script creates a support file.

The output is a zip file with the investigation supporting files. The following example shows the output that results from running this script:

bash-2.05$ ~scmscm/unsupported/getinfo/get_support_files.sh Gathering support files into Sun_14_Oct_2007_06-05-03_PM_JST.zip, please wait................done.The generated support file contains the output of ~scmscm/unsupported/getinfo/get_cm_info.sh


OL-24187-04

Chapter 4 Managing the Collection Manager Configuring the Collection Manager

Configuring the Collection ManagerUse utility scripts to:

• Specify which servers to activate at startup.

• Start or stop the database.

• Start or stop an adapter.

• Drop a Cisco ASR 1000 Series connection.

Use the following scripts to configure the Collection Manager:

• ~scmscm/setup/on-boot.sh

• ~scmscm/scripts/nf-adapterconf.sh

• ~scmscm/scripts/dbconf.sh

For information about scripts for managing the database and the CSV repository, see Chapter 5, “Managing Databases and the Comma Separated Value Repository.”

Use the following files to configure the Collection Manager:

• nf.conf—General configuration of the Collection Manager, including the adapters to turn on when the Collection Manager starts. See the “Enabling Adapters” section on page 4-4.

• nf-queue.conf—Configuration of the adapter queues, including the tags that are associated with a specific adapter. See the “Monitoring System Health” section on page 4-7.

• Activating Servers, page 4-3

• Controlling Adapters, page 4-4

• Enabling Adapters, page 4-4

• Configuring Databases, page 4-5

• Managing INI values for NetFlow, page 4-6

Activating ServersTo specify the servers (Collection Manager or Sybase) that are activated at startup, use the on-boot.sh script:

~scmscm/setup/on-boot.sh --cm=flag --sybase=flag

Changes take effect the next time the system restarts.

Note To view the current startup status of each component, run the script with no parameters.

To restart the Collection Manager, run the following script as the scmscm user:

~scmscm/cm/bin/cm restart_nf

Table 4-1 on-boot.sh Options

--cm={ on | off } Activate or do not activate the Collection Manager at startup.

--sybase={ on | off } Activate or do not activate the Sybase server at startup.


OL-24187-04

Chapter 4 Managing the Collection Manager Controlling Adapters

The following example shows how to set the Collection Manager and Sybase servers to run at startup. (This setting is the default setting of the script.)

>~scmscm/setup/on-boot.sh –-cm=on –-sybase=on

Controlling AdaptersTo shut down or activate a configured adapter, use the adapterconf.sh script. You can use the adapterconf.sh script to list the Collection Manager adapters that are currently running:

~scmscm/scripts/nf-adapterconf.sh --op=action [ --adapter=adapter name ]

To shut down an adapter, as the scmscm user, run the following script:

~scmscm/scripts/nf-adapterconf.sh --op=stop –-adapter=adapter name

To activate an adapter, as the scmscm user, run the following script:

~scmscm/scripts/nf-adapterconf.sh --op=start –-adapter=adapter name

Enabling AdaptersTo configure an adapter to turn on when the Collection Manager starts, remove the remark character at the start of the appropriate line in the cm.conf file.

The following example shows how to configure the CSV adapter to remain off when the Collection Manager starts.

#adapter.2 = com.cisco.scmscm.netflow.adapters.CSVAdapter

Note The value of the adapter.<number> must match the adapter_id parameter value configured in the queue.conf file for the corresponding adapter.

Table 4-2 adapterconf.sh Options

--op=start Activate the adapter specified by the adapter parameter.

--op=stop Shut down the adapter specified by the adapter parameter.

--op=list List the Collection Manager adapters that are currently running.

adapter=adapter name Identify the adapter you want to control. Use only with the start and stop options.

--help Display these options.


OL-24187-04

Chapter 4 Managing the Collection Manager Configuring Databases

Configuring DatabasesTo configure a database, run the dbconf.sh script:

~scmscm/scripts/dbconf.sh --nf

The script prompts you to choose the database type and the corresponding database configuration parameters.

The following example shows how to run the dbconf.sh script to configure the Collection Manager.

$ ~scmscm/scripts/dbconf.sh --nfEnter the DB type:1 - Oracle2 - MySQL3 - Sybase (external not bundled)4 - exitEnter your choice: 1Enter Oracle server host (current is localhost) :10.56.216.80Enter Oracle server listening port (current is 1521) :Enter Oracle server instance id (current is avocado) :Enter CM schema user name (current is pqb_admin) :Enter CM schema user password (current is pqb_admin) :Do you want to test the DB connection? (yes/no): yesPASS:db is upDB connection succeeded.$

Note Restart the Cisco Collection Manager for the change to take place.

Note By default, the Collection Manager handles database related transactions in uppercase table names. However, in a Windows based MySql server, table names are stored in lowercase and name comparisons are not case-sensitive; and, it automatically converts all table names to lowercase. If you are using a Windows based database server, you need to modify the lower_case_table_names server configuration entry.


OL-24187-04

Chapter 4 Managing the Collection Manager Managing INI values for NetFlow

Managing INI values for NetFlowThe Cisco Collection Manager automatically updates the INI_VALUES for a specific Cisco ASR 1000 Series Router with the option template values received from that Cisco ASR 1000 Series Router. A script is included in the Cisco Collection Manager distribution that enables you to manage INI_VALUES for a specific Cisco ASR 1000 Series Router.

In Cisco Collection Manager 3.7.0, the application attributes mapping in NF_INI_VALUES table is populated as y or n format. This format is changed to Yes or No in Cisco IOS XE 3.5 version onwards. While upgrading the Cisco Collection Manager 3.7.0 to later versions along with Cisco IOS XE 3.5, the NF_INI_VALUES will contain both the y/n and Yes/No format data for the Cisco ASR 1000 Series router. This results in duplication while generating reports.

The duplicate entries with (y/n) format needs to be removed from NF_INI_VALUES table by using the below option.

~scmscm/cm/bin/updateNetFlowMap.sh --nf=<< ASR1K IP >> --rem=[value_type]

Example:

~scmscm/cm/bin/updateNetFlowMap.sh --nf=1.1.1.1 --rem=114

This will remove the entries with the value_type as 114 and ip as 1.1.1.1 and value_key as either y or n

~scmscm/cm/bin/updateNetFlowMap.sh --nf=1.1.1.1 --rem=114-119

This will remove the entries with the value_type from 114 to 119 and ip as 1.1.1.1 and value_key as either y or n


OL-24187-04

Chapter 4 Managing the Collection Manager Monitoring System Health

Monitoring System HealthThe Collection Manager contains a small, expandable framework that monitors the system and issues alerts for predefined, potentially problematic conditions.

Use the following scripts to monitor the Collection Manager:

• ~scmscm/setup/monitor/setup-monitor.sh

• ~scmscm/setup/monitor/monitor.sh

Installing the Periodic CheckerTo insert (or remove) an entry for monitor.sh, the periodic checker script, in the cron (periodic scheduler) subsystem, run the setup-monitor.sh script:

~scmscm/setup/monitor/setup-monitor.sh -a flag [ -i flag ]

The following example shows how to install monitor.sh so that it runs every 30 minutes.

$ ./setup-monitor.sh -a install -I 30m

The following example shows how to uninstall monitor.sh.

$ ./setup-monitor.sh -a uninstall

Periodic Checker ScriptThe following sections provide information on the periodic checker script:

• Periodic Checker Script, page 4-7

• Tests, page 4-8

Periodic Checker Script

The periodic checker script, monitor.sh, calls a series of subscripts that monitor different aspects of a running system:

~scmscm/setup/monitor/monitor.sh { -a | TEST NAME } [ -v ] [ -d ]

The script is not intended to be run from the command line, although you can do so. The script sends test results to the syslog subsystem and logs the results in the file /var/log/messages.

Table 4-3 setup-monitor.sh Options

-a { install | uninstall } Insert or remove an entry for monitor.sh in the cron.

-I { 30m | 1h | 12h | 24h } Run monitor.sh every 30 minutes, 1 hour, 12 hours, or 24 hours.


OL-24187-04

Chapter 4 Managing the Collection Manager Periodic Checker Script

Any test returns a result in the following format:

STATUS: Message

• STATUS—PASS or FAIL

• Message—A short informative status message

For example, FAIL: db "avocado" has only 1523 free blocks

The following example shows how to run all available tests and print system output to the window:

$ ./monitor.sh -d -aTest: 01free_db.sh. Status: PASS. Message: db avocado has 1532 free blocksTest: 02cm_is_up.sh. Status: FAIL. Message: cm process is not running

The following example shows how to run one test to check that the installed database has sufficient free space:

$ ./monitor.sh -d free_db Test: 01free_db.sh. Status: PASS. Message: db avocado has 1532 free blocks

Tests

You can run the following tests using monitor.sh:

• db_up—Checks that the Collection Manager database is running.

• cm_up—Checks that the Collection Manager application is running.

• free_db—Checks that the bundled Sybase database has at least 10-percent free space.

• free_log—Checks that the bundled Sybase database transaction log has at least 70-percent free space.

• cm_persistent_buffers—Checks that the persistent buffer of each Collection Manager adapter contains less than 500 files.

The scripts for all these tests are located in the ~/setup/monitor/tests directory.

When calling a test called test_name, the script expects to find a file called NNtest_name.sh, where NN is a number that denotes script priority. For example, the test free_db is mapped to the file 01free_db.sh.

Table 4-4 monitor.sh Options

-a Run all tests.

TEST NAME The names of one or more tests. A test name is the test filename, without the leading digits and trailing .sh.

-v Output results in verbose mode. (Log successful tests.)

-d Print results to window. (By default, results are sent to syslog.)


OL-24187-04

Chapter 4 Managing the Collection Manager Monitoring the Collection Manager

Monitoring the Collection ManagerYou can use scripts to monitor system statistics that are relevant to the Collection Manager, such as:

• Percentage of free space in the database

• Cisco ASR 1000 Series platform connection data

• Viewing database insertion rate statistics per table

• Viewing version information

Use the following scripts to monitor the Collection Manager:

• ~scmscm/scripts/dbfree.sh

• ~scmscm/scripts/nf-rate.sh

• ~scmscm/setup/alive.sh

• ~scmscm/cm/bin/cm version

• ~scmscm/cm/bin/cm dbversion

Use the following scripts to configure the Collection Manager (see the “Configuring the Collection Manager” section on page 4-3). You can also run these scripts to display the relevant configuration:

• ~scmscm/setup/on-boot.sh

• ~scmscm/scripts/nf-adapterconf.sh

Checking the Database CapacityTo display the percentage of free space in the database report tables and the associated transaction log, run the dbfree.sh script:

~scmscm/scripts/dbfree.sh --nf

Use the script only with bundled Sybase and MySQL databases.

Step 1 Run the dbfree.sh script.

Verifying Server OperationTo verify that the server is functioning correctly, run the alive.sh script:

~scmscm/setup/alive.sh

The script verifies that the following components are operational:

• Collection Manager

• Database (in the bundled database case)

• Report tables (in the bundled database case)

If any component is down, the script issues an error message.

Step 1 As the user, run the alive.sh script


OL-24187-04

Chapter 4 Managing the Collection Manager Viewing Version Information

Note It takes time for the components to initialize after a startup. After a restart, wait 5 minutes before running this script.

Viewing Version InformationThe Collection Manager includes two scripts to display the current version of the Collection Manager and of the database:

• cm version

• cm dbversion

The following example shows how to view the current Collection Manager version:

$ ~scmscm/cm/bin/cm versionCM CD Version 3.7.5 Build 104

The following example shows how to view the current database version:

$ ~scmscm/cm/bin/cm dbversionCM:NF DatabaseSybase ase150/


OL-24187-04


C H A P T E R 5
Managing Databases and the Comma Separated Value Repository
IntroductionThis chapter explains how to use utility scripts to manage the Collection Manager (CM) database and the comma separated value (CSV) repository.

Note The Cisco Collection Manager is able to operate with both the Cisco ASR 1000 Series and the Cisco SCE platform at the same time. A separate database is normally created for each platform. Refer to the Cisco Service Control Management Suite Collection Manager User Guide for information on the Cisco SCE platform databases.

Note For instructions on using utility scripts, see the “Using Utility Scripts” section on page 4-2.

Note By default, the Collection Manager handles database related transactions in uppercase table names. However, in a Windows based MySql server, table names are stored in lowercase and name comparisons are not case-sensitive; and, it automatically converts all table names to lowercase. If you are using a Windows based database server, you need to modify the lower_case_table_names server configuration entry.

• Common Database Management Tasks, page 5-2

• Managing the Bundled Sybase Database, page 5-8

• Managing the External MySQL Database, page 5-10

• Partitioning the MySQL Database, page 5-11

• Managing the Oracle Database, page 5-13

• Aggregating Database Data, page 5-14

• Managing the CSV Repository, page 5-18


Chapter 5 Managing Databases and the Comma Separated Value Repository Common Database Management Tasks

Common Database Management TasksThe following database management tasks apply to all the supported databases:

• Generating a list of the database tables

• Defining and applying the schedule for the periodic deletion of old records

• Deleting a table

• Manually deleting old records from a table

Use the following scripts to configure and maintain the database:

• ~scmscm/scripts/dbtables.sh

• ~scmscm/scripts/dbperiodic.sh

• ~scmscm/db_maint/create_periodic_del_procs.sh

• ~scmscm/scripts/droptable.sh

• ~scmscm/scripts/prunetable.sh

• ~scmscm/scripts/sybutfsupport.sh

• ~scmscm/scripts/partitions.sh

• ~scmscm/scripts/createaggevents.sh

Listing Database TablesTo display a list of all of the tables in the database, run the dbtables.sh script:

~scmscm/scripts/dbtables.sh --nf -c

Where applicable, the script displays the number of lines in the table and the earliest and latest timestamps.

Table 5-1 lists the dbtables.sh --nf -c script options.

Table 5-1 dbtables.sh --nf Options

Option Description-l Lists the existing table names only (without statistics).

-a Includes the nonreport tables in the listing.

-c Lists the existing table names with their record counts.

-f Enables fast line counting, using the client rather than JDBC.

-t { sec_num } The maximum waiting time, in seconds, for the response. The default is no timeout.

-h Prints this help message and exits.


OL-24187-04

Chapter 5 Managing Databases and the Comma Separated Value Repository Managing the CM:NFR Periodic Deletion of Old Records

The dbtables.sh script generates the following sample output:

>~scmscm/scripts/dbtables.sh --nfExecuting query ...

name| num_lines| min_time| max_time|----------------+--------+------------------------+------------------------+RPT_USAGE_NF| 4| 2011-03-24 10:07:40.0|| 2007-02-15 2011-03-24 13:44:13.0|RPT_TRANSACTION_NF| 110367| 2011-03-24 09:37:53.0| 2011-03-24 13:44:13.0

Managing the CM:NFR Periodic Deletion of Old RecordsTo manage the periodic deletion of old records, perform the following steps:

1. If they were not installed during the Collection Manager installation, install the periodic delete procedures:

Log in as the scmscm user, start the Collection Manager, wait 1 to 2 minutes for the creation of the database tables, and then run the script:

~scmscm/db_maint/create_periodic_del_procs.sh --nf

2. Edit the periodic delete configuration file. See the “Configuring Periodic Delete” section on page 5-4.

3. To apply the new configuration, run the dbperiodic.sh utility script. See the “Applying the Periodic Delete Configuration File” section on page 5-5.

Caution If you enable both aggregation and the periodic delete feature, be sure to increase their respective config values.

Periodic deletion of a table does not begin while a previous periodic deletion is still running. This prevents excessive load on the database, which would degrade insertion performance in the adapters.

When two or more tables are scheduled to be reduced at the same time, the tables are processed in the order in which they are listed in the periodic delete configuration file.

For ease of configuration, you can schedule periodic deletion for all tables consecutively on one schedule.

All periodic delete activity is recorded in the system log file. For Solaris, the system log file is located in the /var/adm/messages directory; for Linux, the system log file is located in the /var/log/messages directory. The following is an example of the system log file at the time when the periodic delete action occurs.

Feb 17 05:00:07 app-redhat79 logger: /opt/scmscm/db_maint/periodic_del.sh for hourly all - running single del for RPT_USAGE_NF to 14 daysFeb 17 05:00:09 app-redhat79 logger: Starting periodic delete for RPT_USAGE_NF keeping 14 daysFeb 17 05:00:09 app-redhat79 logger: Deleted rows: 0 from the table RPT_USAGE_NF

Caution Be sure to disable the MySQL partitioning process in the Collection Manager when you enable the periodic delete feature.

• Configuring Periodic Delete, page 5-4

• Applying the Periodic Delete Configuration File, page 5-5


OL-24187-04


Configuring Periodic Delete

By default, the periodic delete configuration file (nf-dbperiodic.conf) is located in the directory ~scmscm/db_maint/. The file has a structure similar to an INI file, where each section describes a particular data reduction operation for a specific set of tables, to be performed according to a specified schedule.

Note The name of each section of the file is not used when the file is parsed. Use any desired name.

Each section begins with the section name in brackets, and contains the parameters shown in the following table. (Not all parameters are required in each section of the configuration file.) Separate the parameters and their values by an equal sign (=). Table 5-2 contains examples of periodic delete configuration files.

Note Values for all parameters except active and daystokeep can be either a single value, a list of values separated by commas, a range of values (two values separated by a dash), or an asterisk (*) which signifies all possible values. A range is not possible for tablenames.

In the following example, all fields are set to their default values.

# This nf-dbperiodic.conf file emulates the legacy style for periodic# deletion. All tables are processed every hour on the hour, and# records are kept for 30 days.[4 hourly 30 days]active = truetablenames = RPT_USAGE_NF,RPT_TRANSACTION_NF,RPT_GLB_USAGE_NF daystokeep = 14minute = 0hour = 0,4,8,12,16, 20

Table 5-2 Parameters in the Periodic Delete Configuration File

Parameter Name Explanation Values Default Example

active Specifies whether or not to use this section of the configuration file

true/false true false

tablenames Names of the tables to which this section applies

Names of tables separated by commas, or * for all tables

* (all) RPT_USAGE_NF

daystokeep Number of days to keep records

Positive integers 14 30

minute

hour

day

month

The time at which to perform the deletion in this section of the configuration file

0 … 59, *

0 … 23, *

1 … 31, *

1 … 12, *

30

(all) *

(all) *

(all)

0

0,4,8,12,16,20

1

1,3,5,7,9,11


OL-24187-04


In this example, all tables are reduced at 4:30 A.M., leaving 10 days of data in each table. In addition, the real-time tables are reduced every hour, leaving three days of data in each table.

# This nf-dbperiodic.conf file reduces all tables once a day and# real-time tables once an hour.active = truetablenames = *daystokeep = 10minute = 30hour = 4[hourly real-time]active = truetablenames = RPT_USAGE_NF,RPT_TRANSACTION_NF,RPT_GLB_USAGE_NFdaystokeep = 3minute = 0hour = *

Applying the Periodic Delete Configuration File

To load and apply a new periodic delete configuration file or to view the current file, run the dbperiodic.sh script:

~scmscm/scripts/dbperiodic.sh --nf[ --dump ] [ --load | --loadfile=path_to_dbperiodic.conf ]

When the script is used to load a new configuration file, it parses the file, verifies its validity, and updates the scmscm user's crontab to reflect the changes. Table 5-3 lists the dbperiodic.sh options.

The following example shows how to print the current periodic delete configuration:

~scmscm/scripts/dbperiodic.sh --dump

Note This script prints the loaded periodic delete configuration. If the current periodic delete configuration file is not yet loaded, the configuration can vary from the script output.

The following example shows how to load the periodic delete configuration file from ~scmscm/db_maint/dbperiodic.conf:

~scmscm/scripts/dbperiodic.sh --nf --load

Table 5-3 dbperiodic.sh Options

Option Description

--load Load the periodic delete configuration from /export/home/scmscm/db_maint/ nf-dbperiodic.conf

--loadfile=path to periodic delete configuration file

Load the periodic delete configuration file from the specified directory

--dump Print the periodic delete configuration

--h Display these options


OL-24187-04

Chapter 5 Managing Databases and the Comma Separated Value Repository Deleting a Table

Deleting a TableTo delete one table or all current tables from the database, run the droptable.sh script:

~scmscm/scripts/droptable.sh --nf [ -f ] tableParameter

Table 5-4 lists the droptable.sh options.

The following example shows how to drop a CM:NFR table named RPT_USAGE_NF from the database with no request for confirmation:

~scmscm/scripts/droptable.sh --nf -f RPT_USAGE_NF

The following example shows how to drop all tables from the database:

~scmscm/scripts/droptable.sh --nf ALLTABLES

Deleting Old RecordsTo remove records from a database table based on the timestamps of the records, run the prunetable.sh script:

~scmscm/scripts/prunetable.sh --nf [ -f ] num_days table_name

Table 5-5 lists the prunetable.sh options.

The following example shows how to delete all records that are more than seven days old from a table named RPT_USAGE_NF.

Table 5-4 droptable.sh Options

Option Description

table_name Drop table_name from the database

ALLTABLES Drop all tables from the database

-f Drop by force (no questions asked or errors reported)

-h Display these options

Table 5-5 prunetable.sh Options

Option Description

num_days The maximum age (in days) of records not to deleted.

table_name The table for which to delete records

-f Drop by force (no questions asked or errors reported).

-h Display these options.


OL-24187-04

Chapter 5 Managing Databases and the Comma Separated Value Repository UNICODE Support for Sybase Database

Because the –f flag is not specified, there can be requests for confirmation and errors are reported.

>~scmscm/scripts/prunetable.sh --nf 7 RPT_USAGE_NF

UNICODE Support for Sybase DatabaseTo add UTF support for the Sybase database, run the following script:

~scmscm/scripts/sybutfsupport.sh


OL-24187-04

Chapter 5 Managing Databases and the Comma Separated Value Repository Managing the Bundled Sybase Database

Managing the Bundled Sybase DatabaseManaging the bundled Sybase database includes:

• Backing up and restoring a database

• Updating Sybase with a changed IP address

• Updating the Sybase database statistics

The Cisco Collection Manager adds a timestamp to each NetFlow Record it receives and stores in the database. This timestamp is used when various maintenance operations are performed on the database tables.

Use the following scripts to maintain the bundled Sybase database only:

• ~scmscm/scripts/sybback.sh

• ~scmscm/scripts/sybrestore.sh

• ~scmscm/db_maint/update_statistics.sh

Note The Sybase back up and restore options require that the Sybase database resides on the same server as the Cisco Collection Manager.

Backing up the Sybase DatabaseTo create text file backups of all the tables in the database, run the sybback.sh --nf script:

~scmscm/scripts/sybback.sh --nf path_to_backup_directory

The script converts all tables to ASCII files and copies the files to a backup directory. Table 5-6 lists the sybback.sh --nf options.

Note Modify the path if you want to change the default location. The destination location path is defined as: ~scmscm/cm/config/dbpacks/sybase/ase12.5.1/sybase.properties file.

Restoring a Sybase Database To restore a database from the backup file created by the sybback.sh script, run the sybrestore.sh script:

~scmscm/scripts/sybrestore.sh --nf

Table 5-7 lists the sybrestore.sh options.

Table 5-6 sybback.sh Options

Option Description



OL-24187-04

Chapter 5 Managing Databases and the Comma Separated Value Repository Updating Sybase with a Changed IP Address

Note The scripts sybback.sh and sybrestore.sh are not a viable backup mechanism for Sybase. They are designed for backing up and restoring small amounts of data, for example, transferring small tables between machines.

Note If you require a viable backup mechanism, consult the Sybase Backup Server product documentation.

Updating Sybase with a Changed IP AddressWhen you change the IP address, update the Sybase server. As the root user, issue the following command:

~scmscm/setup/syb_interfaces.sh

Updating Sybase Database StatisticsRun the update_statistics.sh script once a week at a time of low database load, for example, at night.

~scmscm/db_maint/update_statistics.sh

Table 5-7 sybrestore.sh Options

Option Description



OL-24187-04

Chapter 5 Managing Databases and the Comma Separated Value Repository Managing the External MySQL Database

Managing the External MySQL DatabaseManaging the MySQL database includes:

• Backing up and restoring a database

The Cisco Collection Manager adds a timestamp to each NetFlow Record it receives and stores in the database. This timestamp is used when various maintenance operations are performed on the database tables.

Use the following scripts to maintain the MySQL database:

• ~scmscm/scripts/mysqlback.sh

• ~scmscm/scripts/mysqlrestore.sh

Note The MySQL back up and restore options require that the Sybase database resides on the same server as the Cisco Collection Manager.

Backing up the MySQL DatabaseTo create text file backups of all the tables in the database, use the mysqlback.sh script:

~scmscm/scripts/mysqlback.sh --nf

The backup files are stored in a SYBASE directory under the Cisco Collection Manager's home directory. The option to specify a directory path is not supported.

The script converts all tables to ASCII files and copies the files to a backup directory. Table 5-8 lists the mysqlback.sh options.

Restoring a MySQL Database To restore a database from the backup file created by the mysqlback.sh script, run the mysqlrestore.sh script:

~scmscm/scripts/mysqlrestore.sh --nf

The script converts all tables to ASCII files and copies the files to a backup directory. Table 5-9 lists the msqlrestore.sh options.

Table 5-8 mysqlback.sh Options

Option Description

--nf Write backup CM:NF MySQL files

--help Displays these options

Table 5-9 mysqylrestore.sh Options

Option Description

--nf Restore the CM:NF database using the sql files in the specified directory

--help Displays these options


OL-24187-04

Chapter 5 Managing Databases and the Comma Separated Value Repository Partitioning the MySQL Database

Partitioning the MySQL DatabaseYou can store records in an external MySQL database. When you configure MySQL as the database, partitioning is supported.

• Prerequisites, page 5-11

• Configuring CM:NF Partitioning, page 5-11

PrerequisitesThe partitioning feature is supported only when the MySQL server meets the following conditions:

• MySQL version 5.1.06 or later

• thread_stack value in my.cnf file should be set to at least 256k under the [mysqld] section

Restart the MySQL server to enable the configuration changes to take effect.

Configuring CM:NF Partitioning

Caution If a partition is created on the existing data, all existing data is deleted . Back up existing data before running the partitioning script.

The default configuration file (nf-partitions.conf) is located in the ~scmscm/cm/config/dbpacks/mysql/4.0.20/partitions/netflow/directory.

The following is a sample of the configuration file:

[partitions_config1]table_names = RPT_TRANSACTION_NF,RPT_USAGE_NF,RPT_GLB_USAGE_NF no_of_partition = 30 period = 1 scheduler_time = 23:3

Each section begins with a unique section name starting with partitions_config in brackets and contains the parameters shown in Table 5-10. Separate the parameters and their values by an equal sign (=).

Table 5-10 Parameters in the Partitioning Configuration File


table_names Name of the table to partition.

Should start with RPT_ and end with_NF.. Should not be duplicated across sections.

— RPT_USAGE_NF

RPT_GLB_USAGE_NF

no_of_partition The number of partitions to create in the database.

1 - 65 — 30


OL-24187-04

Chapter 5 Managing Databases and the Comma Separated Value Repository Configuring CM:NF Partitioning

To create and view partitions, run the partitions.sh script:

~scmscm/scripts/partitions.sh --nf [ --create ] [ --non_partitioned ] [ --partitioned ] [--partitions [table_name]] [events][ --help ]

Table 5-11 lists the partitions.sh options.

Perform the following steps to enable the partitioning feature:

1. Log in as scmscm user.

2. Change directory to ~scmscm/scripts/.

3. Customize the ~scmscm/cm/config/dbpacks/mysql/4.0.20/partitions//netflow/nf-partitions.conf file based on your requirements.

4. Run ~scmscm/scripts/loadsqlprocedures.sh --nf --par to load the required support files for partitioning.

5. Run ~scmscm/scripts/partitions.sh --nf --create to create the partitioning.

Note Ensure that the periodic deletion process is disabled in Collection Manager when you enable the partitioning feature.

period The number of days that the records are retained in each partition.

1 - 5 — 1

scheduler_time The time at which to trigger the daily partitioning event scheduler in the database.

Hour and minutes. Format: HH:MM

— 23:30

Table 5-10 Parameters in the Partitioning Configuration File (continued)


Table 5-11 partitions.sh Options

Option Description

--create Create partitions for database tables that are not partitioned

--non_partitioned Show details for database tables that are not partitioned

--partitioned Show details for database tables that are partitioned

--partitions [table_name] Show all table partitions details. [table_name] parameter shows mentioned table partitioned details.

--events Show configured partitioning events details

--help Display these options


OL-24187-04

Chapter 5 Managing Databases and the Comma Separated Value Repository Managing the Oracle Database

During the creation of the partitions, the partitions.sh script performs the following actions:

• Enables the event scheduler on the MySQL server.

• If any of the tables (table_names) listed in the configuration file (partitions.conf) are not available in the database, partitioning is ignored for that particular table. The remaining tables are processed.

When the MySQL server is restarted after the partitioning, the event scheduler is disabled. To ensure the event scheduler remains enabled through a server restart, add “event_scheduler = 1” to the [mysqld] section of the my.cnf file.

Note If any missing table is created in the database later, run the partitions.sh script to create new partitioning. You can apply the same event scheduler for the new partitioning in each configuration section. This does not disturb the existing partitioning.

Note If any new tables are created in the database, define new configuration sections for these new tables and then run the partitions.sh script. This does not disturb the existing partitioning.

Although the partitioning event scheduler runs every day, it to creates or deletes a partition based on the configuration parameters no_of_partition and period. Even if the server is restarted, the partitioning event scheduler is not disturbed.

Managing the Oracle Database

Privileges The following are the mandatory Privileges that are to be granted to the CM-DB user: Create Session , Create Table, Create Procedure, Create Table Space, Unlimited Table Space, Create Index, Execute Any Procedure

Syntax:

grant CONNECT ,CREATE TABLE ,CREATE TABLESPACE ,CREATE PROCEDURE ,UNLIMITED TABLESPACE,CREATE ANY INDEX, EXECUTE ANY PROCEDURE to pqb_admin

Note Connect to the Oracle DB as SYSTEM/SYSDBA, log into the Oracle server with specific schema (apricot/avocado) and grant the above privileges:


OL-24187-04

Chapter 5 Managing Databases and the Comma Separated Value Repository Aggregating Database Data

Aggregating Database DataYou can aggregate database data according to varying time periods and store data in the same database tables. Both raw and aggregated data are stored in order to maintain compatibility with reporting tools that pull this data to generate analysis reports. All reports that process raw data for display also can process the aggregated data because the data structure is the same.

During the aggregation process, table rows are aggregated per time period. These values are placed in a temporary table. The source records are then removed to allow insertion of the aggregated records.

The aggregation feature is supported in the following CM:NetFlow tables:

• RPT_USAGE_NF

• RPT_GLB_USAGE_NF

Configuring CM:NetFlow AggregationThe aggregation configuration file ~scmscm/cm/config/nf-aggregation.conf specifies the values used in the aggregation process.

The following is an example configuration file:

[agg_event1]procedure_names = P_RPT_USAGE_NF_AGGREGATE,P_RPT_GLB_USAGE_NF_AGGREGATEno_of_days = 2hourly_agg_days = 14scheduler_time = 23:30

Each section in the aggregation configuration file begins with a unique section name in brackets ([]) starting with agg_event . The file contains the parameters shown in Table 5-12. Separate the parameters and their values with an equal sign (=).


OL-24187-04

Chapter 5 Managing Databases and the Comma Separated Value Repository Configuring CM:NetFlow Aggregation

Update the ~scmscm/cm/config/nf-aggregation.conf file to change the default configuration parameters and run the following script.

~scmscm/scripts/createaggevents.sh --nf [ --create ] [ --help ]

Table 5-13 lists the createaggevents.sh options.

Table 5-12 nf-aggregation.conf Parameters

Parameter Name Explanation Values Default Examples

procedure_names SQL procedure name to aggregate.

Any aggregation procedure name present in the respective database configuration location.

Sybase:

~scmscm/cm/config/dbpacks/

sybase/ase12.5.1/aggregation/netflow/

MySQL:


mysql/4.0.20/aggregation/netflow

Oracle:


oracle/9204e/aggregation/netflow

— P_RPT_USAGE_NF_AGGREGATE

and

P_RPT_GLB_USAGE_NF_AGGREGATE

no_of_days Number of days of raw data to retain in the table.

1 to 31 — 2

Hourly_agg_days Number of days of hourly aggregation to retain in the table.

14 — 14

scheduler_time The time at which to trigger the daily aggregation event scheduler.

Hour and minutes.

Format: HH:MM

— 23:30

Table 5-13 createaggevents.sh Options

Option Description--create Creates the corresponding aggregation events

--help Display these options


OL-24187-04

Chapter 5 Managing Databases and the Comma Separated Value Repository Enabling CM:NF Aggregation in Sybase

Note Use loadsqlprocedure.sh --rdr --agg option to update the config changes into the database.

The Sybase, MySQL, and Oracle databases support aggregation. The following sections describe how to enable the aggregation feature:

• Configuring CM:NetFlow Aggregation, page 5-14

• Enabling CM:NF Aggregation in MySQL, page 5-17

• Enabling CM:NF Aggregation in Oracle, page 5-17

Enabling CM:NF Aggregation in SybaseSybase versions 12 and 15 support data aggregation.

Complete the following steps to enable the aggregation feature:

1. Login as scmscm user.

2. Run ~scmscm/scripts/loadsqlprocedures.sh --nf --agg to load the required support files for aggregation.

Note Data aggregation is a Cisco Collection Manager optional feature. Make sure the Oracle user has sufficient privileges to alter and execute the aggregation events. For example: grant select ,insert ,delete ,update on sys.job$ to pqb_admin;


OL-24187-04

Chapter 5 Managing Databases and the Comma Separated Value Repository Enabling CM:NF Aggregation in MySQL

3. Create a cron job to run ~scmscm/cm/config/dbpacks/sybase/ase12.5.1/aggregation/netflow/cron/nf_run_aggregation.sh to schedule the aggregation process.

Cron job example:

crontab -e

Update the path and expected time to schedule the job:

30 23 * * * /<path>/nf_run_aggregation.sh

4. Refer to ~scmscm/cm/config/dbpacks/sybase/ase12.5.1/aggregation/cron/readme_run_aggregation_procedure.txt for more information.

Enabling CM:NF Aggregation in MySQLMySQL versions 5.1.06 and later support data aggregation.


1. Login as scmscm user.

2. Run ~scmscm/scripts/loadsqlprocedures.sh --nf --agg to load the required support files for aggregation.

Enabling CM:NF Aggregation in OracleOracle versions 9, 10, and 11 support data aggregation.


Step 1 Login as scmscm user.

Step 2 Run ~scmscm/scripts/loadsqlprocedures.sh --nf --agg to load the required support files for aggregation.

Note Ensure that the Oracle user has sufficient privileges to alter and execute the aggregation events (for example: GRANT SELECT, INSERT, DELETE ON sys.job$ TO pqb_admin).


OL-24187-04

Chapter 5 Managing Databases and the Comma Separated Value Repository Managing the CSV Repository

Managing the CSV RepositoryYou can run a utility script to manage the repository of CSV files output by the Collection Manager. the CSV adapter writes these files to disk for use by a service provider’s operational support system (OSS) or by a third-party billing system. To prevent disk overflow, monitor the size of the CSV repository.

Note If the backup parameter is set to true, failure to delete CSV files can result in disk overflow. No CSV files are ever deleted.

Note The third-party application is responsible for managing the CSV files and deleting them as necessary.

To successfully invoke this script, the HTTP adaptor of the Collection Manager must be running. If the adapter is down, an error message is printed.

• CM:NF CSV Repository File Structure, page 5-18

• CM:NF Configuring the CSV File Repository, page 5-18

• CM:NF Configuring the Comma Escape, page 5-20

• CM:NF Configuring Escape of Nonprintable Characters, page 5-20

CM:NF CSV Repository File StructureCSV files are stored in several subdirectories. Each subdirectory is given the number of a NetFlow Record (NFR) tag. NFR tags define the type of the NFR. Each NFR is stored in the subdirectory whose name matches its NFR tag number.

The CSV files are numbered sequentially (automatically), with separate numbering in each directory. You can change the location of the parent directory by editing the ~scmscm/cm/config/nf.conf file.

CM:NF Configuring the CSV File RepositoryUse the ~scmscm/scripts/nf-csvconf.sh script to:

• List the number of NFRs currently stored in the repository.

• Configure the maximum number of CSV files and the maximum permissible number of reports (lines) in each file.

• Control whether a backup occurs when an old CSV file is about to be overwritten.

• Control whether each line in a CSV file contains an indication of the IP of the Cisco ASR 1000 that sent this NFR. (By default, this option is off.)


OL-24187-04

Chapter 5 Managing Databases and the Comma Separated Value Repository CM:NF Configuring the CSV File Repository

Table 5-14 lists the nf-csvconf.sh options.

Note Instead of using this script, you can edit the ~scmscm/cm/config/nf-csvadapter.conf file. For changes in this file to take effect, you must restart the Collection Manager.

Note The same configuration is applied to all subdirectories in the CSV repository.

Note Setting these parameters does not change existing CSV files. Parameter value changes only affect files that are created subsequently.

The following example shows how to set the maximum number of CSV files per subdirectory to 1000.>~scmscm/scripts/nf-csvconf.sh --nf --maxfiles=1000

The following example shows how to set the maximum number records per CSV files to 10,000.

>~scmscm/scripts/nf-csvconf.sh --nf --maxlines=10000

The following example shows how to delete all files from the CSV repository:

~scmscm/scripts/nf-csvconf.sh --nf --clear

The following example shows how to disable backing up old CSV files in the repository:

~scmscm/scripts/nf-csvconf.sh –-backups=false

Table 5-14 nf-csvconf.sh Options

Option Description

--list Display the CSV repository contents (the number of NFRs currently stored in the repository).

--clear Delete all files from the CSV repository. (This option deletes all CSV files, but not the directories in which they are contained.)

--maxlines=N Set the maximum number of NFRs per CSV file to N (an integer between 1 and 20,000).

--maxfiles=M Set the maximum number of CSV files in each subdirectory to M (an integer between 10 and 10,000.)

--backups={true |false} Enable or disable backup of old CSV files.

--recordsource={true |false} Enable or disable the inclusion of the record source in CSV files.


OL-24187-04

Chapter 5 Managing Databases and the Comma Separated Value Repository CM:NF Configuring the Comma Escape

CM:NF Configuring the Comma EscapeWhen a comma is contained within a field in a CSV file, an escape sequence indicates that the comma does not mark the end of the field.

Three escape methods are supported:

• Single quotation marks—Single quotation marks surround any field that contains one or more commas. There is no special treatment of single quotation marks that currently exist in the NFRs.

• URL—%2C replaces each comma contained within a field. There is no special treatment of such sequences that currently exist in NFRs.

• Backslash—A backslash (\) precedes each comma contained within a field. There is no special treatment of backslashes that currently exist in NFRs.

The first two escape methods are compatible with Microsoft Excel. The backslash method is not compatible with Excel, but is retained for backward compatibility.

Single quotation marks are used by default. You can change the escape method by modifying the value of the escapeMethod attribute. This attribute is located in the ~/scmscm/cm/config/csvadapter.conf file. You must use one of the following values: backslash, quote, or url.

CM:NF Configuring Escape of Nonprintable CharactersThe CSV adapter can escape nonprintable characters. Enabling this option incurs a performance penalty for the adapter. By default, the option is disabled.

When the option is enabled, a backslash (\) precedes each character that cannot be printed, such as CR and LF, contained within a field.

Enable this option in the ~/scmscm/cm/config/nf-csvadapter.conf file. For changes to this configuration file to take effect, you must restart the Collection Manager.


OL-24187-04


C H A P T E R 6
Configuring Databases
IntroductionThis chapter describes how to configure the Collection Manager to work with your database. The chapter also describes how to use the database infrastructure to extend the functionality of the Collection Manager.

• Generating SQL Code Using the Velocity Template Language, page 6-2

• Database Configuration Files, page 6-3

• Working Sample, page 6-6

• Testing and Debugging, page 6-9

• Using the JDBC Framework in Scripts, page 6-10

• Scalability Hints for Oracle, page 6-12


Chapter 6 Configuring Databases Generating SQL Code Using the Velocity Template Language

Generating SQL Code Using the Velocity Template LanguageThe JDBC Adapter framework uses macros written in the Velocity Template Language (VTL) to generate all SQL code that is passed to the database server. The following sections describe the configuration file that controls the generation process.

For more information regarding VTL (which is part of the Apache Jakarta Project) go to http://jakarta.apache.org/velocity/vtl-reference-guide.html.

Table 6-1 describes VTL constructs:

Table 6-1 Summary of VTL Constructs

Directive Syntax Example Purpose#foreach #foreach ($item in $collection)

item is $item#end

Iterates over a collection, array, or map.

#if...#else...#elseif

#if ($order.total == 0)No charge#end

Conditional statement.

#parse #parse("header.vm") Loads, parses, and includes the specified template in the generated output.

#macro #macro(currency $amount)${formatter.currency($amount)}#end

Defines a new directive and any required parameters. The result is interpreted when used later in the template.

#include #include("disclaimer.txt") Includes the specified file, as is, in the generated output.

#set #set ($customer = ${order.customer}) Assigns a value to a context object. If the context object does not exist, it is added; otherwise, it is overwritten.

#stop #if ($debug) #stop #end Stops template processing.


OL-24187-04

Chapter 6 Configuring Databases Database Configuration Files

Database Configuration FilesWhen you initialize the Database access framework, the first file the Database access framework searches for is nf-main.vm, which contains definitions or pointers to all the required database SQL definitions. The location used to search for this file depends on the dbpack used in the Collection Manager. A dbpack is a collection of configuration files pertaining to a specific database installation. The adapter (according to its configuration file) selects the dbpack. The following code fragment from the nf-jdbcadapter.conf file configures it to work with an Oracle dbpack:

db_template_dir = dbpacks/oracle/9204e/db_template_file = nf-main.vm

Note The directory location is interpreted relative to the main Collection Manager configuration directory (~scmscm/cm/config).

To make the configuration more modular, the nf-main.vm file generally points to other files; however this mechanism is not strictly necessary. The files can contain arbitrary definitions that can be used later, for example, in scripts. Some definitions are mandatory because the JDBC adapter uses them for its operation. These definitions are listed in Table 6-2:

Some objects representing the Collection Manager configuration in the VTL parsing context can be used in the templates. These objects are described in the following sections.

• Context Objects, page 6-4

• Application Configuration, page 6-5

Table 6-2 Mandatory VM Definitions

Object Name Mandatory Definition

$table.sql.dropTable

$table.sql.createTable

$table.sql.createIndexes

$table.sql.insert

$table.sql.metaDataQuery

For each table, these settings control how SQL is generated for the indicated operation

$dbinfo.driverjarfile

$dbinfo.driver

Location and class name for JDBC driver

$dbinfo.cmdSeparator Pattern used to separate multiple SQL statements

$dbinfo.url

$dbinfo.connOptions

URL for connecting to the database Various connection properties


OL-24187-04

Chapter 6 Configuring Databases Context Objects

Context ObjectsBefore any Collection Manager components (for instance, the JDBC adapter or a script), load and parse the VM templates, the Collection Manager initializes the parsing context with the following Java objects:

• The tables object

• The dbinfo object

• The tools object

tables Object

The tables object describes application-related database configuration, such as the structure of NetFlow Records that are stored in the database, the structure of the database tables and where they are stored, and the structure of any other database tables that the Collection Manager uses. The object is an array in which each row represents one of the database tables used by the Collection Manager. For each table, the row can contain the following information (not all items are relevant to all tables):

• Logical name

• Physical name

• NetFlow Record tag associated with this table

• List of fields/columns in this table, with the following attributes for each:

– Field ID

– Field name

– Field native type

– Free-form field options

• List of indexes for this table, with the following attributes for each:

– Index name

– Names of columns indexed

– Free-form index options

The contents of the tables object can be inspected or manipulated when loading the templates. The tables object is initialized using the application-specific XML configuration file. See the “Application Configuration” section on page 6-5.

dbinfo Object

The dbinfo object describes configuration that is specific to the database, such as the parameters and the SID or schema that is used when opening a database connection. The object holds database-specific configuration options. The object contains the following information:

• The name of the JDBC class that is used as a driver for this database

• The name of the JAR file that contains the driver

• The location of the database expressed as a JDBC URL

• Free-form JDBC connection options, such as authentication data (user and password)


OL-24187-04

Chapter 6 Configuring Databases Application Configuration

tools Object

The tools object is a container for several utility methods that you can find useful when developing templates or manipulating the context data structures.

You invoke the object methods by using $tools.method(arg1, ..., argN), where method is the name of the method.

The methods are listed in Table 6-3:

For a sample that demonstrates how to use these tools, see the “Using the JDBC Framework in Scripts” section on page 6-10.

Application ConfigurationAll application-related configuration is contained in one file (nf-dbtables.xml), which includes the following items:

• Name and version of the application

• Name and properties of each database table, and specifically the structure of application NetFlow Records that are stored in database tables

• For each database table:

– Names and native types of the table/NetFlow Record fields

– Names and properties of the table indexes

This information is used primarily to populate the tables object in the template parsing context. See the “tables Object” section on page 6-4.

Table 6-3 Methods of the tools Object

Method Name and Arguments Function

getTableByName (allTables, name) Locates the database table object whose logical name corresponds to name.

getTableByDbTabName (allTables, name) Locates the database table object whose physical name corresponds to name.

assignParams (sql, list_of_args) Replaces question mark characters in the SQL string with consecutive elements from the list_of_args parameter, represented as a String. This method is useful if working with templates that create SQL insert statements using the JDBC PreparedStatement string as a base.

collapseWhitespace() Converts all instances of more than one consecutive white-space character to one space, and trims beginning and ending white space. This method can be useful for databases that require SQL with a minimum of newline and other white-space characters. (Sybase and Oracle do not require this conversion.)


OL-24187-04

Chapter 6 Configuring Databases Working Sample

Working SampleThe nf-main.vm file can contain references to other VM files to support modularization (see the “Database Configuration Files” section on page 6-3). The names of these other files are arbitrary, except for the VM_global_library.vm file, which name is predetermined. Place macros that must be defined in this file to ensure that they are loaded at the right time. For details about this special file, see the Velocity User Guide.

The following sample illustrates the contents of main.vm for an Oracle setup:

#parse ('nf-dbinfo.vm')

#foreach ($table in $tables) #set ($table.sql.dropTable = "#parse ('drop_table.vm')") #set ($table.sql.createTable = "#parse ('create_table.vm')") #set ($table.sql.createIndexes = "#parse ('create_indexes.vm')") #set ($table.sql.insert = "#parse ('insert.vm')") #set ($table.sql.metaDataQuery = "#parse ('metadata.vm')")#end

In this sample, the mandatory database and SQL definitions (see Table 6-2) are moved to separate files, which are loaded and parsed using the #parse directive.

The following sections list possible contents for the various files in the Oracle dbpack. Some of the definitions use macros that are defined in the VM_global_library.vm file. This file contains all macro definitions that any template uses.

• Macro Definitions, page 6-6

• dbinfo Configuration, page 6-7

• SQL Definitions, page 6-7

Macro DefinitionsThe following sample illustrates definitions for the mapping between native types and SQL types, and utility macros such as the optcomma macro, which inserts a comma between successive elements of lists.

#macro (optcomma)#if ($velocityCount >1),#end#end#macro (sqltype $field)#set ($maxStringLen = 2000)#if ($field.type == "INT8") integer#elseif ($field.type == "INT16") integer#elseif ($field.type == "INT32") integer#elseif ($field.type == "UINT8") integer#elseif ($field.type == "UINT16") integer#elseif ($field.type == "UINT32") integer#elseif ($field.type == "REAL") real#elseif ($field.type == "BOOLEAN") char(1)#elseif ($field.type == "STRING") varchar2(#if($field.size <= $maxStringLen)$field.size #else $maxStringLen #end)#elseif ($field.type == "TEXT") long#elseif ($field.type == "TIMESTAMP") date#end#end


OL-24187-04

Chapter 6 Configuring Databases dbinfo Configuration

dbinfo ConfigurationIn the following code sample, note that the only required fields are the URL and connection options (for authentication).

Blank lines in the code separate the code into distinct fields for readability and to ease later configuration changes.

#set ($dbinfo.driver = "oracle.jdbc.OracleDriver")#set ($dbinfo.driverjarfile = "ojdbc14.jar")#set ($dbinfo.options.host = "localhost")#set ($dbinfo.options.port = "1521")#set ($dbinfo.options.user = "pqb_admin")#set ($dbinfo.options.password = "pqb_admin")#set ($dbinfo.options.sid = "avocado")#set ($dbinfo.url = "jdbc:oracle:thin:@$dbinfo.options.host:$dbinfo.options.port:$dbinfo.options.sid")#set ($dbinfo.connOptions.user = $dbinfo.options.user)#set ($dbinfo.connOptions.password = $dbinfo.options.password)## the vendor-specific piece of SQL that will return the current## date and time:#set ($dbinfo.options.getdate = "sysdate")

SQL DefinitionsThe following sections provide SQL definitions:

• Code for Dropping a Table, page 6-7

• Code for Creating a Table, page 6-7

• Code for Creating Indexes, page 6-8

• Code for Insert, page 6-8

• Code for Metadata Query, page 6-8

Code for Dropping a Table

The following code sample drops a table using normal SQL syntax

drop table $table.dbtabname

Code for Creating a Table

The following code sample creates a table using normal SQL syntax. Use this definition for any customized database configuration that requires special directives to create a table. For example, you can modify the configuration to create the table in some unique tablespace or to use table partitioning.

create table $table.dbtabname (#foreach ($field in $table.fields) #optcomma()$field.name #sqltype($field) #if ("$!field.options.notnull" == "true") not null #end#end)


OL-24187-04

Chapter 6 Configuring Databases SQL Definitions

Code for Creating Indexes

The following code creates the indexes using normal SQL syntax. Use this definition to implement a customized database configuration that requires special directives to create an index. For example, you can modify the configuration to create the indexes in some unique tablespace.

#foreach ($index in $table.indexes) create index $index.name on $table.dbtabname ($index.columns)#end

Code for Insert

The following code creates the JDBC PreparedStatement that corresponds to the table structure.

insert into ${table.dbtabname} (#foreach ($field in $table.fields) #optcomma()${field.name}#end)values (#foreach ($field in $table.fields) #optcomma()?#end)

Code for Metadata Query

The following code defines a simple query that gets the table metadata (column names and types). You can use any query that returns an empty result set.

select * from ${table.dbtabname} where 1=0


OL-24187-04

Chapter 6 Configuring Databases Testing and Debugging

Testing and DebuggingAs you develop a set of templates for your database, you can see directly the results of parsing. To invoke such parsing, the JDBC adapter supports direct invocation using the Collection Manager main script ~scmscm/cm/bin/cm.

The syntax for an invocation is:

~/cm/bin/cm invoke com.cisco.scmscm.netflow.adapters.jdbc.JDBCAdapter

Where argument is one of the parameters described in the following sections. You can use this mechanism whether the Collection Manager is running or not.

Additionally, you can use the query and update execution methods described in the following section to test the template results with a live database.

• Parsing a String, page 6-9

• Obtaining Full Debug Information, page 6-9

Parsing a StringYou can parse any string as a VTL template with the complete context in place. The result of the parsing is displayed on the standard output. To parse a string, call the adapter with the -parse parameter. The following sample code illustrates a parsing mechanism (responses are shown in bold):

$ ~/cm/bin/cm invoke com.cisco.scmscm.netflow.adapters.jdbc.JDBCAdapter -parse 'xxx'xxx $ ~/cm/bin/cm invoke com.cisco.scmscm.netflow.adapters.jdbc.JDBCAdapter -parse '$dbinfo.url'jdbc:oracle:thin:@localhost:1521:avocado $ ~/cm/bin/cm invoke com.cisco.scmscm.netflow.adapters.jdbc.JDBCAdapter -parse '$tools.getTableByName($tables, RPT_USAGE_NF).sql.createTable'CREATE TABLE RPT_USAGE_NF ( TIME_STAMP datetime, NF_HEAD_TIME_STAMP integer , NF_HEAD_SOURCE_ID integer, NF_CLASS_ID integer, NF_APPLICATION_ID integer, NF_INGRESS_IF integer, NF_EGRESS_IF integer, NF_FLOW_DIRECTION integer, NF_FLOW_START_SYSUP_TIME integer, NF_FLOW_END_SYSUP_TIME integer, NF_IN_PKTS integer, NF_IN_BYTES integer, NF_CONNECTION_COUNT_NEW integer, NF_CONN_SUM_DURATION integer)

Obtaining Full Debug InformationTo see a dump of all of the contents of the tables and dbinfo structures as created by the templates, use the -debug parameter. The -debug parameter displays in standard output a detailed view of all the fields, properties, and options of these structures.


OL-24187-04

Chapter 6 Configuring Databases Using the JDBC Framework in Scripts

Using the JDBC Framework in ScriptsYou can send arbitrary SQL commands to the database for execution and view the resulting data. Viewing results helps you to perform periodic database maintenance, to monitor the contents of database tables, and to manage extra database tables.

To perform an update operation, call the adapter using the -executeUpdate parameter. To perform a query and view the results, call the adapter using the -executeQuery parameter.

Viewing and Setting the Time Zone OffsetThe following sample of an update operation demonstrates how to change programmatically the value in the database table that contains the Cisco ASR 1000 time zone offset setting. The name of this table is usually CONF_TZ_OFFSET_NF; because the table can be assigned another name, it is referred to here by its logical name TZ. See the listing in the CM:NetFlow nf-dbtables.xml File, page B-2.

To avoid the need to check that the table exists first, and then update it, the table is dropped (ignoring the error status if it does not exist) and then recreated. The proper values are inserted. Since the table contains a timestamp column, you must get the current date from the database. This operation is specific to each database vendor; therefore, this example uses the preconfigured getdate operation that is defined in the templates.

Note the use of the tools assignParams and getTableByName to generate the SQL.

#! /bin/bash

this=$0tableName=TZ

usage () { cat <<EOFUsage: $this --status - show currently configured TZ offset $this --offset=N - set the offset to N minutes (-1440 <= N <= 1440) $this --help - print this messageEOF}

query () {~/cm/bin/cm invoke com.cisco.scmscm.netflow.adapters.jdbc.JDBCAdapter -executeQuery "$*" }

update () {~/cm/bin/cm invoke com.cisco.scmscm.netflow.adapters.jdbc.JDBCAdapter

-executeUpdate "$*" }

get_tz () { query 'select * from $tools.getTableByName($tables, "TZ").dbtabname'}

set_tz () { update '$tools.getTableByName($tables, "TZ").sql.dropTable' update '$tools.getTableByName($tables, "TZ").sql.createTable' update '$tools.assignParams($tools.getTableByName($tables, "TZ").sql.insert, [$dbinfo.options.getdate, '$1'])'}


OL-24187-04

Chapter 6 Configuring Databases Viewing and Setting the Time Zone Offset

case $1 in --status) get_tz ;; --help) usage exit 0 ;; --offset=*) n=$(echo $1 | egrep 'offset=[-]?[0-9]+$' | sed 's/.*=//') if [ "$n" ]; then if [ "$n" -ge -1440 -a "$n" -le 1440 ]; then set_tz $n &>/dev/null ok=1 fi fi if [ ! "$ok" ]; then usage exit 2 fi get_tz ;; *) usage exit 3 ;;esac

When an executed query returns a result set, it is displayed to standard output in tabular form with appropriate column headers.


OL-24187-04

Chapter 6 Configuring Databases Scalability Hints for Oracle

Scalability Hints for OracleThe following sections demonstrate ways to make database handling more scalable for the Collection Manager. These methods are specific to Oracle, and are provided to illustrate the possibilities.

• Using Custom Tablespaces, page 6-12

• Using Table Partitioning, page 6-14

Using Custom TablespacesIf you create several tablespaces and wish to distribute the Collection Manager tables among them, specify the tablespace to use for each table in the file nf-dbtables.xml. The following sample code represents the definition for one table (note the code in bold):

<rdr name="USAGE_NF" dbtabname="RPT_USAGE_NF" tag="Usage"

createtable="true">

<fields>

<field id="1" name="TIME_STAMP" type="INT32">

<options>

<option property="source" value="timestamp" />

</options>

</field>

<field id="2" name="NF_HEAD_TIME_STAMP" type="INT32">

<options>

<option property="source" value="nf_timestamp" />

</options>

</field>

<field id="3" name="NF_HEAD_SOURCE_ID" type="INT32">

<options>

<option property="source" value="recordsource" />

</options>

</field>

<field id="4" name="NF_CLASS_ID" type="UINT32" /> 

<field id="5" name="NF_APPLICATION_ID" type="INT32DIG10" />

<field id="6" name="NF_INGRESS_IF" type="UINT32" />

<field id="7" name="NF_EGRESS_IF" type="UINT32" />

<field id="8" name="NF_FLOW_DIRECTION" type="UINT8" />

<field id="9" name="NF_FLOW_START_SYSUP_TIME" type="UINT32" />

<field id="10" name="NF_FLOW_END_SYSUP_TIME" type="UINT32" />

<field id="11" name="NF_IN_PKTS" type="UINT64" >

<options>

<option property="source" value="unsigned64" />

</options>

</field>


OL-24187-04

Chapter 6 Configuring Databases Using Custom Tablespaces

<field id="12" name="NF_IN_BYTES" type="UINT64" >

<options>


</options>

</field>

<field id="13" name="NF_CONNECTION_COUNT_NEW" type="UINT32" />

<field id="14" name="NF_CONN_SUM_DURATION" type="UINT64" >

<options>


</options>

</field>

</fields>

<indexes>

<index name="RPT_USAGE_NF_I1" columns="NF_HEAD_TIME_STAMP">

<options>

<option property="clustered" value="true" />

<option property="allowduprow" value="true" />

</options>

</index>

<index name="RPT_USAGE_NF_I2" columns="NF_APPLICATION_ID">

</index>

</indexes>

</rdr>

This sample adds the required tablespaces (tspace1 and tspace2) for the index and for the table. There is no preconfigured meaning to the option tablespace in the Collection Manager. Any new option name could have been used. Its meaning is derived from its subsequent use in the templates.

To create the table in the correct tablespace, modify create_table.vm as follows:

create table $table.dbtabname (#foreach ($field in $table.fields) #optcomma()$field.name #sqltype($field) #if ("$!field.options.notnull" == "true") not null #end#end)#if ("$!table.options.tablespace" != "") TABLESPACE $table.options.tablespace#end

To create the index in its own tablespace, modify create_indexes.vm as follows:

#foreach ($index in $table.indexes) create index $index.name on $table.dbtabname ($index.columns)#if ("$!index .options.tablespace" != "") TABLESPACE $index.options.tablespace #end#end


OL-24187-04

Chapter 6 Configuring Databases Using Table Partitioning

Using Table PartitioningTo implement rolling partitioning for a particular table on a weekly basis, create a partitioned option for the table in the nf-dbtables.xml file in a manner similar to the example in the previous section (see the “Using Custom Tablespaces” section on page 6-12). Then augment the create_table.vm code as follows (note the code in bold ):

create table $table.dbtabname (#foreach ($field in $table.fields)#optcomma()$field.name #sqltype($field)#if ("$!field.options.notnull" == "true")not null#end#end)#if ("$!table.options.partitioned" != "") partition by range (timestamp) (partition week_1 values less than (to_date ('01-JAN-2005 00:00:00','DD-MON-YYYY HH24:MI:SS')), partition week_2 values less than (to_date ('08-JAN-2005 00:00:00','DD-MON-YYYY HH24:MI:SS')) partition week_3 values less than (to_date ('15-JAN-2005 00:00:00','DD-MON-YYYY HH24:MI:SS')) partition week_4 values less than (to_date ('22-JAN-2005 00:00:00','DD-MON-YYYY HH24:MI:SS')) ); #end

Because Oracle does not accept nonconstant expression for the time boundaries, the values must be hardwired for the time the tables are created.

Create a cron job to roll the partitions (delete an old partition and create a new one) on a weekly basis. This cron job runs a script that calls the command-line interface of the JDBC Adapter (as explained in the “Using the JDBC Framework in Scripts” section on page 6-10) to issue the appropriate alter table drop partition and alter table add partition SQL commands.


OL-24187-04

Cisco Application VisibilityOL-24187-04

A
P P E N D I X A NetFlow Field Types and Database Formats
IntroductionThis chapter describes the fields contained in NetFlow records (NFR). It also details the formats and field contents of NetFlow Records data tables:

• NetFlow Field Types, page A-1

• Database Tables: Formats and Field Contents, page A-8

NetFlow Field TypesThe following sections detail the different types of NetFlow fields:

• NetFlow Field Types for RPT_USAGE_NF Table, page A-1

• NetFlow Field Types for RPT_TRANSACTION_NF, page A-3

• NetFlow Field Types for RPT_GLB_USAGE_NF Table:, page A-5

• NetFlow Field Types for CONF_TZ_OFFSET_NF Table, page A-6

• NetFlow Field Types for NF_INI_VALUES Table, page A-6

NetFlow Field Types for RPT_USAGE_NF TableUsage Records are records of the different type of applications running over a specific interface. The operator can use Usage records to monitor how much bandwidth the differ-ent applications use. The Usage Records show this application usage over a specific time period, the peak and average usages, and usage for a specific application type.

Table A-1 describes NetFlow Field Types for RPT_USAGE_NF Table:

Table A-1 Summary of NetFlow Field Types for RPT_USAGE_NF Table

Field Name Value Type Description

time_stamp - TIMESTAMP DB Insertion timestamp value

head_time_stamp - INT32 Packet timestamp from ASR1K

A-1 and Control Collection Manager User Guide

Appendix A NetFlow Field Types and Database Formats NetFlow Field Types for RPT_USAGE_NF Table

head_source_id - INT32 Contains the IP address of the Cisco ASR1K platform that generated the NFR

class_id 51 UINT32 Reserved for future use.

application_id 95 INT32 The unique id for application

ingressInterface 10 UINT32 The index of the IP interface where packets of this Flow are being received.

egressInterface 14 UINT32 The index of the IP interface where packets of this Flow are being sent.

flowDirection 61 UINT8 The direction of the Flow observed at the Observation Point. There are only two values defined(Ingress/Egress).

flowStartSysUpTime 22 UINT32 The relative timestamp of the first packet of this Flow. It indicates the number of milliseconds since the last (re-)initialization of the IPFIX Device (sysUpTime).

flowEndSysUpTime 21 UINT32 The relative timestamp of the last packet of this Flow. It indicates the number of milliseconds since the last (re-)initialization of the IPFIX Device (sysUpTime)

packetDeltaCount 2 UINT64 The number of incoming packets since the previous report (if any) for this Flow at the Observation Point.

in_bytes 3 UINT64 Total number bytes received since the previous report (if any) for this Flow at the Observation Point.

connectionCountNew 278 UINT 32 This information element counts the number of TCP or UDP connections which were opened during the observation period. The observation period may be specified by the flow start and end timestamps.

connectionSumDuration 279 UINT64 This information element aggregates the total time in seconds for all of the TCP or UDP connections which were in use during the observation period. For example if there are 5 concurrent connections each for 10 seconds, the value would be 50 s.



A-2Cisco Application Visibility and Control Collection Manager User Guide

OL-24187-04

Appendix A NetFlow Field Types and Database Formats NetFlow Field Types for RPT_TRANSACTION_NF

NetFlow Field Types for RPT_TRANSACTION_NF A transaction is a set of logical exchanges between endpoints. There is normally one transaction within a flow. The Transaction Record monitors the traffic at transaction levels. Transaction Records provide a detailed analysis of the traffic flows, including extracted Layer 7 fields. Due to the high load of transactions these records are sample or filtered. Transaction Records are bound to the input and output directions of the network side interfaces. These Transaction Records allow the system to capture each unidirectional flow once.

Table A-2 describes NetFlow Field Types for RPT_TRANSACTION_NF Table:

ingressVRFID 234 UINT32 A unique identifier of the VRFname where the packets of this flow are being received. This identifier is unique per Metering Process

ipVersion 60 UINT8 The IP version field in the IP packet header.



Table A-2 NetFlow Field Types for RPT_TRANSACTION_NF




head_src_id - INT32 Contains the IP address of the Cisco ASR1K platform that generated the NFR

conn_tx_id 280 UINT64 A unique ID for the transaction

application_id 95 INT32 The unique id for application




flowStartSysUpTime

22 UINT32 The relative timestamp of the first packet of this Flow. It indicates the number of milliseconds since the last (re-)initialization of the IPFIX Device (sysUpTime).


OL-24187-04

Appendix A NetFlow Field Types and Database Formats NetFlow Field Types for RPT_TRANSACTION_NF


packetDeltaCount 2 UINT 64 The number of incoming packets since the previous report (if any) for this Flow at the Observation Point.


src_ipv4_addr 8 UINT32 The IPv4 source address in the IP packet header.

src_port 7 UINT16 The source port identifier in the transport header.

dst_ipv4_addr 12 UINT32 The IPv4 destination address in the IP packet header.

dst_port 11 UINT16 The destination port identifier in the transport header.

protocol_id 4 UINT8 The value of the protocol number in the IP packet header.

flow_id 48 UINT64 Reserved for future use.

flow_end_reason 136 UINT8 The reason for Flow termination.

biflow_direction 239 UINT8 A description of the direction assignment method used to assign the Biflow Source and Destination.

ingressVRFID 234 UINT32 A unique identifier of the VRFname where the packets of this flow are being received. This identifier is unique per Metering Process


sourceIPv6Address 27 UINT32 The IPv6 source address in the IP packet header.

destinationIPv6Address

28 UINT32 The IPv6 destination address in the IP packet header.

Table A-2 NetFlow Field Types for RPT_TRANSACTION_NF



OL-24187-04

Appendix A NetFlow Field Types and Database Formats NetFlow Field Types for RPT_GLB_USAGE_NF Table:

NetFlow Field Types for RPT_GLB_USAGE_NF Table:Global usage records are the records of many applications running over an interface. You can use them to monitor the total traffic over the network, monitor the usage of all the applications over a specific time period, or monitor the peak and average usages of all the applications over an interface.

Table A-3 describes NetFlow Field Types for RPT_GLB_USAGE_NF Table:

Table A-3 NetFlow Field Types for RPT_GLB_USAGE_NF Table




head_source_id - INT32 Contains the IP address of the Cisco ASR1K platform that generated the NFR

class_id 51 UINT32 Reserved for future use.




flowStartSysUpTime

22 UINT32 The relative timestamp of the first packet of this Flow. It indicates the number of milliseconds since the last (re-)initialization of the IPFIX Device (sysUpTime).


packetDeltaCount 2 UINT 64 The number of incoming packets since the previous report (if any) for this Flow at the Observation Point.



OL-24187-04

Appendix A NetFlow Field Types and Database Formats NetFlow Field Types for CONF_TZ_OFFSET_NF Table

NetFlow Field Types for CONF_TZ_OFFSET_NF TableTable A-4 describes NetFlow Field Types for CONF_TZ_OFFSET_NF Table:

NetFlow Field Types for NF_INI_VALUES TableTable A-5 describes NetFlow Field Types for NF_INI_VALUES Table:

connectionCountNew

278 UINT32 This information element counts the number of TCP or UDP connections which were opened during the observation period. The observation period may be specified by the flow start and end timestamps.

connectionSumDuration

279 UINT 64 This information element aggregates the total time in seconds for all of the TCP or UDP connections which were in use during the observation period. For example if there are 5 concurrent connections each for 10 seconds, the value would be 50 s.

ingressVRFID 234 UINT32 An unique identifier of the VRFname where the packets of this flow are being received. This identifier is unique per Metering Process.


Table A-3 NetFlow Field Types for RPT_GLB_USAGE_NF Table


Table A-4 NetFlow Field Types for CONF_TZ_OFFSET_NF Table

Field Name Type Description

time_stamp TIMESTAMP DB insertion timestamp value

offset_min INT16 Offset value in minutes


OL-24187-04

Appendix A NetFlow Field Types and Database Formats NetFlow Field Types for NF_INI_VALUES Table

:

Table A-5 NetFlow Field Types for NF_INI_VALUES Table

Field Name Type Description

time_stamp TIMESTAMP DB insertion timestamp value

nf_ip STRING Identification of the ASR1K platform where these values were applied.

value_type INT16 Key Name/Value family type. The possible values are:

5 -Source address 32-bit / dotted notation

101 - Input interface ID / Input Interface Name

102 - Input interface ID / Input Interface Description

103 - Output interface ID / Output Interface Name

104 - Output interface ID / Output Interface Description

111 -Application ID / Application Name

112 -Application ID / Application Description

114 - Application ID / Category Name

115 - Application ID / Sub - Category Name

116 - Application ID / Application Group

117 - Application ID / Attribute: p2p-technology

118 - Application ID / Attribute: tunnel

119- Application ID / Attribute: encrypted

131 - Sampler ID / Sampler Info

value_key STRING Key name.

For example: Gold, Silver

value INT32 Numeric reference.


OL-24187-04

Appendix A NetFlow Field Types and Database Formats Database Tables: Formats and Field Contents

Database Tables: Formats and Field ContentsEach NFR is sent to the Cisco Collection Manager. On the Collection Manager, adapters convert the NFRs and store them in database tables. This section details these tables and their columns (field names and types).

The following sections detail the different types of database tables:

• Table CONF_TZ_OFFSET_NF, page A-8

• NetFlow Field Types for RPT_USAGE_NF Table, page A-9

• NetFlow Field Types for RPT_TRANSACTION_NF Table, page A-9

• Columns of Table RPT_GLB_USAGE_NF, page A-10

• NetFlow Field Types for NF_INI_VALUES Table, page A-11

Table CONF_TZ_OFFSET_NF Database table CONF_TZ_OFFSET_NF contains the time-zone offset in minutes for the clock of each Cisco Series ASR 1000 Series router as configured by the select-sce-tz.sh script.

Table Table A-6 lists the columns of table CONF_TZ_OFFSET_NF.

Table A-6 CONF_TZ_OFFSET_NF Table Columns

Field Name Type

TIME_STAMP TIMESTAMP

OFFSET_MIN INT16


OL-24187-04

Appendix A NetFlow Field Types and Database Formats NetFlow Field Types for RPT_USAGE_NF Table

NetFlow Field Types for RPT_USAGE_NF Table

NetFlow Field Types for RPT_TRANSACTION_NF Table

Table A-7 RPT_USAGE_NF Table Field Types

Field Name Type


NF_HEAD_TIME_STAMP INT32

NF_HEAD_SOURCE_ID INT32

NF_CLASS_ID UINT32

NF_APPLICATION_ID INT32

NF_INGRESS_IF UINT32

NF_EGRESS_IF UINT32

NF_FLOW_DIRECTION UINT8

NF_FLOW_START_SYSUP_TIME UINT32

NF_FLOW_END_SYSUP_TIME UINT32

NF_IN_PKTS UINT64

NF_IN_BYTES UINT64

NF_CONNECTION_COUNT_NEW UINT 32

NF_CONN_SUM_DURATION UINT64

NF_INGRESS_VRF_ID UINT32

NF_IP_VERSION UINT8

Table A-8 RPT_TRANSACTION_NF Table Field Types

Field Name Type



NF_HEAD_SRC_ID INT32

NF_CONN_TX_ID UINT64


NF_EGRESS_IF UINT32




NF_IN_PKTS UINT64

NF_IN_BYTES UINT64

NF_SRC_IPV4_ADDR UINT32

NF_SRC_PORT UINT16


OL-24187-04

Appendix A NetFlow Field Types and Database Formats Columns of Table RPT_GLB_USAGE_NF

Columns of Table RPT_GLB_USAGE_NF

NF_DST_IPV4_ADDR UINT32

NF_DST_PORT UINT16

NF_PROTOCOL_ID UINT8

NF_FLOW_ID UINT64

NF_FLOW_END_REASON UINT8

NF_BIFLOW_DIRECTION UINT8


NF_IP_VERSION UINT8

NF_SRC_IPV6_ADDR STRING

NF_DST_IPV6_ADDR STRING

Table A-9 RPT_GLB_USAGE_NF Table Columns

Field Name Type



NF_HEAD_SOURCE_ID INT32

NF_APPLICATION_ID INT32


NF_EGRESS_IF UINT32




NF_IN_PKTS UINT64

NF_IN_BYTES UINT64

NF_CONNECTION_COUNT_NEW UINT 32

NF_CONN_SUM_DURATION UINT64


NF_IP_VERSION UINT8

Table A-8 RPT_TRANSACTION_NF Table Field Types

Field Name Type


OL-24187-04


NetFlow Field Types for NF_INI_VALUES TableTable A-10 NF_INI_VALUES Table NetFlow Field Types

Field Name Type


NF_IP STRING

VALUE_TYPE INT16

VALUE_KEY STRING

VALUE INT32


OL-24187-04



OL-24187-04


A
P P E N D I X B Code Samples
IntroductionThis appendix contains samples of files used to configure the adapters that process the data that the Collection Manager receives.

• Application Configuration, page B-1

Application ConfigurationThe following sections list part of the XML file (nf-dbtables.xml) used to configure the database tables, and the DTD file used to verify the structure of the XML file.

• CM:NetFlow nf-dbtables.xml File, page B-2

B-1 and Control Collection Manager User Guide

Appendix B Code Samples CM:NetFlow nf-dbtables.xml File

CM:NetFlow nf-dbtables.xml FileThe XML file is verified at runtime against a simple DTD, which is reproduced in the following section:

<?xml version="1.0" encoding="ISO8859_1"?><!DOCTYPE dbtabconf PUBLIC "-//P-Cube//Engage DB RDR Configuration 2.1.0//EN" "dbtables.dtd"><dbtabconf> <fileversion/> <application name="NetFlow" version="3.7.0"/> <dbtables> <rdr name="USAGE_NF" dbtabname="RPT_USAGE_NF" tag="Usage" createtable="true"> <fields> <field id="1" name="TIME_STAMP" type="TIMESTAMP"> <options> <option property="source" value="timestamp" /> </options> </field> <field id="2" name="NF_HEAD_TIME_STAMP" type="INT32"> <options> <option property="source" value="nf_timestamp" /> </options> </field> <field id="3" name="NF_HEAD_SOURCE_ID" type="UINT32"> <options> <option property="source" value="recordsource" /> </options> </field> <field id="4" name="NF_CLASS_ID" type="UINT32" />  <field id="5" name="NF_APPLICATION_ID" type="INT32DIG10" /> <field id="6" name="NF_INGRESS_IF" type="UINT32" /> <field id="7" name="NF_EGRESS_IF" type="UINT32" /> <field id="8" name="NF_FLOW_DIRECTION" type="UINT8" /> <field id="9" name="NF_FLOW_START_SYSUP_TIME" type="UINT32" /> <field id="10" name="NF_FLOW_END_SYSUP_TIME" type="UINT32" /> <field id="11" name="NF_IN_PKTS" type="UINT64" > <options> <option property="source" value="unsigned64" /> </options> </field> <field id="12" name="NF_IN_BYTES" type="UINT64" > <options> <option property="source" value="unsigned64" /> </options> </field> <field id="13" name="NF_CONNECTION_COUNT_NEW" type="UINT32" /> <field id="14" name="NF_CONN_SUM_DURATION" type="UINT64" > <options> <option property="source" value="unsigned64" /> </options> </field> </fields> <indexes> <index name="RPT_USAGE_NF_I1" columns="NF_HEAD_TIME_STAMP"> <options>

B-2Cisco Application Visibility and Control Collection Manager User Guide

OL-24187-04


<option property="clustered" value="true" /> <option property="allowduprow" value="true" /> </options> </index> </indexes> </rdr> <rdr name="TRANSACTION_NF" dbtabname="RPT_TRANSACTION_NF" tag="Transaction" createtable="true"> <fields> <field id="1" name="TIME_STAMP" type="TIMESTAMP"> <options> <option property="source" value="timestamp" /> </options> </field> <field id="2" name="NF_HEAD_TIME_STAMP" type="INT32"> <options> <option property="source" value="nf_timestamp" /> </options> </field> <field id="3" name="NF_HEAD_SRC_ID" type="UINT32"> <options> <option property="source" value="recordsource" /> </options> </field> <field id="4" name="NF_CONN_TX_ID" type="UINT64" > <options> <option property="source" value="unsigned64" /> </options> </field> <field id="5" name="NF_APPLICATION_ID" type="INT32DIG10" /> <field id="6" name="NF_INGRESS_IF" type="UINT32" /> <field id="7" name="NF_EGRESS_IF" type="UINT32" /> <field id="8" name="NF_FLOW_DIRECTION" type="UINT8" /> <field id="9" name="NF_FLOW_START_SYSUP_TIME" type="UINT32" /> <field id="10" name="NF_FLOW_END_SYSUP_TIME" type="UINT32" /> <field id="11" name="NF_IN_PKTS" type="UINT64" > <options> <option property="source" value="unsigned64" /> </options> </field> <field id="12" name="NF_IN_BYTES" type="UINT64" > <options> <option property="source" value="unsigned64" /> </options> </field> <field id="13" name="NF_SRC_IPV4_ADDR" type="IPADDRDIG10" > <options> <option property="source" value="ipaddress"/> </options> </field> <field id="14" name="NF_SRC_PORT" type="UINT16" /> <field id="15" name="NF_DST_IPV4_ADDR" type="IPADDRDIG10" > <options>


OL-24187-04

<option property="source" value="ipaddress"/> </options> </field> <field id="16" name="NF_DST_PORT" type="UINT16" /> <field id="17" name="NF_PROTOCOL_ID" type="UINT8" /> <field id="18" name="NF_FLOW_ID" type="UINT64" >  <options> <option property="source" value="unsigned64" /> </options> </field> <field id="19" name="NF_FLOW_END_REASON" type="UINT8" /> <field id="20" name="NF_BIFLOW_DIRECTION" type="UINT8" /> </fields> <indexes> <index name="RPT_TRANSACTION_NF_I1" columns="NF_HEAD_TIME_STAMP"> <options> <option property="clustered" value="true" /> <option property="allowduprow" value="true" /> </options> </index> </indexes> </rdr>

<table name="NF_INI_VALUES" dbtabname="NF_INI_VALUES"> <fields> <field id="1" name="TIME_STAMP" type="TIMESTAMP"/> <field id="2" name="NF_IP" type="STRING" size="20"/> <field id="3" name="VALUE_TYPE" type="INT16"/> <field id="4" name="VALUE_KEY" type="STRING" size="255"/> <field id="5" name="VALUE" type="INT32"/> </fields> </table> </dbtables></dbtabconf>


OL-24187-04

Appendix B Code Samples tables.dtd File

tables.dtd FileThe following sample code represents the DTD file used to verify the nf-dbtables.xml definition file:

<?xml version="1.0" encoding="ISO8859_1"?>

<!ELEMENT dbtabconf (fileversion, application, db?, dbtables)><!ELEMENT fileversion (#PCDATA)><!ELEMENT application EMPTY><!ATTLIST application name CDATA #REQUIRED version CDATA #REQUIRED><!ELEMENT db (options)><!ELEMENT dbtables (rdr*, aggtable*, table*)><!ELEMENT table (options?, fields, indexes?)><!ATTLIST table name CDATA #REQUIRED dbtabname CDATA #REQUIRED createtable (true | false) "true" inserttodb (true | false) "false"><!ELEMENT aggtable (options?, fields, indexes?)><!ATTLIST aggtable name CDATA #REQUIRED dbtabname CDATA #REQUIRED aggperiod CDATA #REQUIRED createtable (true | false) "true"><!ELEMENT rdr (options?, fields, indexes?)><!ATTLIST rdr name CDATA #REQUIRED dbtabname CDATA #REQUIRED tag CDATA #REQUIRED createtable (true | false) "true" inserttodb (true | false) "true"><!ELEMENT fields (field+)><!ELEMENT field (options?)><!ATTLIST field id CDATA #REQUIRED name CDATA #REQUIRED type CDATA #REQUIRED size CDATA #IMPLIED><!ELEMENT indexes (index+)><!ELEMENT index (options?)><!ATTLIST index name CDATA #REQUIRED columns CDATA #REQUIRED create (true | false) "true"><!ELEMENT options (option+)><!ELEMENT option EMPTY><!ATTLIST option property CDATA #REQUIRED value CDATA #REQUIRED>

You can set the location and name of the DTD and XML files separately for each adapter in the adapter configuration file.


OL-24187-04

Appendix B Code Samples tables.dtd File


OL-24187-04


A
P P E N D I X C Configuration Files Descriptions
IntroductionThis appendix describes each of the configuration files used by the Collection Manager along with the configuration parameters in each file.

Each configuration file is divided into sections. A heading identifies each section, for example, [config].

You can comment out any parameter that is not required in your configuration by placing a # at the start of the line.

If you modify the configuration file, restart the Collection Manager to enable the changes to take effect.

Note The configuration files supplied with the Collection Manager software installation contain tested valid settings. You are not expected to change the settings. If it is necessary to change the configuration settings, be careful when you modify a configuration file.

C-1 and Control Collection Manager User Guide

Appendix C Configuration Files Descriptions General Configuration Files

General Configuration FilesThis section describes the cm.conf and queue.conf configuration files.

• Collection Manager Configuration, page C-2

• CM:NetFlow Queue Configuration, page C-5

• CM:NetFlow Adapter Configuration Files, page C-6

Collection Manager ConfigurationThis section describes the Collection Manager configuration file nf.conf.

• [adapter] Section, page C-2

• [adapter_start_timeout] Section, page C-3

• [buffer] Section, page C-3

• [queue] Section, page C-3

• [server] Section, page C-4

• [collector] Section, page C-4

• [csv_adapter] Section, page C-4

• [nf-type] Section, page C-4

[adapter] Section

The [adapter] section contains the following parameters:

• automatic_start

Defines whether starting or resetting the Collection Manager starts the adapters automatically.

Possible values for this parameter are true and false. The default value is true.

• port

Defines the adapter manager server port number. Possible values for this parameter are 1024-65535. The default value is 33002.

• timeout

Defines the timeout value in seconds for client adapters. Setting this parameter to 0 disables the timeout.

• connections

Defines the maximum number of client connections per adapter.

Possible values for the parameter are 2-100. The default value is 20.

• directory

Defines the location to which you want to save buffered NetFlow Record data. You can use the ~/ prefix for this parameter. The default value is ~/cm/adapters/netflow.

• adapter.<number>

C-2Cisco Application Visibility and Control Collection Manager User Guide

OL-24187-04

Appendix C Configuration Files Descriptions Collection Manager Configuration

Defines the full class name of the adapter, where <number> is a unique ID number. The adapter definitions must match the adapter_id parameters of the queues defined in the queue configuration file.

• parestart_start

Defines the time in 24-hour format at which to restart the adapters periodically. An example value for this parameter is 23:30. The default value is 03:45.

• parestart_interval

Defines the interval in minutes between periodic adapter restarts. Possible values for this parameter are 0-2147483647 (0 to 231 - 1), where 0 disables this functionality. The default value for this parameter is 10,080 minutes (18 hours).

[adapter_start_timeout] Section

The [adapter_start_timeout] section contains the following parameter:

• <Adapter class name>

Defines the number of seconds that the Collection Manager waits for a specific adapter to start. The default value is 200 seconds.

[buffer] Section

The [buffer] section contains the following parameters:

• size

Defines the minimum number of accumulated bytes in the queue before a buffer dequeue.

Possible values for this parameter are 1-262,144. The default value is 128,000.

• time

Defines the minimum number of milliseconds between buffer dequeues.


• file_limit

Defines the maximum number of files to store in each adapter queue.


[queue] Section

The [queue] section contains the following parameters:

• upper_hysteresis

Defines the upper hysteresis level as a fraction of the total queue size.

Possible values for this parameter are in the range 0-1. The default value is 0.11.

• lower_hysteresis

Defines the lower hysteresis level as a fraction of the total queue size.

Possible values for this parameter are in the range 0-1. The default value is 0.12.

• config

Defines the queue configuration file. The default value is queue.conf.


OL-24187-04

Appendix C Configuration Files Descriptions Collection Manager Configuration

[server] Section

The [server] section contains the following parameters:

• port

Defines the internal NetFlow port number for the Collection Manager.

Possible values for this parameter are 1024–65,535. The default value is 2055.

• connections

Defines the maximum number of client connections.

Possible values for this parameter are 1–50. The default value is 50.

• timeout

Defines the client timeout in seconds. The default value is 600.

• allow_multi

Defines whether to allow concurrent connections from the same IP.

Possible values for this parameter are true and false. The default value is false.

[collector] Section

The [collector] section contains the following parameters:

• notification_level

Defines the length of time in milliseconds between successive UM notifications.

Possible values for this parameter are 0–60,000. The default value is 10.

• log_directory

Defines the directory in which to store log files. You can use the ~/ prefix for this parameter. The default value is ~/cm/logs.

[csv_adapter] Section

The [csv_adapter] section contains the following parameter:

• csvdir

Defines the parent directory for CSV files. The default value for this parameter is csvdir=~/cm/adapters/netflow/CSVAdapter/csvfiles.

[nf-type] Section

The [nf-type] section contains the following parameters:

<id>.NetFlow Queue Tag : Defines the Netflow queue tags and it's unique field ID's

Ex: 10.Transaction=280,8,12

unknown type : Defines the default NetFlow type,if above

mentioned types was not matched


OL-24187-04

Appendix C Configuration Files Descriptions CM:NetFlow Queue Configuration

The Cisco Collection Manager processes NetFlow IPv4 traffic records by default. To enable IPv6 support please update the Transaction field ids as 280,27,28 and restart the Cisco Collection Manager.

CM:NetFlow Queue ConfigurationThis section describes the queue configuration file. The name of the queue configuration file is provided in the [queue] section of the cm.conf configuration file.

Each section of the queue configuration file defines a queue. The section names, and hence the queue names, are all user defined. Each queue section has the following parameters:

• adapter_id

Defines a unique adapter ID number. The ID number must match the adapters defined in the adapter.<number> parameter in the [adapter] section of the nf.conf configuration file.

• priority

Defines the priority level of the queue. Possible values for this parameter are 1–3. This parameter is deprecated. Do not change it.

• warning_size

Defines the number of NFRs that can be present in the queue before a warning is sent.

• maximum_size

Defines the maximum size of the queue.

• tags

Defines a list of NFR tag numbers sent to this queue by the categorizer. The same tag can be sent to multiple queues.

The following is an example of a queue definition in the queue configuration file:

# JDBCAdapter[dbase-hi]adapter_id=1priority=3warning_size=40000maximum_size=50000tags=Usage,Transaction,GlobalUsage


OL-24187-04

Appendix C Configuration Files Descriptions CM:NetFlow Adapter Configuration Files

CM:NetFlow Adapter Configuration FilesThis section describes the configuration files for all of the possible adapters.

• CSV Adapter

• JDBC Adapter

CSV AdapterThis section describes the CSV adapter configuration file nf-csvadapter.conf.

• [csv_adapter] Section, page C-4

• [bench] Section, page C-6

[csvadapter] Section

The [csvadapter] section contains the following parameters:

• maxCSVFiles

Defines the maximum number of entries in a CSV file. When the count reaches this number, the counting wraps back to 1.

Possible values for this parameter are 1–9,999. The default value is 9,999.

• nfLimit

Defines the maximum number of NFRs that can be written to a single file.

Possible values for this parameter are 1–1,000. The default value is 1,000.

• backupPreviousFiles

Defines whether to back up existing CSV files before overwriting them.


• includeRecordSource

Defines whether to include the record source (the Cisco ASR 1000 IP) in each NFR line.


• escapeNonPrintables

Defines whether to escape characters that cannot be printed.


Note Setting escapeNonPrintables to true can negatively affect adapter performance.

[bench] Section

The [bench] section contains the following parameter:

• rate_period_msec

Defines the time period in milliseconds between write rate updates. The default value is 5,000. Set this parameter to 0 to disable.


OL-24187-04

Appendix C Configuration Files Descriptions JDBC Adapter

• Option_template_timeout: Defines how many seconds between processing of the option template. The default value is 10. Set this parameter to 0 to disable it.

JDBC AdapterThis section describes the JDBC adapter configuration file nf-jdbcadapter.conf.

• [bench] Section, page C-7

• [db] Section, page C-7

• [app] Section, page C-8

[bench] Section

The [bench] section contains the following parameter:

• rate_period_msec

Defines the time period between insert rate updates in milliseconds. The default value is 5,000.

Set this parameter to 0 to disable the insertions.

[db] Section

The [db] section contains the following parameters:

• db_template_dir

Defines the location of the main database configuration template relative to the Collection Manager config directory. The configuration file contains default values for each of the three possible databases; however, only one database can be configured at a time. The possible values are:

– Oracle: dbpacks/oracle/9204e

– Sybase: dbpacks/sybase/ase12.5.1

– MySQL: dbpacks/mysql/4.0.20

The default value is dbpacks/sybase/ase12.5.1.

• db_template_file

Defines the name of the main database configuration template. The default value is nf-main.vm. Do not change this parameter.

• batch_size

Defines how many inserts can be made in a single batch. The default value is 10.

• transaction_size

Defines how many batches can be made in a single transaction. The default value is 15.

• commit_interval

Defines the maximum time in seconds between database commits. The default value is 6.

• operations_timeout

Defines the maximum time in seconds allowed for the completion of database operations such as connecting, insertion, and so on. The default value is 60.

• blocking_connects


OL-24187-04

Appendix C Configuration Files Descriptions JDBC Adapter

Defines whether to block connection attempts to the DB.

Possible values for this parameter are true and false. The default value is true.

[app] Section

The [app] section contains the following parameters:

• app_conf_file

Defines the name of the application configuration XML file. The default value is dbtables.xml.

• app_dtd_file

Defines the name of the application configuration DTD file. The default value is dbtables.dtd.

• app_conf_dir

Defines the location of the application configuration XML and DTD files relative to the Collection Manager configuration directory. The default value is apps/scasbb/latest.


OL-24187-04

cisco application visibility and control collection ... · chapter 1 cisco application visibility...

Documents