with ibm corp. · v apache ambari operational framework. apache ambari is an open framework for pr...

40
IBM Network Performance Insight 1.2.0 Document Revision R2E1 Administering Network Performance Insight IBM

Upload: others

Post on 22-May-2020

26 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

IBM Network Performance Insight 1.2.0Document Revision R2E1

Administering Network PerformanceInsight

IBM

Page 2: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

NoteBefore using this information and the product it supports, read the information in “Notices” on page 23.

This edition applies to version 1.2.0.0 of IBM Network Performance Insight and to all subsequent releases andmodifications until otherwise indicated in new editions.

© Copyright IBM Corporation 2015, 2016.US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contractwith IBM Corp.

Page 3: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

Contents

Administering Network PerformanceInsight . . . . . . . . . . . . . . . vIntended audience . . . . . . . . . . . . vOrganization . . . . . . . . . . . . . . vNetwork Performance Insight architecture . . . . vService Management Connect . . . . . . . . viiiNetwork Performance Insight technical training viiiSupport information . . . . . . . . . . . viiiConventions used in this publication . . . . . viii

Typeface conventions . . . . . . . . . . viii

Chapter 1. Administering NetworkPerformance Insight services throughAmbari . . . . . . . . . . . . . . . 1Adding nodes and components to existing hosts in acluster . . . . . . . . . . . . . . . . 1

Adding new hosts to the cluster . . . . . . . 1Adding components to new hosts in a cluster . . 1Adding ZooKeeper service to new hosts in acluster . . . . . . . . . . . . . . . 3

Deleting masters and slaves hosts in a cluster . . . 3Decommissioning NodeManager and DataNodecomponents. . . . . . . . . . . . . . 3Decommissioning the Kafka Broker component . . 4

Deleting hosts from a cluster . . . . . . . . 8Ambari Metrics System . . . . . . . . . . . 8

Ambari metrics for monitoring NetworkPerformance Insight services . . . . . . . . 9

Chapter 2. Starting and stoppingservices . . . . . . . . . . . . . . 11Controlling the services from Ambari administrationinterface . . . . . . . . . . . . . . . 11Controlling the Ambari server and Ambari agentservices. . . . . . . . . . . . . . . . 12Restarting Tivoli Network Manager Storm Spout . . 12

Chapter 3. Database administration . . 13Network Performance Insight backup and restore . 13

Backing up data - HDFS . . . . . . . . . 14Backing up data - Local backup . . . . . . 15Restoring data - HDFS. . . . . . . . . . 17Restoring data - Local backup . . . . . . . 18

Backing up and restoring Ambari database . . . . 19

Notices . . . . . . . . . . . . . . 23Trademarks . . . . . . . . . . . . . . 25Terms and conditions for product documentation. . 26

© Copyright IBM Corp. 2015, 2016 iii

Page 4: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

iv Administering Network Performance Insight

Page 5: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

Administering Network Performance Insight

Use this information to administer IBM® Network Performance Insight, Version1.2.0 and its integration services.

Intended audienceThe audience who are network administrator or operations specialist responsibleforconfiguring the Network Performance Insight product suite on an enterprisenetwork.

To install Network Performance Insight successfully, you must have a thoroughunderstanding of the following subjects:v Network Performance Insight 1.2.0 systemv Basic principles of network protocols and network managementv Flow conceptsv RHEL Administrationv Jazz for Service Management

OrganizationRead this summary to help you find the information that you need.v Chapter 1, “Administering Network Performance Insight services through

Ambari,” on page 1v Chapter 2, “Starting and stopping services,” on page 11v Chapter 3, “Database administration,” on page 13

Network Performance Insight architectureIBM Network Performance Insight is a network performance monitoring system.

Network Performance Insight provides comprehensive, flexible, and scalable trafficdata management with visualization and reporting to support complex,multi-vendor, multi-technology networks. It offers a range of dashboard viewswith robust security features that are designed to meet the needs of executivemanagement and converging network and IT operations teams.

Network Performance Insight offers near real-time and interactive view on thetraffic data that helps in reduced network downtime and optimized networkperformance.

Network Performance Insight provides IBM Netcool® Operations Insight withcomprehensive IP network device performance monitoring and session trafficanalysis.

The following diagram shows how data is flowing through the variouscomponents in Network Performance Insight:

© Copyright IBM Corp. 2015, 2016 v

Page 6: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

IBM Open Platform with Apache Spark and Apache Hadoop

IBM Open Platform with Apache Spark and Apache Hadoop (IOP) can be used tohelp process and analyze the volume, variety, and velocity of data that continuallyenters your organization every day. Network Performance Insight is installed as aservice extension to the installed IBM Open Platform with Apache Spark andApache Hadoop stack.

The features of IOP that are used in installing Network Performance Insight:v IBM Open Platform with Apache Spark and Apache Hadoopv Default support for rolling upgrades for Hadoop servicesv Support for long-running applications within YARN for enhanced reliabilityv Spark in-memory distributed compute engine for dramatic performance

increasesv Apache Ambari operational framework. Apache Ambari is an open framework

for provisioning, managing, and monitoring Apache Hadoop clusters. Ambariprovides an intuitive and easy-to-use Hadoop management web UI backed byits collection of tools and APIs that simplify the operation of Hadoop clusters.

v Essentially includes the following open source technologies for working withNetwork Performance Insight:– HDFS– Kafka– Ambari– Spark– ZooKeeper

vi Administering Network Performance Insight

Page 7: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

Note: Because Zookeeper requires a majority, it is best to use an odd numberof machines. For example, with four machines ZooKeeper can only handle thefailure of a single machine; if two machines fail, the remaining two machinesdo not constitute a majority. However, with five machines ZooKeeper canhandle the failure of two machines.

Integrated products

The products that are needed to work with Network Performance Insight, V1.2.0are as follows:

Jazz™ for Service Management 1.1.3.0Dashboard Application Services Hub provides visualization and dashboardservices in Jazz for Service Management. It has a single console foradministering IBM products and related applications. Visualization forNetwork Performance Insight is federated into Dashboard ApplicationServices Hub.

Products that are integrated with Network Performance Insight 1.2.0:

IBM Tivoli® Network Manager IP Edition 4.2.0.1Tivoli Network Manager provides network discovery, device polling,including storage of polled SNMP data for reporting and analysis, andtopology visualization. In addition, Network Manager can display networkevents, perform root-cause analysis of network events, and enrich networkevents with topology and other network data.

Tivoli Netcool/OMNIbus component of IBM Netcool Operations Insight 1.4.0.3Netcool Operations Insight is powered by the fault managementcapabilities of IBM Tivoli Netcool/OMNIbus. In Network PerformanceInsight v1.2.0, Tivoli Netcool/OMNIbus 8.1.0.8 is an important part of thesolution for monitoring the network threshold violations.

Network Performance Insight services

Network Performance Insight components are running on microservice architecturethat has the software application as a suite of independently deployable, small,modular services in which each service runs a unique process and communicatesthrough a well-defined, lightweight mechanism.

For more information about these services, see IBM Network Performance Insight:Product Overview.Related information:

IBM Network Performance Insight on IBM Knowledge Center

IBM BigInsights 4.2 documentation

HDFS Architecture

Apache Hadoop YARN

Apache Kafka

Apache Zookeeper

Administering Network Performance Insight vii

Page 8: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

Service Management ConnectConnect, learn, and share with Service Management professionals: product supporttechnical experts who provide their perspectives and expertise.

Access Network and Service Assurance community at https://www.ibm.com/developerworks/servicemanagement/nsa/index.html. Use Service ManagementConnect in the following ways:v Become involved with transparent development, an ongoing, open engagement

between other users and IBM developers of Tivoli products. You can access earlydesigns, sprint demonstrations, product roadmaps, and prerelease code.

v Connect one-on-one with the experts to collaborate and network about Tivoliand the Network and Service Assurance community.

v Read blogs to benefit from the expertise and experience of others.v Use wikis and forums to collaborate with the broader user community.Related information:

IBM Network Performance Insight community on developerWorks

Network Performance Insight technical trainingFor Tivoli technical training information, see the following Network PerformanceInsight Training website at https://tnpmsupport.persistentsys.com/updated_trainings.

Support informationIf you have a problem with your IBM Software, you want to resolve it quickly.IBM provides the following ways for you to obtain the support you need:

OnlineAccess the IBM Software Support site at http://www.ibm.com/software/support/probsub.html .

IBM Support AssistantThe IBM Support Assistant is a free local software serviceability workbenchthat helps you resolve questions and problems with IBM Softwareproducts. The Support Assistant provides quick access to support-relatedinformation and serviceability tools for problem determination. To installthe Support Assistant software, go to http://www.ibm.com/software/support/isa.

Troubleshooting GuideFor more information about resolving problems, see the problemdetermination information for this product.

Conventions used in this publicationSeveral conventions are used in this publication for special terms, actions,commands, and paths that are dependent on your operating system.

Typeface conventionsThis publication uses the following typeface conventions:

Bold

viii Administering Network Performance Insight

Page 9: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

v Lowercase commands and mixed case commands that are otherwisedifficult to distinguish from surrounding text

v Interface controls (check boxes, push buttons, radio buttons, spinbuttons, fields, folders, icons, list boxes, items inside list boxes,multicolumn lists, containers, menu choices, menu names, tabs, propertysheets), labels (such as Tip:, and Operating system considerations:)

v Keywords and parameters in text

Italic

v Citations (examples: titles of publications, diskettes, and CDs)v Words defined in text (example: a nonswitched line is called a

point-to-point line)v Emphasis of words and letters (words as words example: "Use the word

that to introduce a restrictive clause."; letters as letters example: "TheLUN address must start with the letter L.")

v New terms in text (except in a definition list): a view is a frame in aworkspace that contains data.

v Variables and values you must provide: ... where myname represents....

Monospace

v Examples and code examplesv File names, programming keywords, and other elements that are difficult

to distinguish from surrounding textv Message text and prompts addressed to the userv Text that the user must typev Values for arguments or command options

Bold monospace

v Command names, and names of macros and utilities that you can typeas commands

v Environment variable names in textv Keywordsv Parameter names in text: API structure parameters, command

parameters and arguments, and configuration parametersv Process namesv Registry variable names in textv Script names

Administering Network Performance Insight ix

Page 10: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

x Administering Network Performance Insight

Page 11: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

Chapter 1. Administering Network Performance Insightservices through Ambari

The installation of Ambari server host and agents are now complete. Use this basicinformation to help you administer your environment.

Adding nodes and components to existing hosts in a clusterApache Hadoop clusters grow and change with use. You can add more servicesafter you build your initial cluster with a base set of Hadoop services.Related information:

Using Apache Ambari to add new nodes to your existing cluster

Adding new hosts to the clusterYou can add more hosts to an existing cluster and assign these hosts to run forexample as DataNodes and NodeManagers to expand both HDFS storage capacityand YARN processing power.

Procedure1. Open a browser and access the Ambari server dashboard.

Use the following default URL: http://<myserver.ibm.com>:8080The default user name is admin, and the default password is admin.

2. In the Ambari web interface, browse to the Hosts page and click the Actionsbutton.

3. Select Add New Hosts.The ADD HOST WIZARD provides a sequence of prompts similar to the onesin the Ambari installation wizard. Follow the prompts, providing informationsimilar to that provided to define the first set of hosts in your cluster.

4. Select the slave components to include on the host.For example, DataNode and NodeManager.Ambari deploys the software packages on the hosts, push the configuration tothe hosts, and start the components so they join their respective Services.

Related information:

Adding Hosts to a Cluster

Adding components to new hosts in a clusterAfter a new node is added to the cluster, expand your cluster by adding all themaster host components to the slave hosts in your cluster.

About this task

Add the following components:v npi-managerv Kafka Broker

© Copyright IBM Corp. 2015, 2016 1

Page 12: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

Procedure1. Open a browser and access the Ambari server dashboard.

Use the following default URL: http://<myserver.ibm.com>:8080The default user name is admin, and the default password is admin.

2. Click Hosts and browse to the specific host page and click the Add button.3. Start all the added services.

What to do next

If you add a Kafka Broker to your new host, make sure to reassign the topicpartition Leaders. Follow these steps:1. Run the following commands to list all the available topics as follows:

cd /usr/iop/current/kafka-broker/bin./kafka-topics.sh --zookeeper <myserver.ibm.com>:<port> --list

You might see an output as follows:

__consumer_offsetsambari_kafka_service_checkeventsitnm.ncim.network-interfacesnm.datarequestnm.monitoredinstancenm.monitoredobjectnm.polldatanpi-connect-configsnpi-connect-offsetsnpi.agg.win-mgrnpi.aggstatusnpi.cfg.domain_names-mgrnpi.cfg.retention_profile-mgrnpi.cfg.thresholds-mgrnpi.flow-collector-mgrnpi.flow_metric.interface-mgrnpi.ncpolldata.monitored_instancenpi.ncpolldata.monitored_objectnpi.notification.baseline.dynamic_definitionnpi.notification.entity_metric.rawnpi.notification.flow_metric.agg-001daynpi.notification.flow_metric.agg-001minnpi.notification.flow_metric.agg-030minnpi.notification.flow_metric.rawnpi.notification.ncim.network_interfacesnpi.notification.ncpolldata.monitoredinstancenpi.notification.ncpolldata.monitoredobjectnpi.notification.threshold.statenpi.threshold.static_definition-mgrnpi.timeseries-aggstatusnpi.timeseries.rawsnapshot.npi.agg.win-mgrsnapshot.npi.cfg.domain_names-mgrsnapshot.npi.cfg.retention_profile-mgrsnapshot.npi.cfg.thresholds-mgrsnapshot.npi.flow-collector-mgrsnapshot.npi.flow_metric.interface-mgrsnapshot.npi.threshold.static_definition-mgr

2. Use the output to create your topics-to-move.json file.3. Follow the steps as in “Reassigning Kafka topic partition Leaders” on page 5.

2 Administering Network Performance Insight

Page 13: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

Adding ZooKeeper service to new hosts in a clusterAfter a new node is added to the cluster, expand your cluster by adding all themaster host components to the slave hosts in your cluster.

Procedure1. Open a browser and access the Ambari server dashboard.

Use the following default URL: http://<myserver.ibm.com>:8080The default user name is admin, and the default password is admin.

2. Click Services > ZooKeeper > Service Actions.3. Select Add ZooKeeper Server and add it to the specific Ambari agent host.4. Restart the service.

Deleting masters and slaves hosts in a clusterDecommissioning is a process that supports removing a component from thecluster. You must decommission a master or slave before you remove thecomponent or host from service. Decommissioning helps prevent potential loss ofdata or service disruption.Related information:

Managing Hosts

Decommissioning NodeManager and DataNode componentsUse this information to decommission NodeManager and or DataNodecomponents on Ambari.

About this task

HDFS replication rule states that the number of live DataNodes must be equal orless than the replication factor. The dfs.replication is an HDFS global setting toset the replication factor that is available in hdfs-site.xml file. You can set thisvalue on Ambari as follows:1. Open a browser and access the Ambari server dashboard.

Use the following default URL: http://<myserver.ibm.com>:8080The default user name is admin, and the default password is admin.

2. Click Services > HDFS > Advanced.3. Expand the General pane and enter a value for the Block replication.

Note: This value must be equal or less than the number of live DataNodes inyour cluster.

Procedure1. In the Ambari web interface, click Hosts to find the host FQDN on which the

component resides.2. Select the specific host to decommission the components.3. Click Actions > Selected Hosts > <component_type>.

You can decommission NodeManagers and DataNodes.

Chapter 1. Administering Network Performance Insight services through Ambari 3

Page 14: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

Note: Make sure that decommissioning of NodeManagers and DataNodes iscompleted successfully with status that is changed to “Decommissioned”. Thedecommissioning of DataNodes can take a long time based on the size of yourDataNode.

4. Click Decommission.5. Delete the decommissioned component.6. Restart all services from Ambari.Related tasks:“Decommissioning the Kafka Broker component”To remove a Kafka Broker component, first reassign the Kafka topic partitionLeaders from the Kafka Broker by using the kafka-reassign-partitions.sh script,and then shutdown and delete the Kafka Broker component in Ambari.Related information:

Decommissioning Masters and Slaves

Decommissioning the Kafka Broker componentTo remove a Kafka Broker component, first reassign the Kafka topic partitionLeaders from the Kafka Broker by using the kafka-reassign-partitions.sh script,and then shutdown and delete the Kafka Broker component in Ambari.

Before you begin

Ensure that Kafka and ZooKeeper services are up and running.

Procedure1. Connect to ZooKeeper Service by using the following commands:

cd /usr/iop/current/kafka-broker/bin ./zookeeper-shell.sh <zookeeper_server>:<port>

Wherev <zookeeper_server> is the host name where ZooKeeper is running.v <port> is the port number where the ZooKeeper Service is running. By

default, 2182.2. In the ZooKeeper shell, list the brokers and their IDs by using the following

command:ls /brokers/ids

3. In the ZooKeeper shell, get the specific Kafka Broker information with the helpof the broker IDs by using the following commands:get /brokers/ids/<brokerId>

The output looks as follows:

"jmx_port":-1,"timestamp":"1473387779988","endpoints":["PLAINTEXT://<myserver.ibm.com>:6667"],"host":"<myserver.ibm.com>","version":2,"port":6667}cZxid = 0xdefctime = Thu Sep 08 22:22:59 EDT 2016mZxid = 0xdefmtime = Thu Sep 08 22:22:59 EDT 2016pZxid = 0xdefcversion = 0dataVersion = 0aclVersion = 0ephemeralOwner = 0x1570cc10b1d0000dataLength = 127numChildren = 0

4. Exit the zookeeper shell, with the following command:quit

4 Administering Network Performance Insight

Page 15: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

5. Identify the list of topics and partitions that require leadership reassignment byusing the following commands:The kafka-topics.sh script is available in /usr/iop/current/kafka-broker/bindirectory../kafka-topics.sh --zookeeper <zookeeper_server>:2182 --describe

The output looks as follows:

Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:1Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=uncompressedTopic: __consumer_offsets Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001Topic: __consumer_offsets Partition: 1 Leader: 1001 Replicas: 1001 Isr: 1001........Topic:ambari_kafka_service_check PartitionCount:1 ReplicationFactor:1 Configs:Topic: ambari_kafka_service_check Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001Topic:events PartitionCount:1 ReplicationFactor:2 Configs:Topic: events Partition: 0 Leader: 1001 Replicas: 1001,1002 Isr: 1001,1002

Note: Topic partitions that require reassignment are identified with Leadervalues that are equal to the broker ID of the node that is to be decommissioned

6. Isolate the topics for a specific broker ID that you want to decommission byrunning the following commands:For example, to decommission, broker ID 1004:./kafka-topics.sh --zookeeper `hostname`:2182 --describe | grep "Leader: 1004"

You can see the following output:

Topic: topic1 Partition: 0 Leader: 1004 Replicas: 1004 Isr: 1004Topic: topic2 Partition: 0 Leader: 1004 Replicas: 1004 Isr: 1004Topic: topic2 Partition: 2 Leader: 1004 Replicas: 1004 Isr: 1004Topic: topic3 Partition: 0 Leader: 1004 Replicas: 1004 Isr: 1004Topic: topic4 Partition: 1 Leader: 1004 Replicas: 1004 Isr: 1004Topic: topic4 Partition: 3 Leader: 1004 Replicas: 1004 Isr: 1004Topic: topic5 Partition: 1 Leader: 1004 Replicas: 1004 Isr: 1004Topic: topic6 Partition: 1 Leader: 1004 Replicas: 1004 Isr: 1004

7. Reassign partitions to a different Kafka Broker.8. Stop the Kafka Broker Service and delete the component in Ambari.9. Restart all the Network Performance Insight services and HDFS and YARN that

might be consuming from the decommissioned Kafka node in the cluster.Related tasks:“Decommissioning NodeManager and DataNode components” on page 3Use this information to decommission NodeManager and or DataNodecomponents on Ambari.Related information:

How to Delete a Component

Reassigning Kafka topic partition LeadersUse these steps to reassign the Kafka topic partition Leaders to a different KafkaBroker in your cluster.

Before you begin

Ensure that Kafka and ZooKeeper Services are up and running.

Procedure1. Create a topics-to-move.json with the following format:

{"topics": [

{"topic": [topic]}, #Topics of partition to reassign

Chapter 1. Administering Network Performance Insight services through Ambari 5

Page 16: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

...],"version":1

}

For example:{"topics": [{"topic": "topic1"},{"topic": "topic2"},{"topic": "topic3"},{"topic": "topic4"},{"topic": "topic5"},{"topic": "topic6"}],"version":1}

2. To ease the topic partition leader reassignment, run the following commands togenerate a proposed partition reassignment configuration file:The kafka-reassign-partitions.sh script is available in /usr/iop/current/kafka-broker/bin directory. Enter the command in a single line../kafka-reassign-partitions.sh --zookeeper <zookeeper_server>:<port>--topics-to-move-json-file <topics-to-move.json> --broker-list <broker_IDs> --generate

For example, to distribute topic partition leaders to broker IDs 1001 and 1002,run the command as follows:./kafka-reassign-partitions.sh --zookeeper <myserver.ibm.com>:2182--topics-to-move-json-file /tmp/topics-to-move.json --broker-list "1001,1002" --generate

The output looks as follows:Current partition replica assignment{"version":1,"partitions":[{"topic":"topic2","partition":1,"replicas":[1002]},{"topic":"topic5","partition":1,"replicas":[1004]},{"topic":"topic3","partition":1,"replicas":[1002]},{"topic":"topic5","partition":0,"replicas":[1002]},{"topic":"topic2","partition":0,"replicas":[1004]},{"topic":"topic5","partition":2,"replicas":[1002]},{"topic":"topic6","partition":1,"replicas":[1004]},{"topic":"topic6","partition":0,"replicas":[1002]},{"topic":"topic1","partition":1,"replicas":[1002]},{"topic":"topic4","partition":2,"replicas":[1002]},{"topic":"topic4","partition":3,"replicas":[1004]},{"topic":"topic1","partition":0,"replicas":[1004]},{"topic":"topic4","partition":1,"replicas":[1004]},{"topic":"topic2","partition":2,"replicas":[1004]},{"topic":"topic4","partition":0,"replicas":[1002]},{"topic":"topic3","partition":0,"replicas":[1004]}]}Proposed partition reassignment configuration is as follows:{"version":1,"partitions":[{"topic":"topic2","partition":1,"replicas":[1001]},{"topic":"topic5","partition":1,"replicas":[1001]},{"topic":"topic3","partition":1,"replicas":[1002]},{"topic":"topic5","partition":0,"replicas":[1002]},{"topic":"topic2","partition":0,"replicas":[1002]},{"topic":"topic5","partition":2,"replicas":[1002]},{"topic":"topic6","partition":1,"replicas":[1002]},{"topic":"topic6","partition":0,"replicas":[1001]},{"topic":"topic1","partition":1,"replicas":[1002]},{"topic":"topic4","partition":2,"replicas":[1001]},{"topic":"topic4","partition":3,"replicas":[1002]},{"topic":"topic1","partition":0,"replicas":[1001]},{"topic":"topic4","partition":1,"replicas":[1002]},{"topic":"topic2","partition":2,"replicas":[1002]},{"topic":"topic3","partition":0,"replicas":[1001]},{"topic":"topic4","partition":0,"replicas":[1001]}]}

Note: All topics from Broker ID 1004 are now proposed to be reassigned to1001 and 1002.

3. Copy the content that is available in Proposed partition reassignmentconfiguration section of the output from the previous command to</tmp/reassign-partitions.json> fileThe format of the reassign-partitions.json file is as follows:

6 Administering Network Performance Insight

Page 17: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

{"version": 1,"partitions": [{

"topic": "[topic]", #Topic of partition to reassign"partition" : [partition], #partition to reassign"replicas": [brokerIds] #brokerids to reassign partition to

},...

]}

For example:{"version":1,"partitions":[{"topic":"topic2","partition":1,"replicas":[1001]},{"topic":"topic5","partition":1,"replicas":[1001]},{"topic":"topic3","partition":1,"replicas":[1002]},{"topic":"topic5","partition":0,"replicas":[1002]},{"topic":"topic2","partition":0,"replicas":[1002]},{"topic":"topic5","partition":2,"replicas":[1002]},{"topic":"topic6","partition":1,"replicas":[1002]},{"topic":"topic6","partition":0,"replicas":[1001]},{"topic":"topic1","partition":1,"replicas":[1002]},{"topic":"topic4","partition":2,"replicas":[1001]},{"topic":"topic4","partition":3,"replicas":[1002]},{"topic":"topic1","partition":0,"replicas":[1001]},{"topic":"topic4","partition":1,"replicas":[1002]},{"topic":"topic2","partition":2,"replicas":[1002]},{"topic":"topic3","partition":0,"replicas":[1001]},{"topic":"topic4","partition":0,"replicas":[1001]}]}

4. Run the reassignment of topics to replicas by using the following command:Enter the command in a single line../kafka-reassign-partitions.sh --zookeeper <zookeeper_server>:<port>--reassignment-json-file <reassign-partitions.json> --execute

For example:./kafka-reassign-partitions.sh --zookeeper <myserver.ibm.com>:2182--reassignment-json-file /tmp/reassign-partitions.json --execute

5. Verify the reassignment by using the following commands:Enter the command in a single line../kafka-reassign-partitions.sh --zookeeper <zookeeper_server>:<port>--reassignment-json-file <reassign-partitions.json> --verify

For example:./kafka-reassign-partitions.sh --zookeeper <myserver.ibm.com>:2182--reassignment-json-file /tmp/reassign-partitions.json --verify

Review the leadership assignment for topics and partitions by using thefollowing command:./kafka-topics.sh --zookeeper <zookeeper_server>:<port> --describe

Related information:

Reassign Partitions Tool

Chapter 1. Administering Network Performance Insight services through Ambari 7

Page 18: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

Deleting hosts from a clusterDeleting a host removes the host from the cluster.

Before you begin

Before you delete a host, complete the following tasks:v Decommission any DataNodes, NodeManager, and Kafka Broker that are

running on the host.v Stop all components that are running on the host.v Stop Ambari Metrics on each DataNode.v Stop all components that are running on the host.v Stop the Ambari Agent on the host.v Move from the host any master components, such as NameNode or

ResourceManager, running on the host.v Turn Off Maintenance Mode, if necessary, for the host.

Procedure1. Open a browser and access the Ambari server dashboard.

Use the following default URL: http://<myserver.ibm.com>:8080The default user name is admin, and the default password is admin.

2. In the Ambari web interface, browse to the Hosts page and select the specifichost that you want to remove.

3. Click Host Actions > Delete Host.4. Restart all services from Ambari.Related information:

Deleting a Host from a Cluster

Ambari Metrics SystemAmbari Metrics System (AMS) is a built-in system for collecting, aggregating, andserving metrics for various services in Ambari managed clusters.

It contains two components:

Metrics CollectorIt is the stand-alone server that collects metrics, aggregates metrics, servesmetrics from the Hadoop service sinks and the Metrics Monitor.

Metrics MonitorIt is installed on each host in the cluster to collect system-level metrics andforward to the Metrics Collector.

Metrics Hadoop SinksPlug into the various Hadoop components sinks to send Hadoop metricsto the Metrics Collector.

8 Administering Network Performance Insight

Page 19: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

Ambari metrics for monitoring Network Performance Insightservices

The Ambari web interface home page is a dashboard of the operating status ofyour cluster. The dashboard contains metrics widgets that provide statusinformation for each service in your cluster, and the status of your cluster. You canuse some of these Ambari metrics to monitor Network Performance Insightservices.

By default, you can see metrics for HDFS, YARN, MapReduce, and ZooKeeper inaddition to cluster-wide metrics in the Ambari dashboard. You can add andremove individual widgets, and rearrange the dashboard by dragging anddropping each widget to a new location in the dashboard.

Status information appears as pie and bar charts. You can see the following metricsfor Network Performance Insight services:

Widget Description Metrics

NetworkPerformanceInsight service

Cluster membership Number of microservicesthat are in the cluster.This metric is a divisionof MemberCount byStartedInstance.

MemberCountNumber of instances that are able tojoin as a cluster member.

StartedInstanceNumber of instances that aredeployed.

Manager

Total deployedinstances

Number of NetworkPerformance Insightmicroservices that aredeployed in the cluster.

TotalInstance Manager

Flow Agg-IFrecords read

Total records that areread for FlowAggregation Interfacegrouper.

FlowAggIFRecordsRead.sum Flow Analytics

Flow Analytics IPAddress Count

Number of IP addressesthat are resolved by FlowAnalytics Inventory.

FlowInventoryIPAdressCount.sum Flow Analytics

Flow Collectordisabled interface

Number of Interfaces thatare disabled.

FlowCollectorDisabledInterfaceCount.max Flow Collector

Flow Collectoravailableinterface

Number of Interfaces thatare available to be added.

FlowCollectorAvailableInterfaceCount.min Flow Collector

Flow Collectordiscarded rawrecords count

Total Number of RAWrecords that arediscarded for all Flowcollectors.

FlowCollectorFilteredRecords.sum Flow Collector

Flow Collectortotal recordswritten

Total number of RAWrecords that are writtento Storage.

FlowCollectorWrittenRecords.sum Flow Collector

Flow Collectortotal recordsreceive

Sum of RAW records thatare received from allFlow Collectors.

FlowCollectorRecordReceived.sum Flow Collector

Chapter 1. Administering Network Performance Insight services through Ambari 9

Page 20: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

Widget Description Metrics

NetworkPerformanceInsight service

ITNM CollectorNCIM Interfacerecords received

Number of NCIMNetwork Interfacerecords that are receivedfrom Kafka connectthrough Tivoli NetworkManager database.

ITNMCollectorNCIMRecordReceived Tivoli NetworkManager Collector

ITNM monitoredinstance

Number of monitoredinstances that arereceived from Kafkatopic and processed byTivoli Network ManagerCollector.

ITNMCollectorMonitoredInstances. Tivoli NetworkManager Collector

ITNM monitoredobjects

Number of monitoredobjects that are receivedfrom Kafka topic andprocessed by TivoliNetwork ManagerCollector.

ITNMCollectorMonitoredObjects Tivoli NetworkManager Collector

ITNM entitymetrics records

Tivoli Network ManagerEntity Merics RAWrecords that areprocessed by TivoliNetwork ManagerCollector.

ITNMCollectorPollData Tivoli NetworkManager Collector

ITNM PollDataKafka Topic Lag

Tivoli Network Managerpoll data Kafka topic lagoffset by Collector.

CollectorKafkaPolldataLag Tivoli NetworkManager Collector

ITNM NCIMInterfaces

Tivoli Network ManagerNCIM Interfaces that areprocessed by TivoliNetwork ManagerCollector.

ITNMCollectorNCIMInterfaces Tivoli NetworkManager Collector

EventsProcessed/Stored

Threshold events that areprocessed and stored byEvent service.

v EventsProcessed

v EventsStored

Event

DNS NameResolution

Forward resolution(domain name to IPaddress) requests andresponses.

v DNSResolveNameRequest

v DNSResolveNameResponse

DNS

DNS AddressResolution

Reverse resolution (IPaddress to domain name)requests and responses.

v DNSResolveAddressRequest

v DNSResolveAddressResponse

DNS

DNS ServiceAvailable MessageBuffer

DNS Service free bufferqueue to DNS server.Difference betweenDNSBufferMessageCapacityandDNSMessageBufferSize.

v DNSMessageBufferCapacity

Buffer capacity for caching before sendingto DNS.

v DNSMessageBufferSize

Buffer used. This metric shows thecapacity-size.

DNS

Related information:

Ambari metrics

10 Administering Network Performance Insight

Page 21: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

Chapter 2. Starting and stopping services

Use this information to start and stop Network Performance Insight and its relatedcomponents.

About this task

Note: If you must restart your server, stop all the services on that server. After therestart, start all the Network Performance Insight services manually.

Controlling the services from Ambari administration interfaceStop all IBM Open Platform with Apache Spark and Apache Hadoop services, byeither using the Ambari administration interface or command line to start AmbariREST APIs.

Procedure

Stopping the servicesv Click Actions > Stop All from the Ambari web interface.

Then, wait for all of the services to stop.v Optional: Follow this sequence to stop the services on Ambari web interface:

The order in which to stop the services:1. Network Performance Insight2. MapReduce23. YARN4. HDFS5. KAFKA6. Ambari Metrics7. ZooKeeper

Starting the servicesv Click Actions > Start All from the Ambari web interface.v Optional: Follow this sequence to start the services on Ambari web interface:

The order in which to start the services:1. ZooKeeper2. Ambari Metrics3. KAFKA4. HDFS5. YARN6. MapReduce27. Network Performance Insight

© Copyright IBM Corp. 2015, 2016 11

Page 22: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

Controlling the Ambari server and Ambari agent servicesCommands to control the Ambari server and Ambari agent services.

Procedurev To start the Ambari server:

service ambari-server start

v To stop the Ambari server:service ambari-server stop

v To restart the Ambari server:service ambari-server restart

v To check the Ambari server processes:ps -ef | grep Ambari

v To stop the Ambari agent by using the following command:Run this step on the specific Ambari agent server.service ambari-agent stop

v To start the Ambari agent by using the following command:Run this step on the specific Ambari agent server.service ambari-agent start

Restarting Tivoli Network Manager Storm SpoutThe Storm Spout that is available with Tivoli Network Manager provides pollingdata and related metadata through Kafka.

Procedure1. Go to the following directory:

cd $NCCOREHOME/precision/bin

2. Source the environment.source $NCCOREHOME/env.sh

3. Restart Storm by using the following commandsitnm_stop stormitnm_start storm

Related information:

Starting and stopping Apache Storm

12 Administering Network Performance Insight

Page 23: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

Chapter 3. Database administration

Provides information about essential administration tasks such as backing up andrestoring your traffic data that is stored in a specialized, indigenously developeddatabase in IBM Network Performance Insight.

A backup is a safeguard against unexpected data loss and application errors. If youlose the original data, then you can reconstruct it by using a backup.

Important: As a part of database best practices, it is recommended that you runbackup operation regularly. Backup and recovery procedures protect your databaseagainst data loss and reconstruct the data, if there is a data loss.

You can use Network Performance Insight backup and restore scripts.

Network Performance Insight backup and restoreThere are two modes of backup in Network Performance Insight 1.2.0

The following are some typical scenarios of a backup activity.v Perform an ad hoc backup before an application or services upgradev A periodic back up to recover from a disaster or failure scenariosv Single node cluster backup in local directory

It is a best practice to schedule a backup process regularly. You can set up a cronjob to run the backup command.

The following are the Network Performance Insight Backup Contents.v Network Performance Insight Storage datav Domain Objects

The following are the two types of backup modes:v HDFS - Store the Network Performance Insight Backup Contents in HDFS.v Local Backup - Store the Network Performance Insight Backup Contents in

HDFS and creates a tar file to a specified local directory.

Domain Objects

Domain object snapshots, which are maintained in Kafka are critical for managingNetwork Performance Insight Domain processes. Hence important to have abackup setup regularly.

The following list the Network Performance Insight Kafka topics that are backupduring the procedure:v snapshot.npi.cfg.domain_names-mgrv snapshot.npi.cfg.retention_profile-mgrv snapshot.npi.cfg.thresholds-mgrv snapshot.npi.flow.interface-mgrv snapshot.npi.threshold.static_definition-mgr.json

© Copyright IBM Corp. 2015, 2016 13

Page 24: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

Directory Structure

An HDFS cluster NameNode manages the file system namespace and regulatesaccess to files by clients.

Apart from command line, you can access to monitor the HDFS directories byusing Ambari Server user interface. The Domain Objects state is kept as JSONformat stored in HDFS directory.

The output of the backup is located at the following directories:v Network Performance Insight Storage data at /npi/work/storage/.snapshot/

<BACKUP_VERSION>

v Domain Objects at /npi/work/dmbackup/<BACKUP_VERSION>

Note: <BACKUP_VERSION> is the backup directory with the following naming format:yyyyMMddHHmmss

You can access the NameNode UI by using the Ambari Server UI. To access theNameNode UI from Ambari Server UI, select Services > HDFS. Click Quick Links> NameNode UI.

The NameNode UI can also be used to browse the file system. From theNameNode UI, select Utilities > Browse the file system . Enter the directory pathand click Go!.

For more information about deploying clusters, see Deploying Network PerformanceInsight clusters in Installing IBM Network Performance Insight.

Backing up data - HDFSCreate a backup of your Network Performance Insight database to prevent dataloss if there is a database service outage.

Before you begin1. Ensure that the following services are up and running:v ZOOKEEPERv KAFKAv HDFSv Network Performance InsightYou can start the services through the Ambari Server UI.For more information about backup command reference see IBM NetworkPerformance Insight: References.

Note: Resiliency might be better achieved with a combination of a periodic backupto HDFS and a multi-node HDFS setup, for example, on a multi-node clusterenvironment.

Procedure1. Log in as root or netcool user.2. Change to /<NPI_Home>/npi-storage directory:

# cd /<NPI_Home>/npi-storage

3. Run the following command to display the usage for backup command.# ./bin/backup -help

14 Administering Network Performance Insight

Page 25: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

4. Run the following command to start the database backup procedure.# ./bin/backup

The backup script runs the following tasks in the background:v Creates and stores Network Performance Insight backup contents in the

following HDFS file system directories:– Network Performance Insight Storage data at /npi/work/storage/

.snapshot/<BACKUP_VERSION>/*

For example:/npi/work/storage/.snapshot/20130412151029/

– Domain Objects at /npi/work/dmbackup/<BACKUP_VERSION>/*.jsonFor example:/npi/work/dmbackup/20130412151029/snapshot.npi.cfg.domain_names-mgr.json

Note: Only the latest HDFS backup version is retained in the HDFS file system.All the old HDFS backup versions are deleted.

5. Verify the /<NPI_Home>/npi-storage/logs/backup.log file for any issues duringthe backup procedure.All messages during backup procedure are logged in backup.log file.

Backing up data - Local backupCreate a backup of your Network Performance Insight database to prevent dataloss during database service outage.

Before you begin1. Ensure that the following services are up and running:v ZOOKEEPERv KAFKAv HDFSv Network Performance Insight

Note: Depending on the size of your HDFS DataNodes, local backup mode maybe more reasonable for a single HDFS node set up, for example a stand-alonemode of deployment.

Important: Backing up content to LOCAL_BACKUP_DIRECTORY might take a long timeto complete and large amount of disk space depending on the size of the backupcontents.

Procedure1. Log in as root or netcool user.2. Change to /<NPI_Home>/npi-storage directory:

# cd /<NPI_Home>/npi-storage

3. Run the following command to display the usage for backup command.# ./bin/backup -help

4. Run the backup script with –path option to create tar file of the backupcontents to local directory by using the following command:# ./bin/backup -path <LOCAL_BACKUP_DIRECTORY>

Chapter 3. Database administration 15

Page 26: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

Note: Make sure that the <LOCAL_BACKUP_DIRECTORY> has sufficient privilege toread and write by netcool user. The directory should have sufficient disk spaceto contain the backup contents as well.For example:# ./bin/backup -path /opt/IBM/npi/npi-storage/work/backup

You can specify any file path (-path) to store the backup tar file. In thisexample, tar file is created at /opt/IBM/npi/npi-storage/work/backup. Bydefault, the backup script takes the following parameter.

Parameter Default value

-backupTool <NPI_Home>/npi-storage/bin/backup-toolNote: The backup script uses the defaultbackup-tool script that is available in theNetwork Performance Insight package.

The backup script runs the following tasks in the background:v Creates and stores Network Performance Insight backup contents in the

following HDFS file system directories:– Network Performance Insight Storage data at /npi/work/storage/

.snapshot/<BACKUP_VERSION>/*

For example:/npi/work/storage/.snapshot/20130412151029/

– Domain Objects at /npi/work/dmbackup/<BACKUP_VERSION>/*.jsonFor example:/npi/work/dmbackup/20130412151029/snapshot.npi.cfg.domain_names-mgr.json

Creates a copy of Network Performance Insight backup contents in a tar filewithin the specified backup path directory (<LOCAL_BACKUP_DIRECTORY>).For example, the backup contents that are created in /opt/IBM/npi/npi-storage/work/backup:

# cd /opt/IBM/npi/npi-storage/work/backup

# ls -lrttotal 228236-rw-r--r-- 1 netcool hadoop 75581440 Jul 28 17:21 20160728171318.tar

5. Optional: Run the backup script with –backupTool option, to override thedefault backup-tool script, by using the following command:# cd <NPI_Home>/npi-storage# ./bin/backup -path <LOCAL_BACKUP_DIRECTORY> -backupTool <Customised_script>

Where, the Customized_script is the full path of the script.The Customized_script takes the following parameter from the backup script:a. <LOCAL_BACKUP_DIRECTORY> specified with -path.b. The backup version name (<BACKUP_VERSION>), for example 20160806134501.

Note: It depends entirely to the user on how they want to customize theirbackup-tool script.

6. Verify the /<NPI_Home>/npi-storage/logs/backup.log file for any issues duringthe backup procedure.All messages during backup procedure are logged in backup.log file.

16 Administering Network Performance Insight

Page 27: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

Restoring data - HDFSYou can restore the Network Performance Insight database to a previous state byusing the restore option. A backup image of the database must exist before you canuse this script. The database is restored to the same state as the backup copy.

Before you begin1. Ensure that the Network Performance Insight service is down.2. Ensure that the HDFS, Kafka, and ZooKeeper services are running.

You can stop Network Performance Insight services through Ambari Server UI.

About this task

Explains how to restore from HDFS on Network Performance Insight databaseenvironment.

See restore command reference in IBM Network Performance Insight: References.

Procedure1. Log in as root or netcool user.2. Change to the <NPI_Home>/npi-storage directory:

# cd <NPI_Home>/npi-storage

3. Run the following command to display the usage for restore command.# ./bin/restore -help

4. To restore from HDFS, run the following command:# ./bin/restore

After the restore procedure successfully completes, the Network PerformanceInsight backup content is copied into the respective HDFS directories.

5. Start the Network Performance Insight process by using the Ambari Server UI.Ensure that process starts without any errors and your Network PerformanceInsight system works as expected.

6. Verify the /<NPI_Home>/npi-storage/logs/restore.log file for any issuesduring the restore procedure.All messages during restore procedure are logged in restore.log file.a. If the restore procedure fails, the script tries to revert the restore procedure.

The following message is seen in the log when restore procedure fails andthe revert restore is triggered.GYMSB1021E: Begin revert Restore

b. You should seek technical support help for the following cases:1) You are unable to resolve issues that are highlighted in the log where

you cant proceed with a restore rerun.2) If there are issues starting Network Performance Insight process.3) You see the following message, which indicates the revert restore failed:

GYMSB1022E: Failed to revert restore. error: <error stack trace>

4) You see the following message, which indicates that the cleanup ofDomain Object Kafka topics failed:GYMSB1019E: Failed to clean Domain Object Kafka Topics, error: <stack trace>

orGYMSB1027E: Failed to clean Domain Object Kafka Topics, cause: Topics: ${topic} exists after delete

Chapter 3. Database administration 17

Page 28: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

Restoring data - Local backupExplains how to restore from a local backup tar file on Network PerformanceInsight database environment.

Before you begin1. Ensure that the Network Performance Insight service is down.2. Ensure that the HDFS, Kafka, and ZooKeeper services are running.

Restriction: You cannot run an HDFS backup restore immediately after running alocal backup restore. It can only be done after a backup procedure is run after thelocal backup restore procedure.

Procedure

Restore from a Local backup1. Log in as root or netcool user.2. Change to the <NPI_Home>/npi-storage directory:

# cd <NPI_Home>/npi-storage

3. Run the following command to display the usage for restore command.# ./bin/restore -help

4. To restore from a local backup file, run the following command:# ./bin/restore -localBackup <LOCAL_BACKUP_TAR_FILE>

Note: You need to specify the full path of the backup tar file(<LOCAL_BACKUP_TAR_FILE>).For example:# ./bin/restore -localBackup /opt/IBM/npi/npi-storage/work/backup/20160718173010.tar

After the restore procedure successfully completes, the Network PerformanceInsight storage backup tar file is extracted and copied into the respective HDFSdirectories.

5. Start the Network Performance Insight process by using the Ambari Server UI.Ensure that process starts without any errors and your Network PerformanceInsight system works as expected.

6. Verify the /<NPI_Home>/npi-storage/logs/restore.log file for any issuesduring the restore procedure.All messages during restore procedure are logged in restore.log file.a. If the restore procedure fails, the script tries to revert the restore procedure.

The following message is seen in the log when restore procedure fails andthe revert restore is triggered.GYMSB1021E: Begin revert Restore

b. You should seek technical support help for the following cases:1) You are unable to resolve issues that are highlighted in the log where

you cant proceed with a restore rerun.2) If there are issues starting Network Performance Insight processes.3) You see the following message, which indicates the revert restore failed:

GYMSB1022E: Failed to revert restore. error: <error stack trace>

4) You see the following message, which indicates that the cleanup ofDomain Object Kafka topics failed:GYMSB1019E: Failed to clean Domain Object Kafka Topics, error: <stack trace>

18 Administering Network Performance Insight

Page 29: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

orGYMSB1027E: Failed to clean Domain Object Kafka Topics, cause: Topics: ${topic} exists after delete

Backing up and restoring Ambari databaseAmbari server stores cluster configurations in a database. The default database isPostgresSQL.

Before you begin

Ensure that Ambari Server is installed. For more information, see Installing NetworkPerformance Insight guide.

About this task

Describes the steps for backing up and restoring the PostgresSQL database.

Regular backup of Ambari server is a good practice to keep your Ambari serverrunning.

It is important and best practice to create a backup of the Ambari database beforemaking any changes to it. For instances like before cluster upgrades, changingservice configurations, or adding new servers. Backing up enables you to restoreyour cluster to its original state on any unseen circumstances. Careful planning ofbackup operations and reliable equipment makes file recovery easier and faster.

Procedure

Backing up the server configuration1. Log in to the Ambari server host as root user. You need to have the privileges

to start and stop the Ambari server and agent and read the PostgresSQLdatabase.a. From a terminal window on the Ambari server node, stop the Ambari

server.# service ambari-server stop

b. On each Ambari agent node, stop the Ambari agent by using the followingcommand:ambari-agent stop

2. Create a directory to place the database backup SQL files. For example:

# cd /tmp# mkdir dbdumps

3. Change to the dbdumps directory.# cd /tmp/dbdumps

4. Create the database backup SQL files.a. Run the following command to create ambari.sql.

# pg_dump -U ambari ambari > ambari.sql

Enter the Ambari database password in the prompt: bigdata

Note: To check the database password on the Ambari server, open/etc/ambari-server/conf/ambari.properties and check the propertyserver.jdbc.user.passwd, which points to a file that contains the Ambaridatabase password.

Chapter 3. Database administration 19

Page 30: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

b. Run the following command to create ambarirca.sql.# pg_dump -U mapred ambarirca > ambarirca.sql

Enter the password in the prompt: mapred

5. Verify the database backup SQL files are created:

# ls /tmp/dbdumpsambari.sql ambarirca.sql

These database backup SQL files need to be backed up to a server for a futurerestore.

Restoring the server configuration6. Log in to the Ambari server host as root user. You need to have the privileges

to start and stop the Ambari server and read the PostgresSQL database.7. Stop the Ambari server and its agent by using the following command:

# service ambari-server stop# service ambari-agent stop

8. Change to the dbdumps directory.# cd /tmp/dbdumps

9. Connect to PostgresSQL# su – postgres# psql

10. Drop the existing databases by using the following command:# drop database ambari;# drop database ambarirca

For example:

# su postgres# psqlpsql (8.4.18)Type "help" for help

postgres# drop database ambari;DROP DATABASE

postgres# drop databas ambarirca;DROP DATABASE

11. Run the following command to verify that the databases are dropped:\list

The ambari and ambarirca databases are not listed.12. Create new databases by running the following command:

# psqlpsql (8.4.18)Type "help" for help

postgres# create database ambari;CREATE DATABASE

postgres# create databas ambarirca;CREATE DATABASE

13. Exit PostgresSQL: ^d14. Copy the ambari.sql and ambarirca.sql databases backup files from the

backup server (which was created in Step 5) to a new server.15. Restore the files as postgres user.

20 Administering Network Performance Insight

Page 31: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

# su - postgres# psql -d ambari -f ambari.sql# psql -d ambarirca -f ambarirca.sql

16. From a terminal window on the Ambari server node, start the Ambari server:# service ambari-server start

17. On each Ambari agent node, start all the Ambari agents by running thefollowing command:ambari-agent start

18. Open a browser and access the Ambari server dashboard.For example, the default URL: http://<myserver.ibm.com>:8080

19. Verify the configuration.

Chapter 3. Database administration 21

Page 32: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

22 Administering Network Performance Insight

Page 33: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

Notices

This information was developed for products and services offered in the US. Thismaterial might be available from IBM in other languages. However, you may berequired to own a copy of the product or product version in that language in orderto access it.

IBM may not offer the products, services, or features discussed in this document inother countries. Consult your local IBM representative for information on theproducts and services currently available in your area. Any reference to an IBMproduct, program, or service is not intended to state or imply that only that IBMproduct, program, or service may be used. Any functionally equivalent product,program, or service that does not infringe any IBM intellectual property right maybe used instead. However, it is the user's responsibility to evaluate and verify theoperation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matterdescribed in this document. The furnishing of this document does not grant youany license to these patents. You can send license inquiries, in writing, to:

IBM Director of LicensingIBM CorporationNorth Castle Drive, MD-NC119Armonk, NY 10504-1785US

For license inquiries regarding double-byte character set (DBCS) information,contact the IBM Intellectual Property Department in your country or sendinquiries, in writing, to:

Intellectual Property LicensingLegal and Intellectual Property LawIBM Japan Ltd.19-21, Nihonbashi-Hakozakicho, Chuo-kuTokyo 103-8510, Japan

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THISPUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHEREXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIEDWARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESSFOR A PARTICULAR PURPOSE. Some jurisdictions do not allow disclaimer ofexpress or implied warranties in certain transactions, therefore, this statement maynot apply to you.

This information could include technical inaccuracies or typographical errors.Changes are periodically made to the information herein; these changes will beincorporated in new editions of the publication. IBM may make improvementsand/or changes in the product(s) and/or the program(s) described in thispublication at any time without notice.

Any references in this information to non-IBM websites are provided forconvenience only and do not in any manner serve as an endorsement of those

© Copyright IBM Corp. 2015, 2016 23

Page 34: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

websites. The materials at those websites are not part of the materials for this IBMproduct and use of those websites is at your own risk.

IBM may use or distribute any of the information you provide in any way itbelieves appropriate without incurring any obligation to you.

Licensees of this program who wish to have information about it for the purposeof enabling: (i) the exchange of information between independently createdprograms and other programs (including this one) and (ii) the mutual use of theinformation which has been exchanged, should contact:

IBM Director of LicensingIBM CorporationNorth Castle Drive, MD-NC119Armonk, NY 10504-1785US

Such information may be available, subject to appropriate terms and conditions,including in some cases, payment of a fee.

The licensed program described in this document and all licensed materialavailable for it are provided by IBM under terms of the IBM Customer Agreement,IBM International Program License Agreement or any equivalent agreementbetween us.

The performance data discussed herein is presented as derived under specificoperating conditions. Actual results may vary.

The client examples cited are presented for illustrative purposes only. Actualperformance results may vary depending on specific configurations and operatingconditions.

Information concerning non-IBM products was obtained from the suppliers ofthose products, their published announcements or other publicly available sources.IBM has not tested those products and cannot confirm the accuracy ofperformance, compatibility or any other claims related to non-IBM products.Questions on the capabilities of non-IBM products should be addressed to thesuppliers of those products.

Statements regarding IBM's future direction or intent are subject to change orwithdrawal without notice, and represent goals and objectives only.

All IBM prices shown are IBM's suggested retail prices, are current and are subjectto change without notice. Dealer prices may vary.

This information is for planning purposes only. The information herein is subject tochange before the products described become available.

This information contains examples of data and reports used in daily businessoperations. To illustrate them as completely as possible, the examples include thenames of individuals, companies, brands, and products. All of these names arefictitious and any similarity to actual people or business enterprises is entirelycoincidental.

COPYRIGHT LICENSE:

24 Administering Network Performance Insight

Page 35: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

This information contains sample application programs in source language, whichillustrate programming techniques on various operating platforms. You may copy,modify, and distribute these sample programs in any form without payment toIBM, for the purposes of developing, using, marketing or distributing applicationprograms conforming to the application programming interface for the operatingplatform for which the sample programs are written. These examples have notbeen thoroughly tested under all conditions. IBM, therefore, cannot guarantee orimply reliability, serviceability, or function of these programs. The sampleprograms are provided "AS IS", without warranty of any kind. IBM shall not beliable for any damages arising out of your use of the sample programs.

Each copy or any portion of these sample programs or any derivative work mustinclude a copyright notice as follows:

© (your company name) (year).Portions of this code are derived from IBM Corp. Sample Programs.© Copyright IBM Corp. _enter the year or years_.

TrademarksIBM, the IBM logo, and ibm.com are trademarks or registered trademarks ofInternational Business Machines Corp., registered in many jurisdictions worldwide.Other product and service names might be trademarks of IBM or other companies.A current list of IBM trademarks is available on the web at "Copyright andtrademark information" at www.ibm.com/legal/copytrade.shtml.

Adobe, Acrobat, PostScript and all Adobe-based trademarks are either registeredtrademarks or trademarks of Adobe Systems Incorporated in the United States,other countries, or both.

IT Infrastructure Library is a registered trademark of the Central Computer andTelecommunications Agency which is now part of the Office of GovernmentCommerce.

Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo,Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks orregistered trademarks of Intel Corporation or its subsidiaries in the United Statesand other countries.

Linux is a registered trademark of Linus Torvalds in the United States, othercountries, or both

Microsoft and Windows are trademarks of Microsoft Corporation in the UnitedStates, other countries, or both.

ITIL is a registered trademark, and a registered community trademark of TheMinister for the Cabinet Office, and is registered in the U.S. Patent and TrademarkOffice.

UNIX is a registered trademark of The Open Group in the United States and othercountries.

Notices 25

Page 36: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

Java and all Java-based trademarks and logosare trademarks or registered trademarks ofOracle and/or its affiliates.

Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in theUnited States, other countries, or both and is used under license therefrom.

Linear Tape-Open, LTO, the LTO Logo, Ultrium, and the Ultrium logo aretrademarks of HP, IBM Corp. and Quantum in the U.S. and other countries.

Terms and conditions for product documentationPermissions for the use of these publications are granted subject to the followingterms and conditions.

Applicability

These terms and conditions are in addition to any terms of use for the IBMwebsite.

Personal use

You may reproduce these publications for your personal, noncommercial useprovided that all proprietary notices are preserved. You may not distribute, displayor make derivative work of these publications, or any portion thereof, without theexpress consent of IBM.

Commercial use

You may reproduce, distribute and display these publications solely within yourenterprise provided that all proprietary notices are preserved. You may not makederivative works of these publications, or reproduce, distribute or display thesepublications or any portion thereof outside your enterprise, without the expressconsent of IBM.

Rights

Except as expressly granted in this permission, no other permissions, licenses orrights are granted, either express or implied, to the publications or anyinformation, data, software or other intellectual property contained therein.

IBM reserves the right to withdraw the permissions granted herein whenever, in itsdiscretion, the use of the publications is detrimental to its interest or, asdetermined by IBM, the above instructions are not being properly followed.

You may not download, export or re-export this information except in fullcompliance with all applicable laws and regulations, including all United Statesexport laws and regulations.

IBM MAKES NO GUARANTEE ABOUT THE CONTENT OF THESEPUBLICATIONS. THE PUBLICATIONS ARE PROVIDED "AS-IS" AND WITHOUTWARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDINGBUT NOT LIMITED TO IMPLIED WARRANTIES OF MERCHANTABILITY,

26 Administering Network Performance Insight

Page 37: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

NON-INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE.

Notices 27

Page 38: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

28 Administering Network Performance Insight

Page 39: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an
Page 40: with IBM Corp. · v Apache Ambari operational framework. Apache Ambari is an open framework for pr ovisioning, managing, and monitoring Apache Hadoop clusters. Ambari pr ovides an

IBM®

Printed in USA