copyright · commvault ® stream management ... looking at the job in more detail we find that the...

Copyright

Information in this document, including URL and other website references, represents the current view of Commvault

Systems, Inc. as of the date of publication and is subject to change without notice to you.

Descriptions or references to third party products, services or websites are provided only as a convenience to you and

should not be considered an endorsement by Commvault. Commvault makes no representations or warranties, express or

implied, as to any third party products, services or websites.

The names of actual companies and products mentioned herein may be the trademarks of their respective owners. Unless

otherwise noted, the example companies, organizations, products, domain names, e-mail addresses, logos, people,

places, and events depicted herein are fictitious.

Complying with all applicable copyright laws is the responsibility of the user. This document is intended for distribution to

and use only by Commvault customers. Use or distribution of this document by any other persons is prohibited without

the express written permission of Commvault. Without limiting the rights under copyright, no part of this document may

be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic,

mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of

Commvault Systems, Inc.

Commvault may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering

subject matter in this document. Except as expressly provided in any written license agreement from Commvault, this

document does not give you any license to Commvault’s intellectual property.

COMMVAULT MAKES NO WARRANTIES OF ANY KIND, EXPRESS OR IMPLIED, AS TO THE INFORMATION CONTAINED IN

THIS DOCUMENT.

©2018 Commvault Systems, Inc. All rights reserved. Commvault, Commvault and logo, the “C” hexagon logo, Commvault

Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data

Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery

Director, CommServe, CommCell, APSS, Commvault GO, Commvault HyperScale, IntelliSnap, ROMS, Commvault

OnePass, CommVault Edge, Edge Drive, and CommValue, are trademarks or registered trademarks of Commvault

Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the

property of and used to identify the products or services of their respective owners. All specifications are subject to

change without notice.

Confidentiality

The descriptive materials and related information in the document contain information that is confidential and proprietary

to Commvault. This information is submitted with the express understanding that it will be held in strict confidence and

will not be disclosed, duplicated or used, in whole or in part, for any purpose other than evaluation purposes. All right,

title and intellectual property rights in and to the document is owned by Commvault. No rights are granted to you other

than a license to use the document for your personal use and information. You may not make a copy or derivative work

of this document. You may not sell, resell, sublicense, rent, loan or lease the document to another party, transfer or

assign your rights to use the document or otherwise exploit or use the Manual for any purpose other than for your

personal use and reference. The document is provided "AS IS" without a warranty of any kind and the information

provided herein is subject to change without notice.

©1999-2018 Commvault Systems, Inc. All rights reserved.

V11 SP11 Performance April 2018

Commvault® Education Services of 36

ABOUT THIS DOCUMENT This document is intended for all audiences and is current as of the software version and service

pack stated in the top left corner of the page.

This document is updated every three to six months depending on feature changes to Commvault®

software. The date of publish is within the document title, e.g. 170530 indicating a publish date of

May 30, 2017, and the date appears in the top right-hand side of each page. New and updated

sections are indicated in the revision history section with hyperlinks to each section and appear with

a RED heading and darker text for easy identification. For updated versions of this document, contact

us at: [email protected]

Whether an employee, partner, or customer; we all want to work collectively to provide the best

technical education material possible. If you have ideas to improve this document or corrections to

existing content, please contact us as: [email protected]

AUTHORS

Frank Celauro, Irene Grimaldi, Carl Brault

Edited by: Madelyn Moalam

REVISION HISTORY

Link Date Contributing

Author

Notes

May 30, 2017 Initial document release

Unbuffered I/O for UNIX/Linux Network Equipment Optimization

Recommended Filters Tape Chunk Size Graphic

Updated Copyright HyperScale Storage

March 28, 2018 Carl Brault SP10-11 revision

For comments, corrections, or recommendations for additional content,

contact: [email protected]

mailto:[email protected]





Contents About this Document ................................................................................................................................................... 3

Authors ...................................................................................................................................................................... 3

Revision History .......................................................................................................................................................... 3

Performance Overview ............................................................................................................................................. 5

Performance Benchmarks ......................................................................................................................................... 5

Environment Considerations ..................................................................................................................................... 6

Commvault® Stream Management ............................................................................................................................ 9

Commvault® Performance Tunables and Best Practices ............................................................................................ 11

File System Backup ............................................................................................................................................ 11

Virtual Server Agent Backup ................................................................................................................................ 14

Database Agents ................................................................................................................................................ 15

Microsoft Exchange Database Agent .................................................................................................................... 16

Network Settings ................................................................................................................................................ 17

Disk Storage ...................................................................................................................................................... 19

Tape Storage ..................................................................................................................................................... 23

Cloud ................................................................................................................................................................. 26

Auxiliary Copy Performance ................................................................................................................................ 27

Commvault Features Assessment Charts ................................................................................................................. 28

Storage Policy Configurations .............................................................................................................................. 28

Subclient Configurations ..................................................................................................................................... 32

Deduplication Configurations ............................................................................................................................... 33

SILO Storage Configurations ............................................................................................................................... 35



Performance Overview Commvault® software is a high-performance solution for protecting all data in any environment within defined protection windows. The software also provides many settings to improve performance. Before considering tuning Commvault

software, it is important to understand capabilities and limitations of all hardware and software deployed within an

environment.

There is no such thing as a static data center. Network infrastructures are constantly changing, new servers are added, mission critical business systems are moving to hybrid cloud, or public cloud infrastructures. Before considering

Commvault tunables, it is first important to understand your environment including the capabilities and limitations of the

infrastructure; specifically, the ability to transfer large amounts of data of production or backup networks.

When making modifications to an environment, changes that may positively impact one aspect of the environment can negatively affect another aspect. This is also true about Commvault settings. For example, enabling multiplexing when

writing to tape drive can improve backup speeds. However, it may have a negative impact on restores if dissimilar data types are multiplexed to the same tape. Another example is using Commvault deduplication and setting a high number of

data streams. Since client-side deduplication is being used, there will be a low impact to the network. But if the deduplication database needs to be sealed, the next set of backup operations may result in oversaturating the network

while re-baselining blocks in storage.

Performance Benchmarks Benchmarks can be divided into two kinds, component and system. Component benchmarks measure the performance of

specific parts of a process, such as the network, tape or hard disk drive, while system benchmarks typically measure the

performance of the entire process end-to-end.

Establishing a benchmark focuses your performance tuning and quantifies the effects of your efforts. Building a benchmark is made up of the following 5 steps:

Understand the process

Identify the resources involved

Minimize outside influence

Periodic test

Write it down

Understand the process

You can’t document or improve something if you don’t know what’s going on. More importantly, you need to understand

what phases a job goes through and how much each phase affects the overall outcome.

For example, a backup job over a network to a tape library takes two hours to complete. You think it should take a lot

less and you spend time, effort, and money to improve your network and tape drives and parallel the movement of data. The job now takes 1.8 hours to complete. You gained a 10% improvement.

Looking at the job in more detail we find that the scan phase of the job is taking 1.5 hours and the rest is the actual data movement. Switching the scan method reduces the scan phase time to 12 minutes. The job now takes .4 hours. You

gained a 78% improvement.

Knowing what phases a job goes through and how much each phase impacts the overall performance can help you focus

your time, effort, and money on the real problems.

Identify the resources involved

Each hardware component is going to have a theoretical performance limit and a practical one. Attempting to get improvement beyond these limits without changing the resources involved is a waste of time. Consider using newer vs.

older technologies, such as tape drives.



Minimize outside influence

Large data movements are usually done during non-production hours for two reasons – one, they can degrade production

work, and two, production work can degrade the movement of data. You want to minimize competition for resources to get a fair benchmark of what performance is achievable. In those cases, where competition cannot be eliminated, you

must accept the impact to performance or invest in more resources.

Periodic Test

A single measurement is not a benchmark. Tape devices have burst speeds that are not sustainable over the long run. Networks have various degrees of bandwidth availability over a period of time. A single snapshot check of bandwidth will

not give you a realistic expectation. Do periodic testing over the actual usage of a resource to determine its average

performance. Try to level out the peaks and valleys - or at least try to identify what causes these variations.

Multiple measurements scattered over a day can also help in establishing if an unexpected external process is impacting the environment. For example, if you have a database server that is slowly backing up at night, but when you sample

during the day, it is achieving expected performances, you can suspect an external process impacting the backup, such as

a database administrator dumping the database and copying it to another server at the same time in this example.

Write it down

The hardest lessons are the ones you must learn twice. Once you’ve established your acceptable and/or expected performance levels for each resource and end-to-end, write them down and use them as the baseline for comparing

future performance.

Environment Considerations Before modifying Commvault® software settings to improve performance, consider environmental capabilities and

limitations. Ensure the environment is optimized to the best of your team’s abilities. Commvault software can move data

at high rates of speed, but it will ultimately be limited by bottlenecks on servers and network devices.

TCP/IP

TCP/IP is the most common network transmission protocol. Factors that can degrade TCP/IP performance are:

Latency - Packet retransmissions over distance take longer and negatively impact overall throughput for a transmission path.

Concurrency - TCP/IP was intended to provide multiple users with a shared transmission media. For a single user, it is an extremely inefficient means to move data.

Line Quality - Transmission packet sizes are negotiated between sender/receiver based on line quality. A poor line connection can degrade a single link’s performance.

Duplex setting - Automatic detection of connection speed and duplex setting can result in a half-duplex connection. Full duplex is needed for best performance.

Switches - Each switch in the data path is a potential performance degrader if not properly configured.

Firewalls – Firewall is the first line of defense against hackers, malware, and viruses. There are hardware

firewall appliances and software firewalls, such as operating system firewalls. Firewalls can have minor to

moderate impacts on transfer performances.

Vendor documentation for most network interface cards (NIC), switches, and router provide useful

information on optimization of their equipment.



SCSI/RAID

SCSI is the most common device protocol used and provides the highest direct connection speed. An individual SCSI

drive’s speed is determined by spindle speed, access time, latency, and buffer. Overall SCSI throughput is also

dependent on how many devices are on the controller and in what type of configuration. The limitation of SCSI is the

distance between devices and the number of devices per controller.

RAID arrays extend the single addressable capacity and random access performance of a set of disks. The fundamental difference between reading and writing under RAID is this: when you write data in a redundant environment, you must access every place where that data is stored; when you read the data back, you only

need to read the minimum amount of data necessary to retrieve the actual data--the redundant information does

not need to be accessed on a read. Basically – writes are slower than reads.

RAID 0 (striping) or RAID 1 (mirror) or RAID 1+0 with narrow striping are the fastest configurations when it comes to sequential write performance. Wider striping is better for concurrent use. A RAID 5 configured array can

have poor write performance. The tradeoff in slower write performance is redundancy should a disk fail.

Fine tuning a RAID controller for sequential read/write may be counterproductive to concurrent

read/write. If backup/archive performance is an issue, a compromise must be arranged.

iSCSI/Fibre Channel

iSCSI or Fibre Channel protocol (FCP) is essentially serial SCSI with increased distance and device support. SCSI

commands and data are assembled into packets and transmitted to devices where the SCSI command is assembled and

executed. Both protocols are more efficient than TCP/IP. FCP has slightly better statistics than iSCSI for moving

data. Performance tuning is usually setting the correct ‘Host Bus Adapter’ configuration (as recommended by the vendor

for sequential I/O) or hardware mismatch. Best performance is achieved when the hardware involved is from the same

vendor. Given that configuration and hardware is optimum, then for both iSCSI and FCP, performance is inhibited only by

available server CPU resources.

Disk I/O

Performing I/O to disks is a slow process because disks are physical devices that require time to move the heads to the correct position on the disk before reading or writing. This re-positioning of the head is exacerbated by having many files

or having fragmented files. You can significantly improve read performance of the source data by de-fragmenting the

data on a regular basis.

Anti-Virus

Anti-viruses are intelligent software protecting a system against corrupted data by periodically scanning files systems and

ensuring that every file accessed or opened by any processes running on the system is a legitimate file (and not a virus).

You can easily imagine that when a backup runs and protects every system files, the anti-virus validation significantly

decrease backup performances. It might also access and lock Commvault files, such as log files. It is recommended on all

systems on which Commvault software is installed, to add exclusions to the anti-virus software for Commvault® software

folders, so that when Commvault related processes are in action, they do not trigger the anti-virus validation process.



Windows Recommended anti-virus exceptions

For V10 and earlier environments, any Content Store directory must be replaced by ‘Simpana.’ For V11

environments, any Content Store directory must be replaced by ‘ContentStore.’

CommServe,

client, and

MediaAgent

Installation Paths

Software Installation Path *:\Program Files\CommVault\ContentStore\**

Updates Cache folder with CVPackages and CVUpdates subfolders on the CommServe

C:\Program Files\CommVault\ContentStore\SoftwareCache

DR backup set directories on the CommServe

C:\Program Files\CommVault\CS_DR

Job Results folder C:\Program Files\CommVault\ContentStore\iDataAgent\JobResults

Index Directory folder C:\Program Files\CommVault\ContentStore\IndexCache

Additional

MediaAgent

Paths

Magnetic libraries *:\**\CV_MAGNETIC\**

Deduplication databases Obtain the deduplication database location from the CommCell® console, from the Copy Properties dialog box

of the primary copy, located in the Deduplication tab.

SharePoint

Agent

Temp folder path

C:\Users\Commvault Services account\AppData\Local\Temp

Content Indexing

and Search

Exclude entire CI Engine install folders: C:\Program Files\CommVault\ContentStore\CIServer

C:\Program Files\CommVault\ContentStore\CVCIEngineSolr folder path

C:\Program Files\CommVault\ContentStore\CVCIEngine\solr

CI Index folder path C:\Program Files\CommVault\ContentStore\CVCIEngine\solr\CIIndex

UNIX, Linux and Macintosh Recommended anti-virus exceptions

CommServe,

client, and

MediaAgent

Installation Paths

Software install directory

*/opt/commvault/**

Job Results directory /opt/commvault/iDataAgent/jobResults

Index directory /opt/commvault/IndexCache

Directory to extract installation binaries /tmp/.gxsetup

Additional

MediaAgent

Paths

Magnetic libraries

*/CV_MAGNETIC/**

Deduplication databases Obtain the deduplication database location from the CommCell® console, from the Copy Properties dialog

box of the primary copy, located in the Deduplication tab.



Commvault® Stream Management Data Streams are used to move data from source to destination. The source can be production data or Commvault protected data. A destination stream will always move to Commvault protected storage. Understanding the data stream

concept will allow a CommCell® environment to be optimally configured to meet protection and recovery windows.

Stream settings are configured in various places within the CommCell® console including the storage policy, MediaAgent,

subclient, and library. The system always uses the lowest setting. If a MediaAgent is configured to receive as many as 100 streams and one storage policy is writing through the MediaAgent and is configured to use 50 streams, then only 50

streams will be sent through the MediaAgent.

During a data protection job, streams originate at the source file or application that is being protected. One or more read

operations is used to read the source data. The number of read operations is determined by the number of subclients and within each subclient, the number of data readers or data streams, depending on which agent is managing the data.

Once the data is read from the source it is processed by the agent and then sent to the MediaAgent as job streams. The

MediaAgent then processes the data, arranges the data into chunks and writes the data to storage as device streams. The data is written to storage based on the number of writers, for a disk library, or devices (tape drives) for a tape

library.

Stream management high level overview



Stream Settings Summary Table

Features & Functionality

Description

Subclients Subclients are independent jobs, meaning each subclient will have one or more streams associated with each job.

Multi-stream subclients

Most subclients can be multi-streamed. For subclients that do not support multiple streams, multiple subclients are used to multi-stream data protection jobs.

Data readers are configured in the General tab of the subclient. Data Streams are configured in the storage device tab for MS-SQL and Oracle

subclients.

Non-Subclient based agents

Agents such as the new Exchange Mailbox agent manage streams at the object level. For Exchange, each mailbox is protected as a single stream.

The default subclient data readers setting is still used as the primary stream governor for the maximum number of concurrent objects that can be protected.

Job Streams Job streams are active network streams moving from source (client or MediaAgent) to destination (MediaAgent).

The Job controller shows the total number of job streams currently in use in the bottom of the window and the job stream ‘high watermark’ for the CommCell environment.

Add the ‘Number of Readers in Use’ field in the job controller to view the number of streams being used for each active job.

Device Streams Configured in the Storage Policy properties. Determines how many concurrent write operations will be performed to a library. This

number should be set to equal the number of drives or writers in the library to maximize throughput.

Multiplexing is used to consolidate multiple job streams into single device streams.

Drives For a removable media library writing data sequentially to devices, there will be one device stream per drive.

Writers For a disk library where random read/write operations can be performed the number of writers should be set to allow the maximum throughput without creating bottlenecks in your network, MediaAgents, or disks.



Commvault® Performance Tunables and Best Practices

File System Backup

Consider the following key points when backing up the File System Agent:

For backups on Windows operating systems, ensure source disks are defragmented.

Ensure all global and local filters are properly configured. Consult Commvault online documentation for

recommended filters.

If source data is on multiple physical drives, increase the number of data readers to multi-stream protection jobs.

For larger high-speed disk, a maximum of two data readers can be set for an individual disk. Enable ‘Allow

Multiple Data Readers within a Drive or Mount Point’ to allow multiple streams on a single disk.

If source data is on a RAID volume, create subclient(s) for the volume and increase the number of data readers

to improve performance. Enable the ‘Allow Multiple Data Readers within a Drive or Mount Point’ option.

Consider using synthetic full, or better, DASH Full backups over traditional full backups.

Consider using the Commvault OnePass® agent to archive older ‘stale’ data.

For large volumes containing millions of objects use the File System Block-Level Backup.

Consider using multiple subclients and stagger backup operations over a weekly or even monthly time period.

For supported hardware, consider using the Commvault IntelliSnap® feature to snap and backup volumes using a

proxy server.

Increase the ‘Application Read Size’ from the default of 64KB to 512KB.

Data Readers

Disk I/O is the most costly, time-consuming portion of a data movement job. Using multiple data readers (also called data

streams) can improve performance.

Conditions that can degrade performance for the File System Agent:

In some configurations, such as concurrent backups that use embedded agents on multiple virtual machines

(VMs) in a hypervisor environment, using multiple data readers for each backup might overwhelm the disk I/O

and degrade performance. In this situation, using only one data reader for each VM might achieve the best

performance.

Internal algorithms determine the maximum number of data readers that can read concurrently from a single

physical drive. Too many data readers on a single physical drive can degrade performance.

Subclient content is divided between data readers based on physical drives. Thus, the first data reader reads from

the first physical drive, the second data reader reads from the second physical drive, and so on. By default, only one data reader is allowed per physical drive, regardless of how many data readers are configured. Often, a data

reader completes before the other data reader completes, which reduces the performance gain of using multiple

data readers.



Allow Multiple Readers within a Drive or Mount Point

For the File System Agent, the Number of Data Readers value determines the number of parallel read operations from the

data source.

The ‘Allow multiple data readers within a drive or mount point’ option helps you to use data readers more efficiently. For

example, if you have subclient content that spans 4 physical drives, and you configure 8 data readers. Each physical drive

gets 2 data readers. When one data reader completes its task, it assists another physical drive. This process continues

until all data is read. This process maximizes the time that multiple data streams are moving data, which can improve

performance.

Setting the number of readers and multiple readers within a drive or mount point

Application Read Size

The application read size is the size of the application data that is read from the clients during backup jobs.

Values for the application read size must be in the power of 2; the minimum value is 64 KB, and the maximum value is

4,096KB (4MB).

Recommended values for Application Read Size

NTFS volume 512KB

ReFS volume 2,048KB

When the size of the application data that is read during backup jobs matches the source application’s internal buffer

allocation, the overhead is minimized, and performance is improved. To achieve the optimal rate of data transfer during

backup jobs, configure the application read size based on the source application's internal buffer allocation. You can

increase the application read size to reduce the amount of data that is read from the given application. Reducing the

amount of data that is read also reduces the number of I/O jobs that are performed against the application. As a result,

overall backup performance might improve. However, backup memory usage might also increase, which might

inadvertently consume additional resources from the application.



Commvault recommends that you set the application read size at either the default value or at the

cluster size that is directed by the application.

Microsoft NTFS uses a default cluster size (allocation unit) of 4KB by default. The 4KB cluster size was established when

2GB disks were considered large. Today, Microsoft recommends using a cluster size of 16KB or higher for NTFS volumes

on servers. Commvault recommends that you use 64KB clusters, which matches the Microsoft ReFS default cluster size.

With source data on volumes that have a 64KB cluster size, Commvault recommends using an application read size of at

least 2,048KB for NTFS and ReFS.

For information about cluster sizes, see the Microsoft support article “Default cluster size for NTFS,

FAT, and exFAT”.

Setting the Application Read Size



Virtual Server Agent Backup

General guidelines

To optimize virtual environment data protection and recovery performance, contact Commvault Professional

Services for the latest guidance and assistance.

Use the Commvault Virtual Server Agent (VSA) to protect most VMs. Specific I/O intensive VMs may require more

advanced protection methods.

Use backup set or subclient VM filters to filter VMs that don’t require protection.

Use subclient VM rules to group priority VMs for protection. For example, use the power state rule to set

infrequent schedules of VMs that are not powered on.

Maximize VM backup concurrency by increasing the ‘Data Readers’ option. Use caution as setting the readers

option too high can cause performance degradation on backups and datastores or volumes hosting the VMs. As a

general starting point, start with two VM backups per datastore or volume.

It is preferred to use physical VSA MediaAgent proxies versus virtual server MA proxies.

Ensure there are enough proxies to handle data movement load.

Use Commvault® software client-side deduplication and DASH Full backups.

For larger VMs, consider using the Commvault OnePass® feature to archive older ‘stale’ data.

Consider using multiple subclients and staggering schedules for when incremental and full or synthetic (DASH)

full backups run.

Ensure Change Block Tracking (CBT) is enabled for all virtual machines, when applicable.

VMware specific guidelines

Ensure VSA proxies can access storage using the preferred transport mode. SAN transport and HotAdd will fall

back to NBD mode if they cannot access VMs from the SAN or DataStore.

When protecting applications in a virtual environment:

Using the VSA to protect applications without the Application Aware feature or agents installed within the VM may

result in crash consistent backups.

For low to medium I/O applications, use the Application Aware feature. Check the Commvault Online

Documentation for a list of applications supported by the VSA Application Aware feature.

For I/O intensive applications, it is still preferred to use application agents installed in the VMs.

Commvault IntelliSnap® for VSA:

Use IntelliSnap for VSA to protect I/O intensive VMs.

Define subclients by datastore affinity. When hardware snaps are performed the entire datastores is snapped

regardless of whether the VM is being backed up.

For smaller Exchange or MS-SQL databases (less than 500GB), application consistent snapshots can be performed

using the IntelliSnap feature and VSA.



Database Agents

General Guidelines

For large databases that are being dumped by application administrators, consider using Commvault database agents to provide multi-streamed backup and restores.

When using Commvault database agents for instances with multiple databases, consider creating multiple subclients to manage databases.

For large databases, consider increasing the number of data streams for backing up database. For multi-streamed

subclient backups of SQL and Sybase databases, the streams should not be multiplexed. During auxiliary copy operations to tape if the streams are combined to a tape, they must be pre-staged to a secondary disk target

before they can be restored.

For MS-SQL databases using file/folder groups, separate subclients can be configured to manage databases and file/folder groups.

Database Agent Streams

Disk I/O is the most costly, time-consuming portion of a data movement operation. Using parallel data readers (also

called data streams) can improve performance. For databases, the Number of Data Readers value determines the number

of parallel read operations that are requested from the database application.

Before you modify the number of data readers, Commvault recommends recording baseline

throughput performance using the default settings, which are the recommended settings. You can

then modify the number of data readers until you achieve the fastest throughput performance.

SQL data streams configuration



Microsoft Exchange Database Agent

Application Read Size

The performance of both regular backup operations and IntelliSnap backup operations of an Exchange Database can

benefit greatly from an application read size of 4MB (4,096 KB). The default value is 64KB.

For most Data Availability Group (DAG) environments, backup operations are performed on the passive node, and

memory usage for the application read size is not a concern. If production performance problems occur, then you can

decrease the application read size.

Multi-streamed Exchange Database Backups

Multi-streamed backups of Exchange database reduce backup time by allocating streams on a per database level. The

maximum number of streams that is used by a backup is determined by the number of databases in the Exchange

environment. If a subclient’s content contains four databases, then four streams could be used – each stream protecting

one database.

In a DAG environment, the stream allocation is based on the number of nodes. When the job starts, the stream logic

automatically assigns one stream to each node. If there are additional streams remaining, they are allocated based on

which node has the most databases. The stream allocation process continues in order of Exchange servers in the DAG

environment containing the most databases to fewest in a prioritized round-robin method until all streams are allocated.

Configure Multi-Streamed Exchange Database Backups



Network Settings

Pipeline Buffers

By default, Commvault software establishes 30 Data Pipeline buffers for each data movement connection. You can

increase the data transfer throughput from the client by increasing or even decreasing the number of Data Pipeline

buffers. The number of the Data Pipeline buffers depends largely on the transport medium.

To set the number of pipeline buffers, use the ‘nNumPipelineBuffers’ additional setting.

Although the maximum value for ‘nNumPipelineBuffers’ is 1,024, if you use a value that is greater than 300, you should

consult with Commvault Support. When you increase the number of Data Pipeline buffers, the client or MediaAgent

consumes more shared memory. When available memory is low, this consumption of shared memory might degrade the

server performance for other operations.

Recommended values for nNumPipelineBuffers:

Internet - 30 buffers

100BASE - 30 buffers

1000BASE - 120 buffers

Add Pipeline buffers additional setting



Network Agents

Network agents are threads or processes that transfer data to and from the network transport layer. Each network agent

spends half its time reading and half its time writing. For higher speed networks, having multiple networks agents can

improve performance.

Default values and valid values for the number of network agents:

Windows default – 2. Valid options 1 – 4

Unix default – 1. Valid options 1 – 2

Network Agent configuration for a Windows subclient



Disk Storage

Chunk Size

Chunk sizes define the size of data chunks that are written to media and is also a checkpoint in a job. The default size for

disk is 4GB. The default size for tape is 8GB for indexed based operations or 16GB for non-indexed database backups.

The data path ‘Chunk Size’ setting can override the default settings. A higher chunk size results in a more efficient data

movement process. In highly reliable networks, increasing chunk size can improve performance. However, for unreliable

networks, any failed chunks must be rewritten, so a larger chunk size could have a negative effect on performance.

Chunk size recommendation for disk storage

Storage media Job type Default

chunk size

Recommended

chunk size

Disk All data protection jobs 4 GB 512 MB – 8 GB

Direct-attached NDMP All data protection jobs 8 GB N / A

Commvault HyperScale® scale out storage

All data protection jobs 8 GB N / A

Chunk size settings for a disk data path



Chunk size configuration for MediaAgents

Use the ‘DMMBCHUNKSIZE’ additional setting to control the chunk size of the data write jobs that go to the MediaAgent

on which the additional setting is created.

The chunk size that you specify in the additional setting overrides the values that you specify in the chunk size that you

specify for the CommCell® in the Media Management configuration.

Configuring MediaAgent chunk size



Block Size

MediaAgents can write to media that is formatted with different block allocation sizes or file allocation sizes if the

MediaAgent operating system supports those sizes. Using a larger block size for disk library volumes can reduce overhead

and thus increase the speed of write operations to media.

Linux ext3 and Microsoft NTFS use a default block (allocation unit) of 4KB. The 4KB block size was established when 2GB

disks were considered large. Today, Microsoft recommends using at least a 16KB block size or higher for NTFS volumes.

Commvault recommends that you use 64KB, which matches the Microsoft default value for the ReFS block size.

You can increase the Linux ext3 block size only on an Itanium system. For other file systems, consult your OS vendor

documentation for your file system’s available block sizes.

Block size settings for a disk data path



Unbuffered I/O for Windows® MediaAgent

If the source copy is on disk and is managed by a Windows MediaAgent, then enable the Use Unbuffered I/O option for

each mount path. Using unbuffered I/O can significantly improve performance.

To increase the speed of jobs that access the mount path, you can configure the MediaAgent to bypass the Microsoft

Windows file system buffering.

You can make this configuration for Windows MediaAgents and for disks that are mounted directly (not for UNC paths).

Unbuffered I/O configuration for Windows MediaAgent

Unbuffered I/O for Unix/Linux MediaAgent

A similar option is available for UNIX/Linux based MediaAgent, however, it must be enforced at the operating system level

and not through the Commvault® software GUI. It can be achieved using two methods:

Method one – Use the GFS tool provided by most UNIX/Linux based OS. This tool sets a direct I/O flag to a

directory and all its current subdirectories and files. Once enabled, any new directory or files created will also

inherit the direct I/O attribute. It can be turned on (using the setflag parameter) or off (clearflag) as desired.

Method two – Use the Unbuffered I/O configuration for Linux MediaAgent:

Gfs_tool setflag inherit_directio MyDirectory

Mount the NFS filesystem using the force direct I/O flag (forcedirectio). For as long as the filesystem is

mounted, it will bypass the operating system buffer.

For more information on the GFS tool or the mount direct I/O option, refer to your operating system vendor’s

documentation.



Tape Storage

Chunk Size

A chunk is the unit of data that the MediaAgent software uses to store data on media. For sequential access media, a

chunk is defined as data between two file markers. By default, the chunk size is configured for optimal throughput to the

storage media.

Job type

Default chunk size Recommended chunk size

Granular (index based) job

8 GB 8 – 32 GB

Database (non-indexed) job

16 GB 8 – 32 GB

Chunk Size for tape libraries can be modified on the data path for a specific tape library, or globally, using the Media

Management applet. Global chunk size settings are configured per agent type.

Chunk size settings for a tape data path



Global chunk size settings for tape media

Block Size

Before changing tape block size, ensure that the following criteria are satisfied:

Block size is supported by the MediaAgent OS, Host Bus Adapter (HBA), and the tape device.

All the MediaAgents that are associated with a storage policy support the block size that is configured on that

storage policy. Consider the support and the compatibility of MediaAgent platforms at any disaster recovery site.

If you use different MediaAgents for backup operations and restore operations, and if the backup MediaAgent has

a higher block size, then ensure that the restore MediaAgent can read data that is written with a higher block

size.

Many streaming tape drives perform a read-after-write check. If the drive detects a bad block, then the drive puts a

discard token after the block, and repeats the entire buffer write. If the drive detects a discard token, then the read cycle

has corresponding logic to replace the bad block with the replacement block.

All tapes will have media defects. If you write 1,024KB blocks instead of 256KB blocks, then the chance of any block

spanning a media defect are increased by a factor of 4. Because of the larger block size, the rewrite time is 4 times as

long as well.

Increasing block size can improve the performance of writing to tape by minimizing the overhead associated with

accessing and recording each block. If you select the data path’s Use Media Type Setting option, then the data path’s

default block size for tape is 64KB. Refer to the Commvault Online Documentation: Use Media Type Setting section for

more information.

Important notes on configuring tape block size:

Use caution when you select large block sizes. Large block sizes can vastly increase error rates and retries.

Block size applies only to tape media in direct-attached libraries.

Changes to the block size settings take effect when the next spare tape media is used.

Ensure hardware at data center and other location, including DR sites support higher block sizes.



Block size settings for a tape data path



Cloud

Deduplication Block Size and Async IO

Commvault recommends that you always use deduplication for efficiency and performance when writing to cloud libraries.

The default deduplication block size is 128KB, but for the fastest performance to cloud libraries, Commvault recommends

using a deduplication block size of 512KB.

Commvault recommends using the ‘SILookAheadAsyncIOBlockSizeKB’ additional setting to set the block size that is used

by Async IO in the look-ahead reader. The recommended value is 2 times the deduplication block size.

The default value of ‘SILookAheadAsyncIOBlockSizeKB’ is 128KB for cloud libraries. For the recommended deduplication

block size of 512KB, set the AsyncIO block size to 1,024KB.

Configure the ‘SILookAheadAsyncIOBlockSizeKB’ additional setting on all source data mover MediaAgents that are

associated with the source storage policy copy. For instructions about adding additional settings from the CommCell®

console, refer to the Commvault Online Documentation: Add or Modify an Additional Setting section for more information.

Configuring the Async IO look ahead reader block size for cloud libraries



Auxiliary Copy Performance

If the network utilization during a DASH copy job or a backup job with deduplication is low, then apply all the following

settings on all source data mover MediaAgents that are associated with the source storage policy copy. For auxiliary copy

operations, these settings are enabled by default.

For instructions about adding additional settings from the CommCell® console, refer to the Commvault Online

Documentation: Add or Modify an Additional Setting section for more information.

Additional Settings for Auxiliary Copy performance

Additional Setting Description Values

DataMoverUseLookAh eadLinkReader

Enables the reading of multiple

data signatures

from the deduplication

database

Default value: 1 (enabled)

SignaturePerBatch Ensures that the

DDB signature

look-ups are batched with

multiple signatures per look-up

Modify this value only if there is a large latency

between the client and the deduplication

MediaAgent.

Default value: 1

Maximum value: 32

Recommended values: 16 and 32. If the

latency is between 100 and 200 ms, then

use 16. If the latency is more than 200 ms, use 32.

DataMoverLookAhead

LinkReaderSlots

Performs more

lookups on the destination

deduplication database for

deduplication

block signatures

Default value: 16

Valid values: 16, 32, 64, 128, and 256

Recommendation: Set the look-ahead slot

value to as high as the MediaAgent

memory and concurrent operations allow.

Each look-ahead slot uses a memory size of one

block of deduplicated data. So, with a default deduplication block size of 128KB and 16 look-

ahead slots, each reader stream in an auxiliary

copy operation uses 2MB of memory. Setting lookahead slots to 128 uses 16MB of memory for

each reader stream. So, if 200 auxiliary copy streams run on the MediaAgent, then the total

memory overheard for the lookahead slots only is

3.2GB.



Commvault Features Assessment Charts Commvault® software provides many features and settings that can improve data protection and recovery performance. It is important to note that, in some cases, these features or settings can have a positive impact in certain aspects, but

negative impacts in other aspect.

The following section provides several charts that compare and contrast the pros and cons of features and settings within

Commvault software:

Each feature explains potential positive and negative effects on protection windows, media management, and

recovery objectives.

Some features may not have any affect or minimal effect so no information is listed.

Each analysis is rated as ‘May have’, ‘Will have’, or ‘Significant’ effect. An explanation is also provided explaining

why the effect may occur and which setting causes the effect to take place.

Additional considerations are listed for some features.

The following charts highlights the positive and negative effects of various Commvault protection

methods and configuration options. Charts and explanations assume a solid level of competency with

the Commvault Product suite.

Storage Policy Configurations

Feature / Option / Implementation Methods

Protection Windows (Primary / Secondary)

Media Management

Recovery Objectives Other Considerations

Device Streams May have positive effect by running more jobs concurrently. May have negative effect if infrastructure cannot handle higher data loads. Will have positive effect when used in conjunction with client side deduplication.

Library writers or drives should equal number of storage policy device streams. When using deduplication, ensure Media management ‘Maximum number of parallel data transfer operations for deduplication database’ is adjusted accordingly.

Erase Data May have a negative effect on restore as Media Explorer, Catalog or restore by job features cannot be used on media written by an erase data enabled storage policy.

Incremental Storage Policy

May have a positive effect on incremental or log backup performance by writing to a separate disk library.

May have a positive effect on restore performance by restoring inc / logs from fast disk storage.



Primary Copy Configurations

Option / Value


Media Management Recovery Objectives Other Considerations

Device Streams

May have positive effect by running more jobs concurrently. May have negative effect if infrastructure cannot handle higher data loads. Will have positive effect when used in conjunction with client side deduplication.

Library writers or drives should equal number of storage policy device streams. When using deduplication, ensure Media management ‘Maximum number of parallel data transfer operations for deduplication database’ is adjusted accordingly.

Multiplexing Will have positive effect on backups when protecting multiple slower job streams to a single device stream.

May have negative effect on RTO if streams are not required for restore and they are multiplexed to same tape with streams that are required for restore.

Retention May have negative effect on

storage if retention is improperly configured.

Spool Copy Will have positive effect when first spooling to disk then writing to tape when protecting large amounts of small files.

Extended Retention Rules

May have negative effect on tape consumption if jobs with extended retention are mixed on tape containing jobs with no extended retention. Will have a positive effect on retention when grandfather /

father / son tape rotation is required. Significant negative effect if using deduplication. A significant increase for both the disk library size and the deduplication database size that could potentially lead to a deduplication database seal and restart.



Option / Value


Media Management


GridStor™ Round Robin

Will have a positive effect on primary backup by load balancing across multiple MediaAgent.

Will have a negative effect on RTO when using deduplication writing to shared SAN library.

GridStor™ Failover

Will have a positive effect on primary backup by providing a failover path to different MediaAgent, drive pool, scratch pool or library.

Data path chunk size

Will have a positive effect for indexed jobs by increasing chunk

size. May have a positive effect for indexed jobs over unreliable network (WAN) by decreasing chunk size.

Data path block size

May have a positive effect on backup performance by increasing block size.

Significant negative effect on restore if hardware does not support higher block size including NIC, HBA, library, drive and MA OS.

Hardware encryption (LTO4 - LTO7)

May have negative effect on backup performance by additional processing to encrypt data.

Provides high level of security.

May have negative effect on restore performance.



Secondary Copies Configurations

Feature / Option


Media Management


Inline copy May have positive effect by running primary and secondary jobs concurrently. May have negative effect since jobs are streamed and will only run as fast as slowest device.

Combine to

streams using tape media

May have a negative

effect if configured improperly.

Will have a

positive effect on media usage by combining streams to fewer tape media.

May have a negative effect on RTO if streams

for restore are mixed with streams not required for restore. Will have negative effect on RTO for multi-stream backups as opposed to placing each stream on separate tape. Note this only applies if using restore by job. Will have negative effect on RTO if different data sets required for restore are on same tape. Only one data set can be restored at a time.

Multiplexing with combine to streams

May have a positive effect if source of auxiliary copy is disk with multiple mount

paths.

Software encryption

May have a negative effect on performance depending on MediaAgent resources and size of data in secondary copy.

Using Global Secondary Copy

Significant positive effect on tape usage since many storage policy secondary copies are

concatenated on the same set of media.

Significant negative effect on RTO. Less restore jobs can run concurrently as data in stored on same tapes, as opposed to different storage policies tape copies isolated on their own media.



Subclient Configurations

Feature / Option



Custom Subclients Will have positive effect by providing greater scheduling flexibility. Will have positive effect by running multiple streams concurrently.

Will have positive

effect by allowing specific content to

be retained separately.

May have negative effect for

secondary copies if

stream management

settings are improperly

configured.

Data readers Will have positive effect by running multiple streams concurrently.

Multiple readers in drive or mount point

Will have positive effect by running multiple streams concurrently on RAID volumes

Data Streams for SQL, DB2 and Sybase

Will have a positive effect on performance by multi-streaming backup jobs.

May have a negative effect for tape copy since each stream must go to separate tape.

Will have a positive effect on RTO from tape with streams on separate tapes. Significant negative effect if multi-stream backup was combined to tape. Tape copy requires pre-staging to disk prior to recovery.

Filters Will have a positive effect on backup and auxiliary performance by eliminating unwanted data from jobs.

Will have a positive effect on restores since unwanted data was not backed up. May have a negative effect on restorability if filters are improperly configured and filter important data.

UNC path for file system subclient contents

Will have a negative effect on primary backup. Path will be from UNC share to the Client and then to MediaAgent.



Deduplication Configurations




Isolating data in separate policies based on data type and block size

Will have positive effect to improve performance when using multiple MediaAgent using dedicated DDB for each policy. May have negative impact if more than two DDB loaded on single MediaAgent.

May have negative effect with marginal decrease in dedupe efficiency in primary disk storage. May have negative effect for secondary copies if using tape media.

May have positive effect on restore performance by grouping data blocks relative to data type being protected in their own deduplication store folders.

Best practice is to isolate data types to different policies providing greater scalability.

Client side deduplication

Significant positive effect after the first full by sending only change data blocks to MediaAgent. Will have positive effect on secondary copies if using DASH Copy.

Significant positive effect reducing size of data on disk. Will have positive effect on retention by making more efficient use of disk media.

May have negative RTO by slower disk read access (due to fragmentation of data blocks on disk).

For WAN backups using client side deduplication, enable the client side disk cache.

Block size Larger block size may have positive effect by reducing size of dedupe

database making hash comparisons faster.

May have negative effect if using small block size <128k (due to

fragmentation of data blocks on disk).

Follow Commvault best practices when determining block size

settings.

Compression May have negative effect due to increase CPU usage on client. Will have positive effect requiring less bandwidth for backup.

May have positive effect since compression adds considerable space savings on deduplication depending on data type.

Will have positive effect on RTO since data is decompressed on client.

Compression takes place before hash so duplicate blocks will generate same hash. If application compresses data, disable compression in dedupe SP.




Do not deduplicate against objects older than

Will have negative effect at point specified as all duplicate blocks older than specified number will be backed up again.

Temporary negative effect as duplicate block will be written to dedupe store.

This option is for block integrity by periodically (based on days set) rewrite blocks to disk.



Seal deduplication store

Will have negative effect at point store seals

as all blocks will be written to disk.

Temporary negative effect until the old store ages and

pruned.

In a well-designed environment sealing

the store should not be necessary unless it meets specific protection needs. Sealing the store may become necessary if dedupe database grows too large or if Commvault best practices are not followed.

Global deduplication policy

Will have positive effect on performance by deduplicating against blocks from multiple primary copies into single store. May have significant negative effect on scalability and performance if not configured and implemented correctly.

(See other considerations).

Will have positive effect on disk consumption by deduplicating blocks from multiple storage policies into single store. May have negative effect since each individual SP will still have to create individual secondary copies requiring dedicated tape for each policy copy.

May have negative effect on performance that may degrade by slower disk read access (due to fragmentation of data blocks on disk).

Global deduplication policies are intended to consolidate remote site backups to a central disk location. This is not intended to centralize data for all policies in an environment.

DASH Full Significant positive effect on full backups after the first full is performed.

DASH Copy Significant positive effect on off-site copies to secondary disk target.

Will have positive effect on off-site RPO since data can be copied off-site more frequently versus sending tapes off-site.



SILO Storage Configurations




Using SILO Will have positive effect on performance since data does not require rehydration.

Significant positive effect by writing dedupe data to tape without rehydration. Will have positive effect disk consumption by pruning old volume folders from disk.

May have negative impact on RTO since volume folders must be pre-staged to disk before data can be recovered. May have significant negative effect if large amounts of data need to be recovered. Will have positive effect by copying all dedupe data to SILO it can improve RPO with more recovery points available as opposed to periodically performing full auxiliary copies.

SILO & global deduplication policy

Significantly reduce tape consumption if multiple policies are writing to a global dedupe. Using SILO in place of traditional secondary copies will allow data from multiple storage policies to be stored on fewer tapes. Traditional aux copy would require separate tapes for each secondary copy.

May have negative impact on restore since volume folders must be pre-staged to disk before data can be recovered. May have negative impact on RTO. If large amounts of data need to be recovered, RTO performance can be significantly degraded. Significant positive impact on RPO. By copying all dedupe data to tape, more recovery points will be available as opposed to periodically performing full auxiliary copies.

Number of Silos to be kept in cache

May have a negative impact on storage. Will Increase disk consumption by keeping SILO data in disk cache.

Significant positive effect on RTO. Improve RTO if volumes required for restore are in cache.

copyright · commvault ® stream management ... looking at the job in more detail we find that the...

Documents