Integration of GlusterFS in to Commvault data platform

Ram Ankireddypalle Principal Engineer, Commvault Systems

Commvault data platform architecture

Need for software defined storage

Software defined storage requirements

• Support for file system interface

• Support horizontal scale out

• Support for resiliency through erasure coding

• Easy to install/maintain/administer

• Linux based

• Open source

• Hardware agnostic

• Coexist with Commvault software on a server

• Converged data management

StoragePool

• Grid of Commodity Server based Nodes with internal Storage which will be used for configuring a disperse volume

• Converged data management

Nodes run most of Commvault Data Management software along with GlusterFS

• Fast, Easy deployment and expansion

• Erasure Coding for resiliency

Tolerate loss of multiple drives or entire nodes without loss of Data Access

• Mix different types of servers across multiple generations

Easily add higher density drives or faster nodes without rip and replace

• Horizontally Scale-out of Storage as well as Compute

StoragePool

Start small and expand dynamically

• Start the StoragePool with 3 node block

• Usually start with 2 Control nodes

• Additional node blocks in same increments

• Introduce more control nodes as needed

• StoragePool automatically expands to consume new capacity

• Loss of one or more nodes does not affect data availability

● 2U Server, Dual processor 16 total cores, v4

Xeon

● 256GB RAM, 2x10 Gbps

● 2xSSD for ~400 GB usable for OS and software

● 12 X 4TB / 6TB / 8TB NL-SAS/SATA drives for

up to 96 TB Raw Storage

● No RAID Card, dual Power

● 1 TB Flash/NVMe Card for Index

Suggested reference architecture

● 2U Server, Dual processor 16 total cores, v4

Xeon

● 256GB RAM, 2x10 Gbps

● 2xSSD for ~400 GB usable for OS and software

● 12 X 4TB / 6TB / 8TB NL-SAS/SATA drives for

up to 96 TB Raw Storage

● No RAID Card, dual Power

● Max 4 Control Nodes per Policy

● 4 TB Flash/NVMe Card for Metadata

Control node Data node

Tolerance to disk/server loss

GlusterFS Volume

Integration of GlusterFS in to Commvault data platform

❏ Existing OS/Applications will be preserved

❏ Manually configure bricks

❏ More lead time to bring a new node into the

glusterfs volume

❏ Existing OS/Applications will be destroyed

❏ Plug and play kind of solution that

transforms a Bare Metal Server to data

management node in short time

❏ Bare Metal to backup in about 30 minutes

❏ Process of creating a new node completely

automated

Server with installed operating system Bare metal box

Server with installed operating system

• Install the GlusterFS

• Install the Commvault software

• Create the brick mount paths by formatting the disks with XFS file system

and mounting under predefined directories by executing Commvault utility

• The above steps need to be performed for all the nodes that are to become

part of the GlusterFS volume.

• Commvault data platform will orchestrate the creation of GlusterFS

volume, dedupe store and other components needed for creating the backend

store and metadata needed for data management.

Bare metal server

• A single bootable DVD is capable of

imaging any hardware

• DVD can be used for imaging both

BIOS/UEFI firmware based servers

• SSD’s will be used for laying out

❏ Linux system mount points

❏ Dedupe store

• Disks will be used for creating the

GlusterFS bricks

• Imaging configures the following

❏ Centos/RHEL

❏ Commvault data platform

• Imaging completely automated with no

manual intervention

• After imaging

❏ Install GlusterFS

❏ Register with the Commserver

• Commvault data platform will orchestrate

the creation of GlusterFS volume, dedupe store

and other components needed for creating the

backend store and metadata needed for data

management.

• Bare metal to backup in 30 minutes

Horizontal scaling

Adding more nodes to StoragePool

• Install Commvault data platform on a server through one of the following methods:

❏ Server with installed operating system

❏ Bare metal server

• Commvault data platform will orchestrate the addition of new bricks to GlusterFS volume

• Dedupe store and other components will be expanded to utilize the extra capacity

Vertical scaling

Adding more disks to the node

• Utility to detect newly added disks and mount under predefined directories

• Commvault data platform detects the extra bricks on each node and orchestrate the addition of

new bricks to GlusterFS volume

• Dedupe store and other components will be expanded to utilize the extra capacity

Health checks

• Monitor the status of each node in the StoragePool

• Monitor the status of XFS file system under each brick

• Monitor the status of block device underlying each brick

• Monitor the SMART parameters reported by each block device

• Periodically update the health information and send alerts

Interested features/enhancements

• Support for Hole punching in the file system

• API/Interface to list the files and directories under a sub volume

• DR tools to reconstruct the GlusterFS volume if the underlying

disks are available

Thank you