drbd

DRBD

Pierre Mavro

www.enovance.com

September 11, 2013

http://www.enovance.com

DRBD: Summary

Summary

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Basic usages and understandings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3 Use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Pierre Mavro www.enovance.comDRBD 2 / 62


DRBD: Introduction

How it worksDRBD refers to block devices designed as a building block to form high availability(HA) clusters. This is done by mirroring a whole block device via an assigned network.DRBD can be seen as network based raid-1.



DRBD: Introduction

Synchronisations

DRBD supports three distinct replication modes, allowing three degrees of replication :

I A (Asynchronous replication) : Master node disk write + local TCP sendbuffer → operation completed

I B (Semi synchronous replication) : Master node disk write + replicationpackets reached other node → operation completed

I C (Synchronous replication protocol) : Written to both the local and theremote disk to get operation completed

The Syncronous replication protocol is the most robust protocol tested inproduction.http://www.drbd.org/home/mirroring/


http://www.drbd.org/home/mirroring/


DRBD: Introduction

Data accessibility

A consequence of mirroring data on block device level is that you can access your data(using a file system) only on the active node. This is not a shortcoming of DRBD butis caused by the nature of most file systems (ext3, XFS, JFS, ext4, ...).These file systems are designed for one computer accessing one disk, so theycannot cope with two computers accessing one (virtually) shared disk.In spite of this limitation, there are still a few ways to access the data on the secondnode:

I Use DRBD on logical volumes and use LVM’s capabilities to take snapshots onthe standby node, and access the data via the snapshot.

I DRBD’s primary-primary mode with a shared disk file system (GFS, OCFS2).These systems are very sensitive to failures of the replication network.

I Mount in read only mode the partition

We are using DRBD in master/slave at eNovance.



DRBD: Basic usages and understandings

Plan

2 Basic usages and understandings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6I Configuration and initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6I Check replication status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15I Node switching and manual synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22I Remove a DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26




Configuration

The DRBD configuration should be the same on both nodes. This drbd.conf file isthe global configuration of the DRBD service with fine tuning configuration :

/etc/drbd.confglobal { <-- Global configuration

usage-count no; <-- Do not report statistics usage to LinBit}common { <-- All resources inherit the options set in this section

protocol C; <-- C (Synchronous replication protocol)

startup {wfc-timeout 1 ; <-- Wait for connection timeout (in seconds)degr-wfc-timeout 1 ; <-- Wait for connection timeout, if this node was a degraded

} cluster (in seconds)




Configuration

/etc/drbd.confnet {

max-buffers 8192; <-- Maximum number of requests to be allocated by DRBDmax-epoch-size 8192; <-- The highest number of data blocks between two write barrierssndbuf-size 512k; <-- The size of the TCP socket send bufferunplug-watermark 8192; <-- how often the I/O subsystem’s controller is forced to

process pending I/O requestscram-hmac-alg sha1; <-- The HMAC algorithm to enable peer authentication at allshared-secret "xxx"; <-- The shared secret used in peer authentication# Split brainsafter-sb-0pri disconnect; <-- Split brain, resource is not in the Primary role on any hostafter-sb-1pri disconnect; <-- Split brain, resource is in the Primary role on one hostafter-sb-2pri disconnect; <-- Split brain, resource is in the Primary role on both hostrr-conflict disconnect; <-- Helps to solve the cases when the outcome of the resync

} decision is incompatible with the current role assignment

handlers {pri-on-incon-degr "echo node is primary, degraded and the local copy of the data is

inconsistent | wall "; <-- If the node is primary, degraded and if the} local copy of the data is inconsistent




Configuration

/etc/drbd.confdisk {

on-io-error pass_on; <-- The node downgrades the disk status to inconsistent on io errorsno-disk-barrier; <-- Disable protecting data if power failure (done by hardware)no-disk-flushes; <-- Disable the backing device to support disk flushesno-disk-drain; <-- Do not let write requests drain before write requests of a new

reordering domain are issuedno-md-flushes; <-- Disables the use of disk flushes and barrier BIOs when accessing

} the meta data device

syncer {rate 300M; <-- The maximum bandwidth a resource uses for background

re-synchronizational-extents 3833; <-- Control how big the hot area (= active set) can get

}}




Configuration

Now the resources can be defined. One resource should match a DRBD device. Bothnodes informations should be filled :

/etc/drbd.d/resources.confresource drbd1 { <-- DRBD block device name

syncer {after drbd0 } <-- Start drbd1 after drbd0 is up & running

on srv1 { <-- Master node namedevice /dev/drbd1; <-- Block device name of the resource being describeddisk /dev/sda1; <-- Block device to store and retrieve the dataaddress x.x.x.x:7789; <-- IP address and port of the local hostmeta-disk internal; <-- The last part of the backing device is used to store

the meta-data}

on srv2 { <-- Slave node namedevice /dev/drbd2;disk /dev/sda1;address y.y.y.y:7789;meta-disk internal;

}}




Initialisation

Here is a summary to understand the process to create a drbd replication. You needto follow that rule to get a working DRBD synchronization :

Init metadatadevice

AttachDRBD device

ConnectDRBD device

Start syn-chronization




Init metadata device

This step must be completed only on initial device creation. It initializes DRBD’smetadata (replace <drbd_volume> by the drbd device name you want to initialize).You should complete those steps on both nodes :

Create device metadata on server 1srv1~$ drbdadm create -md <drbd_volume >Writing meta data ...initialising activity logNOT initializing bitmapNew drbd meta data block sucessfully created .

Create device metadata on server 2srv2~$ drbdadm create -md <drbd_volume >...




Attach and connect DRBD device

Attach a local backing block device to the DRBD resource’s device :

Attach DRBD device$ drbdadm attach <drbd_volume >

If the peer device is already configured, the two DRBD devices will connect :

Sets up the network configuration of the resource’s device$ drbdadm connect <drbd_volume >




Start synchronization

To start the first synchronization, you need to ask to the first server to sync all blocksto the secondary server.This will erase all blocks on the other server, so all data will be lost :

Start the first synchronization$ drbdadm -- --overwrite -data -of -peer primary <drbd_volume >

This action may take a while, depending on the network bandwidth and DRBDvolume size.




Plan





Check replication status

To check the replication status, simply look at DRBD file in /proc :

Check$ cat /proc/drbdversion : 8.3.7 (api :88/ proto :86 -91)srcversion : EE47D8BF18AC166BE219757

0: cs: SyncSource ro: Primary / Secondary ds: UpToDate / Inconsistent C r.ns :912248 nr :0 dw :0 dr :920640 al :0 bm :55 lo :1 pe :388 ua :2048 ap :0

[=== >................] sync ’ed: 21.9% (3283604/4194304) Kfinish : 1:08:24 speed : 580 (452) K/sec





Another solution consists to launch this command that will provide informations of allDRBD devices :

Check$ drbd - overview0: home Connected Primary / Secondary

UpToDate / UpToDate C r--- /home xfs 200G 158G 43G 79%




Check replication statusSeveral informations need be checked to know the status replication of a DRBDdevice :

Replicationstatus

Connectionstate (cs)

Resourcesroles (ro)

Diskstates (ds)





Connection states (cs) :

I StandAlone : No network configuration available. The resource has not yetbeen connected, has been administratively disconnected or has dropped itsconnection due to failed authentication/split brain.

I Unconnected : Temporary state, prior to a connection attempt.

I WFConnection : This node is waiting until the peer node becomes visible onthe network.

I Connected : A DRBD connection has been established, data mirroring is nowactive. This is the normal state.

I PausedSync : The local node is the source or target of an ongoingsynchronization, but synchronization is currently paused. This may be due to adependency on the completion of another synchronization process, or due tosynchronization having been manually interrupted.





Resource roles (ro) :

I Primary : The resource is currently in the primary role, and may be read fromand written to. This role only occurs on one of the two nodes, unlessdual-primary mode is enabled.

I Secondary : The resource is currently in the secondary role. It normally receivesupdates from its peer (unless running in disconnected mode), but may neither beread from nor written to. This role may occur on one or both nodes.

I Unknown : The resource’s role is currently unknown. The local resource rolenever has this status. It is only displayed for the peer’s resource role, and only indisconnected mode.





Disk states (ds) :

I UpToDate : Consistent, up-to-date state of the data. This is the normal state.

I Inconsistent : The data is inconsistent. This status occurs immediately uponcreation of a new resource, on both nodes (before the initial full sync). Also, thisstatus is found in one node (the synchronization target) during synchronization.




Plan





Node switching

You may need to perform maintenance tasks. For instance you need to switch aDRBD volume to the other node.Simply connect to the secondary node and launch :

Set secondary as primary$ drbdadm primary <drbd_volume >

You can also switch all volumes at once. Double check if you really can do thisbefore proceed :

Set secondary as primary$ drbdadm primary all




Node switching

You also can switch a Primary node to Secondary :

Set primary as secondary$ drbdadm secondary <drbd_volume >

Notes

This will not automatically set the old secondary node to primary state




Manual synchronization

To start a manual synchronization, you need to invalidate the DRBD device on thecurrent host :

Invalidate data on current host$ drbdadm invalidate <drbd_volume >

You can do the same from one node to the other node :

Invalidate data on remote host$ drbdadm invalidate_remote <drbd_volume >

You can then check the status in /proc/drbd




Plan





Remove a DRBD device

Before removing a DRBD device, be sure you’ve dumped all data ! Then disconnectthe device on both nodes :

Disconnect DRBD device$ drbdadm disconnect <drbd_volume >

And delete it :

Remove DRBD device$ drbdsetup <volume_number > down

Change volume_number by the number seen with drbd-overview command. To finish,remove the config files.



DRBD: Use cases

Plan

3 Use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28I Stop, upgrade and commit changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28I Stop, upgrade and rollback changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31I Up, connect...but doesn’t want to come UpToDate . . . . . . . . . . . . . . . . . . . . . . . 35I How to promote a primary when having a dual secondary . . . . . . . . . . . . . . . . . . 39



DRBD: Use cases

Stop, upgrade and commit changes

Sometimes, you may need to perform maintenances tasks on a DRBD device. Forexample, you need to upgrade stuffs on a DRBD device and if success, commitchanges to the secondary :

DRBD Primary DRBD Secondary

Set secondary nodein maintenance mode

Resume sync

Upgrade success

Normal state



DRBD: Use cases

Stop, upgrade and commit changes

To set the secondary node in maintenance, simply stop the synchronisation on thesecondary node :

Disconnect drbd device$ drbdadm disconnect <drbd_volume >

Do what you have to do on the DRBD primary device and once you have finished andeverything looks fine, resume the synchronization to get back to the normal state :

Connect drbd device$ drbdadm connect <drbd_volume >



DRBD: Use cases

Plan




DRBD: Use cases

Stop, upgrade and rollback changes

Sometimes, you may need to perform maintenances tasks on a DRBD device. Forexample, you need to upgrade stuffs on a DRBD device and if fail, rollback changesfrom the secondary :

DRBD Primary DRBD Secondary

Set secondary nodein maintenance mode

Upgrade failed

Normal state

Rollback sync



DRBD: Use cases


To set the secondary node in maintenance, simply stop the synchronisation on thesecondary node :

Disconnect drbd device$ drbdadm disconnect <drbd_volume >

Then you’ve upgraded and want to rollback. You have to switch your secondary nodeto master node :

Promote device as primary$ drbdadm primary <drbd_volume >



DRBD: Use cases


Then you need to invalidate the datas on the old primary :

Invalidate drbd device data$ drbdadm invalidate <drbd_volume >

To finish, adjust and connect the device to perform the rollback on the new masternode :

Adjust and connect$ drbdadm adjust <drbd_volume >$ drbdadm connect <drbd_volume >



DRBD: Use cases

Plan




DRBD: Use cases

Up, connect...but doesn’t want to come UpToDate

If you don’t understand why after several up, connect etc.... you don’t see anythingchanging and still got this kind of event :

Check$ drbd - overview0: drbd0 StandAlone Primary / Unknown UpToDate / DUnknown r-----

You need to instigate more. The first thing to do is to look at the logs :

/var/log/syslog[ 6375.849509] block drbd0: Split-Brain detected but unresolved, dropping connection![ 6375.850025] block drbd0: helper command: /sbin/drbdadm split-brain minor-0[ 6375.852859] block drbd0: helper command: /sbin/drbdadm split-brain minor-0 exit code 0 (0x0)[ 6375.852867] block drbd0: conn( WFReportParams -> Disconnecting )[ 6375.852873] block drbd0: error receiving ReportState, l: 4![ 6375.853274] block drbd0: asender terminated[ 6375.853277] block drbd0: Terminating drbd0_asender[ 6375.853506] block drbd0: Connection closed[ 6375.853514] block drbd0: conn( Disconnecting -> StandAlone )[ 6375.853523] block drbd0: receiver terminated[ 6375.853526] block drbd0: Terminating drbd0_receiver



DRBD: Use cases


Anyway, it may not be clear enough for you. You can use that command to get abetter understanding on the situation :

Show all drbd informations$ drbdadm show -gi <drbd_volume >

+--< Current data generation UUID >-| +--< Bitmap ’s base data generation UUID >-| | +--< younger history UUID >-| | | +-< older history >-V V V V

B63AABEFCFADD87D :2 C30515E7D05B35D :82 C4146DCB8E6BFF :82 C3146DCB8E6BFF :1:1:1:0:0:0:0^ ^ ^ ^ ^ ^ ^

-< Data consistency flag >--+ | | | | | |-< Data was/is currently up -to -date >--+ | | | | |

-< Node was/is currently primary >--+ | | | |-< Node was/is currently connected >--+ | | |

-< Node was in the progress of setting all bits in the bitmap >--+ | |-< The peer ’s disk was out - dated or inconsistent >--+ |

-< This node was a crashed primary , and has not seen its peer since >--+

flags : Primary , StandAlone , UpToDate



DRBD: Use cases


Here the problem is clear : we’re in a split brain situation. So there is no othersolutions than doing a full sync.But what if no split brain was detected ? There’s a last solution :

Adjust and connect the drbd device$ drbdadm adjust <drbd_volume >$ drbdadm connect <drbd_volume >

Now you can check the replication which is resuming.



DRBD: Use cases

Plan




DRBD: Use cases

How to promote a primary when having a dual secondary

If you get the dual secondary state after node’s restart, check which node as the’UpToDate’ state :

Check$ drbd - overview0: drbd0 Connected Secondary / Secondary UpToDate / UpToDate C r-----

Here, both are up to date. Simply select one and promote it as primary :

Set as primary$ drbdadm primary <drbd_volume >

You should now see one primary and a secondary synced.



DRBD: Ninja tricks

Plan

4 Ninja tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41I Create DRBD replication without syncing data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41I What to do in Split brain case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45I Mount secondary node as read only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50I All is good, but still doesn’t want to connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53I Can’t "create-md" on an old DRBD device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55I My DRBD device is still there but not the configuration . . . . . . . . . . . . . . . . . . . 58I Speed up sync transfer rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60



DRBD: Ninja tricks

Create DRBD replication without syncing data

When you have big devices to sync on a DRBD initialization, it could take severalhours or days in worst cases. There is a solution to only sync bitmap and have yourDRBD device available in a few seconds instead !Create on both nodes the DRBD device :

Create device metadata$ drbdadm create -md <drbd_volume >Writing meta data ...initialising activity logNOT initializing bitmapNew drbd meta data block sucessfully created .



DRBD: Ninja tricks


Then bring it up (still on both nodes) :

Connect the volume$ drbdadm up <drbd_volume >

Only on the master node, initialize the bitmap and uuid of the DRBD device :

Only sync bitmap$ drbdadm -- --clear - bitmap new -current -uuid <drbd_volume >



DRBD: Ninja tricks


Promote this master node as primary node to replicate bitmap :

Set primary volume$ drbdadm primary <drbd_volume >

Now you can check the DRBD is ready to use :-)

Check$ drbd - overview1: drbd1 Connected Primary / Secondary UpToDate / UpToDate C r-----



DRBD: Ninja tricks

Plan




DRBD: Ninja tricks

What to do in Split brain case

If you get this kind of problem in /proc/drbd :

Set secondary as primary$ cat /proc/drbdprimary / unknown

that mean that you are in a split-brain case. First of all you need to check the versionon both nodes ! You absolutely need to have the same version to avoid errors(8.0-8.3/8.4/9.0) :

Set secondary as primary$ drbdadm --version | grep DRBDADM_VERSION =DRBDADM_VERSION =8.3.13



DRBD: Ninja tricks


You will need to discard all data on the secondary node and reimport all of them fromthe master node.

Umount all drbd

Umount all failed DRBD mounted devices before continuing !!!

On the master node, we need to connect all failed DRBD devices first :

Connect DRBD$ drbdadm connect <drbd_volume >



DRBD: Ninja tricks


And then sync all data to the secondary node :

Sync from master$ drbdadm -- --discard -my -data connect <drbd_volume >

Check the sync state in /proc/drbd :

Check sync state$ cat /proc/drbdversion : 8.3.7 (api :88/ proto :86 -91)srcversion : EE47D8BF18AC166BE219757

0: cs: SyncSource ro: Primary / Secondary ds: UpToDate / Inconsistent C r.ns :912248 nr :0 dw :0 dr :920640 al :0 bm :55 lo :1 pe :388 ua :2048 ap :0

[=== >................] sync ’ed: 21.9% (3283604/4194304) Kfinish : 1:08:24 speed : 580 (452) K/sec

Mount your DRBD device when finished.



DRBD: Ninja tricks


Note : there is a difference between "invalidate" and "discard-my-data" :

I drbdadm invalidate : This forces the local device of a pair of connected DRBDdevices into SyncTarget state, which means that all data blocks of the device arecopied over from the peer. This command will fail if the device is not part of aconnected device pair.

I drbdadm – –discard-my-data : Use this option to manually recover from asplit-brain situation. In case you do not have any automatic after-split-brainpolicies selected, the nodes refuse to connect. By passing this option you makethis node a sync target immediately after successful connect.



DRBD: Ninja tricks

Plan




DRBD: Ninja tricks

Mount secondary node as read only

There are several ways to mount secondary node in a read only state but several waysto break your data on the secondary node as well. This method is the best to avoid afull resync.First, create a snapshot on your logical volume on top of the DRBD device :

Create logical volume$ lvcreate -s -n <lv >- snapshot -L 2G /dev/<vg >/<lv >

You’re now able to mount the snapshot as a read only device. To make it work, youabsolutely need to specify the filesystem :

Mount filesystem$ mount -t ext4 -o ro /dev/<vg >/<lv >- snapshot / mountpoint

Backup all your desired data.



DRBD: Ninja tricks

Mount secondary node as read only

You can check any time that your data are still under sync with ’drbd-overview’command. When you’ve finished, you need to umount the mountpoint :

Unmount$ umount / mountpoint

and remove the logical volume :

Delete LV$ lvremove /dev/<vg >/<lv >- snapshot



DRBD: Ninja tricks

Plan




DRBD: Ninja tricks

All is good, but still doesn’t want to connect

All your configuration is ok, your logs doesn’t give any errors but the connect methoddoesn’t seam to do something.The last chance is to try the ’adjust’ command :

Adjust$ drbdadm adjust <drbd_volume >$ drbdadm connect <drbd_volume >

This should work now



DRBD: Ninja tricks

Plan




DRBD: Ninja tricks

Can’t "create-md" on an old DRBD device

If you get this kind of error when you want to recreate and md device on an oldexisting one :

Initiate DRBD device$ drbdadm create -md <drbd_volume >

Found some data==> This might destroy existing data! <==

Do you want to proceed ?[need to type ’yes ’ to confirm ] yes

drbdadm create -md nameResource : exited with code 40

To resolve the problem, erase all data on the current device :

Flush disk data$ drbdadm wipe -md <drbd_volume >

Now restart the create procedure.



DRBD: Ninja tricks

Can’t "create-md" on an old DRBD device

If it doesn’t work, because you don’t have your configuration any more for example,you can do it manually :

Flush disk data$ dd if =/ dev/zero of =/ dev/sdaX bs =1M count =128

Now restart the create procedure.



DRBD: Ninja tricks

Plan




DRBD: Ninja tricks

My DRBD device is still there but not the configuration

You may want to remove the DRBD device but unfortunately you did it in the wrongway or an automatic script didn’t did it well.The solution consist to recreate the configuration file to get the device recovered.

Warning

Before creating a configuration file in the drbd.d folder, check that you’ve configuredthe correct device and ports to avoid conflicting with already used DRBD devices.

You can check that you’ve recovered your device with drbd-overview.Then you’ll be able to follow the remove procedure.



DRBD: Ninja tricks

Plan




DRBD: Ninja tricks

Speed up sync transfer rate

If you need to speed up the transfer rate because you need to do a full sync betweento hosts, you can change it on the fly. It does not make sense to set a synchronizationrate higher than the maximum write throughput on your secondary node, so set it upto a correct rate on the secondary node :

Set sync ratedrbdsetup /dev/< drbd_volume > syncer -r 900M

Then, when you’ve finished and want to revert to the default value :

Set sync ratedrbdadm adjust <drbd_volume >



www.enovance.com


drbd

Technology

drbd conguration

drbd device

use drbd

drbd service

shortcoming of drbd

drbd pierre mavro

replication network

replication packets