build ha nfs cluster based cephfsbos.itdks.com/da6f82321d2d4b2ea87a30b40104e0ea.pdfhow virtual ip...
TRANSCRIPT
![Page 1: Build HA NFS Cluster Based CephFSbos.itdks.com/da6f82321d2d4b2ea87a30b40104e0ea.pdfHow Virtual IP Pool CTDB Node1 CTDB Node2 CTDB Node3 Cluster File System RIP: 100.100.100.102 100.100.100.251~100.100.100.253](https://reader036.vdocuments.net/reader036/viewer/2022071401/60eb6a7e3ef95274f41adaac/html5/thumbnails/1.jpg)
Build HA NFS Cluster Based CephFS
![Page 2: Build HA NFS Cluster Based CephFSbos.itdks.com/da6f82321d2d4b2ea87a30b40104e0ea.pdfHow Virtual IP Pool CTDB Node1 CTDB Node2 CTDB Node3 Cluster File System RIP: 100.100.100.102 100.100.100.251~100.100.100.253](https://reader036.vdocuments.net/reader036/viewer/2022071401/60eb6a7e3ef95274f41adaac/html5/thumbnails/2.jpg)
Why01
What 02
How03
04 Future
![Page 3: Build HA NFS Cluster Based CephFSbos.itdks.com/da6f82321d2d4b2ea87a30b40104e0ea.pdfHow Virtual IP Pool CTDB Node1 CTDB Node2 CTDB Node3 Cluster File System RIP: 100.100.100.102 100.100.100.251~100.100.100.253](https://reader036.vdocuments.net/reader036/viewer/2022071401/60eb6a7e3ef95274f41adaac/html5/thumbnails/3.jpg)
Why
NFS(Network File System):
1) Widely used network protocol2) Many enterprises still heavily depend on NFS to access their data from different operating systems and applications3) Traditional storage system(SAN/NAS) is being replaced with SDS
![Page 4: Build HA NFS Cluster Based CephFSbos.itdks.com/da6f82321d2d4b2ea87a30b40104e0ea.pdfHow Virtual IP Pool CTDB Node1 CTDB Node2 CTDB Node3 Cluster File System RIP: 100.100.100.102 100.100.100.251~100.100.100.253](https://reader036.vdocuments.net/reader036/viewer/2022071401/60eb6a7e3ef95274f41adaac/html5/thumbnails/4.jpg)
Why
CephFS(Ceph File System):
1) Ceph widely used 2) Ceph uniquely delivers object, block, and file storage in one unified system3) Ceph-fuse need to be installed on client, which limits its application scope
![Page 5: Build HA NFS Cluster Based CephFSbos.itdks.com/da6f82321d2d4b2ea87a30b40104e0ea.pdfHow Virtual IP Pool CTDB Node1 CTDB Node2 CTDB Node3 Cluster File System RIP: 100.100.100.102 100.100.100.251~100.100.100.253](https://reader036.vdocuments.net/reader036/viewer/2022071401/60eb6a7e3ef95274f41adaac/html5/thumbnails/5.jpg)
What
MDS(Active)
MDS(Standby)
MON MON MON
OSD
OSD
OSD OSD
OSD OSDMDS(Standby)
libcephfs librbd librgw
![Page 6: Build HA NFS Cluster Based CephFSbos.itdks.com/da6f82321d2d4b2ea87a30b40104e0ea.pdfHow Virtual IP Pool CTDB Node1 CTDB Node2 CTDB Node3 Cluster File System RIP: 100.100.100.102 100.100.100.251~100.100.100.253](https://reader036.vdocuments.net/reader036/viewer/2022071401/60eb6a7e3ef95274f41adaac/html5/thumbnails/6.jpg)
What
NFS Server mandatory requirements:
1) Reliability2) Performance 3) Scalability
![Page 7: Build HA NFS Cluster Based CephFSbos.itdks.com/da6f82321d2d4b2ea87a30b40104e0ea.pdfHow Virtual IP Pool CTDB Node1 CTDB Node2 CTDB Node3 Cluster File System RIP: 100.100.100.102 100.100.100.251~100.100.100.253](https://reader036.vdocuments.net/reader036/viewer/2022071401/60eb6a7e3ef95274f41adaac/html5/thumbnails/7.jpg)
What
Reliability:Multiple active NFS servers(Redundant NFS servers)
MDS(Active)
MDS(Standby)
MON MON MON
OSD
OSD
OSD OSD
OSD OSDMDS(Standby)
libcephfs librbd librgw
![Page 8: Build HA NFS Cluster Based CephFSbos.itdks.com/da6f82321d2d4b2ea87a30b40104e0ea.pdfHow Virtual IP Pool CTDB Node1 CTDB Node2 CTDB Node3 Cluster File System RIP: 100.100.100.102 100.100.100.251~100.100.100.253](https://reader036.vdocuments.net/reader036/viewer/2022071401/60eb6a7e3ef95274f41adaac/html5/thumbnails/8.jpg)
What
Performance:1) Ceph and CephFS3) Multiple active NFS servers(Load Balance)
MDS(Active)
MDS(Standby)
MON MON MON
OSD
OSD
OSD OSD
OSD OSDMDS(Standby)
libcephfs librbd librgw
![Page 9: Build HA NFS Cluster Based CephFSbos.itdks.com/da6f82321d2d4b2ea87a30b40104e0ea.pdfHow Virtual IP Pool CTDB Node1 CTDB Node2 CTDB Node3 Cluster File System RIP: 100.100.100.102 100.100.100.251~100.100.100.253](https://reader036.vdocuments.net/reader036/viewer/2022071401/60eb6a7e3ef95274f41adaac/html5/thumbnails/9.jpg)
What
Scalability:1) Ceph and CephFS2) Add more NFS Servers
MDS(Active)
MDS(Standby)
MON MON MON
OSD
OSD
OSD OSD
OSD OSDMDS(Standby)
libcephfs librbd librgw
![Page 10: Build HA NFS Cluster Based CephFSbos.itdks.com/da6f82321d2d4b2ea87a30b40104e0ea.pdfHow Virtual IP Pool CTDB Node1 CTDB Node2 CTDB Node3 Cluster File System RIP: 100.100.100.102 100.100.100.251~100.100.100.253](https://reader036.vdocuments.net/reader036/viewer/2022071401/60eb6a7e3ef95274f41adaac/html5/thumbnails/10.jpg)
What
HA NFS Cluster:
1) NFS server based CephFS2) HA No SPOF(Single Point Of Failure) LB(Load Balance)
![Page 11: Build HA NFS Cluster Based CephFSbos.itdks.com/da6f82321d2d4b2ea87a30b40104e0ea.pdfHow Virtual IP Pool CTDB Node1 CTDB Node2 CTDB Node3 Cluster File System RIP: 100.100.100.102 100.100.100.251~100.100.100.253](https://reader036.vdocuments.net/reader036/viewer/2022071401/60eb6a7e3ef95274f41adaac/html5/thumbnails/11.jpg)
How
Kernel NFS Server:
1) Performance – User mode can be slow2) Hard to debug kernel module
![Page 12: Build HA NFS Cluster Based CephFSbos.itdks.com/da6f82321d2d4b2ea87a30b40104e0ea.pdfHow Virtual IP Pool CTDB Node1 CTDB Node2 CTDB Node3 Cluster File System RIP: 100.100.100.102 100.100.100.251~100.100.100.253](https://reader036.vdocuments.net/reader036/viewer/2022071401/60eb6a7e3ef95274f41adaac/html5/thumbnails/12.jpg)
How
User Space NFS Server(NFS-ganesha):
1) Flexible and Plug-able FSAL2) Clustering becomes natural and easy3) Easy restarts, failover, failback implementation
![Page 13: Build HA NFS Cluster Based CephFSbos.itdks.com/da6f82321d2d4b2ea87a30b40104e0ea.pdfHow Virtual IP Pool CTDB Node1 CTDB Node2 CTDB Node3 Cluster File System RIP: 100.100.100.102 100.100.100.251~100.100.100.253](https://reader036.vdocuments.net/reader036/viewer/2022071401/60eb6a7e3ef95274f41adaac/html5/thumbnails/13.jpg)
HowSy
slog
API
Exte
rnal
Con
trol
API
Logg
ing
Mod
ule
Adm
inis
trat
ion
Mod
ule
Has
h Ta
bles
Mem
ory
Man
ager
File Content CacheMetadata Cache
RPC Dispatcher
RPCSEC_GSS Dup Req Module
Mount V1/V3 NFS v2/v3 NFS v4
File System Abstraction Layer
GSSA
PI
Secu
rity
cache fs operations
fs operations
Clients requests
libcephfs VFS RGW GLUSTER ......
NFS-ganesha Architecture
![Page 14: Build HA NFS Cluster Based CephFSbos.itdks.com/da6f82321d2d4b2ea87a30b40104e0ea.pdfHow Virtual IP Pool CTDB Node1 CTDB Node2 CTDB Node3 Cluster File System RIP: 100.100.100.102 100.100.100.251~100.100.100.253](https://reader036.vdocuments.net/reader036/viewer/2022071401/60eb6a7e3ef95274f41adaac/html5/thumbnails/14.jpg)
How
CephFS Architecture
MDS(Active)
MDS(Standby)
MON MON MON
OSD
OSD
OSD OSD
OSD OSDMDS(Standby)
Ceph Storage Cluster Protocol(librados)
Ceph FS Library(libcephfs)
CephFS Kernel Object CephFS FUSE
![Page 15: Build HA NFS Cluster Based CephFSbos.itdks.com/da6f82321d2d4b2ea87a30b40104e0ea.pdfHow Virtual IP Pool CTDB Node1 CTDB Node2 CTDB Node3 Cluster File System RIP: 100.100.100.102 100.100.100.251~100.100.100.253](https://reader036.vdocuments.net/reader036/viewer/2022071401/60eb6a7e3ef95274f41adaac/html5/thumbnails/15.jpg)
How
NFS ganesha based CephFS
MDS(Active)
MDS(Standby)
MON MON MON
OSD
OSD
OSD OSD
OSD OSDMDS(Standby)
Ceph Storage Cluster Protocol(librados)
Ceph FS Library(libcephfs)
NFS ganesha NFS service
![Page 16: Build HA NFS Cluster Based CephFSbos.itdks.com/da6f82321d2d4b2ea87a30b40104e0ea.pdfHow Virtual IP Pool CTDB Node1 CTDB Node2 CTDB Node3 Cluster File System RIP: 100.100.100.102 100.100.100.251~100.100.100.253](https://reader036.vdocuments.net/reader036/viewer/2022071401/60eb6a7e3ef95274f41adaac/html5/thumbnails/16.jpg)
How
HA(High Availability) Solution:1) Pacemaker + corosync2) CTDB + LVS3) HAproxy4) Piranha(RedHat)5) Heatbeat + mon6) Keepalive......
![Page 17: Build HA NFS Cluster Based CephFSbos.itdks.com/da6f82321d2d4b2ea87a30b40104e0ea.pdfHow Virtual IP Pool CTDB Node1 CTDB Node2 CTDB Node3 Cluster File System RIP: 100.100.100.102 100.100.100.251~100.100.100.253](https://reader036.vdocuments.net/reader036/viewer/2022071401/60eb6a7e3ef95274f41adaac/html5/thumbnails/17.jpg)
How
CTDB:CTDB is a cluster implementation of the TDB database used by Samba and other projects to store temporary data.
CTDB features:1) provides a TDB that has consistent data and consistent locking across all nodes in a cluster.2) CTDB is very fast.3) In case of node failures, CTDB will automatically recover and repair all TDB databases that it manages.4) provides HA features such as node monitoring, node failover, and IP takeover.
CTDB requirements:a clustered filesystem, it must be mounted and available on all nodes.
![Page 18: Build HA NFS Cluster Based CephFSbos.itdks.com/da6f82321d2d4b2ea87a30b40104e0ea.pdfHow Virtual IP Pool CTDB Node1 CTDB Node2 CTDB Node3 Cluster File System RIP: 100.100.100.102 100.100.100.251~100.100.100.253](https://reader036.vdocuments.net/reader036/viewer/2022071401/60eb6a7e3ef95274f41adaac/html5/thumbnails/18.jpg)
How
Virtual IP Pool
CTDB Node1 CTDB Node2 CTDB Node3
Cluster File System
RIP: 100.100.100.102
100.100.100.251~100.100.100.253
LAN
CTDB Architecture
RIP: 100.100.100.103 RIP: 100.100.100.104
VIP: 100.100.100.251 VIP: 100.100.100.252 VIP: 100.100.100.253
![Page 19: Build HA NFS Cluster Based CephFSbos.itdks.com/da6f82321d2d4b2ea87a30b40104e0ea.pdfHow Virtual IP Pool CTDB Node1 CTDB Node2 CTDB Node3 Cluster File System RIP: 100.100.100.102 100.100.100.251~100.100.100.253](https://reader036.vdocuments.net/reader036/viewer/2022071401/60eb6a7e3ef95274f41adaac/html5/thumbnails/19.jpg)
How
MDS(Active)
MDS(Standby)
MON MON MON
OSD
OSD
OSD OSD
OSD OSDMDS(Standby)
libcephfs librbd librgw
NFS Service based CTDB
VIPs: 100.100.100.251~100.100.100.253
![Page 20: Build HA NFS Cluster Based CephFSbos.itdks.com/da6f82321d2d4b2ea87a30b40104e0ea.pdfHow Virtual IP Pool CTDB Node1 CTDB Node2 CTDB Node3 Cluster File System RIP: 100.100.100.102 100.100.100.251~100.100.100.253](https://reader036.vdocuments.net/reader036/viewer/2022071401/60eb6a7e3ef95274f41adaac/html5/thumbnails/20.jpg)
How
NFS Service based CTDB:1) Multiple VIP2) Load Balance
Solutions:1) Round Robin DNS2) LVS
![Page 21: Build HA NFS Cluster Based CephFSbos.itdks.com/da6f82321d2d4b2ea87a30b40104e0ea.pdfHow Virtual IP Pool CTDB Node1 CTDB Node2 CTDB Node3 Cluster File System RIP: 100.100.100.102 100.100.100.251~100.100.100.253](https://reader036.vdocuments.net/reader036/viewer/2022071401/60eb6a7e3ef95274f41adaac/html5/thumbnails/21.jpg)
LAN(10/100G)
How
LVS(Linux Virtual Server):Virtual server is a highly scalable and highly available server built on a cluster of real servers.It was in the offical kernel 2.6.10 released on December 25, 2004.
VirtualServer(Load Balancer)
RealServer 1
Client
RealServer 2 RealServer 3 RealServer 4
![Page 22: Build HA NFS Cluster Based CephFSbos.itdks.com/da6f82321d2d4b2ea87a30b40104e0ea.pdfHow Virtual IP Pool CTDB Node1 CTDB Node2 CTDB Node3 Cluster File System RIP: 100.100.100.102 100.100.100.251~100.100.100.253](https://reader036.vdocuments.net/reader036/viewer/2022071401/60eb6a7e3ef95274f41adaac/html5/thumbnails/22.jpg)
How
NFS Service based CTDB + LVS
MDS(Active)
MDS(Standby)
MON MON MON
OSD
OSD
OSD OSD
OSD OSDMDS(Standby)
libcephfs librbd librgw
RIPs: 100.100.100.251~100.100.100.253
VIP: 100.100.100.100
![Page 23: Build HA NFS Cluster Based CephFSbos.itdks.com/da6f82321d2d4b2ea87a30b40104e0ea.pdfHow Virtual IP Pool CTDB Node1 CTDB Node2 CTDB Node3 Cluster File System RIP: 100.100.100.102 100.100.100.251~100.100.100.253](https://reader036.vdocuments.net/reader036/viewer/2022071401/60eb6a7e3ef95274f41adaac/html5/thumbnails/23.jpg)
Future
1) Performance2) CTDB depends on a clustered file system(CephFS).