![Page 1: 7 - Red Hat - SDCIndia 2018 Gluster Scale Atin - Repaired · Elastic volume management Configuration consistency Distributed command execution (orchestration) Service management (manages](https://reader035.vdocuments.net/reader035/viewer/2022071012/5fcae459036c8f3b3d3f77f2/html5/thumbnails/1.jpg)
Gluster can’t scale - Is it a reality or a past?
Atin MukherjeeEngineering Manager, Red Hat Gluster Storage
![Page 2: 7 - Red Hat - SDCIndia 2018 Gluster Scale Atin - Repaired · Elastic volume management Configuration consistency Distributed command execution (orchestration) Service management (manages](https://reader035.vdocuments.net/reader035/viewer/2022071012/5fcae459036c8f3b3d3f77f2/html5/thumbnails/2.jpg)
Agenda
rBrief about GlusterFSrScaling bottlenecks in GlusterrHow did (and are) we overcoming themrProjects/features like GlusterD2, brick multiplexing, sub-directory fuse mount, flexi sub vol.rQ&A
![Page 3: 7 - Red Hat - SDCIndia 2018 Gluster Scale Atin - Repaired · Elastic volume management Configuration consistency Distributed command execution (orchestration) Service management (manages](https://reader035.vdocuments.net/reader035/viewer/2022071012/5fcae459036c8f3b3d3f77f2/html5/thumbnails/3.jpg)
What is GlusterFS
● A general purpose scale-out distributed file system.
● Aggregates storage exports over network interconnect to provide a single unified namespace.
● Filesystem is stackable and completely in userspace.
● Layered on disk file systems that support extended attributes.
![Page 4: 7 - Red Hat - SDCIndia 2018 Gluster Scale Atin - Repaired · Elastic volume management Configuration consistency Distributed command execution (orchestration) Service management (manages](https://reader035.vdocuments.net/reader035/viewer/2022071012/5fcae459036c8f3b3d3f77f2/html5/thumbnails/4.jpg)
Some key terminologies
rTrusted Storage Pool (TSP) - Trusted Storage Pool (cluster) is a collection of storage servers.rPeer – Node in the clusterrVolume - A volume is a logical collection of bricksrBrick - A brick is the combination of a node and an export directory – for e.g. hostname:/dir
![Page 5: 7 - Red Hat - SDCIndia 2018 Gluster Scale Atin - Repaired · Elastic volume management Configuration consistency Distributed command execution (orchestration) Service management (manages](https://reader035.vdocuments.net/reader035/viewer/2022071012/5fcae459036c8f3b3d3f77f2/html5/thumbnails/5.jpg)
Present..
● RHGS cluster of ~150 nodes run in production deployments (with some caveats)
● Community version - ~200-250 nodes (with some caveats)
● With OpenShift Deployments, ~600 volumes on a 3 node cluster. (Yes! Thanks to brick multiplexing feature)
![Page 6: 7 - Red Hat - SDCIndia 2018 Gluster Scale Atin - Repaired · Elastic volume management Configuration consistency Distributed command execution (orchestration) Service management (manages](https://reader035.vdocuments.net/reader035/viewer/2022071012/5fcae459036c8f3b3d3f77f2/html5/thumbnails/6.jpg)
Scaling bottlenecks
● Node scalability
– Is 150 nodes deployment in hybrid cloud sufficient with the
growing storage demand?
– 1000 nodes cluster even be a reality?
● GD1 (GlusterD 1) ====> GD2 (GlusterD2)
● GD2 (released as tech preview with GlusterFS 4.0) will
become the default management plane for Gluster in
coming releases
![Page 7: 7 - Red Hat - SDCIndia 2018 Gluster Scale Atin - Repaired · Elastic volume management Configuration consistency Distributed command execution (orchestration) Service management (manages](https://reader035.vdocuments.net/reader035/viewer/2022071012/5fcae459036c8f3b3d3f77f2/html5/thumbnails/7.jpg)
Scaling bottlenecks – Node scalability
● GlusterD - Manages the cluster configuration for Gluster
Peer membership management
Elastic volume management
Configuration consistency
Distributed command execution (orchestration)
Service management (manages Gluster daemons)
![Page 8: 7 - Red Hat - SDCIndia 2018 Gluster Scale Atin - Repaired · Elastic volume management Configuration consistency Distributed command execution (orchestration) Service management (manages](https://reader035.vdocuments.net/reader035/viewer/2022071012/5fcae459036c8f3b3d3f77f2/html5/thumbnails/8.jpg)
Scaling bottlenecks – Node scalability
● Limitations of GD1
– Non linear node scalability
– Lack of native ReST APIs
– Code maintainability and easier feature integrations in the form of plugins
● What’s in for GD2
– Centralized store (etcd)
– Better transaction framework
– ReST API support
– Intelligent Volume Provisioning
![Page 9: 7 - Red Hat - SDCIndia 2018 Gluster Scale Atin - Repaired · Elastic volume management Configuration consistency Distributed command execution (orchestration) Service management (manages](https://reader035.vdocuments.net/reader035/viewer/2022071012/5fcae459036c8f3b3d3f77f2/html5/thumbnails/9.jpg)
Node scalability - GD2 Architecture
![Page 10: 7 - Red Hat - SDCIndia 2018 Gluster Scale Atin - Repaired · Elastic volume management Configuration consistency Distributed command execution (orchestration) Service management (manages](https://reader035.vdocuments.net/reader035/viewer/2022071012/5fcae459036c8f3b3d3f77f2/html5/thumbnails/10.jpg)
Scaling bottlenecks
● Volume scalability
– Thousands of volumes in cluster?
● Brick Multiplexing
![Page 11: 7 - Red Hat - SDCIndia 2018 Gluster Scale Atin - Repaired · Elastic volume management Configuration consistency Distributed command execution (orchestration) Service management (manages](https://reader035.vdocuments.net/reader035/viewer/2022071012/5fcae459036c8f3b3d3f77f2/html5/thumbnails/11.jpg)
Volume scalability – Brick Multiplexing
● Brick Multiplexing (GlusterFS 3.10 onwards)
– 1:1 process & brick ===> 1:Many process & bricks
– Reduced resource (Port, Memory, CPU) consumption
– Not a performance enhancer!
– One of the gluster core salient features for container native storage’s success.
![Page 12: 7 - Red Hat - SDCIndia 2018 Gluster Scale Atin - Repaired · Elastic volume management Configuration consistency Distributed command execution (orchestration) Service management (manages](https://reader035.vdocuments.net/reader035/viewer/2022071012/5fcae459036c8f3b3d3f77f2/html5/thumbnails/12.jpg)
Other ways to scale
● Fuse based sub directory export (GlusterFS 3.12 onwards)
– Single volume, sub directory based isolation for different clients
– Namespace access control through auth.allow
– Gluster snapshots need to work at volume level
![Page 13: 7 - Red Hat - SDCIndia 2018 Gluster Scale Atin - Repaired · Elastic volume management Configuration consistency Distributed command execution (orchestration) Service management (manages](https://reader035.vdocuments.net/reader035/viewer/2022071012/5fcae459036c8f3b3d3f77f2/html5/thumbnails/13.jpg)
Other ways to scale
● Gluster sub-vol (https://github.com/gluster/gluster-subvol)
– Use subdirectories of Gluster volumes as persistent volumes in openshift.io
– glfs-vol – a flex volume plugin to allow mounting Gluster subdirectories into containers.
– Uses xfs quota for setting up quotas for the directory based space control.
![Page 14: 7 - Red Hat - SDCIndia 2018 Gluster Scale Atin - Repaired · Elastic volume management Configuration consistency Distributed command execution (orchestration) Service management (manages](https://reader035.vdocuments.net/reader035/viewer/2022071012/5fcae459036c8f3b3d3f77f2/html5/thumbnails/14.jpg)
Other scaling challenges
● Resource control
● Debugg-ability
● User experience
![Page 15: 7 - Red Hat - SDCIndia 2018 Gluster Scale Atin - Repaired · Elastic volume management Configuration consistency Distributed command execution (orchestration) Service management (manages](https://reader035.vdocuments.net/reader035/viewer/2022071012/5fcae459036c8f3b3d3f77f2/html5/thumbnails/15.jpg)
Q & A
![Page 16: 7 - Red Hat - SDCIndia 2018 Gluster Scale Atin - Repaired · Elastic volume management Configuration consistency Distributed command execution (orchestration) Service management (manages](https://reader035.vdocuments.net/reader035/viewer/2022071012/5fcae459036c8f3b3d3f77f2/html5/thumbnails/16.jpg)
Thank you!
● Reach us @– [email protected] gluster-
[email protected]– IRC : #gluster, #gluster-dev @freenode