bitrot detection in glusterfs - events.static.linuxfound.org · gaurav garg. we are, gluster...
TRANSCRIPT
![Page 1: BitRot detection in GlusterFS - events.static.linuxfound.org · Gaurav Garg. We are, Gluster developers at Red Hat... participate in meetups, open source events... hang out on #freenode:](https://reader033.vdocuments.net/reader033/viewer/2022043023/5f3f487d4376b6086f2f78a7/html5/thumbnails/1.jpg)
BitRot detection in GlusterFS
Venky ShankarGaurav Garg
![Page 2: BitRot detection in GlusterFS - events.static.linuxfound.org · Gaurav Garg. We are, Gluster developers at Red Hat... participate in meetups, open source events... hang out on #freenode:](https://reader033.vdocuments.net/reader033/viewer/2022043023/5f3f487d4376b6086f2f78a7/html5/thumbnails/2.jpg)
We are, Gluster developers at Red Hat
... participate in meetups, open source events
... hang out on #freenode: gluster, gluster-dev nick: overclk, ggarg... interact with community: [email protected] [email protected]
![Page 3: BitRot detection in GlusterFS - events.static.linuxfound.org · Gaurav Garg. We are, Gluster developers at Red Hat... participate in meetups, open source events... hang out on #freenode:](https://reader033.vdocuments.net/reader033/viewer/2022043023/5f3f487d4376b6086f2f78a7/html5/thumbnails/3.jpg)
OK, enough. Let’s get started...
![Page 4: BitRot detection in GlusterFS - events.static.linuxfound.org · Gaurav Garg. We are, Gluster developers at Red Hat... participate in meetups, open source events... hang out on #freenode:](https://reader033.vdocuments.net/reader033/viewer/2022043023/5f3f487d4376b6086f2f78a7/html5/thumbnails/4.jpg)
GlusterFS Quick Tour
![Page 5: BitRot detection in GlusterFS - events.static.linuxfound.org · Gaurav Garg. We are, Gluster developers at Red Hat... participate in meetups, open source events... hang out on #freenode:](https://reader033.vdocuments.net/reader033/viewer/2022043023/5f3f487d4376b6086f2f78a7/html5/thumbnails/5.jpg)
Where’s my data?
● Distributed● Local filesystem (brick)
○ XFS○ EXT3, EXT4○ BTRFS
● Prerequisite○ POSIX compatible○ Xattr support
![Page 6: BitRot detection in GlusterFS - events.static.linuxfound.org · Gaurav Garg. We are, Gluster developers at Red Hat... participate in meetups, open source events... hang out on #freenode:](https://reader033.vdocuments.net/reader033/viewer/2022043023/5f3f487d4376b6086f2f78a7/html5/thumbnails/6.jpg)
Understanding data corruption
![Page 7: BitRot detection in GlusterFS - events.static.linuxfound.org · Gaurav Garg. We are, Gluster developers at Red Hat... participate in meetups, open source events... hang out on #freenode:](https://reader033.vdocuments.net/reader033/viewer/2022043023/5f3f487d4376b6086f2f78a7/html5/thumbnails/7.jpg)
Corruption?How ?
● Direct “brick” manipulation○ Script bug○ Admin○ Malicious
![Page 8: BitRot detection in GlusterFS - events.static.linuxfound.org · Gaurav Garg. We are, Gluster developers at Red Hat... participate in meetups, open source events... hang out on #freenode:](https://reader033.vdocuments.net/reader033/viewer/2022043023/5f3f487d4376b6086f2f78a7/html5/thumbnails/8.jpg)
Corruption?How ?
(cont..)
● Silent corruption○ Disk itself
■ Firmware bug■ Mechanical wear■ Ageing
![Page 9: BitRot detection in GlusterFS - events.static.linuxfound.org · Gaurav Garg. We are, Gluster developers at Red Hat... participate in meetups, open source events... hang out on #freenode:](https://reader033.vdocuments.net/reader033/viewer/2022043023/5f3f487d4376b6086f2f78a7/html5/thumbnails/9.jpg)
Illustration
![Page 10: BitRot detection in GlusterFS - events.static.linuxfound.org · Gaurav Garg. We are, Gluster developers at Red Hat... participate in meetups, open source events... hang out on #freenode:](https://reader033.vdocuments.net/reader033/viewer/2022043023/5f3f487d4376b6086f2f78a7/html5/thumbnails/10.jpg)
![Page 11: BitRot detection in GlusterFS - events.static.linuxfound.org · Gaurav Garg. We are, Gluster developers at Red Hat... participate in meetups, open source events... hang out on #freenode:](https://reader033.vdocuments.net/reader033/viewer/2022043023/5f3f487d4376b6086f2f78a7/html5/thumbnails/11.jpg)
Solution: Integrity checks
![Page 12: BitRot detection in GlusterFS - events.static.linuxfound.org · Gaurav Garg. We are, Gluster developers at Red Hat... participate in meetups, open source events... hang out on #freenode:](https://reader033.vdocuments.net/reader033/viewer/2022043023/5f3f487d4376b6086f2f78a7/html5/thumbnails/12.jpg)
Integrity CheckConsistency
● Track data modifications○ Checksum (signature)○ Persistent
● Verify during access○ Recompute and check
● Repair if corrupted
![Page 13: BitRot detection in GlusterFS - events.static.linuxfound.org · Gaurav Garg. We are, Gluster developers at Red Hat... participate in meetups, open source events... hang out on #freenode:](https://reader033.vdocuments.net/reader033/viewer/2022043023/5f3f487d4376b6086f2f78a7/html5/thumbnails/13.jpg)
Enough of theory, show me how it’s done.
![Page 14: BitRot detection in GlusterFS - events.static.linuxfound.org · Gaurav Garg. We are, Gluster developers at Red Hat... participate in meetups, open source events... hang out on #freenode:](https://reader033.vdocuments.net/reader033/viewer/2022043023/5f3f487d4376b6086f2f78a7/html5/thumbnails/14.jpg)
ImplementationConstraints on choices
● Big fat-file story● Deployments
○ Distribute + Replicate○ Stripe, now [3.7+] sharding○ Erasure coded
![Page 15: BitRot detection in GlusterFS - events.static.linuxfound.org · Gaurav Garg. We are, Gluster developers at Red Hat... participate in meetups, open source events... hang out on #freenode:](https://reader033.vdocuments.net/reader033/viewer/2022043023/5f3f487d4376b6086f2f78a7/html5/thumbnails/15.jpg)
ImplementationConstraints on choices (cotd..)
● In-band data signing○ Costly○ RMW cycle○ Degraded I/O performance
● Verification○ “Ditto”
![Page 16: BitRot detection in GlusterFS - events.static.linuxfound.org · Gaurav Garg. We are, Gluster developers at Red Hat... participate in meetups, open source events... hang out on #freenode:](https://reader033.vdocuments.net/reader033/viewer/2022043023/5f3f487d4376b6086f2f78a7/html5/thumbnails/16.jpg)
ImplementationDetails
● Out-of-band data signing○ Daemon○ Asynchronous
■ Policy■ Strong hash (reason ?)
● Verification○ Daemon (scrubber)○ On-demand○ Pre-scrubbed
![Page 17: BitRot detection in GlusterFS - events.static.linuxfound.org · Gaurav Garg. We are, Gluster developers at Red Hat... participate in meetups, open source events... hang out on #freenode:](https://reader033.vdocuments.net/reader033/viewer/2022043023/5f3f487d4376b6086f2f78a7/html5/thumbnails/17.jpg)
ImplementationDetails (cotd..)
● Object versioning○ Versioned upon modification○ Versioning xattr (64 bit)○ Reflect “object state”
● Signature○ xattr○ Attached to a “version”
![Page 18: BitRot detection in GlusterFS - events.static.linuxfound.org · Gaurav Garg. We are, Gluster developers at Red Hat... participate in meetups, open source events... hang out on #freenode:](https://reader033.vdocuments.net/reader033/viewer/2022043023/5f3f487d4376b6086f2f78a7/html5/thumbnails/18.jpg)
ImplementationDetails (cotd..)
● Integrity checking○ Periodic
■ daily, weekly, etc..○ Filesystem scan
■ Signature mismates■ Matching version
○ QoS■ Controlled crunching
● Corrupted objects○ Denies access (EIO)○ Repairable
■ Replica, Codes
![Page 19: BitRot detection in GlusterFS - events.static.linuxfound.org · Gaurav Garg. We are, Gluster developers at Red Hat... participate in meetups, open source events... hang out on #freenode:](https://reader033.vdocuments.net/reader033/viewer/2022043023/5f3f487d4376b6086f2f78a7/html5/thumbnails/19.jpg)
Use cases
![Page 20: BitRot detection in GlusterFS - events.static.linuxfound.org · Gaurav Garg. We are, Gluster developers at Red Hat... participate in meetups, open source events... hang out on #freenode:](https://reader033.vdocuments.net/reader033/viewer/2022043023/5f3f487d4376b6086f2f78a7/html5/thumbnails/20.jpg)
Use Cases● Small files● Long lived data
○ Archival storage
○ WORM workload
![Page 21: BitRot detection in GlusterFS - events.static.linuxfound.org · Gaurav Garg. We are, Gluster developers at Red Hat... participate in meetups, open source events... hang out on #freenode:](https://reader033.vdocuments.net/reader033/viewer/2022043023/5f3f487d4376b6086f2f78a7/html5/thumbnails/21.jpg)
Future
● Replica consistency● Metadata checksumming● Offloading
○ BTRFS● Sharding adaption● GlusterFS 4.0 [Interesting!]
○ In-band (weaker hash)○ Checksum everything○ Default○ Lost (phantom) writes
![Page 22: BitRot detection in GlusterFS - events.static.linuxfound.org · Gaurav Garg. We are, Gluster developers at Red Hat... participate in meetups, open source events... hang out on #freenode:](https://reader033.vdocuments.net/reader033/viewer/2022043023/5f3f487d4376b6086f2f78a7/html5/thumbnails/22.jpg)
3.7
● Bitrot detection● No recovery● In comes sharding
3.7.2
● Recovery support
3.7.4
● Bug fixes● Scrub status
3.8
● Sharding ready● Bitrot adaption
4.0
● Hell of a change● Sharding by default● Checksum everything
![Page 23: BitRot detection in GlusterFS - events.static.linuxfound.org · Gaurav Garg. We are, Gluster developers at Red Hat... participate in meetups, open source events... hang out on #freenode:](https://reader033.vdocuments.net/reader033/viewer/2022043023/5f3f487d4376b6086f2f78a7/html5/thumbnails/23.jpg)
Q & A