managing terabytes on aws
DESCRIPTION
JFrog is handling huge amount of binaries files for all our customers. Since each customer has its own space and domain, using a global Object Store can be tricky. Also our application cannot work with an “eventually consistent” storage, and cannot deliver customer requirements with current S3 performance. Learn in this session how we managed fast upload, critical replication and backups, and global download availability of the terabytes of JFrog customer binaries files.TRANSCRIPT
![Page 1: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/1.jpg)
![Page 2: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/2.jpg)
![Page 3: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/3.jpg)
![Page 4: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/4.jpg)
![Page 5: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/5.jpg)
![Page 6: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/6.jpg)
![Page 7: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/7.jpg)
![Page 8: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/8.jpg)
![Page 9: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/9.jpg)
![Page 10: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/10.jpg)
![Page 11: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/11.jpg)
![Page 12: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/12.jpg)
![Page 13: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/13.jpg)
Request
Storage File Listing Checksum Storage
![Page 14: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/14.jpg)
Request
Storage File Listing Checksum Storage
Metadata requests
Cache check, FS request
DB request
![Page 15: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/15.jpg)
Request
Storage File Listing Checksum Storage
Metadata requests
Cache check, FS request
DB request
HEAD requests Cache check, FS request
DB request
![Page 16: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/16.jpg)
Request
Storage File Listing Checksum Storage
Metadata requests
Cache check, FS request
DB request
HEAD requests Cache check, FS request
DB request
GET request Cache check, FS request
DB + FS request
![Page 17: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/17.jpg)
Request
Storage File Listing Checksum Storage
Metadata requests
Cache check, FS request
DB request
HEAD requests Cache check, FS request
DB request
GET request Cache check, FS request
DB + FS request
PUT request FS request DB + FS request
![Page 18: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/18.jpg)
![Page 19: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/19.jpg)
![Page 20: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/20.jpg)
![Page 21: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/21.jpg)
![Page 22: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/22.jpg)
![Page 23: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/23.jpg)
![Page 24: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/24.jpg)
![Page 25: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/25.jpg)
![Page 26: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/26.jpg)
![Page 27: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/27.jpg)
![Page 28: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/28.jpg)
![Page 29: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/29.jpg)
![Page 30: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/30.jpg)
![Page 31: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/31.jpg)
![Page 32: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/32.jpg)
![Page 33: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/33.jpg)
![Page 34: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/34.jpg)
![Page 35: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/35.jpg)
![Page 36: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/36.jpg)
![Page 37: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/37.jpg)
![Page 38: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/38.jpg)
EBS + LVM
= WIN!
![Page 39: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/39.jpg)
/data
250GB 250GB
![Page 40: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/40.jpg)
/data
250GB 250GB 250GB 250GB
![Page 41: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/41.jpg)
/data
250GB 250GB 250GB 250GB 250GB 250GB
![Page 42: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/42.jpg)
![Page 43: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/43.jpg)
![Page 44: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/44.jpg)
![Page 45: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/45.jpg)
![Page 46: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/46.jpg)
![Page 47: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/47.jpg)
![Page 48: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/48.jpg)
![Page 49: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/49.jpg)
![Page 50: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/50.jpg)
![Page 51: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/51.jpg)
Backup files
![Page 52: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/52.jpg)
Backup files
Backup database
![Page 53: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/53.jpg)
Backup files
Backup database
Keep them in sync!
![Page 54: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/54.jpg)
Backup files
Backup database
Keep them in sync!
![Page 55: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/55.jpg)
Backup only deltas
![Page 56: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/56.jpg)
Backup only deltas
Manage threads to
ec2 efficiently
![Page 57: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/57.jpg)
Backup only deltas
Manage threads to
ec2 efficiently
Keep the data
isolated
![Page 58: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/58.jpg)
Backup only deltas
Manage threads to
ec2 efficiently
Keep the data
isolated
![Page 59: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/59.jpg)
![Page 60: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/60.jpg)
![Page 61: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/61.jpg)
![Page 62: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/62.jpg)
![Page 63: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/63.jpg)
![Page 64: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/64.jpg)
![Page 65: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/65.jpg)
Customer EBS
Customer EBS
Customer EBS
![Page 66: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/66.jpg)
De-duped
EBS
HA Storage
Customer EBS
Customer EBS
Customer EBS
![Page 67: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/67.jpg)
![Page 68: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/68.jpg)
De-duped Glacier
De-duped
EBS Backup
HA Storage
Customer EBS
Customer EBS
Customer EBS
![Page 69: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/69.jpg)
![Page 70: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/70.jpg)
![Page 71: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/71.jpg)
![Page 72: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/72.jpg)
![Page 73: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/73.jpg)
![Page 74: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/74.jpg)
Create empty Artifactory instance
![Page 75: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/75.jpg)
Create empty Artifactory instance
Populate from back
storage on demand
![Page 76: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/76.jpg)
Create empty Artifactory instance
Populate from back
storage on demand
Drink
Cosmopolitan
![Page 77: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/77.jpg)
Conclusions time
1. Be lazy.
![Page 78: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/78.jpg)
![Page 79: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/79.jpg)
Conclusions time
1. Be lazy.
2. Be smart.
![Page 80: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/80.jpg)
Conclusions time
1. Be lazy.
2. Be smart.
3. Be agile.
![Page 81: Managing terabytes on aws](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a786cd8b42abb628b45c5/html5/thumbnails/81.jpg)
Drink
Cosmopolitan