webinar presentation: stories of accidental data loss
TRANSCRIPT
![Page 1: Webinar Presentation: Stories of Accidental Data Loss](https://reader036.vdocuments.net/reader036/viewer/2022070517/58cf9df21a28ab6b088b45bf/html5/thumbnails/1.jpg)
Confidential and Proprietary1
Did I Do That? Stories of Accidental Data Loss
![Page 2: Webinar Presentation: Stories of Accidental Data Loss](https://reader036.vdocuments.net/reader036/viewer/2022070517/58cf9df21a28ab6b088b45bf/html5/thumbnails/2.jpg)
Confidential and Proprietary2
Agenda Real-world Data Loss Incidents
Lessons Learned
Business Impacts of Inadequate Data Protection
Case Studies on Data Protection
Architecture Considerations for Backup & Recovery
![Page 3: Webinar Presentation: Stories of Accidental Data Loss](https://reader036.vdocuments.net/reader036/viewer/2022070517/58cf9df21a28ab6b088b45bf/html5/thumbnails/3.jpg)
Confidential and Proprietary3
Incident #1 – Multiple Backup Solutions Do Not Ensure Protection
Production – db1Issues, Merge Requests, Code Snippets
Standby– db2
• 6 Hours of Data Lost
• ~40 Hours of Downtime
Data Protection Production Issue Restore Issue• Snapshots• NFS Backups• S3 Backups• Database Dump• Replication
• Spammers destabilized DB• Replication stopped
working• DBA deleted files on
production DB instead of standby
• Replication not working• Snapshot 24 hours old• NFS backup could not be
located• Database dump failing• Backup to S3 not working
![Page 4: Webinar Presentation: Stories of Accidental Data Loss](https://reader036.vdocuments.net/reader036/viewer/2022070517/58cf9df21a28ab6b088b45bf/html5/thumbnails/4.jpg)
Confidential and Proprietary4
Incident #2 – Multiple Replicas Do Not Ensure Protection
• 2 weeks of downtime• Resources working
24x7• Total Cost - $1.1M Hadoop Cluster
Multi-PetabyteHIVE
Data
Inge
st
Data Protection Production Issue Restore Issue
• Relying on 3 replicas in Hadoop
• Developer cleaning old tables• Accidentally dropped active
table• 500 TB data lost
• All replicas deleted• No backups• Rebuilt table by re-ingesting
all data
![Page 5: Webinar Presentation: Stories of Accidental Data Loss](https://reader036.vdocuments.net/reader036/viewer/2022070517/58cf9df21a28ab6b088b45bf/html5/thumbnails/5.jpg)
Confidential and Proprietary5
Incident #3 – Immutable Storage Systems Do Not Ensure Protection
• Regulatory certification denied• Large fines
CRM Application
Secure Storage for Contracts
Retention Management
Data Protection Production Issue Restore Issue• Secure storage
environment with retention management
• Sales rep changed retention for the wrong customer from 24 mos. to 6 mos.
• 18 months of contracts deleted from storage
• No backups resulting in permanent data loss
![Page 6: Webinar Presentation: Stories of Accidental Data Loss](https://reader036.vdocuments.net/reader036/viewer/2022070517/58cf9df21a28ab6b088b45bf/html5/thumbnails/6.jpg)
Confidential and Proprietary6
Lessons Learned
Production Environment
Use descriptive database names
No testing in production databases (limited access)
Use multiple replicas to protect from hardware failure – NOT BACKUP
Use remote replication for Disaster Recovery – NOT BACKUP
Backup Architecture Design
Not all data is equalDesign based on Recovery-Time-Objectives & Recovery-Point-Objectives (RPO)Store backups off-hostKeep multiple restore pointsKnow the cost of recreating lost data
Operations
Create and maintain a playbook
Monitor backup processes
Regularly practice recovery scenarios
![Page 7: Webinar Presentation: Stories of Accidental Data Loss](https://reader036.vdocuments.net/reader036/viewer/2022070517/58cf9df21a28ab6b088b45bf/html5/thumbnails/7.jpg)
Confidential and Proprietary7
Business Impact of Inadequate Data Protection
Extended Application Downtime• Revenue loss• Productivity loss• Lose customers to competitors• Damage to reputation/brandData Loss• Valuable resources spent on rebuilding data• Regulatory & compliance violations resulting in
fines• Customer frustration• Lawsuits
![Page 8: Webinar Presentation: Stories of Accidental Data Loss](https://reader036.vdocuments.net/reader036/viewer/2022070517/58cf9df21a28ab6b088b45bf/html5/thumbnails/8.jpg)
Confidential and Proprietary8
Data Protection Case Study #1
Amazon EC2Cassandra Production Cluster
52 nodes, 58 Terabytes
BACKUP
Backup/Recovery Challenges• Backing up and restoring a 58 TB
table• Updating QA/DR cluster with
production data
Solution Characteristic• Combined backup & DR• Incremental-forever backups & restores• Storage de-duplication & compression• Backup to Amazon S3
Amazon EC2Cassandra DR Cluster
36 nodes, 58 Terabytes
DR
![Page 9: Webinar Presentation: Stories of Accidental Data Loss](https://reader036.vdocuments.net/reader036/viewer/2022070517/58cf9df21a28ab6b088b45bf/html5/thumbnails/9.jpg)
Confidential and Proprietary9
Talena GUI
Five node Talena
cluster usingdirect-attached
storage
Vertica Cluster10-node
Vertica Cluster10-node
HadoopCluster
Data Protection Case Study #2
Backup/Recovery Challenges
• Native backup tool had limitations
• Multiple scripts hard to manage
Solution Characteristics• Hadoop Namenode backup• Topology-agnostic Vertica
backup/recovery• Single pane of glass
![Page 10: Webinar Presentation: Stories of Accidental Data Loss](https://reader036.vdocuments.net/reader036/viewer/2022070517/58cf9df21a28ab6b088b45bf/html5/thumbnails/10.jpg)
Confidential and Proprietary10
Data Protection Case Study #3
Hadoop Cluster37 nodes, 425
TB
Hadoop Cluster8-node, 50 TB
Hadoop Cluster16 nodes, 180
TB
PROTECT
ARCHIVE
BACKUP
ARCHIVEBACKUP
BACKUP
Backup/Recovery Challenges• Complexity of managing
600+TB of backups• Unable to meet RPO
requirements
Solution Characteristics• Single pane of glass for all Hadoop
clusters• Hourly backup to meet stringent RPO
needs• Scale-out solution to accommodate
scale
Talena GUI
![Page 11: Webinar Presentation: Stories of Accidental Data Loss](https://reader036.vdocuments.net/reader036/viewer/2022070517/58cf9df21a28ab6b088b45bf/html5/thumbnails/11.jpg)
Confidential and Proprietary11
Architectural Considerations for Backup & Recovery
Dealing with Distributed Deployments• Scale-out solution• Parallel data transfers between primary & backup
Storage• An agentless modelDealing with Volume & Velocity• De-duplicated, compressed backups• Scalable, searchable catalog that can store millions of
objects• Incremental-forever backups with single-step restore• Fast & granular recoveryDealing with Data Variety• Application-aware backups and restores• Content-aware storage optimization
![Page 12: Webinar Presentation: Stories of Accidental Data Loss](https://reader036.vdocuments.net/reader036/viewer/2022070517/58cf9df21a28ab6b088b45bf/html5/thumbnails/12.jpg)
Confidential and Proprietary12
Q&AWe’ll send you a link to our eBook “Compendium of Data Loss Horror Stories”
Additional resources: talena-inc.com/resources and talena-inc.com/blog
Ping us with any additional questions: [email protected]
![Page 13: Webinar Presentation: Stories of Accidental Data Loss](https://reader036.vdocuments.net/reader036/viewer/2022070517/58cf9df21a28ab6b088b45bf/html5/thumbnails/13.jpg)
Confidential and Proprietary13
THANK YOU