do you have a dr plan in place: so, don't let a disaster defeat your business
Post on 20-Aug-2015
449 Views
Preview:
TRANSCRIPT
You Have a Disaster Recovery Plan:
So, Don’t Let a Disaster Defeat Your
Business
MWLUG
August 22, 2012
93% of businesses that suffer a significant loss of data are out of business within 4 years.
The Bureau of Labor
Sobering Statistic
Disaster Preparedness Checkpoint #1:Are Disasters Viewed as a Real Threat?
Disasters never really happen.
It’s just a vendor sales gimmick…
August 29, 2012 Velocity Proprietary and Confidential4
Why Is This Area “ Vital ” ?
� Expectations of the IT Services are demanding
� Technology is an enabler of business
� Penalties are becoming more severe
� Business is becoming more competitive
� Businesses rely more heavily on their enterprise applications
than any other time in history
5
What is a Disaster ?
ANYTHING !!!!
That stops your business from functioning & that cannot be corrected within an acceptable amount of time….
What is Disaster Recovery?
Reaction to a sudden, unplanned event that enables an organization to continue critical business functions until
normal business operations resume.
“…It is not enough to arrange for hardware replacement;…planning must address continuation of business operations, or
business continuation.”
8
Common Issues with Disaster Recovery Planning
We tend to let our guard down when times improve.
As planners, we must always be ready and prepared.
• Are the expectations of your DR plan realistic?
• Has your plan kept up to date with your IT integrations?
• Do you have a qualified staff to execute the plan?
• Is DR integrated into your Change Control process?
August 29, 2012 Velocity Proprietary and Confidential8
9
Protect Important Assets
Primary assets needed to operate Information Systems:
• Hardware and networks can be replaced
• Facilities can be rebuilt or relocated
• Your Data is Priceless !
• People are assets too!
August 29, 2012 Velocity Proprietary and Confidential9
Checkpoint # 2: Of course we have a DR plan… Somewhere
Plan it yourself or get someone to do it, but whatever you do.. Plan!
August 29, 2012 Velocity Proprietary and Confidential10
11
DR Plans Come in Many Flavors
� To be on holidays when the
big one hits
� Cross your Fingers
� RESUME
� Resume stored offsite
12
No / Incomplete Disaster Recovery Plan
Guarantees:
• Confusion
• Lack of direction
• Conflict
• Lost Customers or credibility
13
The Products of the Plan
Who will execute recovery actions
What is needed to continue, resume, recover or restore business functions
When business functions and operations must resume
Where to go to resume corporate, business & operational functions
How detailed procedures for continuity, resumption, recovery or restoration
CLASSIC: WHO-WHAT-WHERE-WHEN-HOW
Checkpoint # 3: Ensuring Your Plan Meets Your Business Needs
Does your business know what your IT department can deliver?
August 29, 2012 Velocity Proprietary and Confidential14
Recover data to last known good
point before outage
RECOVERY POINT
OBJECTIVE
The amount of time it takes to
recover production systems
and resume operations
RECOVERY TIME
OBJECTIVE
Days MinsHrsWks Secs
Recovery Point
Mins DaysHrsSecs Wks
Recovery Time
What Are Your Recovery Objectives?
August 29, 2012 Velocity Proprietary and Confidential15
RTO - Recovery Time Objective
48 hours RTO – Systems Availability
RPO - Recovery Point Objective
24 Hours RPO - Tape Recovery
Recovery Solution Examples
August 29, 2012 Velocity Proprietary and Confidential16
Categorize Your Technology Inventory as Critical vs. Necessary vs. Not at All
Typically look for a 60, 20, 20 breakdown.
• 60% of all your servers are critical for recovery• 20% are necessary but not required immediately• Remaining 20% are optional or not required at all.
This means you can focus on 60% not the 100%.
August 29, 2012 Velocity Proprietary and Confidential17
It has become Mission Critical.
It represents a way you communicate with customers, partners, employees, and the world.
Imagine no Email !!
Should EMAIL be an H/A application?
August 29, 2012 Velocity Proprietary and Confidential18
Traditional Backup Process
Offsite Storage
Facility
Traditional Backup & Recovery Process
Offsite Storage
( Bonded Provider )
Backup
Software
August 29, 2012 Velocity Proprietary and Confidential19
How Much Data Can You Afford to Lose?
• Lost data is lost money!
• Most IT shops depend on tape backups to protect their data
• Data considerations for tape
• Minimum of 24 hrs. of lost data
• Application access may be more critical
August 29, 2012 Velocity Proprietary and Confidential20
• Successfully deliver RTO objectives
Configure hardware
Install OS
Configure OS
Install backup agent
Load each and Every application from CD / Tape, FTP etc.
RestoreVM
Poweron VM
True recovery process comparison
24 + hrs
< 2+ hrs
Recovery Timeline Sample
August 29, 2012 Velocity Proprietary and Confidential21
August 29, 2012 Velocity Proprietary and Confidential22
Does your Business Know What IT Can Deliver?
August 29, 2012 Velocity Proprietary and Confidential23 23
Management Awareness
� Management shocked that the IT folks lost all orders
from previous day.
� 48 Hour RTO - Who agreed to this ?
� Do you have a new Senior Management Team ?
Checkpoint #4: Backups
Build it and they will come.
Put into practice what you say you will do.
August 29, 2012 Velocity Proprietary and Confidential24
Tape Backups - Pros
� Easy to run
� Easy to manage – Tape Management Products
� Easy to Automate – Automation Software Solutions
� High Speed & High Capacity reduced backup windows
� Capacity of tapes reduces number of Tapes
� It’s the way we have always done things
� Virtual tape libraries
� Local Electronic Backups
• Media, interface, or drive errors
• Seldom validated – How do you know your backups are valid?
• Time to ship, travel and restore tapes at Hotsite in a disaster
• Loss of data because backup only runs once per day
• Backups omitted in lieu of system access
• Lost tapes – Misplaced!
Tape Backups - Cons
27August 29, 2012 Velocity Proprietary and Confidential27
August 29, 2012 Velocity Proprietary and Confidential28 28
Backup = Recovery
How many people backup their system ?
38% are in-complete
19 % iSeries ( AS400) are un-recoverable
32 % Windows are un-recoverable
30 % Oracle DB are un-recoverable
15 % Virtual Infrastructure un-recoverable
29
“Backups ran fine. I never heard of any issues.”
• With a sign off, the process implies correctness
• Many backup solutions are partially broken
• Backup Success Rate = 97%
93
94
95
96
97
98
99
1 2 3 4 5 6 7 8 9 10 11 12
Failure rate = 3 %
This equals 11 days per year
with NO Backups
August 29, 2012 Velocity Proprietary and Confidential29
30
Sample Log
• Starting SAVDLO of folder *ANY to devices TAP01.
• 2574 document library objects saved.
• Starting save of list *LINK to devices TAP01.
• 43917 objects saved. 342 not saved.
• Save of list *LINK completed with errors.
• Starting save of media information at level *OBJ to device
• 18 objects saved from library QUSRBRM.
• Save of BRM media information at level *OBJ complete.
• DAILY *BKU 0070 *EXIT CALL PGM(BBSYSTEM/ENDDAYBU).
• Control group DAILY type *BKU completed with errors.
Backup/Recovery Best Practices
• Examine your backup strategy and policies for data
• Review backup strategy and policies for server infrastructure
Document server related information with signoff
• Are your network configurations backed up?
• Review off-site tape and vital records storage strategies
• Have you audited your backups?
• Can you demonstrate recovery procedures for daily, weekly, and monthly backups?
August 29, 2012 Velocity Proprietary and Confidential31
Checkpoint #5: Building a Recovery Team
Getting the right people:
Do they have the Right Stuff?
August 29, 2012 Velocity Proprietary and Confidential32
Perspective
Too often, companies populate their DR teams with raw or inexperienced staffers
Volunteers to satisfy an auditor or the sacrificial lamb
Ideal Team Member Characteristics
• Considered an expert by his or her peers
• A go-to person for anything and/or everything
• Works well under pressure
• Sense of urgency, willing to fix problems created by others
• Confident
• Trusted by peers
• Dedicated – A company person
August 29, 2012 Velocity Proprietary and Confidential34
Recovery is only possible if your staff is available
to put IT back together again.
Disasters affect people in unpredictable ways:
• It can devastate people both physically and mentally
• Can effect others around them
• Make the individual unable to function
• Emotional breakdown
Respect the situation!
The Effects of an Unprepared Recovery Staff Are Real
August 29, 2012 Velocity Proprietary and Confidential35
The Role of Executives
Executives are not typically involved and should NOT be.
Why:
• Executive Pressure is hindering
• Intimidating
• Often Nasty
• Glorified Experts
The team must provide regular status reports & the
Executives should be accessible when required.
• Feed your staff - Have snacks handy
• Stay away from High Energy Drinks
• Enforce no shift duration to exceed 12 hours
• Encourage people to take breaks
• Provide distractions during breaks
Best Practices During a Disaster
Family Comes First
Until the basic personal needs are met
• Family comes first. Always !!
• Staff members will not focus on the Enterprise recovery
• Staff members may or will not be available
Irene was no lady!
People Evacuated
2, 465, 000
Checkpoint #6: Is Your Data Offsite?
It’s eleven o’clock.
Do you know where your data is?
August 29, 2012 Velocity Proprietary and Confidential39
Recipe for Disaster
• One average filing cabinet
• Your recovery tapes/CD’s
• One average business fire
August 29, 2012 Velocity Proprietary and Confidential40
Recipe for Disaster
Directions:
• Place media in filing cabinet.
• Bake in a fire for 20 minutesat 800°
• Let cool.
• Open filing cabinet.
August 29, 2012 Velocity Proprietary and Confidential41
Storing Backup Media on Site …Your Business Is Toast!
August 29, 2012 Velocity Proprietary and Confidential42
Offsite Strategy
DISASTER
STRIKES
10 AM
W T F S S M T W T F S S M T W T F S
FullBackup
S
OPERATIONS RECOVERY
POSSIBLEUNPROCESSED
DATA
PLANNED
OFFSITE
BACKUP
DAILY EXPOSURE
Incremental backup
August 29, 2012 Velocity Proprietary and Confidential43
Checkpoint # 7: Virtualization
Practice smart virtualization
August 29, 2012 Velocity Proprietary and Confidential44
46
– OS & applications have 1:1 dependencies on hardware configuration
– Complex to physically recover OS, applications & data
– Separate processes for system and application data
Slow and Unreliable Process,
Expensive Infrastructure
Physical Wintel Recovery Challenges Today
cd, tape or ghost image
Application
OS
OS files
localstorage
Storage
WAN
Application
OS
OS files
localstorage
Storage
Production Server “Boot & Pray”
47
DR Scenario – Without Virtualization
Hardware Gap - Your Production & DR must always be equal
Gap grows as new severs are deployed
49
� Break dependencies between OSand hardware
� Manage OS and application assingle unit by encapsulatingthem into VMs
� VM’s are hardware-independent: they can be provisioned anywhere
With Virtualization
50
Physical Servers Converted into a Virtual Solution
Production Site
Convert images to virtual machines
Image Repository
Import virtual machines to ESX Servers
DR Site
51
Physical to Virtual (P-V) then Recovery
How:
• Converter creates virtual machines matching physical machines
• Copy virtual machines to Denver
If Recovery is required:
• Boot virtual machines on any hardware
• Start data recovery of application data
imaging
imaging
imaging
conversion
conversion
conversion Replication
P2V
P2V
P2V
JAS
Deploy
Load-Balance
Checkpoint # 8: Testing your DR Plan
It worked last year.
Nothing has really changed…
August 29, 2012 Velocity Proprietary and Confidential52
It’s not all bad news if your plan fails during a test…
Frequent testing identifies gaps in your recovery process
August 29, 2012 Velocity Proprietary and Confidential53
Passive Testing
• Hands on plan review
• Paper walkthrough
– Assessment of current Recovery Plan
– Assessment of risk areas
– Involve every member of your recovery team. This includes the business and management!
August 29, 2012 Velocity Proprietary and Confidential54
Active Testing
Validates the Recovery Plan in terms of:
1. Recovery capabilities and abilities
2. Alternate site configuration
3. Network recovery
4. Offsite records
5. Identify weaknesses in the plan
6. Provides training
This all equals success during a disaster
August 29, 2012 Velocity Proprietary and Confidential55
Testing / Hotsite Best Practices
• Examine your hardware schedule in your DR contract
— Check for back level compatibility
— Do you need additional capacity?
— Performance Profile – Have database requirements changed?
— Travel and expenses
• Avoid testing from “special” backups
• Test your entire technology infrastructure
• Train your alternates
• Evaluate vendor performance
• Make sure your “hot box” is current for vital records
August 29, 2012 Velocity Proprietary and Confidential56
Disaster Recovery Planning Is an Ongoing Process
Plan
ImplementTest
Business
Requirements
…To be prepared, you must
regularly go through the cycle
August 29, 2012 Velocity Proprietary and Confidential57
Remember to Consider All the Key Checkpoints
1. Recognize that disasters do happen. Know how your business units
may be impacted and understand their expectations
2. Ensure you have an actionable and repeatable DR plan
3. Confirm your DR plan supports your business
4. Backup: Put into practice what you say you will do
5. Build a recovery team
6. Keep your data stored safely offsite and know how to access it.
7. Practice smart virtualization
8. Test your plan, and then update the plan accordingly
August 29, 2012 Velocity Proprietary and Confidential58
StaffingStaffing
TestingTesting
Disaster RecoveryDisaster Recovery
RiskRisk
StandardizationStandardization
CareerCareer
Aging InfrastructureAging Infrastructure
GovernanceGovernance
What’s on the Mind of CIOs . .
DR Preparedness for 21st Century
Understand and are Ready
– Top Gun
Understand and are not Ready
– Sleepless in Seattle
Understand but don’t want DR
- Dumb and Dumber
Don’t Understand. Why Bother !
– Clueless or Armageddon
top related