19.10.2017
1
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
A practical approach to implement a high
available data protection solution with virtualized
IBM Spectrum Protect Servers in a Disk based
environment
How to set up and operate an IBM Spectrum
Protect System virtualized without Tape, but
Container Storage Pools and Replication
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany 2
Personal Data
Remo Leuppi is a Storage and Backup Engineer for Baloise Group. His area of expertise includes HDS, EMC, NetApp storage knowhow and IBM
Tivoli Storage Manager. His the Projectmanager for the following implementation at Baloise Group
Some words about Baloise / Basler Versicherung
The Baloise Group is focused on continental Europe with insurance, pension solutions and banking services. Baloise Group is an attractive,
modern employer represents solidity since 1863. The Company with about 7400 employees is listed on SIX Swiss Exchange, Main Market
Segment.
• Remo Leuppi
• Baloise Group
19.10.2017
2
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany 3
Personal Data
Gerd Becker is a Project Manager for EMPALIS Consulting GmbH, an IBM Business Partner in Germany. He has more than 40 years of IT
experience, including over 20 years experience with storage management products such as DFSMS and Tivoli Storage Manager. His areas of
expertise include IBM Tivoli Storage Manager implementation projects and education at customer sites, including mainframe environments
(OS/390®, VSE, VM, and Linux® for zSeries®).
In the context of data protection he has certifications and experience in project management.
He holds several certifications, including technical and sales, and is an IBM Tivoli Certified Instructor. He has developed and taught several
storage classes for IBM Education Services in Germany, Switzerland and Austria and for qSkills in Nuernberg. He has been Chairman of the
Guide Share Europe (GSE) Storage-Usergroup for more than 15 years. He is author of the Redbooks „IBM Tivoli Storage Manager Technical
Guide 5.3, IBM Tivoli Storage Manager Technical Guide 6.1, IBM Tivoli Storage Manager Certification Guide 6.1, IBM Tivoli Storage Manager as
a Dataprotection Solution“, did the betatest for TSM Version since 5.3 to 7.1.x , IBM Spectrum Protect to 8.1.x and is member of the IBM
Spectrum Protect Early Access Program.
• Gerd Becker
• Empalis Consulting GmbH
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany 4
Agenda
• Baloise Group – existing Environment
• Propositions
• Initial Design, architectural Concept
• Implementation
• Project Experiences
• Outlook
• Hints and Tips
19.10.2017
3
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Baloise Group Server Landscape
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Baloise Group Matrix of competences
19.10.2017
4
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Baloise Group TSM Workload
• MS SQL
• MySQL
• Domino
• Mainframe
• SAS
• DB2
Domain Summary Total Data GB Summary Total files
CLUSTER 14.223,36 598.948
DB2 41.423,64 2.333.921
DOMINO 1.209,95 2.344.888
DOMINO_DAOS 5.168,16 7.051.932
DOMINO_DB 103.762,05 408.609
FILER 75.275,08 276.778.124
LINUX-OS 30.456,52 45.521.779
ORACLE_DB 11.017,78 26.724
OS 98.098,31 51.764.118
SAP_DB 19.056,64 1.110.052
SAS 29,31 170.047
SHAREPOINT 24,65 623
SQL 49.295,91 883.235
Result 449.041,36 388.993.000
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Propositions – where we come from
• 3 Node Cluster
• HP DL380 G8
• 128 GB Memory
• 2 TSM Instances
• Max 64 GB Memory
• Primary and Copy Pool
• Primary on HDS HUS-VM (DC1)
• Secondary on HDS HUS-VM
(DC3)
• DB / LOG / ARCH on NetApp • with 4 snapshots per day
• Snapshot replication to DC3
19.10.2017
5
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Propositions – why we will change the world
Challenges and Motivation
• Hardware out-of-life
• Software out-of-support
• Data amount growth
• New Functions
• Less complexity
• Upgrading DataCenter 2• before only data position (storage)
• Now with compute power (storage and servers)
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Propositions – guidelines for the new design
• Windows Operating System
• Disk based solution – no Tape
• Virtualized servers (if possible)
• High level backup and recovery performance
• High available environment – cluster solution
• Disaster protection – distant datacenters
• Ransomware attack protection – air gap (without tape ?)
• Hardware efficient solution
• Cost optimized solution
• Bidirectional replication of data
19.10.2017
6
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Initial Design – TSM Architecture Concept
• TSM VM
• Win 2012 R2
• 64 GB Memory
• SAN Raw Device• DB (Flash)
• Activelog (Flash)
• Archivelog (SAS and NL-SAS)
• Datapool (SAS and NL-SAS)
• Hyper-V Cluster
• Win 2012 R2
• Ressource shared with DPM
• 4 Host DC1 / 2 Host DC2• DL380 G9
• 256 GB Memory
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Initial Design – Hyper-V Architecture Concept
• Cluster DC1• 4 Host
• DL380 G9
• 256 GB Memory
• 2X 40 Gb Network
• 2x 16 Gb FC (1per Fabric)
• VM• OS Data on shared Datastore
• Virtual WWN for SAN Disk
• Hyper-V Datastore• CSV (NetApp)
• Mirrored to DC2
• Snapshot all 6h
• Presented on all Host
• TSM Data • HDS HUS VM
• Raw Device (FC)
• Direct to VM
19.10.2017
7
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Implementation – Step by Step
Step overview:
1. Installing test TSM Instances – Test installation procedure and basic functions
2. Working with IBM Blueprint – where you can find information
3. Performance Test (Blueprint) – scripts for I/O testing and server setup, what we used and what not
4. Installing Mgmt Server (HUB and TSMManager) – Administrators view
5. Installing production TSM Instances – implementation guidelines
6. Generate Housekeep and other scripts managed by Mgmt-Server – Enterprise Management
7. Server-to-Server Connection – New instances, old instances
8. Data Migration Approach
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Implementation – Step 1
Installation Procedure
Hardware preparation
Software Installation with the Installation Manager
Installation latest good level (July 2016) IBM Spectrum Protect Version 7.1.5
Setting up the Sever Instances Test1 (Basel), Test2 (Zürich)
Basic Tests, functionality of container pools and node replication
Testing benefits: Compression and Deduplication
19.10.2017
8
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Implementation – Step 2
IBM Blueprint:
- Design for IBM Hardware
- Design for „real“ Servers, not virtualized
- Naming conventions not variable, will not fit to every environment
+ very good basic information
+ values can be adapted to every hardware
+ guidelines for the setup
https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli%20Storage%20Manager/page/IBM%20Spectrum%20Protect%20Blueprints
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Implementation – Step 3 - Performance
Containerpool performance:Blueprint baseline:
19.10.2017
9
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Implementation – Step 3 - Performance
DB performance:
Blueprint baseline:
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Implementation – Step 4 – Admins view
IBM: Operations Center JamoDat: TSM-Managerpros
• Free of charge – included in the product
• Helpful for beginners
• New functions and features available with the
server-level
• Web based application
cons
• Dependency to the hub server
• Resource hungry
• Functionality limited – implemented via
commandbuilder
pros
• Easy to use
• Compatible to all server levels
• New functions and features near time
available with new server-level
• Long term reports included
• Resource saving
cons
• License fee
• Dependency between collector and
viewer
• Only Microsoft Windows based
• New functions like replication, cloud
rare implemented
19.10.2017
10
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Implementation – Step 5 – Implementation Guidelines
• Windows 2012 R2 virtualized with Hyper-V
• Disk Storage FC direct attached to the Guest System (Raw Device Mapping for STGPools, Database and Log)
• Only one Spectrum Protect Instance per OS
• Fast Network Connection – 10 Gb per VM (Host 2x 40 Gb)
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Implementation – Step 6 – Housekeep and Protection
Relationship between instances for Disaster
protection:
- Data: Protect Stgpool
- Metadata: Replicate node
- Database Data Instances: s2s virtual volumes
- Database management Instance: local copy
on Netapp, Snap Mirror to the remote site
- Recovery Planfiles Data Instances: s2s virtual
volumes
- Recovery Planfiles management instance:
local copy on Netapp, Snap Mirror to the
remote site
19.10.2017
11
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Implementation – Step 7 – Server – to – Server Relationship
• Old Servers TSM Version 6.3
• New Servers IBM Spectrum Protect Version 7.1.5
• High Speed Network Connection (10 Gb)
• Disk Storage FC direct attached to the Guest System (Raw Device Mapping)
• Only one Spectrum Protect Instance per OS
• Fast Network Connection – DWDM Basel - Zurich
40 Gb between Hyperv Hosts
10 Gb between HyperV Host and SP Server
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Implementation – Step 8 – Data Migration
Possible Approaches:
• „big bang“ Switch: dry out old systems during data retention
• Export / Import: move all data from all old systems to the new systems
• Mix of 1. and 2.: initial backup to the new systems, dry out old data with short retention, export/import data with long term retention and archive objects (mergefilespace)
Challenge:
• Initial backup with a huge amount of data
• Import not direct in container pool, must be a sequential pool
• Convert stgpool necessary, file-pool after conversion unusable
Solution:
• Migration Node by Node with migration plan
• Additional resources temporary available (File Stgpool)
• Automated procedure for export/import and Stgpool conversion
• No NDMP-Data to migrate
• No Backupsets to migrate
19.10.2017
12
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Implementation – Step 8 – Data MigrationPossible Approaches:
• „big bang“ Switch: dry out old systems during data retention
• Export / Import: move all data from all old systems to the new systems
• Mix of 1. and 2.: initial backup to the new systems, dry out old data with short retention, export/import data with long
term retention and archive objects (mergefilespace)
Challenge:
• Initial backup with a huge amount of data
• Import not direct in container pool, must be a sequential pool
• Convert stgpool necessary, file-pool after conversion unusable
Solution:
• Migration Node by Node with migration plan
• Additional resources temporary available (File Stgpool)
• Automated procedure for export/import and Stgpool conversion
• No NDMP-Data to migrate
• No Backupsets to migrate
Migration: Control of the Import-Process
/* ------------------------------------------------------------------*/
/* Script Name: ++IMPORT_nodes */
/* Description: Workflow for Migration-process */
/* with Export / Import */
/* controlling script variable ZZ_node_xxxx */
/* 1. Check Status */
/* 2. Start Export without data */
/* 3. Connect Node in Import Domain */
/* 4. Start Export with Data */
/* 5. End Part 1 */
/*--------------------------------------------------------------------*/
/*----->>>>>>>>>>>>> convert Filepool if necessary <<<<<<<<----------*/
/*--------------------------------------------------------------------*/
/* 6. Check Status */
/* 7. connect node in target domain */
/* 8. associate schedule */
/* 9. delete node admin */
/* 10. >>>>>>> to do manually */
/* - >>>>>>> associate schedules */
/* - >>>>>>> change client option file */
/* - >>>>>>> generate password (dsmcutil) */
/* - >>>>>>> restart scheduler */
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Project Experiences
1. Security Guidelines for Node access
2. New node registration Procedure
3. Disaster Recovery Procedure
4. Daily Ingest Procedures
5. Storage Pool savings (dedup and compression
6. Storage Pool protection
7. Node Replication
8. Overall Data Occupancy
9. Database Occupancy
19.10.2017
13
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Project Experiences – Security Guidelines• Access to Clients without Password:
the password has to be stored encrypted –
password access generate
• The node password set during registration
will expire in a short term (5 days)
• No access to other client‘s data
• No web access with node admin
• Only Server administrators can do backups
and restores (via RDP)
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Migration: activate Node after Export, Conversion and Registration
/* -----------------------------------------*/
/* Script Name: ++activate_node */
/* Description: Migration: aktivate Node after Export, Conversion and Registration */
/* Parameter1 node-Name */
/* Parameter2 Domaene */
/* Parameter3 Schedule */
/* example1: run ++activate_node node1 os*/
start:
upd node $1 dom=$2 replstate=ena maxnummp=5 passexp=5 cloptset=""
if (rc_ok) goto Step2
goto Fehler
Step2:
unlock node $1
if (rc_ok) goto Step3
goto Fehler
Step3:
remove admin $1
if (rc_ok) goto Step4
if (rc_notfound) goto step4
goto Fehler
Step4:
define association $2 $3 $1
if (rc_ok) goto Ende
if (rc_exists) goto Ende
goto Fehler
Ende:
issue message i "node $1 ready for use"
exit
Fehler:
issue Message e "node $1 not ready for use, please check"
exit
Project Experiences – New Node registration
Procedure to register new nodes:
Requirement open registration
• Graphical Interface for Node registration
• Creation of dsm.optnode-name, tcpserver, tcpport
• Run dsmc, enter password and contact
• Setup scheduler with password (passwordaccess generate)
• Node activation:- connect node to target domain- set passwordexpiration- remove node admin-define association to the schedule
19.10.2017
14
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Project Experiences – Disaster RecoveryProcedure to prevent disaster situationsDifferent Levels of Disaster
• Temporary outagenode failover to the target siteonly restore and retrieveno backup, e.g. for database logs
• Permanent outage of the primary datacenterConfiguration change, Target <-> Sourcemust be manually done – tested scriptsway back probably complicated
• Replacement of the primary datacenterrecover virtualized Server (binaries and definitions)restore the databaserecover the container storage pools
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Project Experiences – Daily ingest Procedures
Approximately 11 to 20 TB per Day
19.10.2017
15
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Project Experiences – Storage Pool Savings
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Project Experiences – Storage Pool Protection
SOURCE_POOL
TARGET_
SERVER START_TIME END_TIME Duration
GBYTES_TO_
PROTECT
GBYTES_
PROTECT
ED
EXTENTS_TO
_PROTECT
EXTENTS_
PROTECTE
D
EXTENTS_TO_
DELETE
EXTENTS_
DELETED
COMP_
STATE
CONTAINERPOOL TSMP031 09.08.2017 02:00 09.08.2017 03:59 1:59:17 1.231.713 1.231.713 16074343 16074343 2296814 2296814 Success
CONTAINERPOOL TSMP031 17.08.2017 02:00 17.08.2017 04:18 2:18:22 1.001.617 1.001.617 17059265 17059265 29706896 29706896 Success
CONTAINERPOOL TSMP031 12.08.2017 02:00 12.08.2017 08:51 6:51:13 964.971 964.971 13483738 13483738 28843040 28843040 Success
CONTAINERPOOL TSMP031 04.08.2017 02:00 04.08.2017 03:57 1:57:39 939.475 939.475 12817938 12817938 19018680 19018680 Success
CONTAINERPOOL TSMP031 18.08.2017 02:00 18.08.2017 04:47 2:47:06 912.685 912.685 16488847 16488847 8842000 8842000 Success
CONTAINERPOOL TSMP031 14.08.2017 02:00 14.08.2017 07:15 5:14:31 865.929 865.929 12265702 12265702 12577193 12577193 Success
CONTAINERPOOL TSMP031 16.08.2017 02:00 16.08.2017 03:39 1:39:23 816.214 816.214 12398328 12398328 6908774 6908774 Success
19.10.2017
16
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Project Experiences – Node Replication
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Project Experiences – Overall Occupancy
Savings with deduplication / compression
Less amount of data to license
Upgrade V8.1.1 ???
19.10.2017
17
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Project Experiences – Database OccupancyMassive database growth with container pool deduplication
DB Backup to virtual Volumes with compression
Performance and Tuning Guide:
For a rough estimate, plan for 100 GB of database
storage for every 50 TB of data that is to be
protected in deduplicated storage pools.
Protected data is the amount of data before data
deduplication, including all versions of objects
stored.
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Outlook – Future and Functions
• Upgrade Spectrum Protect Server to Version 8.1.1 with eFix
8.1.1.9
• Upgrade Operations Center to Version 8.1.1
• Update Spectrum Protect Clients to Version 8
• Enhanced Security with Version 8.1.2 +
19.10.2017
18
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Outlook – Obscurities and Questions
• Why does the node replication takes longer than protect
stgpool ?
• When will we be able, to do backups to the target server in
case of a disaster?
See RFE 107699 and 109500
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Hints and Tips
With Air-Gap against ransomware !
• Disk only solution
before
after
19.10.2017
19
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Questions
ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks
26.-29. September 2017, Cologne Marriott Hotel, Germany
Thank you
Remo Leuppi
Dipl. Techniker HF
Wirtschaftsinformatik
Corporate IT
Basler Versicherungen,
Aeschengraben 21, Postfach,
4002 Basel
Telefon +41 58 285 76 70,
Mobile +41 76 451 94 08
www.baloise.com