The Next FrontierOpportunities With the Cloud and Virtualization

Kellyn Pot’Vin-Gorman

Kellyn Pot’Vin-Gorman

Technical Intelligence Manager for the Office of CTO,

Delphix

• Multi-platform DBA, (Oracle, MSSQL, MySQL, Sybase,

PostgreSQL…..)

• Oracle ACE Director, (Alumni), Oak Table

• APEX Women in Technology Award, CTA

• STEM education with Raspberry Pi and Python

• Liaison for Denver SQL Server User Group

• President, Rocky Mountain Oracle User Group

• Author, blogger, (http://dbakevlar.com)

Management of Non-production Environments:

Virtualized sanity for the DBA Realist.

Provisioning: Patching, refreshing and if you

ask me one more time!

Cloud: Cloudy with a chance of failures.

Security: Yo Developer- Is that the SA password

taped to your monitor??

The Life of a DBA

Provision Databases

Refresh and provide data to reporting, testing

and development

Secure database environments

Optimize data access

Collaborate to solve business challenges

What is Copy Data Management, (CDM)

The management of all non-production databases.

Broad Term- Physical and virtual clones

Managed or unmanaged

Command line or User Interface, (or both)

Administrative, Infrastructure, security

Why Virtualize- This Scientific Reason

• The Economics of Data- the natural life of a database is growth. It’s only

going to get bigger.

• Von Newmann’s Bottleneck- the limiting factor of computing speed

dependent on where the data resides and how much data there is.

• Data Gravity- Jim McCrory coined this term for the gravitational pull of

applications and services to data.

These may be theories, these may be viewed as technology challenges to be

overcome another day, but physics an important consideration in technology.

Storage costs

Thin-provisioning storage avoidance

Data transfer costs

Far less data transferred during provisioning/refresh

operations

As opposed to the volume of data transferred using

traditional cloning techniques

Simplifies Provisioning vs. archaic processes to copy data

Why Use Virtualize- the Business

Reason

8

▶▶▶

Virtualize and Deployed▶ ▶ ▶

80% of Environemnts are Repeat Data

Storage Pool for Delphix

QADEV PATCH TEST

PRODUCTION

Database/App Tier

1 TB

1 TB

0.6 TB

Read From Production

Spin a VIRTUAL database up a patch test, without having to remove a current development or test one.

TEST

Example Environment- 1TB

Data Virtualization: From Prod to

Virtual

Validated Sync Environment

Source Environment

SCSI/SSL

Data Virtualization: From Prod to

Virtual

Validated Sync

Environment

Source Environment

SC

SI/S

SL

Any StorageCreate as many VDBs as needed!

Data Virtualization: Space Savings

Delphix Virtualization EngineValidated Sync Target

Dozen Virtual Database Copies, (VDBs)

Data Virtualization: How is it possible?

Delphix Virtualization Engine

Any Storage

Virtual Database

• Using any storage and only fraction of space

• Syncs with native or third-party SQL Server backups

• Can maintain two weeks of data changes

• Managed just like any SQL Server database

• Users can instantly provision a read/write virtual copy

of a database

• Can be used for replication, mirroring, change data

capture (CDC), and maintenance.

This is Data Version Control

Administration Ease

Bookmarking and Branching

Release 1.2 Release 1.3 Branch 1.3

Branch 1.1

17

Spin up a new

VIRTUAL DB and SQL

bin Files

Night Time ETL/Maintenance Challenges

Finance

Dev

Test

QA

ETL

BI

Reporting

DBCC’s

Scripted out to spin up new VDBs

to run nightly jobs, maintenance,

etc.

18

Epiphany

e·piph·a·ny

əˈpifənē/

noun

a (1) : a usually sudden manifestation or perception of the essential nature or meaning of

something (2) : an intuitive grasp of reality through something (as an event) usually simple and

striking (3) : an illuminating discovery, realization, or disclosure

b : a revealing scene or moment

19

Patching- Got the T-shirt

20

How Do We Build Out? How Long?

Finance

HR

CRM

21

22

Patching and Upgrading Databases

Each patch, would need to be applied to a

development database, requiring outage

to development teams and then tested

before applying to test, UAT and then

onto production.

This has to be performed to EACH

environment, every SQL Server, each

quarter.

Finance

HR

CRM

UAT Test DevProd

23

Downtime for valuable resources.

DBAs working afterhours

Each database must have it done and…

The tedious task must be performed over and over

again.

Little opportunity for advanced learning.

Each database may experience different bugs.

Risks/Challenges Of This Approach

24

Spin up a new

VIRTUAL DB and SQL

bin files and apply

patch to it.

Environment Virtualization, DB Style

CRM

Finance

HR

25

No need to keep the

extra VDBs post

patch to prod.

After Testing, Apply to Production

CRM

Finance

HR

26

The Compressed Copies

in the Delphix Engine

are Upgraded!

Environment Virtualization, DB Style

HR

Finance

CRM

27

In minutes,

refresh and

complete!

Environment Virtualization, DB Style

CRM

Finance

HR

28

29

• I didn’t have to take away a valuable resource’s database environment to test

the patches.

• I didn’t have to apply the patches to subsequent environments, as they are

virtualized copies of the source, simply requiring a refresh from production,

post final patch.

• I save significant time that commonly has to be allocated to quarterly and

annual maintenance for patching.

• I apply the patch twice- once to test, once to production. I only need to

refresh my environments after I’m done.

• For releases, this can be “containerized”, simplifying release and if required,

rollback.

Patching and Upgrading with

Virtualization

30

The quickest way to complete a task is not

having to do it.

31

Pro

ject

Tim

elin

e

Optimize

Migrate

Initial Testing

Requirements

Cloud Projects

Just copy data and applications into the cloud…• Straightforward approach

• Inefficient, non-incremental for large environments

• Open-source “bcp” uses encryption, compresses, and multi-threads

• Archaic processes recommended by vendors

Start with backups to IaaS storage, then populate re-hosted applications by restoring from those backups

• Cloud backups are easy, known technology

How Are Companies Migrating to

the Cloud?

Cost Estimates for Azure

https://azure.microsoft.com/en-us/pricing/details/storage/blobs/

Storage Capacity LRS

First 1 TB / Month $0.024 per GB

Next 49 TB (1 to 50 TB) / Month $0.0236 per GB

Next 450 TB (50 to 500 TB) / Month $0.0232 per GB

Next 500 TB (500 to 1,000 TB) / Month $0.0228 per GB

ZRS

$0.03 per GB

$0.0295 per GB

$0.029 per GB

$0.0285 per GB

Backups cloud service: from on-

premises to the Cloud

Database server

Cloud Storage

Migration Complete…Not so Fast…• What if you only want dev and test in the cloud?

• What about the application, support files and other data sources?

• Data is migrated, but this doesn’t count for ongoing data loads,

application connectivity across the network.

• To refresh will take considerable time to perform with traditional tools or

cloning methods.

• Rarely a consideration for the difference in cost structure for processing

large amounts of data from on-premise to the cloud.

https://blogs.msdn.microsoft.com/brunoterkaly/2013/09/26/how-to-migrate-an-on-premises-sql-server-2012-database-to-windows-azure-sql-database/

What’s the right

configuration??

https://docs.microsoft.com/en-us/azure/sql-database/sql-database-cloud-migrate

• Database must be MSSQL 2005 or higher, (easy)

• Ensure that the database is compatible with Azure SQL DB, (correct

any incompatible functions, etc.)

• Must have identified all performance issues that will be impacted

beforehand.

• Ensure there is as little physical distance between the cloud data

center and bacpac files to be used for migration.

• Disable management jobs that will hinder migration processing.

• Drop any objects or historical data that can impact migration time

and can be performed post migration.

Cloud Performance

Let’s Discuss Network Latency

• Network has been the bottleneck of every cloud project I’ve been a part of.

• There’s a reason that AWS has invested in the Snowball and Snowmachine.

• We can’t break the law of physics.

• Let’s talk about Shannon’s Law…

• In layman’s terms- the data is only going to go as fast as it can do so without hitting

a error threshold.

How We Migrate it All

Database server

Cloud Storage

Database server

Much Improved Option-

1. Data virtualization

Easy, secure, revolutionary

Simple movement via virtualized environments- movement of

storage encompassing approximately one environment,

no matter how many.

Extended time to “rehydrate” environments to physical if

desired.

Ability to containerize data sources, applications and support

files as Data Pods and move easily as one.

Migrating from on-prem to IaaS with

Virtualization

Data Virtualization, OnPrem & Data Pods

Source DB serverSQL Server 2008-2016

1 TB

Delphix Virtualization Engine2 TB storage

Target DB serverSQL Server 1008-2016No database storage

SCCI/SSL

Backups

via SMB

Bin files, flat files,

data sources

Virtualized and

now

containerized as

a Data Pod

Data virtualization: from on-premises

into the Cloud

Source DB serverSQL Server 2008-2016

2TB storage

Delphix Virtualization Engine2 TB storage

Target DB serverSQL Server 2008-2016No database storage

Optimized for the cloud in the first place…not after!

Different cost structures

Much smaller storage footprint, much less data-transfer

How Does Data Virtualization Enhance

this?

Traditional copy data management techniques

Developed without concerns about infrastructure chargeback

This corresponds to higher cost.

IaaS vendors monitor storage and data transfers

Help meet SLAs, garner profits

It’s not just the data that exists in the end, so transformations can

equal big money for cloud vendors.

Different cost structures

Know Thy Enemy…

• Tune SQL and Apps to perform efficiently as possible- natural

life of database is growth, (in processes, resources, etc.) before

migrating.

• The less network latency, the better- network tracing to

eliminate database blame is important.

• Many of the same tools and data provides value- DMVs provide

data internally to SQL Server.

• Look at management tools such as Cloudmonix, (formerly

AzureWatch) AppDynamics, Dynatrace, Zabbix or Logic Monitor.

For Non-Production Systems…

Change the way you’ve always performed tasks.

Performing common tasks the same way as previously might end up costing more.

Secure Data

All IaaS alternatives promote encryption for data in-flight and for data at-rest, but encryption may not be the right answer…

Secondary Considerations

All IaaS solutions provide encryption in-flight and encryption at-rest

But encryption doesn’t protect data as much as it needs to be .

Europe already requires data masking, not just data encryption for any

confindential data, (GDPR):

http://ec.europa.eu/justice/data-protection/article-

29/documentation/opinion-

recommendation/files/2014/wp216_en.pdf

Confidential data

Encryption is reversible data obfuscation, which is very different from

masking data.

• Data masking is non-reversible.

It solves the issue at the data level.

Is authentication and authorization in non-production in compliance with

security goals?

All organizations will soon need to review if critical data in non-

production environments be accessible to developers, testers and

users.

Confidential data

Masking personally-identifiable, (PII, HIPPA, PCI, etc.) information

renders it useless from a security standpoint

Resolves both the technical and personal responsibility issue.

The data can be masked before it moves to non-production,

removing unnecessary risk.

Why Masking is Part of the Answer

Confidential data

Production

Non-Production

Exposure

Masking- All the Data

SQL Server Validated Sync Environment +

applications/flat files

Delphix Masking Engine

Delphix Virtualization Engine2TB storage

SQL Server TargetNo database storage

Simple Masking and then to the Cloud

Delphix Virtualization Engine2 TB storage

SQL Server TargetStorage Only for

Masked Data

Delphix Virtualization Engine2TB storage

Delphix Masking Engine

SQL Server Validated Sync Environment

SQL Server TargetStorage Only for

Masked Data

Confidential data

Production

Non-

Production

Exposure

Encryption

Masking

Solution

Masked Data Pods Migrated to Cloud

Delphix Virtualization Engine4 TB storage

SQL Server Validated Sync Environment

SQL Server TargetStorage Only for Masked

Data

SQL Server TargetStorage Only for Masked

Data

Delphix Virtualization Engine4 TB storage

SQL Server TargetStorage Only for Masked

Data

Delphix Masking Engine

Virtualization Makes Management of non-

production environments simple.

Security with masking and encryption is best

Cloud migrations are more successful when

virtualized and planned accordingly.

Patching and Maintenance can be done with less

effort and resources.

Want to try it out, download the Delphix Azure Trial! https://www.delphix.com/products/free-trial-

request

Twittter: @DBAKevlar

Linked in: http://linkedin.com/in/kellynpotvin

Blog: http://dbakevlar.com

59

Delphix with SQL Server- the Basics

https://docs.delphix.com/docs/delphix-administration/sql-server-environments-and-data-

sources/managing-sql-server-environments/overview-of-setting-up-sql-server-

environments

Delphix Upgrade Workflow: https://community.delphix.com/delphix/topics/tip-of-the-day-

upgrading-a-sql-server-dsource

Upgrading the Dsource after an Upgrade: https://docs.delphix.com/docs/delphix-

administration/sql-server-environments-and-data-sources/virtualizing-databases-using-

delphix-with-sql-server/managing-sql-server-dsources/additional-dsource-

topics/upgrading-a-dsource-after-a-sql-server-upgrade

Delphix in the Cloud

https://www.delphix.com/solutions/cloud-migration-virtual-data

References and Tips