Lynn Langit

lynnl@microsoft.com

Moving Your Data to the Cloud

Migrating Applications to SQL Azure

Sr. Developer Evangelisthttp://blogs.msdn.com/SoCalDevGal

April 2011

Migrating Applications to SQL Azure

OverviewSQL Azure – now and NEW

App Migration Data Access Tier Considerations

Database MigrationMigrating to SQL Azure from MySqlMoving Schema and Data into SQL AzureTransferring Large Data into SQL Azure

Future

Windows Azure PlatformGENERAL PURPOSE PROGRAMMING

LANGUAGES

Compute

Storage

Management

Marketplace

App FabricCDN

Windows Azure Platform

Relational

databaseData Sync

When Migrating Applications

Data Access APIs SupportSecurity ModelConnections & SizingAdmin and Programmability Surface SupportCollation Behavior

Data Access APIs

Supported APIs

Connection StringADO.Net - Encrypt=True and add @servername to User IDODBC - Encrypt=yes and add @servername to Uid

Supported Unsupported

ADO.Net .Net 3.5 SP1 and 4.0 ODBC - SNAC 10Entity Framework .Net 3.5 SP1 and 4.0SQL Server 2008 Driver for PHP v1.1

OleDB

Data Access APIs cont.

Protocols

Authentication Mode

Supported Unsupported

TCP/IP over port 1433 Dynamic portsNamed pipesShared memory

Supported Unsupported

SQL Auth Windows Auth

Connectivity & Sizing

Robust Connection Management through Retry Logic

FailoverHW Failures or Upgrades

Resource Management OperationsThrottling and Load Balancing

ALTER DATABASE (T-SQL)Changes to Database Name Changes to MAXSIZE/EDITION

ALTER DATABASE [TestDB] MODIFY (MAXSIZE=50GB)

GO

Security ManagementManage Logins - Authentication - SQL Server (only)

CREATE LOGIN [testlogin] WITH PASSWORD='password‘

Manage UserCREATE USER testuser FROM LOGIN testlogin;

Manage Server RolesEXEC sp_addrolemember ‘dbmanager', 'testuser'; [securityadmin role]EXEC sp_addrolemember ‘loginmanager', 'testuser'; [dbcreator]

Granting Database Access to a LoginCreate Login and Database in MasterCreate User and Grant Role in User Database

Viewssys.sql_logins. sys.databases, sys.database_principal

Firewall Management

Portal AccessProgrammatic Access

System Views and Stored Procedures - hereView: sys.firewall_rulesStored Procedure: sp_set_firewall_ruleStored Procedure: sp_delete_firewall_rule

API Implementation Example- here

CollationsDefault Collation - Set to SQL_LATIN1_GENERAL_CP1_CI_AS

Metadata, TempDb and Default for User Data Object Names (tables, sprocs etc), String variables and Columns when no collation is specified. Use column and/or expression level collation when needed

CREATE TABLE [dbo].[ErrorLog]([ErrorLogID] [int] IDENTITY(1,1) NOT NULL,[ErrorTime] [datetime] NOT NULL,[UserName] [nvarchar(4000)] COLLATE Turkish_100_CI_AI NOT NULL,[ErrorNumber] [int] NOT NULL,[ErrorMessage] [nvarchar](4000) COLLATE Japanese_CI_AS NOT NULL,

CONSTRAINT [PK_ErrorLog_ErrorLogID] PRIMARY KEY CLUSTERED ([ErrorLogID] ASC) )declare @str nvarchar(128) COLLATE Greek_CS_AI

Feature Parity

Admin Surface – no physical administrationYou have a master database You do NOT have server level constructs (sp_configure, endpoints, DBCC commands, server level DMVs and System Views…)

Programmability SurfaceMost RDMS features availablePartially available - USE, XML processing, deprecated T-SQL etc. – list hereSome Features are NOT available today

Full-text Search, Remote Data Access and Linked Servers, Distributed Transactions, Change Tracking, Service Broker etc.List is available here

Migrating ApplicationsApplication Data Access Tier Just Works Need to

Investigate

Data Access APIs Support ADO.Net, ODBC, EF and PHP

OleDB

Security Model SQL Auth Windows Auth

Connection Management Apps with transaction retry logic, 3-tiered, stateless or partially stateful apps

Legacy Apps2 tiered apps with online connections

Admin and Programmability Surface Support

Tables, Views, Sprocs, Functions, Triggers, built-in data types, etc.

Server level options, FTS, Service Broker, Distributed Transactions

Collation Behavior Apps explicit with collation options

Non-English apps depending on db/server collation defaults

Database Migration

Moving Schema and / or DataSource Database types

SQL ServerMySQLAccessOther RDMS

Data Migration Tools Overview

WizardsSQL Azure Migration Wizard (CodePlex) – hereMigration Assistant for Access or for MySQL – here or hereImport/Export WizardGenerate Script Wizard

Copy Database commandCREATE DATABASE DB2A AS COPY OF Server1.DB1A

Other Tools SSIS BCPDataSync (CTP)OData (CTP)

From SQL Server to SQL AzureGUI Tools

SSMS for R2 > Generate Script WizardSSMS for R2 > Import / Export WizardSSIS for R2SQL Azure Migration Wizard

Command-line ToolsT-SQL script & bcpCopy Database commandImport/Export Wizard for SQL Azure (command -> GUI)3rd Party tools

SSMS– Generate Script Wizard

Migration of schema and/or data w/ fine grain controlPros - Native support for SQL Azure Schema options: “Engine Type = SQL Azure”

Ensures correct options and settings are applied for the TSQL script generation.Requires explicit action on unsupported objects.

Cons - Verbose – INSERT Statements instead of raw data

Data is scripted with a fixed 100 row batch size.Edit “GO” statements between small batches

Round-trip EfficiencyUse “SET NOCOUNT ON”

Remove unsupported features, such as…

Migrating By Hand

USE DATABASE ON PRIMARY

(Filegroups) Extended Properties NOT FOR REPLICATION Various Index options (WITH) Windows logins Unsupported XML features Remove DEFAULT NEWSEQUENTIALID() Encrypted Stored Procedures No Heap tables (all tables must have clustered indices) UDTs

Generate Script Wizard - DEMO

Migrating Databases to SQL Azure

SQL Azure Migration Wizard

http://sqlazuremw.codeplex.com

What Could Go Wrong?

Success!

Migration Assist for MySql / Access (CTP)

Scenario - Auto porting of schema, database code and data from MySql and Access to SQL AzureSQL Server Migration Assistant for MySql and Access

Supports MySQL 4.1 and upSupport Access v 97 and upSQL Server versions supported (all editions)

SQL Azure, SQL Server 2005, SQL Server 2008 and 2008 R2

MySQL to SQL Azure - DEMO

SQL Server Migration Assistant Version for MySQLVersion for Access

MySQL Migration Tool

Access to SQL Azure Migration Wizard - DEMO

SQL Azure - Import/Export

DAC v2.0 adds two new DAC services: import and exportImport and export services let you deploy & extract both schema & data from a single file identified with the “.bacpac” extension

DAC (Data-tier Application) PackageSingle unit for authoring, deploying, and managing data-tier objects (schema)

Development Lifecycle (VS 2010)

Creating / Editing DACsSchema and DB Code DevelopmentCode Analyses, Deployment Policy SettingsSchema Comparison and more…

Management Lifecycle (SSMS 2008 R2)

Extracting DACs (as files)Managing DACs (deploy or upgrade [coming later this year for SQL Azure)Registering existing database as DACs

If ‘regular’ into MSDBIf ‘Azure’ into masterIf using UCP into UCP database

DACPAC – DEMO in SSMS

In SSMS• Extract• Register• Monitor

w/UCPs

In VS 2010• Import• Create

DACPAC – DEMO in VS

Transferring Large Data to SQL Azure

Best Practices for efficient Data movement to SQL Azure

Use the right tools is key BCPSSIS

Network performance considerationsLatency vs Bandwidth

BCP & Bulk Copy APIs for Large Data Transfer

Optimize Databases for Data ImportDisable/Delay creation of none Clustered IndexesOrder Data on the Clustered IndexDisable Triggers and Constraint checking

-N Native mode so no data type conversion needed.

-c Character mode for interoperability

-b batchsize will let you control transaction size-h”TABLOCK, ORDER(…)” optimize writes

Import/Export Wizard – for data transfer

Scenario - Simplified wizard for migrating schema and data through SSISPros

Great performance out of the boxAllows ‘save as package’ for full control

SSIS Data Flow Task (Large Data Xfer)Diverse Source and destinations -

To/From Flat Files, ADO.Net, OleDBFully programmable flow

Loops, Sorts, Conditional operators, XML/WebServices Processing etcVS Debugging support with data viewers, watches and conditional breakpoints

Optimize Databases for Data ImportDisable/Delay creation of none Clustered IndexesOrder Data on the Clustered IndexDisable Triggers and Constraint checkingBatch Size: Adjust the transaction sizeBuffer and Blob temp storage area; spread over to multiple drives for large data transfersParallelization – based on execution trees, task will auto parallelize

SSIS package - DEMO

Use supported features only, such as bcp, but not BULK INSERT, etc…

Parallelization & Network Perf

Latency & Bandwidth for high performance import & export

Latency is about laws of physicsLong distances means high latencyParallelization can help mitigate some latency

Bandwidth is key! Once you reach the bandwidth limit parallelization won’t help

Datacenter Locations

BCP Performance – test set up exampleCREATE TABLE [dbo].[table1](

[c01] [int] NOT NULL, [c02] [int] NOT NULL, [c03] [smallint] NOT NULL, [c04] [int] NOT NULL, [c05] [smallint] NOT NULL, [c06] [smallint] NOT NULL, [c07] [char](50) NOT NULL, [c08] [char](24) NOT NULL, … [c17] [char](24) NOT NULL, PRIMARY KEY CLUSTERED ([c01] ASC,[c02] ASC))

CREATE NONCLUSTERED INDEX [index1] ON [dbo].[table1] ([c07] ASC)

Data Distribution:Row Size: 600BytesRow Count: 100KPages: ~7700Total Size: ~60MBEven Distribution on PkeyConstraints and Indexes

Compound Pkey (int+int) None-clustered key on char(50)No triggers or relationships

BCP Upload Perf - Rows/Sec

Low Bandwidth, Medium Latency - 114ms

Low Bandwidth, Very High Latency - 289ms

High Bandwidth, High Latency - 169ms

High Bandwidth, Medium Latency - 54ms

0 1000 2000 3000 4000 5000 6000 7000 8000 9000

Parallel Upload Batch Size 1K (sum rows/sec)

Native Sorted - Batch 10K (rows/sec)

Native - Batch 10K (rows/sec)

Native - Batch 100 (rows/sec)

BCP Download Perf – Rows/sec

Medium Bandwidth, Medium Latency - 114ms

Medium Bandwidth, Very High Latency - 289ms

High Bandwidth, High Latency - 169ms

High Bandwidth, Medium Latency - 54ms

0 2000 4000 6000 8000 10000 12000 14000 16000

4x Native Sorted - Batch Size 1K (sum rows/sec)

1x Native Sorted - Batch 10K (rows/sec)

Guidance on Data Transfer to SQL Azure

Split data transfer into multiple concurrent streamsTune BCP batch size and network packet size option Add non clustered indexes after loading data to SQL Azure.

Two additional indexes created before loading the data increased the final database size by ~50% and increased the time to load the same data by ~170%.If, while building large indexes, you see a throttling-related error message, retry using the online option.

COPY Database (same or different servers)CREATE DATABASE destination_database_name AS COPY OF [source_server_name.]source_database_nameCREATE DATABASE Database2A AS COPY OF Server1.Database1A

Check via DMVProgresssys.dm_database_copies

Database COPY

SQL Azure Data Sync

Elastic ScaleService scales as resources requirements grow

No-Code Sync ConfigurationEasily define data to be synchronized

Schedule SyncChoose how often data is synchronized

Conflict HandlingHandle issues where same data is changed in multiple locations

Logging and MonitoringAdministration capabilities for tracking data and monitoring potential issues

CTP 1

CTP 2

Data Sync

Table Order in Sync

Tracking Objects Created

• Tracking Tables• On Source / on

Destination• Triggers• Stored Procedures

SQL Azure Data Sync CTP3

• New UX integrated with the Windows Azure management portal.

• Ability to select a subset of columns to sync as well as to specify a filter so that only a subset of rows are synced.

• Ability to make certain schema changes to the database being synced without having to re-initialize the sync group and re-sync the data.

• Conflict resolution policy can be specified.• General usability enhancements.

Monitoring Tasks• Space Used• Monitor space used via

‘sys.dm_db_partition_stats’• Trap ‘out of space exception’ – SQL

Error 40544• Synchronized Copies• Database copy• SQL Azure Synchronization Provider• Database backup / restore (and point-

in-time restore) on 2011 Q2 Roadmap

Data Migration Tools Summary

WizardsSQL Azure Migration Wizard (CodePlex): http://sqlazuremw.codeplex.com/

Migration Assistant for Access or for MySQL:http://www.microsoft.com/downloads/en/details.aspx?FamilyID=5abe098d-c7e1-46c6-994a-09a2856eef0b&displaylang=en

Import/Export WizardGenerate Script Wizard

Copy Database commandCREATE DATABASE DB2A AS COPY OF Server1.DB1A

Other Tools SSIS BCPDataSync (CTP)OData (CTP)Import / Export Tool(coming spring 2011)

Database Migration Tools

Tools SSMA

Generate Script Wizard DACs SSIS BCP

MySql and Access to SQL Azure

√ (Schema

and Data)

√ (Data Only)

Move SQL Schema √ √ √

Move Data √ √ √

Move Large Data √ √

Try SQL Azure for FREE

www.sqlazure.com/getstarted

Activate your free trial!

What’s Your Idea?

www.sqlazure.com/communityCheck out tutorials and hear from MVPs…

Track Resources

Team Blog – http://blogs.msdn.com/sqlazure

MSDN Dev Center – http://msdn.microsoft.com/azure/sqlazure

Follow us on Twitter - @sqlazure

Windows Azure Platform Training Kit - http://bit.ly/dgvMv0

SQL Azure Labs – http://www.sqlazurelabs.com

TeachingKidsProgramming.org

• Do a Recipe Teach a Kid (Ages 10 ++)• Microsoft SmallBasic Free Courseware (recipes)

© 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft

cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.