designing and developing your database for application...
TRANSCRIPT
1
Designing and Developing your Database for Application Availability Presented to the Melbourne CBD SQL PASS Chapter. Saxons - Level 8, 500 Collins Street. Melbourne, Australia 12:30 – 14:00, July 26th 2010
Charley Hanania, QS2 AG
B.Sc (Computing), MCP, MCDBA, MCITP, MCTS, MCT, Microsoft MVP: SQL Server
Senior Database Specialist
My Background
• Now: • Microsoft MVP: SQL Server • Database Consultant (again, and very happy) at QS2 AG
• Formerly: • Production Product Owner of MS SQL Server Platform at UBS Investment Bank • Technical Team Lead
• ITIL v3 Certified
• SQL Server Certified since 1998 • On SQL Server since 1995 • Version 4 on OS/2
• IT Professional since 1992
• PASS • Chapter Leader – Switzerland • Regional Mentor – Europe • European PASS Conference Lead • Event Speaker
2
26 July, 2010
De
sign
ing
& D
eve
lop
ing
Dat
abas
es
for
Ap
plic
atio
n A
vaila
bili
ty
Contact Info
• Email: [email protected]
• Website: http://www.sqlpass.ch <- Slides here
• Twitter: http://www.twitter.com/CharleyHanania
• Blog: http://blogs.mssqltips.com/blogs/charleyhanania
• Linked-in: http://www.linkedin.com/in/charleyhanania
3
26 July, 2010
De
sign
ing
& D
eve
lop
ing
Dat
abas
es
for
Ap
plic
atio
n A
vaila
bili
ty
Agenda
• General Barriers to Application Availability
• Standard High Availability Methodology
• General Standard - Clustering
• One up on the Standard – Database Mirroring
• Decreasing component start-up time
• Instant File Initialisation
• Partitioning your Database Infrastructure
• Eureka! Partitioning for Availability (PfA)
• Basic PfA Methodology
• Code Samples
• Questions
De
sign
ing
& D
eve
lop
ing
Dat
abas
es
for
Ap
plic
atio
n A
vaila
bili
ty
4
26 July, 2010
General Barriers to Availability
De
sign
ing
& D
eve
lop
ing
Dat
abas
es
for
Ap
plic
atio
n A
vaila
bili
ty
6
26 July, 2010
Simplistic Network Topology View:
Slightly more realistic view:
And that’s just scratching the surface of the Peripherals! Think about the layered complexity of the supporting software and drivers…!
Environmental Complexity
Standard HA Methodology
• Keep it up for as long as possible.
• Increase redundancy
• Decrease Single Points of Failure
• Bring it back as quickly as possible
• Decrease the dependency sequence
• Decrease component start-up time
De
sign
ing
& D
eve
lop
ing
Dat
abas
es
for
Ap
plic
atio
n A
vaila
bili
ty
7
26 July, 2010
General Standard - Clustering
• Keep it up for as long as possible.
• Increase redundancy • Additional Servers waiting in readiness
• Increased redundancy = increased cost
• Increased redundancy = decreased utilisation
• Decrease Single Points of Failure • Virtual IP Addresses
• Virtual Network Names
• Additional Network Cards etc
• Storage
• Bring it back as quickly as possible
• Decrease the dependency sequence • Server is Started
• SQL Instance is an exclusive Cluster Resource
• Decrease component start-up time • Instant File Initialisation
• App Database isn’t useable until Primary Data File is opened and recovered
26 July, 2010
De
sign
ing
& D
eve
lop
ing
Dat
abas
es
for
Ap
plic
atio
n A
vaila
bili
ty
8
Primary Standby
Database Mirroring
• We could discuss the architecture, benefits and uses of DB Mirroring, but I’m told you’re mainly architects and developers, and the doors here aren’t locked…
• If you’d like to discuss DB Mirroring, we can chat after the session…
De
sign
ing
& D
eve
lop
ing
Dat
abas
es
for
Ap
plic
atio
n A
vaila
bili
ty
9
26 July, 2010
Decreasing Component Startup Time
(Another DBA point, but one that is Golden and not to be missed!)
• Data and log files are first initialized by being filled and with zeros when created or restored.
• When SQL Server Starts, the tempdb database is recreated. • Application databases do not come online until tempdb has.
• If large, then startup times can be quite long!
• Action: • Ensure that instant file initialisation is set (Windows:
SE_MANAGE_VOLUME_NAME right)
• The group that the SQL Server Service Account is in needs to be added to the Perform Volume Maintenance Tasks security policy, in the “Local Security Policy” Tool.
• I’ve seen this reduce SQL Server Startup from 8 minutes to 1 second (max)
De
sign
ing
& D
eve
lop
ing
Dat
abas
es
for
Ap
plic
atio
n A
vaila
bili
ty
10
26 July, 2010
INSTANT FILE INITIALISATION
Decreasing Component Startup Time
• Msdn lists the following (summarised) • Hardware Partitioning
• Multiprocessors that enable multiple threads of operations.
• RAID (redundant array of independent disks) devices • A table striped across multiple drives can typically be scanned faster than
the same table stored on a single drive.
• Horizontal Partitioning • Divide tables into multiple tables containing the same number of
columns, but fewer rows.
• Partitioning data horizontally based on age and use is common.
• Vertical Partitioning • Divide tables into multiple tables containing fewer columns through
normalization and row splitting.
De
sign
ing
& D
eve
lop
ing
Dat
abas
es
for
Ap
plic
atio
n A
vaila
bili
ty
11
26 July, 2010
PARTITIONING DATABASE COMPONENTS
Additional Partitioning Options
• Logical Schema Partitioning
• Used mainly for securing and grouping objects by department, role, application etc…
• Physical Partitioning
• Of Files and Filegroups
• Used mainly for performance
• Files placed on multiple different data volumes
• Multiple files created for round-robin writes and extent allocations
De
sign
ing
& D
eve
lop
ing
Dat
abas
es
for
Ap
plic
atio
n A
vaila
bili
ty
12
26 July, 2010
Eureka! Partitioning for Availability
• Each database has an area similar to a disk’s boot record • For a database this resides in the Primary data file
• If this file is corrupt or unavailable, the database will not be loaded and opened regardless of other data files
• A good practice is to not put any objects in this data file, and to put all your tables and objects into secondary data files.
• Within the DB you can distribute objects into files to achieve one or more of the following • Enable an application to continue working when other data files
are missing or corrupt by grouping needed objects together
• Restore the most critical files for your application system to start up first, allowing partial functionality of your application to continue without issue.
De
sign
ing
& D
eve
lop
ing
Dat
abas
es
for
Ap
plic
atio
n A
vaila
bili
ty
13
26 July, 2010
14
De
sign
ing
& D
eve
lop
ing
Dat
abas
es
for
Ap
plic
atio
n A
vaila
bili
ty
26 July, 2010
Open Discussion Which ways could we split the adventureworks db,
focussed on client facing / downstream application functionality, criticality and startup sequence for Availability?
Adventureworks Schema
PfA Methodology
• Conceptual extension from Logical Schema Partitioning and Physical File Partitioning:
1. Don’t put user defined objects in the Primary FileGroup
i. Allows the Primary Filegroup to load faster
2. Schema Separate user defined objects
i. For clarity of purpose and operational use
ii. Try not to use dbo schema (good design/operational practice)
iii. DBA’s will schematize objects in a similar fashion to what was in the Adventureworks sample. For PfA you should look to map your schema’s around your application’s core functionality and use cases and criticality
3. Create Filegroups for each Schema
i. Multiple if you want to partition historical data horizontally etc
ii. Name them in a fashion that allows easy understanding of what is in them, or how they should be used.
De
sign
ing
& D
eve
lop
ing
Dat
abas
es
for
Ap
plic
atio
n A
vaila
bili
ty
16
26 July, 2010
PfA Methodology(2)
• Conceptual extension from Logical Schema Partitioning and Physical File Partitioning: 4. Design your application to modularise its data access needs
i. This is critical to segmenting your application to specific data files/filegroups.
ii. This includes stored procedure and function internals, use try/catch within sp’s for example to decide whether rollback or other functionality is needed when a filegroup is offline.
iii. The key flows could be to either contain data access within the necessary filegroups, or
determine whether a function can be used to catch that an object is in an offline filegroup and skip it / do something else / roll back, or
Catch “Msg 8653, Level 16, State 1, Line 3”* and do as per point above.
5. Document application’s modules, with emphasis on criticality and data dependencies.
i. Allows stepped sequencing of apps by criticality during start up
De
sign
ing
& D
eve
lop
ing
Dat
abas
es
for
Ap
plic
atio
n A
vaila
bili
ty
17
26 July, 2010
*[Tambs] While the database is in a state of partial availability, the filegroups that remain online can support queries. However, queries that depend on data that resides in filegroups that are offline return error 8653: Msg 8653, Level 16, State 1, Line 3
The query processor is unable to produce a plan for the table or view 'X' because
the table resides in a filegroup which is not online.
18
De
sign
ing
& D
eve
lop
ing
Dat
abas
es
for
Ap
plic
atio
n A
vaila
bili
ty
26 July, 2010 Adding Files and File Groups to a Database
USE [master]
GO
CREATE DATABASE [PfA]
ON PRIMARY
(
NAME = N'PfA_Primary'
,FILENAME = N'E:\MSSQL10.SQL2008INST01\MSSQL\Data\PfA_Primary.mdf'
,SIZE = 400MB
),
FILEGROUP [PfA_FG1]
(
NAME = N'PfA_F1_FG1'
,FILENAME = N'E:\MSSQL10.SQL2008INST01\MSSQL\Data\PfA_F1_FG1.ndf'
,SIZE = 300MB
),
(
NAME = N'PfA_F2_FG1'
,FILENAME = N'E:\MSSQL10.SQL2008INST01\MSSQL\Data\PfA_F2_FG1.ndf'
,SIZE = 300MB
),
FILEGROUP [PfA_FG2]
(
NAME = N'PfA_F1_FG2'
,FILENAME = N'E:\MSSQL10.SQL2008INST01\MSSQL\Data\PfA_F1_FG2.ndf'
,SIZE = 300MB
),
FILEGROUP [PfA_FG3]
(
NAME = N'PfA_F1_FG3'
,FILENAME = N'E:\MSSQL10.SQL2008INST01\MSSQL\Data\PfA_F1_FG3.ndf'
,SIZE = 300MB
)
LOG ON
(
NAME = N'PfA_log'
,FILENAME = N'F:\MSSQL10.SQL2008INST01\MSSQL\Logs\PfA_PfA_log.LDF'
,SIZE = 100MB
)
GO USE [PfA]; GO SELECT * FROM sys.database_files; SELECT * FROM sys.filegroups;
19
De
sign
ing
& D
eve
lop
ing
Dat
abas
es
for
Ap
plic
atio
n A
vaila
bili
ty
26 July, 2010 Create and Allocate Objects to Schemas
USE [PfA];
GO
CREATE SCHEMA [Sales];
GO
CREATE SCHEMA [Production];
GO
CREATE SCHEMA [Person];
GO
-- moving tables from dbo schema to
-- [Production] and [Sales] schemas
ALTER SCHEMA [Sales]
TRANSFER [dbo].[Customer];
-- Note that triggers move as well...
GO
ALTER SCHEMA [Production]
TRANSFER [dbo].[BillOfMaterials];
GO
ALTER SCHEMA [Person]
TRANSFER [dbo].[ContactInsert];
GO
ALTER SCHEMA [Person]
TRANSFER [dbo].[Contact];
GO
-- change stored proc definitions to focus on right schemas
ALTER PROC [dbo].[ContactInsert]
(
@VbiContactID BIGINT
,@VvcSalutation VARCHAR(4) = NULL
,@VnvcForename NVARCHAR(100),@VnvcMiddlename NVARCHAR(100) = NULL
,@VnvcSurname NVARCHAR(100),@VvcJobTitle NVARCHAR(100) = NULL
,@VvcCompany NVARCHAR(100) = NULL,@VnvcAddressLine1 NVARCHAR(100)
,@VnvcAddressLine2 NVARCHAR(100) = NULL
,@VnvcAddressLine3 NVARCHAR(100) = NULL
,@VvcPostCode VARCHAR(12),@VnvcCity NVARCHAR(50)
,@VnvcState NVARCHAR(50) = NULL,@VinCountryID INT
,@VvcEmailAddress VARCHAR(100),@VvcBusinessPhone VARCHAR(25)
,@VvcBusExt VARCHAR(10) = NULL,@VvcFacsimile VARCHAR(25) = NULL
,@VvcMobilePhone VARCHAR(25) = NULL,@VvcPassword VARCHAR(20)
)
AS
INSERT INTO [Person].[Contact]
([ContactID],[Salutation]
,[Forename],[Middlename]
,[Surname],[JobTitle]
,[Company],[AddressLine1]
,[AddressLine2],[AddressLine3]
,[PostCode],[City]
,[State],[CountryID]
,[EmailAddress],[BusinessPhone]
,[BusExt],[Facsimile]
,[MobilePhone],[Password])
VALUES
(@VbiContactID,@VvcSalutation
,@VnvcForename,@VnvcMiddlename
,@VnvcSurname,@VvcJobTitle
,@VvcCompany,@VnvcAddressLine1
,@VnvcAddressLine2,@VnvcAddressLine3
,@VvcPostCode,@VnvcCity
,@VnvcState,@VinCountryID
,@VvcEmailAddress,@VvcBusinessPhone
,@VvcBusExt,@VvcFacsimile
,@VvcMobilePhone,@VvcPassword);
GO
* Use Synonyms to help the transition from dbo native schema to proper schema usage
20
De
sign
ing
& D
eve
lop
ing
Dat
abas
es
for
Ap
plic
atio
n A
vaila
bili
ty
26 July, 2010 Allocate an Object to a Filegroup
CREATE TABLE [Sales].[SalesInvoice]
(
[SalesInvoiceID] INT IDENTITY(1,1) PRIMARY KEY CLUSTERED
,[Amount] MONEY
,[CustomerID] INT
,[DateInvoiced] DATETIME
,[DatePaid] DATETIME
) ON [PfA_FG1];
GO
• Creating tables on a filegroup is easy as per the code sample. • Moving existing tables can be a little more complicated:
• Use Create index (clustered) with Drop Existing • Rebuild indexes etc to move them
• It may take many iterations to get the sequencing and code here right, but its worth it in the long run.
21
De
sign
ing
& D
eve
lop
ing
Dat
abas
es
for
Ap
plic
atio
n A
vaila
bili
ty
26 July, 2010 Retry Logic (1)
ADO.NET Code Example (http://technet.microsoft.com/en-us/library/cc917713.aspx#ECAA)
This code example uses ADO.NET 2.0 (SqlClient). The following parameters are used in the example.
strConn is the database connection string that includes the failover partner. strCmd is the command, such as Transact-SQL or a stored procedure, to execute. iRetryInterval is the number of seconds to wait (sleep) between retries. iMaxRetries is the number of times to retry the command before failing.
Example connection string ( strConn ) Data Source=SQLA\INST1;Failover Partner=SQLB\INST1;
Initial Catalog=DBMTest;Integrated Security=True
using System.Data;
using System.Data.SqlClient;
using System.Threading;
bool ExecuteSQLWithRetry_NoResults(string strConn,
string strCmd,
int iRetryInterval,
int iMaxRetries)
{
// Step 1: Enter the retry loop in case an error occurs
for (int iRetryCount = 0; iRetryCount < iMaxRetries; iRetryCount++)
{
try
{
// Step 2: Open the database connection
conn = new SqlConnection(strConn);
conn.Open();
// Step 3: Execute the command
if (null != conn && ConnectionState.Open == conn.State)
{
cmd = new SqlCommand(strCmd, conn);
cmd.ExecuteNonQuery();
// Step 4: If there were no errors, clean-up & return success
return true;
}
}
catch (Exception ex)
{
// Step 5: Catch Error on Open Connection or Execute Command
// Error Handling to be added here: ex
}
• I expect that the code here will only need slight Adjustments for use with Clustering
• Connection string would use virtual name • Timeout retry delay would cater for longer
startup/failover
22
De
sign
ing
& D
eve
lop
ing
Dat
abas
es
for
Ap
plic
atio
n A
vaila
bili
ty
26 July, 2010 Retry Logic (2)
finally
{ // Step 6: Clean up the Command and database Connection objects
try
{
// Clean Command object
if (null != cmd)
cmd.Dispose();
cmd = null;
// Clean up Connection object
if (null != conn && ConnectionState.Closed != conn.State)
conn.Close();
conn = null;
}
catch (Exception ex)
{
// Error handling to be added here
}
}
// Step 7: If the maximum number of retries not exceeded
if (iRetryCount < iMaxRetries)
{
// Step 7a: Sleep and continue (convert to milliseconds)
Thread.Sleep(iRetryInterval * 1000);
}
} // end for loop: iRetryCount < iMaxRetries
// Step 7b: If the max number of retries exceeded, return failure
return false;
}
Continued…
23
De
sign
ing
& D
eve
lop
ing
Dat
abas
es
for
Ap
plic
atio
n A
vaila
bili
ty
26 July, 2010 Query Returning Files, Filegroups and Sizes
SELECT
s.data_space_id as Id
,g.name as Filegroup_name
,s.name as Logical_name
,physical_name
,g.is_default AS [IsDefault]
,g.is_read_only AS [ReadOnly]
,CAST(
ISNULL
(
(
SELECT
SUM(gs.size)*CONVERT(float,8)
FROM
sys.database_files gs
WHERE
gs.data_space_id = g.data_space_id
), 0
) AS float
) AS [Size]
FROM
sys.filegroups AS g
INNER JOIN
sys.master_files AS s
ON (s.data_space_id = g.data_space_id)
WHERE
s.type = 0
AND
s.database_id = db_id()
AND
s.drop_lsn IS NULL; From “Partial Database Availability” (Danny Tambs, May 2007)
Links and Resources
• AdventureWorks Database Diagram • http://www.microsoft.com/downloads/details.aspx?FamilyID=0f6e0bcf-a1b5-4760-8d79-
67970f93d5ff&DisplayLang=en
• Whitepaper: “Partial Database Availability” (Danny Tambs, May 2007)
• http://technet.microsoft.com
• Whitepaper: “Partitioned Table and Index Strategies Using SQL Server 2008” (Ron Talmage, March 2009)
• http://technet.microsoft.com
De
sign
ing
& D
eve
lop
ing
Dat
abas
es
for
Ap
plic
atio
n A
vaila
bili
ty
24
26 July, 2010
Questions?
De
sign
ing
& D
eve
lop
ing
Dat
abas
es
for
Ap
plic
atio
n A
vaila
bili
ty
25
26 July, 2010