table partitioning in sql server: a magic solution for better performance? (pragmatic works)
TRANSCRIPT
Table Partitioning in SQL ServerA Magic Solution for Better Performance?
Cathrine Wilhelmsen
@cathrinew
cathrinewilhelmsen.netData Warehouse Architect
Business Intelligence Developer
You?New to table partitioning
"Everything is slow"
Today?Basic Introduction
What, Why & How
A Magic Solution for Better Performance?
Spoiler Alert!
A Magic Solution for Better Performance?
Implementing table partitioning is not a trivial task
and can actually cause worse performance...
A Magic Solution for Better Performance?
...but don't worry, I made plenty of mistakes
so you can avoid them
Enterprise Edition only
What?Partition Key
Partition Function
Partition SchemeWhy?
Backup & Restore
Maintenance
Load & ArchiveHow?
Partition Elimination
Switch, Split & Merge
Sliding Windows
Table Partitioning Basics
What is a partitioned table?
2012-01-01 ... ...
2012-12-31 ... ...
2013-01-01 ... ...
2013-12-31 ... ...
2014-01-01 ... ...
2014-12-31 ... ...
2015-01-01 ... ...
2015-12-31 ... ...
Data is physically stored in groups of rows called partitions
Each partition can be accessed and maintained separately
Partitioning is not visible to users, it behaves like one logical table
Partition Key
2012-01-01 ... ...
2012-12-31 ... ...
2013-01-01 ... ...
2013-12-31 ... ...
2014-01-01 ... ...
2014-12-31 ... ...
2015-01-01 ... ...
2015-12-31 ... ...
Data is partitioned based on a single column, the Partition Key
The Partition Key should always be used as a filter in queries
This ensures Partition Elimination:only relevant partitions are accessed
Partition Function
2012-01-01 ... ...
2012-12-31 ... ...
2013-01-01 ... ...
2013-12-31 ... ...
2014-01-01 ... ...
2014-12-31 ... ...
2015-01-01 ... ...
2015-12-31 ... ...
The Partition Function defines how to partition the data
It specifies boundary values, the points between two partitions
It specifies if the boundary value belongs to its left (upper) partition or its right (lower) partition
Partition Function: Range Left and Range Right
2012-01-01 ... ...
2012-12-31 ... ...
2013-01-01 ... ...
2013-12-31 ... ...
2014-01-01 ... ...
2014-12-31 ... ...
2015-01-01 ... ...
2015-12-31 ... ...
2012-01-01 ... ...
2012-12-31 ... ...
2013-01-01 ... ...
2013-12-31 ... ...
2014-01-01 ... ...
2014-12-31 ... ...
2015-01-01 ... ...
2015-12-31 ... ...
Partition Function: Range Left and Range Right
Range Left means the boundary value is the last value in the left partition
CREATE PARTITION FUNCTIONpfLeft (INT) AS RANGE LEFTFOR VALUES (20,30,40);
Range Right means the boundary value is the first value in the right partition
CREATE PARTITION FUNCTIONpfRight (INT) AS RANGE RIGHT
FOR VALUES (20,30,40);
...20 21-30 41...31-40 ...19 20-39 40...30-39
20 30 40 20 30 40
Partition Scheme
2012-01-01 ... ...
2012-12-31 ... ...
2013-01-01 ... ...
2013-12-31 ... ...
2014-01-01 ... ...
2014-12-31 ... ...
2015-01-01 ... ...
2015-12-31 ... ...
The Partition Scheme maps logical partitions to physical filegroups
Filegroups?
Data files on one or more disks
Backed up and restored individually
Can be Read-Only
Map all partitions to one filegroup
FILEGROUP
2012-01-01 ... ...
2012-12-31 ... ...
2013-01-01 ... ...
2013-12-31 ... ...
2014-01-01 ... ...
2014-12-31 ... ...
2015-01-01 ... ...
2015-12-31 ... ...
Map partitions to separate filegroups
FILEGROUP1(Read-Only)
FILEGROUP2(Read-Only)
2012-01-01 ... ...
2012-12-31 ... ...
2013-01-01 ... ...
2013-12-31 ... ...
2014-01-01 ... ...
2014-12-31 ... ...
2015-01-01 ... ...
2015-12-31 ... ...
FILEGROUP3
FILEGROUP4
How are partitions mapped to filegroups?
...20 21-30 41...31-40 ...19 20-39 40...30-39
20 30 40 20 30 40
The partition function specified the boundary values and partitions:
How are partitions mapped to filegroups?
CREATE PARTITION SCHEMEpsLeft AS PARTITION pfLeftTO (FG1, FG2, FG3, FG4);
CREATE PARTITION SCHEMEpsRight AS PARTITION pfRight
TO (FG1, FG2, FG3, FG4);
...20 21-30 41...31-40 ...19 20-39 40...30-39
20 30 40 20 30 40
The partition scheme uses the partition function...
How are partitions mapped to filegroups?
CREATE PARTITION SCHEMEpsLeft AS PARTITION pfLeftTO (FG1, FG2, FG3, FG4);
CREATE PARTITION SCHEMEpsRight AS PARTITION pfRight
TO (FG1, FG2, FG3, FG4);
...20 21-30 41...31-40 ...19 20-39 40...30-39
20 30 40 20 30 40
FG1 FG2 FG4FG3 FG1 FG2 FG4FG3
...to map each partition to filegroups:
Filegroups
Partition Scheme
Partitioned Table
2012-01-01 ... ...
2012-12-31 ... ...
2013-01-01 ... ...
2013-12-31 ... ...
2014-01-01 ... ...
2014-12-31 ... ...
2015-01-01 ... ...
2015-12-31 ... ...
A partitioned table is created on a partition scheme instead of directly on a filegroup
The partition scheme uses the partition key to store rows in the correct partition and filegroup based on the definition specified in the partition function
Table Partitioning Basics
Why Table Partitioning?
Partition Elimination
SELECT COUNT(*) FROM TableWHERE Year = 2012;
SELECT COUNT(*) FROM Table;
Partition Elimination
SELECT COUNT(*) FROM TableWHERE Year = 2012;
SELECT COUNT(*) FROM Table;
Backup & Restore Partitions
Filegroups can be backed up and restored individually
If each partition is mapped to a separate filegroup, partitions with the most critical data can be restored first
Backup & Restore Partitions
Filegroups can be read-only
If each partition is mapped to a separate filegroup, partitions with historical, unchanging data can be excluded from regular backups
Index Maintenance per Partition
Rebuild and reorganize indexes per partition
Rebuild indexes online per partition was introduced in SQL Server 2014
Set data compression per partition
ALTER INDEX IndexNameON TableNameREBUILD PARTITION = 2WITH (ONLINE = ON);
Statistics Maintenance per Partition
Update statistics on specific partitions instead of scanning and updating the whole table
UPDATE STATISTICSTableName (StatisticsName)WITH RESAMPLEON PARTITIONS (3,5);
CREATE STATISTICSStatisticsName ONTableName (ColumnName)WITH INCREMENTAL = ON;
Incremental Statistics was introduced in SQL Server 2014
Load & Archive: Partition Switching
Partitions can be switched between tables, called switching in or switching out
Partition switching is a metadata operation that updates the location of the data, no data is physically moved
Extremely fast compared to inserting into or deleting from a large table
2013-01-01 ... ...
2013-12-31 ... ...
2014-01-01 ... ...
2014-12-31 ... ...
2015-01-01 ... ...
2015-12-31 ... ...
2012-01-01 ... ...
2012-12-31 ... ...
ALTER TABLE Table1 SWITCH PARTITION 5
TO Table2 PARTITION 5;
SELECT$PARTITION.pf(2014);
Load & Archive: Switch out
2012-01-01 ... ...
2012-12-31 ... ...
2013-01-01 ... ...
2013-12-31 ... ...
2014-01-01 ... ...
2014-12-31 ... ...
2015-01-01 ... ...
2015-12-31 ... ...
Called switch out when you move data out of a table (archive)
Load & Archive: Switch out
2013-01-01 ... ...
2013-12-31 ... ...
2014-01-01 ... ...
2014-12-31 ... ...
2015-01-01 ... ...
2015-12-31 ... ...
2012-01-01 ... ...
2012-12-31 ... ...
Called switch out when you move data out of a table (archive)
Load & Archive: Switch in
2013-01-01 ... ...
2013-12-31 ... ...
2014-01-01 ... ...
2014-12-31 ... ...
2015-01-01 ... ...
2015-12-31 ... ...
2016-01-01 ... ...
2016-12-31 ... ...
Called switch in when you move data into a table (load)
Load & Archive: Switch in
2013-01-01 ... ...
2013-12-31 ... ...
2014-01-01 ... ...
2014-12-31 ... ...
2015-01-01 ... ...
2015-12-31 ... ...
2016-01-01 ... ...
2016-12-31 ... ...
Called switch in when you move data into a table (load)
Sliding Windows
The Sliding Windows technique automates data loading, data archiving and partition management
It keeps a certain number of partitions by continuously switching out the oldest data and switching in new data
2013-01-01 ... ...
2013-12-31 ... ...
2014-01-01 ... ...
2014-12-31 ... ...
2012-01-01 ... ...
2012-12-31 ... ...
2015-01-01 ... ...
2015-12-31 ... ...
Sliding Windows: Split & Merge
2012-01-01 ... ...
2012-12-31 ... ...
2013-01-01 ... ...
2013-12-31 ... ...
2014-01-01 ... ...
2014-12-31 ... ...
2015-01-01 ... ...
2015-12-31 ... ...
Partitions are not actually added or removed, they are split or merged
Be careful!Splitting and merging partitionscan cause data movement!!
Sliding Windows: Split & Merge
2012-01-01 ... ...
2012-12-31 ... ...
2013-01-01 ... ...
2013-12-31 ... ...
2014-01-01 ... ...
2014-12-31 ... ...
2015-01-01 ... ...
2015-12-31 ... ...
Split one partition in two by adding a new boundary value
ALTER PARTITION FUNCTION pf ()
SPLIT RANGE ('2013-06-01');
Data movement will occur if there is data on both sides of the new boundary value!
Sliding Windows: Split & Merge
2012-01-01 ... ...
2012-12-31 ... ...
2013-01-01 ... ...
2013-12-31 ... ...
2014-01-01 ... ...
2014-12-31 ... ...
2015-01-01 ... ...
2015-12-31 ... ...
Merge two partitions to one by removing a boundary value
ALTER PARTITION FUNCTION pf ()
MERGE RANGE ('2014-01-01');
Data movement will occur if there is data on both sides of the old boundary value!
Sliding Windows Steps
1. Add new filegroup and file
2. Create switch out (archive) table
3. Create switch in (load) table and load it with new data
4. Split to add new partition
5. Switch in new partition
6. Switch out old partition
7. Merge to remove old partition
8. Delete switch out and switch in tables
9. Delete old file and filegroup
Sliding Windows Demo
Alternative Sliding Windows Steps
1. Add new filegroup and file
2. Create switch out (archive) table
3. Switch out old partition
4. Merge to remove old partition
5. Create switch in (load) table and load it with new data
6. Split to add new partition
7. Switch in new partition
8. Delete switch out and switch in tables
9. Delete old file and filegroup
Why Table Partitioning?
Case Study: What Went Wrong?
Background: Financial Data Warehouse
Periodic Snapshot Fact TablesDaily, Weekly and Monthly Balances
SQL Server, DB2, Oracle, .csv, .txtLoaded at different times during the day
1 2 3 4 5 6 7
Pension Insurance Bank Fund
First version
Daily fact table:Keep 1 day per source
1 2
Pension
6 7
Bank Fund
Monthly fact table: Keep 36 months per source
3 4 5
Insurance
Second version
Daily fact table:Keep 1 day per source
1 2
Pension
Monthly / Weekly fact table:Keep 36 months for some sourcesKeep 106 weeks for some sources
3 4 5
Insurance
!
6 7
Bank Fund
Third version
Daily fact table:Keep 1 day per source
1 2
Pension
Monthly / Weekly / Daily fact table:Keep 36 months for some sourcesKeep 106 weeks for some sources
Keep 7 days for some sources
3 4 5 6 7
Insurance Bank Fund
!!
"Everything is slow"
"Let's try partitioning"
Daily fact table partitioning
1 ... ...
2 ... ...
3 ... ...
4 ... ...
5 ... ...
6 ... ...
7 ... ...
Partition Key: source number
One partition per source
Helped with deadlock issues
Easy to switch data in and out
Monthly / Weekly / Daily fact table partitioning
12011 ... ...
12012 ... ...
12013 ... ...
12014 ... ...
19990 ... ...
19991 ... ...
19992 ... ...
Partition Key: Source number + Period type
Period type:YYYY for monthly data9991 for weekly data9992 for daily data
Helped with deadlock issues, but…
Monthly / Weekly / Daily fact table partitioning
12011 ... ...
12012 ... ...
12013 ... ...
12014 ... ...
19990 ... ...
19991 ... ...
19992 ... ...
Partition key was difficult to remember and was not used in queries
Data had to be inserted into and deleted from partitions with existing data, instead of switching partitions in and out
What went wrong?
"The usual suspects": Conflicting priorities, changing requirements and high pressure to deliver on time
Fact tables were expanded, not changed, when new sources and requirements were implemented
Partitioning was implemented on fact tables with multiple grains, instead of correcting the fact tables first
What did we learn?
test, test, test, test
References and Resources: Table Partitioning
Brent Ozar Unlimited: SQL Server Table Partitioning Resourcesbrentozar.com/sql/table-partitioning-resources/
Bradley Ball: Partitioning in SQL Server 2012pragmaticworks.com/Training/FreeTraining/ViewWebinar/WebinarID/541
References and Resources: Further Reading
Partial Backups in SQL Server 2014msdn.microsoft.com/en-us/library/ms191539.aspx
Piecemeal Restores in SQL Server 2014msdn.microsoft.com/en-us/library/ms177425.aspx
Benjamin Nevarez: SQL Server 2014 Incremental Statisticsbenjaminnevarez.com/2015/02/2014-incremental-statistics/
Thank you!
@cathrinew
cathrinewilhelmsen.net
no.linkedin.com/in/cathrinewilhelmsen