enhance database performance and...
TRANSCRIPT
Enhance Database Performance and Scalability
Progress OpenEdge: Database Table Partitioning Workshop
Paul Koufalis Progresswiz informatique www.progresswiz.com
© 2014 Progress Software Corporation. All rights reserved. 2
Who Is This Paul Koufalis? Where’s Rich Banville?
Progress DBA and UNIX sysadmin for 20 years
Wide spectrum of clients
• Financial services providers, distributors, manufacturers, retailers
• $10M to $Billions in revenue
• DBs from 1Mg to 100’s of Gb
• 10 to almost 2000 concurrent users
© 2014 Progress Software Corporation. All rights reserved. 3
Agenda
Getting Connected to Your Virtual Machines
Introduction to Table Partitioning
LABS! LABS! LABS!
I Changed My Mind: Changing/Deleting
Geek BONUS: ABL API
Connecting to Your VM
© 2014 Progress Software Corporation. All rights reserved. 5
LAB 0 – Connecting to Your VM
First thing’s first: make sure you can access your private virtual machine
Requirements:
• Internet access (the VMs are in the cloud)
• Web Browser to access OpenEdge Management
• SSH tool like putty
Introduction
We will not be going into every nitty-gritty detail of partitioning No time. You will listen to me for 3h instead of doing labs
Don’t miss…
Getting Started with Table Partitioning Tuesday, October 7, 2014, 11:00 AM — 12:00 PM
Havard Danielsen, Software Engineer; Rich Banville, Software Fellow
Table Partitioning—Application and Design Wednesday, October 8, 2014, 1:30 PM — 2:30 PM
Rich Banville, Software Fellow
© 2014 Progress Software Corporation. All rights reserved. 8
The Steps to Table Partitioning
1. Identify tables
2. Prepare
a. Type II Storage Areas
b. Recid/rowid usage
c. Aligned field assignments
d. Partition aligned (local) indexes
3. Plan strategy
a. Dump / Load
b. “In place” migration
4. Enable partitioning
a. Database
b. Index rebuild
c. Tables / indexes / lobs
5. Define policies
6. Split out existing data
7. Truncate / de-allocate “initial” partition
© 2014 Progress Software Corporation. All rights reserved. 9
Identify Tables
“Stable” data logically grouped by • Chronological events (range partitions)
• Discrete list of values (list partitions)
• Sub-partition partitioning – Partitioning the same table according to
the values of more than one column
Table / index maintenance too high • Operational time
• Data availability
See white paper “Selecting Tables for Partitioning”: community.progress.com/technicalusers/f/18/t/9294.aspx
© 2014 Progress Software Corporation. All rights reserved. 10
Select Partition Columns
Best if application queries on partitioned-aligned columns • Less use to partition by State if no one ever queries by State
Partition columns should not change [often] • Otherwise may require the record to change partitions
Partition column(s) are leading component(s) of an index • Partition-aligned local index
• Consider adding such an index if it does not exist
REMEMBER: Every column value must “fit” in a partition
There is no “default” partition
© 2014 Progress Software Corporation. All rights reserved. 11
Prepare
[Optionally] create Type II S.A for each partition
• Decide where to put each physical extent
• Example: Historical data on cheap SATA drives
Clean your data
• No UNKNOWN values in partition columns
Clean your code
• Assign partition columns in same ASSIGN when creating records
– Avoids default value assignments followed by record to move to another partition
• Watch out for ROWID()
– Will change if record moves to a different partition
• RECID() has NOT been updated to support TP
© 2014 Progress Software Corporation. All rights reserved. 12
Example: List + Range
9 partition example
Sub-partition by region & order-date w/in region
Order Table
12/31/2012 AMER
12/31/2013 AMER
12/31/2014 AMER
12/31/2012 APAC
12/31/2013 APAC
12/31/2014 APAC
12/31/2013 EMEA
12/31/2014 EMEA
12/31/2012 EMEA
9 B-trees supporting 3 index definitions
Index Components Partition Aligned Index #1 (local) {Region, Order-Date, Name} YES
Index #2 (local) {Region, Order-Date, S-rep} YES
Index #3 (global) {Cust-num} NO
Index #2
Global indexes
span partitions
Indexed field need
not be partition aligned
Index #2 Index #1
Local indexes
on order-date, S-rep
Local indexes
on order-date, name
© 2014 Progress Software Corporation. All rights reserved. 13
Migration Strategy – Dump & Load
Dump data Delete data (truncate area hopefully) Define partitions Load data
May be faster than in-place migration BUT…data will be unavailable for longer
Best during a maintenance window Obligatory if you want to change an already partitioned table
© 2014 Progress Software Corporation. All rights reserved. 14
Migration Strategy – In Place
Composite partition created in area where data resides Data is logically segregated (no physical changes) Ready to be split into physical partitions
AKA “Partition 0”
06/30/2014 EMEA
06/30/2014 AMER
06/30/2014 APAC
Order Table
© 2014 Progress Software Corporation. All rights reserved. 15
Split out data Example: 3 of 9 partitions have been split out of composite partition
Migration Strategy – In Place
Order Table AMER
12/31/2014 AMER
APAC 12/31/2014
APAC
EMEA 12/31/2014
EMEA 06/30/2012 06/30/2014
06/30/2012 06/30/2014
06/30/2012 06/30/2014
10
Local Index #10 Partition #1
10
Local Index #10 Partition #2
10
Local Index #10 Partition #3
10
Composite Index #10 Partition #0
© 2014 Progress Software Corporation. All rights reserved. 16
Enable Partitioning
Command line
Or OpenEdge Management/Explorer • ONLY can enable TP
Online requires exclusive schema access
proutil <db> -C enabletablepartitioning
proutil <db> -C enabletpidxbld
© 2014 Progress Software Corporation. All rights reserved. 17
Define Policies
Use OpenEdge Management/Explorer
Advanced users can use new set of ABL classes
© 2014 Progress Software Corporation. All rights reserved. 18
Split Out Existing Data
Identify created partition as a “split target” Move data into target partitions
Recovery of operation restarts where it left off Online operation with full access to non-split data
• New split transitional state for partitions Multiple concurrent operations allowed OpenEdge Replication fully supported
proutil <db> -C partitionmanage split table <name> partition <name> | composite "initial“ useindex <name> recs <number>
© 2014 Progress Software Corporation. All rights reserved. 19
Truncate / Deallocate “Initial” Partition
proutil <db> -C partitionmanage truncate table <tname> composite initial recs <#recs per txn> [ deallocate ]
Order Table
12/31/2013 AMER
06/30/2014 AMER
12/31/2014 AMER
12/31/2013 APAC
06/30/2014 APAC
12/31/2014 APAC
06/30/2014 EMEA
12/31/2014 EMEA
12/31/2013 EMEA
Deallocated Partition 0
LAB 1: Simple List Partitioning
© 2014 Progress Software Corporation. All rights reserved. 21
Lab 1 – Simple List Partition
Let’s do the first lab together
We are going to partition the customer table by country
Open up your Lab Guide
LAB 2: Simple List Partition Continued
© 2014 Progress Software Corporation. All rights reserved. 23
Lab 2 – Simple List Partition Continued
Try it yourself
This time we will physically partition data to distinct storage areas
LAB 3: List + Range Partitions
© 2014 Progress Software Corporation. All rights reserved. 25
Lab 3 – List + Range Partitions
Split Order by Region and by OrderDate (calendar year)
Region = LIST partition
Year = RANGE sub-partition
REMEMBER: Both the LIST and the RANGE must cover ALL the keys in the partition columns!
Merge –Truncate - Idxbuild
© 2014 Progress Software Corporation. All rights reserved. 27
Tools
Four new proutil options • PARTITIONMANAGE SPLIT • PARTITIONMANAGE MERGE • PARTITIONMANAGE TRUNCATE • TPIDXBUILD
We already saw split
© 2014 Progress Software Corporation. All rights reserved. 28
PARTITIONMANAGE TRUNCATE
Truncate and optionally deallocate a partition
Used to free storage space of a partition
DELETES THE DATA
CAREFUL: [recs] governs number of deletes per transaction
• Affects lock table usage and BI size
proutil <db> -C partitionmanage truncate table <name> partition <name> | composite initial [recs <number>] [deallocate]
© 2014 Progress Software Corporation. All rights reserved. 29
PARTITIONMANAGE TRUNCATE
© 2014 Progress Software Corporation. All rights reserved. 30
DB Analysis – Truncate
CHAIN ANALYSIS FOR AREA "Order EMEA 2013": 67 RM CHAIN ANALYSIS --------------------------- Number of Blocks Object Type Object Partition/Tenant/Group ------------------------------------------------------------------------------- 508 Table PUB.Order:13 P:Order_EMEA_2013-12-31:5 AREA "Order EMEA 2013": 67 BLOCK ANALYSIS ------------------------------------------------- 2047 block(s) found in the area. Current high water mark: 2047 1529 free block(s) found in the area 508 record block(s) found in the area 0 index block(s) found in the area 0 empty block(s) found in the area 2 object block(s) found in the area 2 cluster list block(s) found in the area 1 object list block(s) found in the area 2 cluster map block(s) found in the area
© 2014 Progress Software Corporation. All rights reserved. 31
DB Analysis – Truncate and Deallocate
CHAIN ANALYSIS FOR AREA "Order EMEA 2013": 67 RM CHAIN ANALYSIS --------------------------- Number of Blocks Object Type Object Partition/Tenant/Group ------------------------------------------------------------------------------- AREA "Order EMEA 2013": 67 BLOCK ANALYSIS ------------------------------------------------- 2047 block(s) found in the area. Current high water mark: 2047 2041 free block(s) found in the area 0 record block(s) found in the area
Notice no more RM blocks
© 2014 Progress Software Corporation. All rights reserved. 32
Clean Up Time (or Is It?)
Let’s look at the _StorageObject table (area 67 from htplab.st)
for each _StorageObject where _area-number = 67:
displ _storageobject with 1 col no-box
Object-type: 1 Object-number: 13 (This is the Order Table)
Object-associate-type: 2 Object-associate: 39 (This is the Primary Index) Area-number: 67 Object-attrib: 264 Object-system: 0 Create-Limit: 150 Toss-Limit: 300 CollationId: 0 Object-state: 2
© 2014 Progress Software Corporation. All rights reserved. 33
Delete the Partition
In Partition Policy Manager Details
Delete
No more _StorageObject record
Now you can remove the physical area
© 2014 Progress Software Corporation. All rights reserved. 34
Free Up the Space
© 2014 Progress Software Corporation. All rights reserved. 35
CAREFUL
YOU MAY BE TEMPTED to run “proutil htplab –C truncate area”
• DO NOT. You have been warned (KABOOM!)
© 2014 Progress Software Corporation. All rights reserved. 36
PARTITIONMANAGE MERGE
Merge two or more range or list-range partitions
Specify the start and end partitions
Optionally specify a new target partition
All partitions between the start and end will be merged
proutil <db> -C partitionmanage merge table <name> partition <name start> partition <name end> [ partition <name target> ] useindex <name> recs <number>
© 2014 Progress Software Corporation. All rights reserved. 37
PARTITIONMANAGE MERGE
$ _proutil htplab -C partitionmanage merge table order partition "Order_AMER_2013-12-31" partition "Order_AMER_2014-12-31" partition "Order_AMER_2015-12-31" recs 1000 BEGIN: Merge Operation For Table order ... Do you want to continue (y/n)? Partition Order_AMER_2013-12-31[2]: records removed: 28138. Partition Order_AMER_2013-12-31[2]: has been deleted. Partition Order_AMER_2014-12-31[1]: records removed: 5473. Partition Order_AMER_2014-12-31[1]: has been deleted. Partition Order_AMER_2015-12-31[3]: Total records merged: 33611. END: Merge Operation For Table order was successful.
© 2014 Progress Software Corporation. All rights reserved. 38
PARTITIONMANAGE MERGE
Refresh Partition Policy Detail Screen
• Partitions for 2013 and 2014 are gone
Check _StorageObject table for AMER areas (see htplab.st for area #s)
• Only area 51 "Order AMER 2015“ and 52 "Order AMER 2015 Idx“ have objects
You can delete unused storage areas using prostrct remove
© 2014 Progress Software Corporation. All rights reserved. 39
TPIDXBUILD
Index rebuild by partition
Can run multiple tpidxbuild in parallel
Index not available during build
Index does not become available until older clients logout
• TPIDXBUILD notifies you and prompts you to wait or cancel
proutil <db> -C tpidxbuild table <name> [ area <name> | index <name> ] partition <name > | composite initial [ -T <tmpdir> | -SS <srt-file> ] [ -TB # ] [ -TM # ] [ -B # ] [ -SG # ] [ -pfactor # ]
LAB 4: Merge, Truncate & TPIDXBUILD
© 2014 Progress Software Corporation. All rights reserved. 41
Lab 4 – Merge, Truncate & TPIDXBUILD
Merge Order Partitions for AMER
Truncate Order Partition 0
Multiple, concurrent partition index rebuilds
BONUS: ABL API OO Classes
© 2014 Progress Software Corporation. All rights reserved. 43
ABL API OOC
That’s a lot of TLAs (Three Letter Acronyms)
Documentation: “OpenEdge Development: Programming Interfaces”
Package OpenEdge.DataAdmin
© 2014 Progress Software Corporation. All rights reserved. 44
The DataAdminService
Before doing anything, you must instantiate a new DataAdminService using OpenEdge.DataAdmin.DataAdminService from propath.
define variable service as DataAdminService no-undo.
service = new DataAdminService("htplab").
Remember to delete the object at the end delete object service no-error.
© 2014 Progress Software Corporation. All rights reserved. 45
using OpenEdge.DataAdmin.IPartitionPolicy from propath. define variable partitionPolicy as IPartitionPolicy no-undo. partitionPolicy = service:NewPartitionPolicy("OrderRegnYear"). ASSIGN partitionPolicy:Table = service:GetTable("Order") partitionPolicy:HasRange = yes partitionPolicy:DefaultDataArea = service:GetArea("Order") partitionPolicy:DefaultIndexArea = service:GetArea("Order Idx") partitionPolicy:DefaultLobArea = service:GetArea("Order") partitionPolicy:DefaultAllocation = "None". /* Add fields to the fields collection */ partitionPolicy:Fields:Add(partitionPolicy:Table:Fields:Find("Region")). partitionPolicy:Fields:Add(partitionPolicy:Table:Fields:Find("OrderDate")). /* Add RegnDate to the local index collection */ partitionPolicy:Indexes:Add(partitionPolicy:Table:Indexes:Find("RegnDate")).
Create New Partition Policy
© 2014 Progress Software Corporation. All rights reserved. 46
Create New Partition Policy Detail
using OpenEdge.DataAdmin.IPartitionPolicyDetail from propath. partitionPolicyDetail = service:NewPartitionPolicyDetail("Order_EMEA_2013"). /* Add the new partitionPolicyDetail to the Details collection. */ partitionPolicy:Details:Add(partitionPolicyDetail). ASSIGN partitionPolicyDetail:IsSplitTarget = no partitionPolicyDetail:Description = "" partitionPolicyDetail:DefaultDataArea = service:GetArea("Order") partitionPolicyDetail:DefaultIndexArea = service:GetArea("Order Idx") partitionPolicyDetail:DefaultLobArea = service:GetArea("Order"). /* Set the values for the policy detail */ partitionPolicyDetail:SetValue("EMEA",1). /* Region as character */ partitionPolicyDetail:SetValue(12/31/2013,2). /* OrderDate as date */ service:UpdatePartitionPolicy(partitionPolicy).
© 2014 Progress Software Corporation. All rights reserved. 47
Split the Data
using OpenEdge.DataAdmin.Util.PartitionSplitUtility from propath.
define variable split as PartitionSplitUtility no-undo.
split = NEW PartitionSplitUtility(partitionPolicyDetail).
service:ExecuteUtility(split).
© 2014 Progress Software Corporation. All rights reserved. 48
Merge and Truncate
Not yet…sorry
But there are A LOT of cool things you CAN do:
• Manage storage areas and physical extents
• Manage authentication systems, domains and users
• Manage schema changes
• Manage Tenants in a MT DB
© 2014 Progress Software Corporation. All rights reserved. 49
Import and Export
partitionPolicy:Export("Order.json"). {"partitionPolicies": [ { "name": "OrderRegnYear", "schemaName": "PUB", "tableName": "Order", "table_url": "\/schemas\/PUB\/tables\/Order", "description": null, "url": "\/partitionpolicies\/OrderRegnYear", "hasRange": true, "type": "List-range", "partitionPolicyDetails_url": "\/partitionpolicies\/OrderRegnYear\/partitionpolicydetails", "defaultDataAreaName": "Order", "defaultDataArea_url": "\/areas\/Order", "defaultIndexAreaName": "Order Idx", "defaultIndexArea_url": "\/areas\/Order%20Idx", "defaultLobAreaName": "Order", "defaultLobArea_url": "\/areas\/Order", "defaultAllocation": "None" } ]}
Q&A
Get session details & presentation downloads Complete a survey Access the latest Progress product literature
www.progress.com/exchange2014
Visit the Resource Portal