archiving and purging peoplesoft data - quest international users
TRANSCRIPT
QuestDirect.org
Dhan Patel :- Senior Product Manager Data Governance, IBM
Archiving and Purging PeopleSoft Data: Scripts vs.
Tools vs. Storage
QuestDirect.org
What we’ll discuss• Challenges managing the lifecycle of
application data • What’s at Stake• Leveraging an Information Governance
Approach• Optimizing the data lifecycle
– Discover & Define– Develop & Test– Optimize, Archive & Access– Consolidate & Retire
QuestDirect.org
Information GovernanceGovern
Quality Security & Privacy
Lifecycle Standards
Transactional & Collaborative Applications
Business Analytics Applications
External Information Sources
Success requires governance across the “Information Supply Chain”
AnalyzeIntegrate
ManageCubes
Big Data Master Data
Content
Data
StreamingInformation
Data Warehouses
ContentAnalytics
QuestDirect.org
Survey: Data Management Pitfalls
High Capital High Capital ExpendituresExpenditures
Missed Service Missed Service Level AgreementsLevel Agreements
Increased capital expenditures for server CPU’s and disk storage systems
Revenue lost to unresponsive databases and applications
Decreased Decreased ProductivityProductivity
Business users losing productivity because systems are not available IT staff consumed with tedious manual processes
Ad Hoc Ad Hoc Performance Performance ManagementManagement
Performance problems addressed only after business impact
*2009-2010 IBM Business Value Assessment Study
QuestDirect.org
Peoplesoft - General Ledger Pain
- General accounting- Journal entries (Debit, Credit)- Calculate balances- Financial consolidation- Print reports at month end and year end- Receive accounting entries from sub ledgers on a regular basis (month end)- Manage translations (currencies), calendars, periods
Financial team cannot complete the Year end closings which can results in a potential disciplinary action ? Loss of confidence in company reporting? Loss of share price value?
QuestDirect.org
Peoplesoft - Account Payables Pain
- Manage business with vendors- Vouchers, payments management- Payment management (print checks, build electronic payments…)- Posting journals to GL (creation)
For those clients that have automated electronic vouchering from suppliers, the voucher staging process and associated tables can be a major hurdle to performance, which in turn may lead to key suppliers not getting paid on time.
- What Happens to the Key Suppliers or Partners when the month end timely payments are not paid.- What happens to those Manufacturers rely on suppliers for critical parts who in turn rely on timely payments.- What happens to those Insurance companies who rely on financial advisors who also rely on timely payments.
QuestDirect.org
Peoplesoft - Accounts Receivables Pain
- Customers repository- Manage Items, Payments, receipts, credit memo management, collections- Interact with BI (Billing)- Posting journals to GL (creation)
ARUPDATE has been traditionally one the nightmare processes in PeoplesoftFinancials. As ARUPDATE impacts accounts receivable and revenue recognition, can a company afford not to have this process finishing within its batch window?Take a typical Fortune 500 company, customer statements and Dunning are sent out everyday, AR positions are also reported daily or weekly, cash balances are planned daily, bank reconciliations are done daily. Can such a company really afford for the follow-on impacts of not having this process performing up to expectations??
QuestDirect.org
Peoplesoft- HCM Pain
- Manage Employees- Manage Salary, Benefits, Compensation, Recruitment, workforce management- Interact with other Apps and Financials.- Print Checks
Sensitive data like Employee Names, SSN, DOB, Addresses, Credit Card, Compensation are stored in the HCM tables. If nor properly masked can create a exponential Threat and Cost of the Organization.
- Legal Cost.- Loss of Revenue, Sales and dip in Stock price.- Loss of Employee , Customer, Vendor confidence.- Loss of Company Reputation.
QuestDirect.org
Managing the Lifecycle of Data in the Information Supply Chain
Discover & DefineDiscover & Define
Consolidate & RetireConsolidate & Retire
Optimize, Archive & AccessOptimize, Archive & AccessDevelop & TestDevelop & Test
• Understanding the “what & where” of enterprise data• Developing models and code to store and access enterprise data
including the configuration of data for test environments• Optimizing performance through identification of bottlenecks and building the right strategy for
managing applications and data growth • Implementing a consistent process for retiring or consolidating applications
Test/DevTest/Dev
QuestDirect.org
You can’t govern what you don’t understand
• Data can be distributed over multiple applications, databases and platforms
– Where are those databases located?• Complex, poorly documented data
relationships– Which data is sensitive, and which can be
shared?– Whole and partial sensitive data elements can be
found in hundreds of tables and fields• Data relationships not understood because:
– Corporate memory is poor– Documentation is poor or nonexistent– Logical relationships (enforced through
application logic or business rules) are hidden
???
??
??
?????
??
???
?
???
? ??
??
?
?
?
?
Distributed Data Landscape
Discover &Discover &DefineDefine
QuestDirect.org
Discover data & define relationships
• Locate and inventory the databases across the enterprise• Define business objects across heterogeneous databases & applications• Define enterprise-standard data models• Understand transformation rules to discover data relationships• Understand relationships required for identifying compound sensitive data• Define and document the privacy & masking rules and propagate to ensure
sensitive data will be protected• Leverage unified schema builder to create prototypes before deployment
Discover &Discover &DefineDefine
QuestDirect.org
Discover business rules, overlaps & lineage
Row Member SS # Phone Sex
1 595846226 123-45-6789 (123) 456-7890 M
2 567472596 138-27-1604 (138) 271-6037 F
3 540450091 154-86-4196 (154) 864-1961 M
4 514714372 173-44-7900 (173) 447-8996 F
5 490204164 194-26-1648 (194) 261-6476 F
6 466861109 217-57-3046 (217) 573-0453 M
987,623 444629628 243-68-1812 (243) 681-8107 F
987,624 423456789 272-92-3629 (272) 923-6280 M
ID Demo1
595846226 0
567472596 1
540450091 0
514714372 1
490204164 1
466861109 0
444629628 1
423456789 0
Can also detect complex data relationships (e.g. relationships leveraging IF THEN ELSE logic)
Table 1 Table 25
Sex = M then Demo1 = 0Sex = F then Demo1 = 1
Discover &Discover &DefineDefine
Case:
QuestDirect.org
Capture related business objects from across the enterprise
Understanding data relationships across the enterprise
Discover &Discover &DefineDefine
CRM onOracle database
ERP / Financials on DB2
Custom Inventory Mgmt on DB2
QuestDirect.org
Challenges of exponential data growth• Missed SLAs and declining customer satisfaction
due to eroding or unpredictable application performance
• Transaction & data volume growth drive up infrastructure costs
• Multiple instances of production data – back-up, training, testing – can compound data growth
• Layered applications stack obscures visibility for problem isolation
• Archival processes are complex requiring full business context for e-discovery
Are you controlling your data, or is it controlling you?
Develop &Test
Discover&Define
Consolidate &RetireOptimize, Archive
& Access
Information Governance Core DisciplinesLifecycle Management
QuestDirect.org
Requirements for effective data management• Proactive optimization of database performance• Ensure application-independent access of
archived data– Extract the complete business object– Provide automated, business rule-driven process
• Support storage strategies for aging data– Ability to store in multiple formats– Open standards for access of data
• Native application access• 3rd party reporting tools, Mashups• Restore to production
Develop &Test
Discover&Define
Consolidate &RetireOptimize, Archive
& Access
Information Governance Core DisciplinesLifecycle Management
QuestDirect.org
Archive the Right Set of DataReferential Integrity – the Complete Business Object
• Represents application data record – payment, invoice, customer
– Referentially-intact subset of data across related tables and applications; includes metadata
• Provides “historical reference snapshot” of business activity• Federated object support across enterprise data stores
QuestDirect.org
Archive the Right Set of Data Example Criteria
General Ledger
Archive Journals and Balances for any Accounting Period, Set of Books values
Accounts Payable
Archive Invoices and Payments for any Accounting Date, Business Unit, Vendors
Accounts Receivable
Archive Invoices and Receipts for any Accounting Date and Business Unit values
Inventory Archive Inventory Transactions for anyInventory Org and Date
QuestDirect.org
Seamless Integration With PeopleSoft
QuestDirect.org
Business Rules and Validations Example – Purchasing
• Purchase Order– Purchase Orders must have a status of Closed or Cancelled– Purchase Orders must have a receipt status of Close– Within one or more specified Business Units– Blanket and Recurring Purchase Orders not archived– PO date precedes date criteria specified.
• Requisitions– Requisitions must have a status of Complete or Cancelled– Within one or more specified Business Units– Requisitions date precedes date criteria specified.
QuestDirect.org
Align Service Levels with Business UseFunctional Usage / Access Requirements Over Time
Functional Data Frequent, Application-
Based Access
Infrequen,t Ad-Hoc, Query-based Access
(Self-Help)
Exception-based, Application-Independent
Access(24-hour IT response)
Complete Deletion (Dictates storage
planning)
Ledgers (GL) Current – 2Y Years 3 - 5 Years 6 - 10 Year 11
Journals (GL) Current – 2Y Years 3 – 5 Years 6 - 10 Year 11
Vouchers (AP) Current – 2Y Years 3 – 5 Years 6 - 10 Year 11
Payments (AP) Current – 2Y Years 3 – 5 Years 6 - 10 Year 11
Invoices (AR) Current – 2Y Years 3 – 5 Years 6 - 10 Year 11
Items (AR) Current – 2Y Years 3 – 5 Years 6 - 10 Year 11
Invoices (BI) Current – 2Y Years 3 – 5 Years 6 - 10 Year 11
Billing Input (BI) Current Year Year 2 Years 3 - 10 Year 11
Retired Assets (AM) Current – 2Y Years 3 – 5 Years 6 - 10 Year 11
Asset Input (AM) Current Year Year 2 Years 3 - 10 Year 11
QuestDirect.org
Functional Archive Policies: FinancialsApplication Retention
(Years)Archiving Recovery / Access
RequirementsLead Time
Type of Data to Archive
GL 3 Yearly Audit; Trend analysis Y Ledgers, Journals, fully posted
AP 3 Yearly Audit; Trend analysis Y Vouchers, Payments, fully paid and posted
AR 3 Yearly Audit; Trend analysis Y Invoices, items
Billing 3 Yearly Audit; Trend analysis Y Invoices
Billing Interface 1 Quarterly Troubleshooting Y Billing input
AM 3 Yearly Audit; Trend analysis Y Retired assets
AM Interface 1 Quarterly Troubleshooting Y Asset input, GL interface
Payroll 2 Yearly Audit Y Paycheck processing data and balances
QuestDirect.org
Archive Process
Compressed Archives
Compressed Archives
2 -4
Yea
rsAc
tive/H
istor
ical O
nlin
e
XMLXML
ArchiveDatabase
Non DBMS Retention Platform
ATA File ServerEMC Centera™, DR550, Etc.4
-6 Y
ears
On/N
ear-L
ine
Arch
iveOff-line Retention PlatformCD,Tape,Optical, WORMHP StorageWorks™, IBM TSMNetApp NearStore® SnapLock™,IBM Total Storage® solutions
(including the DR550) EMC Centera™.
6+ Y
ears
Off-L
ine A
rchi
ve
• Access Definitions• Complete
Business Object• Business Rules• Validations
ExtractExtract
UNIVERSAL
ACCESS
RestoreRestore
1 - 2 YearsCurrent Data
ProductionDatabase
Additional Options
ODBC / JDBCXMLSQL
Excel / Access
Native Application Access
QuestDirect.org
Peoplesoft Archive Data Access
b
Production User
Archive & Production User
NAA Schema
Enterprise Database SYSADMSchema
Archive TablesArchive Data Schema
- Tables- Views- Packages
- Synonyms- Union Views- Etc
ArchiveFiles
1. Archive & Delete Transactions from Production
2. Insert Archives Into History Database for Transparent Access
- Tables
Preserved & Compressed Historical Snapshot
QuestDirect.org
Production Data Access – Pre Archive
QuestDirect.org
Production Data Access – Post Archive
QuestDirect.org
Archive Data Access – Archive Only - NAA
QuestDirect.org
Test Data Management: The Challenge
1
Production 1 TB
Training 1 TB
Unit Test 1 TB
System Test 1 TB
UAT 1 TB
Integration 1 TB
Total 6 TB
Production 1 TB
Training 1 TB
Unit Test 1 TB
System Test 1 TB
UAT 1 TB
Integration 1 TB
Total 6 TB
23
45
6
QuestDirect.org
Leverage solution delivery and test data management industry practices
• Create database structures• Develop re-useable database code to enhance productivity• Develop high performance queries and improve performance & problem
determination of existing queries• Manage data in non-production environments
– Create /subset– Mask– On demand Refresh– Dynamic Dataset filtered refresh– Automated testing– Browse/edit– Compare
Develop &Develop &TestTest
QuestDirect.org
Production Environment
Baseline Subset
Test
Dev
QA
Extract/ Archive File
Dynamically loadrelational intact data sets & objects based on selection criteria
IBM Optim
Optim Test Data Management Solution• Load • Insert / Update• Compare• Browse• Edit• Refresh
QuestDirect.org
Test Data Management – Archiving Impact
Production
Production 400 GB
Training 400 GB
Unit Test 400 GB
System Test 400 GB
UAT 400 GB
Integration 400 GB
Total 2.4 TBStorage reduced by 60%
Training
Unit Test
System Test
UAT
Integration
QuestDirect.orgCreating right-sized targeted test environments
saves storage costs & speeds testing
TDM: Archiving + Subsetting Impact
Production
Production 400 GB
Training 100 GB
Unit Test 200GB
System Test 200 GB
UAT 200 GB
Integration 100 GB
Total 1.2 TBStorage reduced by 80%
Training
Unit Test
System Test
UAT
Integration
QuestDirect.org
TestProduction
Optim Data Privacy with TDM
Validate and Compare
De-identify for privacy protectionDeploy multiple masking algorithms Substitute real data with fictionalized yet contextually accurate dataProvide consistency across environments and iterationsNo value to hackersEnable off-shore testing
Subset Mask Propagate
QuestDirect.org
Before Data Masking
QuestDirect.org
After Data Masking
QuestDirect.org
Optim Application Retirement
Before Decommissioning After Decommissioning
• Remove an application or system from active service But…
• Retain access to its business-critical data– Retire an application– Consolidate redundant systems into single “enterprise standard”– Migrate portfolio to lower-cost platform – Consolidate and eliminate unsupported databases and versions
ArchiveArchiveFilesFiles
QuestDirect.org
About the Client
Client Success: Data Growth and Compliance
• Challenges:– Ensuring application availability by shortening nightly batch
processing cycles that were close to extending into “open for business” hours
– Supporting business growth and SLAs for PeopleSoft SCM queries, degraded by managing multiple SCM environments and by storing years of historical data
– Addressing new industry and regulatory standards for HR data retention, while supporting the client’s own unique HCM requirements.
• Client Value:– Improved SCM performance by 60% by archiving high volume
tables, reducing overnight processing time from 6.5 hours to 2.2hours.
– Achieved data retention regulatory compliance and improved queryperformance across the HCM modules.
Industry
Industrial Manufacturing
Annual Revenue$3.7 Billion
ApplicationPeopleSoft® Enterprise for SCM, Financials & HCM
QuestDirect.org
About the Client
Client Success: Data Growth• Challenges:
– Meeting SLAs to support payroll processing for 150,000 employees by addressing batch and online performance and availability issues affecting mission critical PeopleSoft HCM V8.8 (HR, Base Benefits and Payroll)
– Defining full-lifecycle data management strategies and supporting data retention requirements for HCM records that had more than doubled in less than 2 years
– Providing technical and functional business users with access to current and historical data to support all audit, legal, contractual and reporting requirements
• Client Value:– Improved SLAs for timely payroll processing and check issuance by archiving
50 percent of PeopleSoft HCM historical payroll records– Defined self-documenting and auditable processing rules to archive HR data to
support retention requirements throughout the data lifecycle– Enabled technical and business functional users to interact with current and
archived historical data, readily accessible for ad hoc queries and reports using familiar tools, like PS_Query
IndustryAircraft Manufacturing
Annual Revenue$55 Billion
ApplicationPeopleSoft® Enterprise for HCM
QuestDirect.org
Enterprise Architecture
Single, scalable, interoperable data management solution provides a central point to deploy policies to extract, store, port, and protect application data records from creation to deletion
Application Metadata Repository Analysis for Oracle Packaged Applications
Data GrowthData PrivacyTest Data Management Application Retirement
QuestDirect.org
Why IBM Optim?
• Most comprehensive, robust, solution in market– Comprehensive discovery and Complete Business Object provides relationally
intact data set that can be accessed independent of application and serves as audit ready snap shot of data in time
– Archive directly to compressed archive files– Comprehensive data restore capabilities including selective restore and restore
directly from compresses archive files– Wide array of access methods for archive data - Native application access as
well as application independent methods– Safeguards such as compare row before delete to ensure data integrity
• Efficient Archive Data Storage for reduced overall TCO– Pioneered compressed archive file format; 7th generation of product using this
format– Also supports XML and database formats
• Single, Scalable, Enterprise Solution– Widest support in terms of databases, operating systems, applications
(custom/legacy and packaged), and storage system interfaces– Test Data Management, Data Privacy, Data Masking, and Application
Retirement using same platform– Addressing mainframe data sources for over 20 years
QuestDirect.org
Why IBM Optim?
• Strong Oracle partnership– Oracle Certified Advantage Partner with a strong alliance with Oracle– Optim is the only solution in market that provides validated solutions for
all major Oracle applications – Siebel, E-Business, PeopleSoft and JD Edwards
– IBM GBS is world’s largest provider of services for Oracle applications
• Leading Market position– Market leader as defined by Gartner Group (46%)– Over 2500+ customers worldwide with largest installed base of
customers for Oracle applications
• Power of IBM behind the solution– Optim is foundation component of IBM strategy for Information On
Demand– World-class software support– Integrations/synergies with other IBM products for data relationship
discovery, content management, data transformation, data viewing, etc.
QuestDirect.orgwww.ibm.com/software/data/infosphere