how to build and promote a successful mdm solution on a shoestring
TRANSCRIPT
Pablo Riboldi, PhDSolution Manager for Information Governance & Quality
The Church or Jesus Christ of Latter-day Saints
How to build and promote a successful MDM solution on a shoestring
Master Data Management (MDM) is a set of processes, technologies, strategies to securely provide accurate and consistent enterprise data.
2© IRI 2011
Master Data Management - Definition
• Large– 13 million members in 160 countries– 53,000 full-time missionaries– Materials translated in 83 languages– 264 centrally managed databases– Largest Genealogy system in the world
• www.lds.org or www.mormon.org
© IRI 2010 3
My Organization
• Large• Organization (Hierarchical)– Governing Council (18 members)– Departments (23)– Silo Organization– Independent information systems
• www.lds.org or www.mormon.org
© IRI 2010 4
My Organization
• Large• Organization (Hierarchical)• Non Profit– Invite people to come to Jesus Christ by
• Proclaiming the gospel• Perfecting the saints• Redeeming the dead
– Clearly defined common purpose and– Great people to work with
• www.lds.org or www.mormon.org
© IRI 2010 5
My Organization
Common Method MDM Sharing
MDM
Systems of Record (Enterprise info)
Pass-through accounts
2800+
ViewsDB Links
Master Data Management
Repository
Consumer Systems
6© IRI 2011
Data Management
• Provide accurate data to Church applications • Improve consistency of data reporting across the Church • Ensure the security of data • Enforce policies for use of data • Reduce development time • Eliminate the need to maintain master data in each
application • Decrease maintenance costs
7© IRI 2011
MDM Goals
• Designate authoritative sources of master data and select stewards
• Provide interfaces for cleansing and enrichment of data • Share master data from the source with other applications • Create a common data model and interfaces for
using master data in applications • Centralize administration of security and data use policies
(at the application level)
8© IRI 2011
MDM Strategy
Table 1
Relational data models Dimensional data models
Table3
Rel 1 2
Table4
Table 2
Rel 1 2 3
Dim 1
Dim 2
Dim 3
FactETL
How to store data in the MDM RepositoryRelational – Dimensional – Canonical Models
• Used for transactional systems• Normalized (at least 3NF, usually at 5NF)• Optimized for CRUD transactions (insert,
update, delete)
• Used for data warehouses• Denormalized: Star schema with
dimensions (entities), facts (measures)• Optimized for Reporting (ad-hoc queries)
Canonical data models
Table 1 2
Table 3 4
ETL
How to store data in the MDM RepositoryRelational – Dimensional – Canonical Models
Table 1
Relational data models
Table3
Rel 1 2
Table4
Table 2
Rel 1 2 3
• Used for transactional systems• Normalized (at least 3NF, usually at 5NF)• Optimized for CRUD transactions (insert,
update, delete)
• Used for reference by other systems• Semi-denormalized (1NF or 2NF)• Optimized for fast access by systems
(usually used in dropdowns)
MDM_ORG
ORG_ID ORG_NAME ORG_STATUS PARENT_ORG …
1 HR Department Active CPBO …
2 ICS Department Active CPBO …
3 ISD Department Discontinued CPBO …
… … … …MDM_ORG
ORG_ID ORG_NAME ORG_STATUS PARENT_ORG …
1 HR Department Active (NULL) …
2 ICS Department Active (NULL) …
MDM Repository
Target Application
• The target receives only “Active” organizations and it does not receive PARENT_ORG.
• Every target receives the same data model.• The target receives the information via Replication,
Database Link, or Web Service.
Need to deliver only the information (rows andcolumns) authorized by the Data Steward…
• We need to constrain the results at both the row and column level to meet data sharing agreement constraints.
• The MDM Repository is hosted on an Oracle database.
• Oracle Virtual Private Database (VPD) permits the creation of policies to control database access at the row and column level.– These policies essentially add a dynamic WHERE clause to
the SQL statement issued against the table, view, or synonym.
Why use Oracle VPD capabilities?
• Establishing the repository:– Selection of a data source (MDM_User or another source)
• Publishing the repository:– Create MDMR schema with a common data model in each
consuming application database • Build applications that use foreign keys to the master data instead of
copying the data– Use Data Integrator to replicate data to MDMR schemas
• Re-use same code for every application– Use VPD to enforce data use policies specific to each application
• Create a passthrough account on the source– Administer MDMR data privileges for each application centrally
(Integration team)
13© IRI 2011
MDM Implementation includes:
Systems of Record (Master data)
MDM_USERschema
MDM Source
OrganizationsLeaders
Finance
Other systems
14© IRI 2011
1) Master data is integrated, cleansed, enriched, and transformed to the MDM data structures from the systems of record.
Master Data Management – Architecture
Systems of Record (Master data)
MDM_USERschema
MDM Source
MDM_SECschemaOrganizations
Leaders
Finance
Other systems
After Login
Trigger
15© IRI 2011
1) Master data is integrated, cleansed, enriched, and transformed to the MDM data structures from the systems of record.
2) A set of security packages is defined to restrict columns and rows.
Master Data Management – Architecture
Systems of Record (Master data)
Business Objects
Data Integrator
MDM_USERschema
MDM Source
MDM_SECschemaOrganizations
Leaders
Finance
Other systems
After Login
TriggerAPP_PTM
3) The After-login trigger applies security policies to the MDM accounts.
16© IRI 2011
1) Master data is integrated, cleansed, enriched, and transformed to the MDM data structures from the systems of record.
2) A set of security packages is defined to restrict columns and rows.
Master Data Management – Architecture
Systems of Record (Master data)
Business Objects
Data Integrator
MDM_USERschema
MDM Source
MDM_SECschemaOrganizations
Leaders
Finance
Other systems
After Login
TriggerAPP_PTM
4) Security policies limit the rows and columns available to each MDM account.
3) The After-login trigger applies security policies to the MDM accounts.
17© IRI 2011
1) Master data is integrated, cleansed, enriched, and transformed to the MDM data structures from the systems of record.
2) A set of security packages is defined to restrict columns and rows.
Master Data Management – Architecture
Systems of Record (Master data)
Business Objects
Data Integrator
Requesting Application Instance
MDMRschemaMDM_USER
schema
MDM Source
MDM_SECschemaOrganizations
Leaders
Finance
Other systems
After Login
TriggerAPP_PTM
4) Security policies limit the rows and columns available to each MDM account.
3) The After-login trigger applies security policies to the MDM accounts.
5) Data is published to the MDMR at set frequencies.
18© IRI 2011
1) Master data is integrated, cleansed, enriched, and transformed to the MDM data structures from the systems of record.
2) A set of security packages is defined to restrict columns and rows.
Master Data Management – Architecture
Systems of Record (Master data)
App.schema
Business Objects
Data Integrator
Requesting Application Instance
MDMRschemaMDM_USER
schema
MDM Source
MDM_SECschemaOrganizations
Leaders
Finance
Other systems
After Login
TriggerAPP_PTM
APP_PTC Application
4) Security policies limit the rows and columns available to each MDM account.
3) The After-login trigger applies security policies to the MDM accounts.
5) Data is published to the MDMR at set frequencies.
6) Requesting App can use IDs as FKs, but should not copy data.
.
19© IRI 2011
1) Master data is integrated, cleansed, enriched, and transformed to the MDM data structures from the systems of record.
2) A set of security packages is defined to restrict columns and rows.
Master Data Management – Architecture
• A database package is defined for each object to be protected.
• The database package functions reference information stored in the Application Context of the current database session and returns the SQL fragment to be either:– Appended to the Where Clause or– Used to determine whether or not to show the
column contents in the result.
How are the policies defined?
CREATE OR REPLACE package MDM_ORG_SEC_PKG as function SET_PREDICATE (OBJECT_SCHEMA VARCHAR2,OBJECT_NAME VARCHAR2) return VARCHAR2; -- row policies... function VIEW_TIMEZONE (OBJECT_SCHEMA VARCHAR2,OBJECT_NAME VARCHAR2) return VARCHAR2; -- column policiesend MDM_ORG_SEC_PKG;/CREATE OR REPLACE package body MDM_ORG_SEC_PKG as CTX_VALUE VARCHAR2(2000) :=NULL; CTX_NAME VARCHAR2(30) :='MDM_SEC_CTX'; -- defines the application context function SET_PREDICATE (OBJECT_SCHEMA VARCHAR2,OBJECT_NAME VARCHAR2) return VARCHAR2 as begin CTX_VALUE:=NULL; CTX_VALUE:=SYS_CONTEXT('USERENV','SESSION_USER'); -- identifies the user associated with this session IF CTX_VALUE = OBJECT_SCHEMA THEN RETURN '1=1'; -- if the user is the same as the object owner, enable all access END IF; CTX_VALUE:=NULL; CTX_VALUE:=SYS_CONTEXT(CTX_NAME,'ORG_PREDICATE',2000); -- gets the where clause fragment to be appended IF CTX_VALUE IS NULL THEN RETURN '1=0'; -- if there is no information in the context, deny all access ELSE RETURN CTX_VALUE; -- return the row constraints END IF; EXCEPTION WHEN OTHERS THEN RETURN '1=0'; end;... function VIEW_TIMEZONE (OBJECT_SCHEMA VARCHAR2,OBJECT_NAME VARCHAR2) return VARCHAR2 as begin…
Sample policy package
• First, we define the objects and protected column lists in the MDM repository security tables.
• Then we execute a set of scripts to:– Create a stored procedure that will be used to
populated the application context when the user connects.
– Create the application context and tie to the above stored procedure.
– Tie the VPD policies to the database objects.
How do we connect the policy to thedatabase object?
Defines the objects (tables, views, synonyms) to be protected
Defines the columns to be protected
MDM_VPD_TABLE_PK TABLE_SCHEMA TABLE_NAME DESCRIPTION APP_CONTEXT POLICY_GROUP
POLICY_FUNCTION_SCHEMA PREDICATE_POLICY_NAME PREDICATE_FUNCTION_NAME LAST_UPDATE
1… MDM_ORG MDM Organization Table MDM_SEC_CTX MDM_ORG_SEC_POLICY_GRP … ORG_PREDICATE MDM_ORG_SEC_PKG.SET_PREDICATE …… … … … … … … … … …
MDM_VPD_COL_SET_PK MDM_VPD_TABLE_PK COL_SET_POLICY_NAME COL_SET_FUNCTION_NAME SECURED_COLUMN_LIST LAST_UPDATE68 1 ORG_TIMEZONE MDM_ORG_SEC_PKG.VIEW_TIMEZONE TIME_ZONE_CODE, TZ_DATABASE_CODE …
… … … … … …
MDM Repository Security Tables (part 1)
For the specific user, thisdefines the where clausefragment …
and the column sets to includewith the default columns.
USER_NAME MDM_VPD_TABLE_PK VPD_PREDICATE LAST_UPDATE DSA_NAMESOME_USER 1 ORG_STATUS = 'Active' … …… … … … …
USER_NAME MDM_VPD_COL_SET_PK LAST_UPDATE RULESOME_USER 68 … org_sensitive_0_1_flag=0… … … …
MDM Repository Security Tables (part 2)
• Now, when the SQL statement is executed:SELECT * FROM MDM_ORGOnly those rows / columns authorized are returned in the result set.
We’re done!
• Reference Data Sets– Languages– Geopolitical Locations (Countries, Regions, etc.)– Currencies– Exchange Rates
• Master Data Sets– Organizations– Leaders– Employees & reporting hierarchy – in progress– Physical Facilities (Churches, Temples, Seminaries, etc.) – in
progress
What do we have in our MDM?
• We replicate authorized master data to 54 other productions systems (230+ instances) daily.
• Developers become familiar with the canonical models for master data, which reduces the development cost of using master data.
• Web services deliver master data from the MDM repository.
• It takes about 20 minutes to provision master data to a new consumer.
• Changes in source systems are completely transparent to the consuming systems.
• Reduce number of DB links to source systems.
How successful is our MDM?
• Questions & Answers
• Pablo Riboldi Solution Manager for Information Governance & Quality [email protected] The Church or Jesus Christ of Latter-day Saints Visit us at www.mormon.org
Thank you!