Download - JPMC - Integrated Consumer DW - ICDW
1 TCS ConfidentialCopyright © 2013 Tata Consultancy Services Limited
JPMC – Integrated Consumer Data Warehouse(ICDW)TCS Digital Enterprise – Analytics, Big Data and Information Management
PLEASE, CONTACT THE ACCOUNT OWNER FOR USING CUSTOMER INFORMATION AND REFERENCES.
2 TCS Confidential
JPMC
The ICDW Customer Migration initiative seeks to move Customer data and processes available in the Enterprise Data Warehouse (EDW) framework into the new ICDW framework
Migration of “Customer” subject areas (data & processes) into the ICDW framework Customer Data Mart (CST), Cross Customer Data (XCS), Relationship Data Mart (RDM), Weekly Relationship Mart (WRM), Prospect Data Mart (PDM), Ad Hoc Matching Model (AMM), Compliance Data Mart (CMP), Know Your Customer (KYC)
Migration of 76 CDW applications from IDW to ICDW environment. CDW Applications includes all load streams , datamarts, engines and other applications like Retail and Stored Value
Business Case
ICDW
ETL: Ab-Initio, Informatica, Reporting: Hyperion, Cognos, Tableau, SSRS Database: Teradata, DB2, Oracle, Data Modeling: Erwin
Technology Stack
Customer Overview
One of the largest Global Bank headquartered in USA. It is one of the leading global financial services firm in Asset & Wealth Management Chase Card, Services Commercial Banking, Investment Banking, Retail Financial, Services Treasury & Securities Services.
Business resources spend a disproportionate amount of time on data integration, movement and transformation rather than being focused on business analysis to drive insights
Cross-LOB analytical data is challenging to locate and access Lack of confidence in data quality & lineage impedes effective business
analysis causing business units to create their own transformations The analytical environment stability issues are causing extended timelines
and rework Data fragmentation has become expensive and complex to do repeatable
historical analysis
Challenges
ICDW solution brings in tactical & strategic solution, factory approach, strong governance focus
Process Driven – Reverse Engineering KIT Process Driven – Informatica Development KIT Process Driven - Data Modelling Development KIT Process Driven – QA Tool KIT Technology Driven – Ab-Initio RE Automation
TCS’ Solution ICDW program brings in benefits like Development Speed to Market: faster
analytic application development, improved analytic development Deeper Analytic Capability: sandbox enables greater experimentation,
greater ability to run scores Data Integrity: Single version of analytical truth, Stability & Performance: Platform designed for business analytics,
improved reporting timeliness( e.g. dashboards, management reporting) Scalability: Longer data retention where appropriate, continues
improvement
Benefits
3 TCS Confidential
JPMC - ICDW
Business Case
Scope comprises capturing data to the ICDW Integration area and Semantic layer.
Attributes from landing pad & Data Mart will be identified based on the attribute usage and develop the Integration model
Semantic views to be developed for all marts based on the transformation rules reverse engineered from EDW application
Scope comprises Migration of 76 CDW applications from IDW to ICDW environment. CDW Applications includes all load streams , datamarts, engines and other applications like Retail and Stored Value.
4 TCS Confidential
RE Inputs• AI Graphs• TWS / Shell Scripts• DB2 Components• Data Models• RE Templates
Reverse Engineering Automation
Data Lineage/
Usage Analysis
Reverse Engineering Factory
Inputs
Base Attribute Derivation
*Business Rules
TCS SME Review
BDW SME Review
EDW SME Review
Reverse engineering Deliverables
*Business Rules will be derived for all other marts except CDM
• BDWSME Review• EDW SME Review• Business User Review
• EDWSME Review• Business User Review
• EDW SME Review• ETL Architecture Review• Business User Review
• EDWSME• Business User
BRD,
Sou
rce
to T
arge
t Map
ping
Business Requirement Analysis
Deliverables
Interaction with Business / SME
Document Business Requirements / Use Cases Source to Logical Mapping Source to Target Mapping
Base Attribute Derivation(Need Basis) Reverse Engineering for• Validation of Attributes • Technical & some Business Validations Rules• Process Orchestration Rules
JPMC ICDW Solution Approach – Business Requirements
5 TCS ConfidentialTCS OwnedActivities
JPMC OwnedActivities
TCS supported Activities
Reverse Engineering (RE) & New Business Requirement Analysis
Team 1
Team 2
Team 3
Team 4
Data Modeling (DM)
Team 1
Team 2
Team 3
Team 4
Conceptual
Logical
Physical
Development(Modified Agile)
EDW to Source Mapping
Usage Analysis
1
2
3
4
Sprint Teams
Sprint Deliverables
Team Leads
RE Solution Lead
DM Leads
SME Reviews
SME Reviews
IQM Reviews
DRB Reviews
1
2
Integration Build
SemanticViewsBuild
Data Architect
Semantic
Code Reviews
Business Reviews
3
CST XCS
XPD/RDM PDM
WRM AMM
4
Integrated Testing
Test Reviews
CST XCS
XPD/RDM PDM
WRM AMM
User Acceptance Testing
UAT
Sign Off
5
ProductionDeployment
6
CST XCS
XPD/RDM PDM
WRM AMM
Warranty Support
7
Central Testing team to create and execute IST test cases
Business / SME / Architect execute UAT test cases
Sprint ExecutionDev
QA Test2 weeks
2 weeks
SOR Populate Structure & Data from SOR to
Clean Staging Data Structure to be made ready before
start of ETL Development
IQM Responsible for Data profiling,
classification and Quality Data Architects work with IQM team
during Data Modeling Phase
Infa Admin
Central QA
Supports code move from Dev to QA, IST, UAT and Prod Environments
Request sent from Dev team with code package details and approvals from ADM
Part of Agile team and responsible for QA and conducts sprint level testing
Responsible for Integrated testing across datamarts, UAT and Go-live testing
Factory Model
Factory Model
Modeling Deliverables
Analysis Deliverables
RE templates, guidelines Documents + New Requirements
DM templates, guidelines documents
Teradata Guidance for Physical Modeling DB space determination
Business Team Semantic view requirement analysis User Acceptance Testing
BusinessAnalysis
New Req. Doc.
JPMC ICDW Solution Approach – Development Approach
6 TCS Confidential
Analysis, SequencingAgile AwarenessBuilding the FoundationProduct planning, Decomposing the Migration into atomic stories
Sprint 0
ETL Team
QA Team
ETL & QA of ICDW Integration
ETL Team
QA Team
CDM
2 – 4 weeks Sprint Cycles depending on the scope
Shippable Software
ETL & QA of ICDW Integration
ETL Team
QA Team Sprint 1….n
Roles RequiredProduct Owner :- Ensures business valueScrum Master :- Facilitates team success by efficient running of process Scrum Team :- Builds shippable functionality per iterationPerforms various tests to ensures high quality
Process
Sprint Planning Meeting Daily Standup Sprint Burn down
Sprint Review & Retrospection
Sprint Goal Sprint Backlog Blocks List Product BacklogShippable Product
Production Deployment
Agile Team 1
CVM, EWS, FDM
HFDM, HPI
Agile Team 2
Agile Team 4
Modified Agile
ETL & QA of ICDW IntegrationETL Team
QA Team
Agile Team 3
COGDM
4 Agile Teams 20 day Sprints Each agile team has a
scrum master
ETL & QA of ICDW Integration
JPMC ICDW Solution Approach – Modified Agile Methodology
7 TCS Confidential
Challenges
Component Volumetric– Database Size– Interfaces
Challenges Listed in Slide#2
9 TCS Confidential
Logical Architecture
BI, Data Access & Query Tools
Met
adat
a (A
b In
itio
Met
a H
ub)
Integrated Consumer Banking Data Model
System Of Record Data
Public Area
Integration[INTG]
SOR
Staging[ACQN]
Layer
File System Landing Pad
Application
Access “Semantic”
[PRSN]
1:1 Views of Integrated Tables
Gov
erna
nce
Discovery Area
Interim Data Repository
Temporary User Workspace
ETL
& E
LT S
ervi
ces
(Info
rmat
ica
9 / T
erad
ata
TPT)
Shared Dimensions and Metrics
Application & Process Specific Views
Source System Representation of Data
Subject Area Interfaces
Load Ready Tables
Supporting Services
Extract Views
TABLES
VIEWS
WORK
IDR
IDR_CDI
Dat
a Q
ualit
y, C
ontr
ol, A
udit
& B
alan
cing
Arc
hite
ctur
e &
Dat
a M
odel
ing
Infr
astr
uctu
re S
ervi
ces
Ref
eren
ce D
ata
Man
agem
ent
Purpose Built Data Marts Analytical / SAS Datasets
Base and Context Security Views
Involved Party
Arrangement
Location
Classification
Event
Resource Item 3rd Party Data
Product
Reference
Audit & Control
11 TCS Confidential
Sample Screen Shots (esp. Reports)
Contact Performance Reporting• Used SSIS to extract and transform data from
heterogeneous sources like raw files and load it into the data warehouse
• Build Tools to do Data Validations for the staging data that comes from the Voice channel and then Push it into the warehouse
• Developed Summary tables for the Data that comes in• Tools to capture the details of Lookup (Dimensional
Values) Values and their Changes• Developed SSRS Reports for the Summarized Data
(Tabular, Matrix, Drill Down, Drill Through, Graphical)• Integrated SSRS reports with Web Portal build using ASP.
Net• Developed SSAS Cubes and Report Models for Adhoc
Users and for deeper dive into the data.
TCS Solution