change data capture for ssis in sql server 2012 in sql server 2012 ssis - allan... · who am i? •...
TRANSCRIPT
![Page 1: Change Data Capture for SSIS in SQL Server 2012 in SQL Server 2012 SSIS - Allan... · Who am I? • Allan Mitchell • Joint author on 2005/2008 SSIS Book by Wrox • Websites –](https://reader034.vdocuments.net/reader034/viewer/2022042517/5ac12e207f8b9ae45b8d0fbe/html5/thumbnails/1.jpg)
Change Data Capture for
SSIS in SQL Server 2012
![Page 2: Change Data Capture for SSIS in SQL Server 2012 in SQL Server 2012 SSIS - Allan... · Who am I? • Allan Mitchell • Joint author on 2005/2008 SSIS Book by Wrox • Websites –](https://reader034.vdocuments.net/reader034/viewer/2022042517/5ac12e207f8b9ae45b8d0fbe/html5/thumbnails/2.jpg)
Who am I? •Allan Mitchell
• Joint author on 2005/2008 SSIS Book by Wrox
•Websites
–www.CopperBlueConsulting.com
•StreamInsight and SQL Azure Advisory Boards
•Specialise in Data and Process Integration
•Microsoft SQL Server MVP
•Twitter: allanSQLIS
![Page 3: Change Data Capture for SSIS in SQL Server 2012 in SQL Server 2012 SSIS - Allan... · Who am I? • Allan Mitchell • Joint author on 2005/2008 SSIS Book by Wrox • Websites –](https://reader034.vdocuments.net/reader034/viewer/2022042517/5ac12e207f8b9ae45b8d0fbe/html5/thumbnails/3.jpg)
Why?
![Page 4: Change Data Capture for SSIS in SQL Server 2012 in SQL Server 2012 SSIS - Allan... · Who am I? • Allan Mitchell • Joint author on 2005/2008 SSIS Book by Wrox • Websites –](https://reader034.vdocuments.net/reader034/viewer/2022042517/5ac12e207f8b9ae45b8d0fbe/html5/thumbnails/4.jpg)
Data
Data, Data Everywhere
Everyone has it
Lots of it
Dispersed around our environment
Heterogeneous Sources
The Problem
![Page 5: Change Data Capture for SSIS in SQL Server 2012 in SQL Server 2012 SSIS - Allan... · Who am I? • Allan Mitchell • Joint author on 2005/2008 SSIS Book by Wrox • Websites –](https://reader034.vdocuments.net/reader034/viewer/2022042517/5ac12e207f8b9ae45b8d0fbe/html5/thumbnails/5.jpg)
Problem with data – Part 1
What is data?
Raw
Doesn’t mean a lot
Needs context
Needs enriching
Need to take next steps
Information
Knowledge
Wisdom
![Page 6: Change Data Capture for SSIS in SQL Server 2012 in SQL Server 2012 SSIS - Allan... · Who am I? • Allan Mitchell • Joint author on 2005/2008 SSIS Book by Wrox • Websites –](https://reader034.vdocuments.net/reader034/viewer/2022042517/5ac12e207f8b9ae45b8d0fbe/html5/thumbnails/6.jpg)
Problem with data – Part 2
Timeliness
When do people need it
How do they need it
“Out of date” data is next to useless
Not everyone needs real-time
Everyone should make “in date” their goal
![Page 7: Change Data Capture for SSIS in SQL Server 2012 in SQL Server 2012 SSIS - Allan... · Who am I? • Allan Mitchell • Joint author on 2005/2008 SSIS Book by Wrox • Websites –](https://reader034.vdocuments.net/reader034/viewer/2022042517/5ac12e207f8b9ae45b8d0fbe/html5/thumbnails/7.jpg)
Problem with data – Part 3
Quantity
No longer only for big Corps
1TB of disk is ~ $50
Moving it around
Windows of operation
Only transfer what we need
![Page 8: Change Data Capture for SSIS in SQL Server 2012 in SQL Server 2012 SSIS - Allan... · Who am I? • Allan Mitchell • Joint author on 2005/2008 SSIS Book by Wrox • Websites –](https://reader034.vdocuments.net/reader034/viewer/2022042517/5ac12e207f8b9ae45b8d0fbe/html5/thumbnails/8.jpg)
What are people doing?
As soon as data in source changes
Extract
Cleanse
Conform
Load
Only moving what needs to be moved
Right?
![Page 9: Change Data Capture for SSIS in SQL Server 2012 in SQL Server 2012 SSIS - Allan... · Who am I? • Allan Mitchell • Joint author on 2005/2008 SSIS Book by Wrox • Websites –](https://reader034.vdocuments.net/reader034/viewer/2022042517/5ac12e207f8b9ae45b8d0fbe/html5/thumbnails/9.jpg)
Why?
Hard
No tracking on Source Systems
Costs money to design
Need new skillsets
![Page 10: Change Data Capture for SSIS in SQL Server 2012 in SQL Server 2012 SSIS - Allan... · Who am I? • Allan Mitchell • Joint author on 2005/2008 SSIS Book by Wrox • Websites –](https://reader034.vdocuments.net/reader034/viewer/2022042517/5ac12e207f8b9ae45b8d0fbe/html5/thumbnails/10.jpg)
How?
![Page 11: Change Data Capture for SSIS in SQL Server 2012 in SQL Server 2012 SSIS - Allan... · Who am I? • Allan Mitchell • Joint author on 2005/2008 SSIS Book by Wrox • Websites –](https://reader034.vdocuments.net/reader034/viewer/2022042517/5ac12e207f8b9ae45b8d0fbe/html5/thumbnails/11.jpg)
CDC in SQL Server 2012
• Introduced in SQL Server 2008 • Not the easiest thing to code • No integration with SSIS • Uses Transactional replication
Agent to read log LSNs
![Page 12: Change Data Capture for SSIS in SQL Server 2012 in SQL Server 2012 SSIS - Allan... · Who am I? • Allan Mitchell • Joint author on 2005/2008 SSIS Book by Wrox • Websites –](https://reader034.vdocuments.net/reader034/viewer/2022042517/5ac12e207f8b9ae45b8d0fbe/html5/thumbnails/12.jpg)
Setup
• Enable DB for CDC • sys.sp_cdc_enable_db
• Enable table for CDC • sys.sp_cdc_enable_table
![Page 13: Change Data Capture for SSIS in SQL Server 2012 in SQL Server 2012 SSIS - Allan... · Who am I? • Allan Mitchell • Joint author on 2005/2008 SSIS Book by Wrox • Websites –](https://reader034.vdocuments.net/reader034/viewer/2022042517/5ac12e207f8b9ae45b8d0fbe/html5/thumbnails/13.jpg)
Components
• Control Flow • CDC Control Task: Manages state and
recovery of CDC Packages.
• Data Flow • CDC Source: Reads change data from
capture tables • CDC Splitter: A Conditional Split for CDC
![Page 14: Change Data Capture for SSIS in SQL Server 2012 in SQL Server 2012 SSIS - Allan... · Who am I? • Allan Mitchell • Joint author on 2005/2008 SSIS Book by Wrox • Websites –](https://reader034.vdocuments.net/reader034/viewer/2022042517/5ac12e207f8b9ae45b8d0fbe/html5/thumbnails/14.jpg)
Packages
• Initial: This package does the initial “Full Pull” • Incremental: This does the incremental extracts
![Page 15: Change Data Capture for SSIS in SQL Server 2012 in SQL Server 2012 SSIS - Allan... · Who am I? • Allan Mitchell • Joint author on 2005/2008 SSIS Book by Wrox • Websites –](https://reader034.vdocuments.net/reader034/viewer/2022042517/5ac12e207f8b9ae45b8d0fbe/html5/thumbnails/15.jpg)
CDC Control Task
• Initial Load • Mark initial load start • Mark initial load end • Mark CDC start
• CDC • Get processing range • Mark processed range • Reset CDC state
![Page 16: Change Data Capture for SSIS in SQL Server 2012 in SQL Server 2012 SSIS - Allan... · Who am I? • Allan Mitchell • Joint author on 2005/2008 SSIS Book by Wrox • Websites –](https://reader034.vdocuments.net/reader034/viewer/2022042517/5ac12e207f8b9ae45b8d0fbe/html5/thumbnails/16.jpg)
CDC Source
• All • All with old values • Net • Net with update mask • Net with merge
![Page 17: Change Data Capture for SSIS in SQL Server 2012 in SQL Server 2012 SSIS - Allan... · Who am I? • Allan Mitchell • Joint author on 2005/2008 SSIS Book by Wrox • Websites –](https://reader034.vdocuments.net/reader034/viewer/2022042517/5ac12e207f8b9ae45b8d0fbe/html5/thumbnails/17.jpg)
Demo
Putting it together