idug - syncsort sponsorship march 29, 2018...idug - syncsort sponsorship march 29, 2018 vincent...
TRANSCRIPT
IDUG-SyncsortSponsorshipMarch29,2018
VincentMannolini–SeniorAccountExecutive(O)8455356937(M)[email protected]
TrustedIndustryLeadershipFor“50”Years
Syncsort 2
800+Experienced&Talented
DataProfessionals
>7,000CustomersWorld-Wide
196850YearsofMarketLeadership
&Award-WinningCustomerSupport
86ofFortune100areCustomers
3xGrowthOrganic/Acquisition
ThegloballeaderinBigIrontoBigData
TheSyncsortSolutionSuiteDifferentiatedProductPortfolio&TechnicalExpertise
3 Syncsort
DataInfrastructureOptimization
DataAvailability
DataIntegration
DataQuality
Market-leadingdataqualitycapability
Best-in-classresourceutilizationandperformance,onpremise
orinthecloud
#1inhighavailabilityforIBMiandAIXPowerSystems
Industry-leadingmainframedataaccessandhighestperforming
ETL
• TrilliumSoftwareSystem• TrilliumQualityforBigData• TrilliumPrecise• TrilliumCloud• TrilliumGlobalLocator
• MFX®forz/OS• ZPSaverSuite• EZ-DB2• EZ-IDMS• DL/2• ZenSuite
• athene®• atheneSaaS®
• Distributed&z/OS
• MIMIXAvailability&DR• MIMIXMove• MIMIXShare• Quick-EDD/HA• iTeraAvailability• Enforcive,Cilasoft,CSI
• Ironstream®forSplunk• Ironstream®TransactionTracing
• DMX&DMX-h• DMXChangeDataCapture
BigIrontoBigDataAfast-growingmarketsegmentcomposedofsolutionsthatoptimizetraditionaldatasystemsand
delivermission-criticaldatafromthesesystemstonext-generationanalyticenvironments.
Industry Leader in Mainframe and Distributed Software Products and Professional Services
DataReplicationTechnicalOverview
John Gay Sr. Manager, Sales Engineering Americas [email protected] 817-291-6440
INDUSTRYTRENDSANDPRACTICES
Today’sBusinessesHaveMultipleDatabases
Multipledatabasesarethenorm– Mergeroracquisition– Choiceofmultipleappsordatabasesfor
bestofbreedsolutions– Combinationoflegacyandnewdatabases– Multi-organizationsupplychain
ITinfrastructuresareheterogeneous– Databaseplatforms– Operatingsystems– Hardware
7
Does your organization rely on multiple databases?
Yes No I don't know.
83%
10% 8%
73% of those with multiple databases share data among databases
Does your organization share data between multiple databases?
Source: Vision Solutions 2017 State of Resilience Report
VariedBusinessandITGoalsforDataSharing
OffloadingreportingtoprotectProductionSystemProvidedataformaintenance,backup,ortestingwithoutproductionimpactConsolidatingdataintoreal-timedatamartsSharingdatawithoutsideconsumersMigratingdatatonewdatabasesorapplicationversionsReplatformingdatabasestonewdatabaseoroperatingsystemplatforms
8
Source: Vision Solutions 2017 State of Resilience Report
For what business purpose does your organization share data between databases?
Consolidating data from multiple sources into a single data warehouse
Performing Business Intelligence on data offloaded from the production
Reporting on data offloaded from the production data
Integrating data used by applications/databases selected for different
Synchronizing data between distributed applications
Dashboard generation
Testing on offloaded data
Feeding real time data to customer or partner facing applications
Running business processes on offloaded data
Master Data Management (MDM)
I don’t know
Other (please specify)
0% 10% 20% 30% 40% 50% 60% 70%
TRADITIONALMETHODSOFDATASHARING
TraditionalMethodsforSharingData
Directnetworkaccess– Reportingonproductionserversacrossthe
networkduringbusinesshours– Issue:Negativelyimpactsnetworkanddatabase
performance–resultinginusercomplaints!
Off-hoursreportsandextractions– Runreportsoff-hoursor
performnightlyextractprocessestomovedatatoareportingserver– Issue:Businessoperatesonagingdatauntilnextextraction– Issue:Difficulttofindacceptabletimetoperformanextraction
ETL(Extract-Transform-Load)Processes– FTP/SCP/filetransferprocessesor
ManualscriptsorBackup/restoreorIn-housetools
– Issue:Periodic,notreal-time,deliveryofdata– Issue:Laborintensivetocreateprocessesandtools– Issue:Expensivetodevelopandmaintain– Issue:Pronetoerrors
10
In-HouseETLScriptsandProcessesAreNotFree
Upfrontdevelopmentcosts– Developmentofcodetoperformdatabaseextraction,transformation,andload– Additionalrequirementsforadditionalpairings,schemas,etc.
Testsystemexpenses– Hardwareandstorageresources– Databaselicensesfortestsystems– Add-onproducts,e.g.gateways
Maintenancecosts– Ongoingenhancementsforalteredschemas,additionalplatforms– TestingnewdatabaseandOSreleases– Crosstraininganddocumentationtoreduceturnoverrisk
Lostopportunitycostsforotherinitiatives
11
REAL-TIMEREPLICATIONOVERVIEW
MIMIXShareforEasy,AutomatedDataSharing
Breaksdownbarriersbetweendatabases– Sameordifferentdatabasemanagementsystems– Sameordifferentoperatingsystems– Physical,virtualorcloudplatforms– Acrossanydistance
Makesdatasharingeasy– Replicatesdatabasechangesinrealtime– Transformsandenhancesdataduringreplication– Supportsleadingdatabaseandoperatingsystems– Offersavarietyofreplicationarchitectures– EasygraphicalUI–noprogrammingrequired!
Quicklyreturnsyourinvestment– Strongerdecisionmaking– Greaterbusinessproductivity– Abilitytochoosemorecost-effectiveinfrastructure– FreesITtofocusonotherbusinessinitiatives
13
ChangeDataCapture(CDC)forReal-TimeReplication
ChangeDataCapture(CDC)capturesdatabasechangesimmediatelyandquicklyreplicatesthemtoanotherdatabase(s)
Onlychangeddataisreplicatedtominimizebandwidthusage
Automaticallyextracts,transformsandloadsdataintotargetdatabasewithoutmanualinterventionorscripting
Ensureswriteorderconsistencyandguaranteeddelivery
Ensuresdataintegritywithconflictresolutionandcollisionmonitoring
Enablestrackingandauditingoftransactionsforcompliance
14
Real-Time Replication
with Transformation
Change Data Capture (CDC)
Conflict Resolution, Collision Monitoring,
Tracking and Auditing
Source Database
Target Database
ReplacesManualProcesses
Point&clickgraphicaluserinterface
Singleviewofdataacrossdatabasesandoperatingsystems
Simple,model-basedconfiguration
Automaticallycreatestargettablesfromthesourcetabledefinition
80+pre-built,click-and-godatatransformations
TransformationscanbeaddedthroughJava-likescripting
Noprogrammingrequired
15
SupportsaBroadRangeofPlatforms
16 * Target only
Leading Operating Systems Leading Databases
• IBM i
• IBM AIX
• HP-UX
• Solaris
• IBM Linux on Power
• Linux SUSE Enterprise
• Linux Red Hat Enterprise
• Microsoft Windows, including Microsoft Azure
• IBM DB2 for i
• IBM DB2 for LUW
• IBM Informix
• Oracle
• Oracle RAC
• MySQL*
• Microsoft SQL Server
• Teradata*
• Sybase
• PostgreSQL*
SupportsaBroadRangeofPlatforms
17
• IBM DB2 for z (DMX-CDC)
• IBM DB2 for i
• IBM DB2 for LUW
• IBM Informix
• Oracle
• Oracle RAC
• Microsoft SQL Server
• Sybase
High-LevelArchitecture
18
Log-BasedDataCapture
19
Did you know?
MIMIX Share can leverage published Log or Journal standards to identify and capture the change before copying to the MIMIX Share Queue.
The MIMIX Share Queue ensures that data integrity is maintained and zero data loss occurs in the event of a dropped connection during file transmission.
Queue
Source DBMS
Retrieve/Transform/Send
Apply
Changed Data
1
1 3
4
Log / Journal
Change Selector
2
1. Use of transaction logs or triggers eliminates the need for invasive actions on the DBMS.
2. Selective extracts from the logs and a defined queue space ensures data integrity.
3. Transformation in many cases can be done off box to reduce impact to production.
4. The apply process returns acknowledgment to queue to complete pseudo two-phase commit.
Target DBMS
FlexibleReplicationOptions
20
One Way Two Way
Cascade
Bi-Directional Distribute
Consolidate
Choose a topology or combine them to
meet your data sharing needs
GuaranteesInformationAccuracy
21
Ensures ongoing integrity § Changes collected in queue on source § Moved to target only after committed
on source § Ensures write-order-consistency retained § Queues retained until successfully applied § No database table locking
Ensures failure integrity § Automatically detects communications errors § Automatically recovers the connection and
processes § Alerts administrator § No data is lost
SMTP Alerting
AccurateTracking&DataAuditing
22
Audit Journal Mapping tracks all updates and changes § Records
§ Before and after values for every column § Type of transaction § Type of sending DBMS § Table name § User name § Transaction information
§ Records to flat file or to database table § Can assist with SOX, HIPPA audit requirements
Detects and resolves conflicts § Maintains data integrity
Model verification § Validates date movement model
LetsYouShareExactlyWHATYouNeed
Filtersdeterminewhatdatagetsmoved– Selectspecificcolumnandtable
– Selectspecificrowsandtable
23
LetsYouTransformtheDataExactlyHOWYouNeedTo
Transformsdataintousefulinformation 80+built-intransformationmethods Fieldtransformations,suchas:
– DECIMAL(5,2)– nulltostring(ZIP_CODE,'00000')
Tabletransformation,suchas:– Columnmerging– Columnsplitting– Creatingderivedcolumns
Customlookuptables CreatecustomdatatransformationsusingpowerfulJavascriptinginterface
24
MIMIXSHAREUSECASES
IBM System i DB2 Lawson M3 (Movex)
MS SQL Server
Query reports
Data Warehouse load
Real time CDC replication
with transformation
Reduce CPU and I/O overhead on production system
improve user response times
Many cost effective tools available on MS SQL server
platform for query reports
Data is already partially ‘scrubbed’ and available
for loading data warehouses and data marts without performance impact on
production system
Production System Offload Query System
UseCase:OffloadReportingfromProductionDatabase
26
UseCase:CentralizedReporting
27
Casino 1
IBM System i DB2
Casino 2
IBM System i DB2
Casino 3
IBM System i DB2
Casino 4
IBM System i DB2
Casino 5
IBM System i DB2
Casino 6
IBM System i DB2
Single Data Warehouse Database Windows Cluster MS SQL Server
Customer loyalty Amounts paid Amounts won
Time at the table Time at the machine
Business intelligence
Real time CDC replication
with transformation
1. Customers enter new banking transactions on line.
They get captured in SQL Server
Microsoft SQL Server IBM System i DB2
On-Line Banking
replicates the transactions in
real time to DB2/400
3. A back-office batch application
processes incoming transactions and updates
data.
4. replicates the processed transactions back to SQL Server
in real time.
1
2
3
4
5. Customers view processed transactions
on-line
5
Bi-directional replication
UseCaseApplicationIntegration
28
Replicated data is merged into a single, consolidated reporting database in the cloud (AWS, Azure, etc.)
3
UseCase:ReportingintheCloud
29
End users enter data into their
local applications.
Target
MIMIX Share captures, transforms,
and replicates database changes in
real-time.
Business Intelligence apps and reporting tools access the
consolidated information from the cloud-based database.
1
2
4
Source
UseCase:DatabaseMigration
30
IBMiDB2fori
JDE(standard)
OldSystem
IBMiDB2fori
JDE(Unicode)
NewSystem
Manufacturing Company
Real time CDC replication
with transformation
UseCase:DatabaseReplatforming
31
IBMiDB2
OldSystem
SunOracleRAC
NewSystem
Large Insurance Company
Two-way Active-Passive replication to enable
application server switching
Near-zero downtime for cutover to new
systems
Transformation between different OS and database
platforms Users are moved to new server in phases over a
period of time
UseCase:GradualDatabaseReplatforming
32
IBMiDB2
OldSystem
WindowsSQLServer
NewSystem
America II Corp
Active-Active replication eliminated need for hard
cutover and enabled partners to move back and
forth between systems
True zero downtime for migration to new
systems
Transformation between different OS and database platforms with completely
different schemas
100s of partners moved to new server after training
nd at their own pace
WHATTOLOOKFORWITHCDCREPLICATION
WhyMIMIXShare?
1. SuperiorDataIntegrity– ZeroDataLossEVER– AutonomicRecovery
• Network• Server• Data
2. SuperiorAdministration– GraphicalInterface– Single-StepPromotiontoProduction– ReplicationSchemaLanguage(RSL)– ModelBased
34
WhyMIMIXShare?
3. SuperiorPerformance– FastThroughput– Off-BoxOption
4. SuperiorMonitoring– Client,Web-based,Command-line– Network,Server,Database,Transactions– ProactiveSMTPE-MailAlerts
5. IntegrationwithOtherProcesses– SharingofJournals/Logs– CoupledRole-Swap– Downstreamdataprocessing
35
36
ThankYou!
37
Tak Bedankt
Takk
Kiitos Merci Danke Grazie
Dziękuję Multumesc Gracias Tack Hvala
감사합니다 謝謝您 Благодаря Thank You
www.visionsolutions.com
Terima Kasih Köszönet
MappingColumns
38
column mappings
Customer number Customer name
Numeric (10)
Customer address line 1 Customer address line 2
Alpha-numeric (10) Alpha-numeric (25) Alpha-numeric (25)
Customer address line 3 Customer address line 4
Alpha-numeric (25)
Customer telephone Customer credit limit
Alpha-numeric (25)
Numeric (10) Numeric (10,2)
Customer address line 5 Alpha-numeric (25)
Column Name Data Type
Target Server
Database Server MS SQL Server
customer_master (SQL table)
CUNUM
CUNAM
Numeric (10)
CUAD1 CUAD2
Alpha-numeric (20) Alpha-numeric (25) Alpha-numeric (25)
CUAD3 CUAD4
Alpha-numeric (25)
CUTEL
CUCLM
Alpha-numeric (25) Numeric (10)
Numeric (10,2)
Column Name Data Type
Source Server
Database Server DB2
CUSTPF
table mapping
HowDoesYourBusinessNeedtoMoveData?
Enablingqueries,reports,businessintelligenceoranalyticswithoutproductionimpact?
Populatingcentralizeddatabases,datamartsordatawarehouses?
Feedingreal-timedatatoemployees,customersorpartners?
Keepingdatafromsiloeddatabasesinsync?
Reducingtheimpactofdatabasemaintenance,backuportesting?
Movingdatatonewdatabases?
Consolidatingdatabases?
Replatformingtonewdatabaseoroperatingsystems?
39
AdditionalUseCases
40
ERP SYSTEM
Customer Orders Payment Details
Product Catalogue Price List
eCOMMERCE & WEB PORTALS
DR / BACKUP
DATA EXCHANGE WITH OUTSIDE VENDOR
(FLAT FILE)
Outside Vendor
TEST & AUDIT ENVIRONMENT
MIMIXShareTransformsBusiness
Breaksdownbarriersbetweendatabases– SameordifferentDBMSorOS– Physical,virtualorcloudplatforms– Acrossanydistance
Makesdatasharingeasy– Replicatesdatabasechangesinrealtime– Transformsandenhancesdataduringreplication– EasygraphicalUI–noprogrammingrequired!
Quicklyreturnsyourinvestment– Strongerdecisionmaking– Greaterbusinessproductivity– Abilitytochoosemorecost-effectiveinfrastructure– FreesITstafftofocusonotherinitiatives
41
ConnectwithVision!
42
Website:visionsolutions.com
Twitter:twitter.com/VSI_Power
@VSI_Power
Facebook:facebook.com/VisionSolutionsInc
YouTube:youtube.com/c/VisionSolutionsInc
LinkedIn:linkedin.com/company/vision-solutions
Blog:http://blog.visionsolutions.com
HowtoLearnMore
Readcasestudies
Downloadawhitepaper
ReadtheMIMIXSharetechbrief
Watchproductsdemoonourwebsite
Listentoanon-demandwebcast
ConnectwithVision’ssocialmediasites
43