scribe insight 02 need for speed
TRANSCRIPT
CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved.
Need For Speed
Message Queuing vs. Bulk & Upsert Capabilities
CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved. 3
Session Abstract
There was a time when including message queuing as part of your Insight integration design was the recommended design pattern to get the most “bang for your buck”. But with new bulk and upsertcapabilities available in endpoints and in Insight now you can get better performance by using those capabilities instead of message queuing. This session will focus on when you would use bulk capability instead of message queuing, and we will even teach you a thing or two about using bulk.
CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved. 4
The Pallet
Bulk LoadMulti
ThreadingParallel
Upsert PreSeekMessage Queue
CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved. 5
Dynamics CRM Bulk Load
Scribe Insight
Payload(200
Rows of Data)
AdaptersCRM
CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved. 6
Bulk Setting
Dynamics CRM only takes 250
at a time
Best for Insert operations
SQL???Challenges
with variable steps
CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved. 7
Scribe Fast Load – Process Threading
Workbench
Payload(1000
Rows of Data)
CRM Adapter
TargetCRM API
DLL
Dynamics CRM
TCP Threads
Process Threads
CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved. 8
Parallel Workbench
0
500
1000
1500
2000
2500
3000
3500
0
5
10
15
20
25
30
35
40
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Worckbench to Error
CPU Error
CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved. 9
Optimal Settings
Turn Off Any Plugins / Workflows
Let CRM automatically generate GUID (3% improvement in write)
Move Source to Microsoft Azure VM (15% over on Premise, and Amazon)
Run Workbench in Parallel 7.9
Run Workbench from Command Line
You will break CRM at pome point
Migration of Large Data into CRM
CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved. 10
Parallel OperationInsight
Payload(200
Rows of Data)
AdaptersCRM
Payload(200
Rows of Data)
AdaptersCRM
Payload(200
Rows of Data)
AdaptersCRM
64 Times
CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved.
Upsert
Scribe Insight Provide Keys via Xref or Lookup if Null then Insert.
Moving to Server Side Operation with Microsoft Fall Update 2015
CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved.
Pre Seek - Option
We thought we were helping…….
SQL impact is very high in CRM 2011 and 2013
Turn on if not using target variables
CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved. 13
Message Queuing
Transaction Processing
Reserved Processors
(7.9)
Message Priority In the
Queue
Recovery / Retry
Error Management
CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved.
Integration Patterns/Use Cases
One time Migration
Update Million Records a day
Ongoing Transactions
CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved. 15
Massive Data Migration
Stage SQL or Source the SQL directly from Old CRM
CRM On Prem
• Turn off Auditing
• Workflows
• Plugins
• SQL Database Indexing during
Read Microsoft White Paper
Stage Migration weeks ahead Then Build Integration
CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved. 16
By The Numbers 100,000 Account Records
Design Rows Per Minute Time Complete
Queue -32 Message Processor
1 Workbench File
10 Workbench File
20 Workbench File
Scribe Online
Entities have different performance
characteristics
Have a large sample –10,000 records
minimumWatch CRM server
CRM Online Limits 2 threads per organization
CONFIDENTIAL | © 2015 Scribe Software Corporation. All rights reserved. 17
Pierre’s Tips and Tricks
Use Staging Tables for Large Migration Jobs.
Resolve lookups references in the source query
Chain The Workbench files
Run multiple Workbench files
17