SnapLogic Best Practices:Tips and Tricks in 30 Minutes

Praneal NarayanJason Slater

2

Goals for the Tech Talk Series

1) Share Best Practices

2) Connect with the Cloud Integration Experts

3) Learn Tips and Tricks not Covered in Basic Training

Use the Chat Panels for Q&A

3

Agenda

Introductions– Praneal Narayan

– Jason Slater

Today’s Topic: 2 Approaches to Scale Your Integrations– Eliminate bottlenecks

– Improve pipeline performance

Wrap Up and Topics for the Next Session

4

Before We Get Started…..

Documentation

Download the following:– Snaplex

– JSDK

– Snap Packs

– Mac, Linux and Windows Installers

Community Board and Idea Exchange

Don’t forget to use our Developer Community

developer.snaplogic.com

5

Scaling Your Integrations

1) Child Pipelines– Split work into parent and child pipelines

– Distributes execution across multiple nodes

– Requires that you configure a ForEach Snap

2) Splitting/Joining data flow and (optionally) embedding a pipeline– Useful for performing things like multiple web service

calls in parallel

– Requires a Router Snap to fork data-flow

– Use the forked data, either directly using Snaps or by embedding a pipeline

Two approaches we’ll be talking about today…

6

Recommendation

Use Child Pipelines:– When the work you want to parallelize is compute

heavy

Use a Splitter and Join Snap:– When it’s computationally light

7

A Few Tips

✔ Parameters in child start with underscore (_) and you often need to use a Data Snap to copy it to a $ variable

With the Router Snap, use numeric ranges, hash character, etc.

Don't pass parameters to embedded pipelines, since they normally receive documents

Name each embedded pipeline instance so you can better understand run statistics later

Be aware of limitations such as throttling and concurrent requests

Be aware of mutually exclusive things you do in parallel pipelines, like writing to a file w/ overwrite option selected; DB is better approach.

✔

✔

✔

✔

✔

8

Connecting with the SnapLogic Team

@SnapLogic

Facebook/SnapLogic

www.SnapLogic.com

developer.snaplogic.com

9

Appendix – ForEach

Setting up your child pipeline using ForEach– Split work into parent and child pipelines

– Distributes execution across multiple nodes

– May alleviate CPU and NIC resource contention for computation-intense pipelines

– Pipelines have spin up/down overhead: may not be best approach for very short-lived pipelines

– Configuring the ForEach Snap• Choose the Snaplex: where child pipelines will run

• Synchronous vs. Fire-and-Forget

• Batch size for Synchronous Mode: how many child pipelines to run at a time

• Pass parameters from parent to child

10

Appendix – Router, Forked, Union

Use a Router snap to fork data-flow to:– Increase number of outputs to have more tines

– Choosing and using an expression for each tine of the fork

– Select First Match

– Inspect output of each tine to validate that your fork is working properly, given limited preview data

Use the forked data, either directly using Snaps or by embedding a pipeline– Depends on number of times, if you need to process the

forked data before/after

– If you modify an embedded pipeline's first or last snaps, you will need to re-add each instance to the parent pipeline

Use a Union snap to join the data– This merges the flow of documents back together