datax platform for marketing integration
TRANSCRIPT
DataX Marketing Integration PlatformSimplifying Marketing Technology Integrations
DataX Whitepaper w w w. s y p t u s . c o m A u g , 2 0 1 6
As the number and complexity of marketing integration processes increases, the utility of ad-hoc,
manually coded integration scripts decreases and the requirement
becomes that of adopting a holistic integration approach that addresses both functional and non-functional
requirements in a flexible, scalable, and cost-effective manner.
The DataX Marketing integration platform addresses this challenge by allowing Marketers to rapidly build,
deploy, configure and control simple to complex integration processes at scale and with minimal internal IT knowhow.
With marketing data increasingly spread across multiple silos, the need for effectively integrating multiple tools has rightly become one of the top priorities for companies across the board. Effective marketing technology integrations address both functional (business logic) and non-functional (speed of development, ease of business use, scalability, error handling, audit logging, job monitoring etc.) requirements in a coherent, cost-effective manner. A tactical, ad-hoc approach to building such integrations is to deploy custom coded scripts (typically php, python etc. code run manually or via crude cron jobs) but such piecemeal approaches quickly run out of utility as the number of integrations increases or when non-functional constraints outlined above become too difficult to ignore.
It is in this regard that we attempt to build a case for using a holistic integration platform that is built from grounds up for use in marketing and which provides Marketers with an all encompassing tool/solution to quickly launch, configure and monitor simple to complex marketing integration processes. We introduce the conceptual architecture and key features of our DataX Marketing Integration Platform-a robust technology infrastructure designed from grounds-up for deploying and running marketing integration workflows at scale.
Before proceeding with building an outline of the DataX platform, here is a brief overview of the challenges associated with use of ad-hoc, tactical integration approaches using custom coded scripts.
Complexity of manual codingAs data processing gets more complex, it becomes increasingly difficult to implement manual code for writing data transformation rules. Imagine the amount of code (and hence development/testing time) required for manually parsing a complex JSON in php and where multiple lookups are required for implementing the final source to target data mapping. Using a GUI based environment with drag and drop features for creating data transformations and which has access to a large number of pre-built components for common utility tasks is a far more preferable approach in terms of speed of development and long term code maintenance.
Lack of holistic error handling and notificationMore often than not, the scripts are written for performing ad-hoc tasks with little regard to handling business logic or technical API level errors. Consider a simple integration that pulls out Facebook Ad Insights data into a data warehouse on a daily basis. Facebook implements rigorous API rate limiting and a poorly designed script would simply stop execution midway when such an error occurs. What happens to the potentially large amount of data that was already pulled? It may or may not have been sent to target but notwithstanding that, the application needs to implement additional checks to accept/discard partial data loads. For example, when you restart, do you fetch data for the whole day again? If not, how do you remember the last record fetched successfully? In some business scenarios, partial data loads may not even be acceptable. Sure, error handling can be implemented for each individual script but what if you have tens if not hundreds of such scripts running across your marketing tool stack?
Lack of end user controlRunning command line operations such as changing a cron schedule, manually editing script configuration rules, checking audit log entries, cleaning up database tables etc. are just some of the tasks required to be routinely conducted when running ad-hoc, hand coded scripts. Understandably, the appetite for such solutions from business user community is pretty low.
Pre-built integrations with leading Marketing platforms
Lack of advanced triggering optionsTwo types of triggers are commonly encountered in commercial marketing integrations. Time and event driven. Time based triggering is a simple mechanism for launching a script based on a time schedule. Most hand coded scripts are implemented using this mode (via cron jobs or task scheduler). Event driven triggering tends to be more complex in the sense that executions are triggered in response to external trigger events. In theory, trigger based execution can be implemented using polling (whereby a scheduled listener script polls an external data source indefinitely) but this approach is severely inefficient and can often result in API quota limit violations.
Lack of easy monitoring and management capabilityWith hand coded scripts, there is usually no centralized script monitoring and management GUI that business users can use to create, configure and monitor the status of their script execution runs. This is by far the biggest drawback of adopting an ad-hoc integration development strategy and the total cost of ownership of such an approach over a longer duration of time actually works out to be significantly higher.
The discussion above hopefully brings out the need for deploying a standard software tool/solution for implementing such marketing technology integrations. This is the domain of Extract Transform and Load tools such as Talend, Informatica, Pentaho or in some cases EAI/ESB tools. Large enterprises that already deploy such middleware should certainly consider using it to standardize their integration development processes. However, using a dedicated, expensive software tool may not be an option for all. True, most ETL tools provide robust capabilities for building and managing complex integrations but the effort, time and costs involved in actually developing and managing these integrations would likely ruin the business case for using such tools in the first place.
This is why we developed DataX. We define it as a Marketing Integration Platform with 4 key components.
App RepositoryA collection of out of the box integrations covering popular
marketing tools and providing basic integration use cases. To deploy these, Marketers simply provide their login credentials
to end-points and setup some basic configurations (e.g. scheduling, fields to integrate etc.). Each app is
designed with a specific use case in mind and provides users with a number of configuration and scheduling options.
App Development FrameworkA flexible development framework that allows us to rapidly
customize pre-built integrations or connect to new APIs and implement complex data processing logic with minimal
coding. Built on top of the industry leading Talend ETL platform (www.talend.com), this framework allows us to
quickly create executable scripts (java code) encapsulating integration business logic.
Platform RuntimeA set of components that work together to provide robust
non-functional capabilities to the executable scripts generated by App development framework. By decoupling features such as error handling, audit logging, scheduling, configuration management and execution monitoring from
actual business logic, the platform provides a common technology infrastructure for efficiently running integrations
at scale.
Management ConsoleThis is a web based GUI that allows Marketers to easily
configure, run/stop/schedule jobs and monitor job executions using feature packed dashboards. Rich access control
features allow setting granular permissions to control script configuration and execution.
For more information about DataX and to get a guided tour of the product, integration demos, pricing options and deployment details, please contact us today.
Test drive sample integrations
See sample integrations in action for major tools including Facebook Ads, Twitter, Google AdWords, Google Analytics, Eloqua Marketing Cloud, Optimizely and many more. Get first hand insight into customization opportunities and key runtime features.
Find out about pricing options
DataX platform follows a simple, pay monthly pricing model with different tiers depending upon the number of integrations activated. Find out more about our tiered pricing structure and details of integrations available for each.
REQUEST A DEMO
USA OFFICE111, North Market Street Suite 300,
San Jose, CA, 95113(408) 786-5219
INDIA OFFICEPlot 23, Maruti Industrial Area, Sector 18,
Gurgaon, Haryana [email protected]
HOW IT WORKS
BUILD
Our team of expert consultants work with you to develop your
integration using our proprietary implementation method and software tools.
Some integrations we provide are plug and play, and can be configured to run without any
customization.
CONFIGURE AND CONTROL
Each integration we build has a web based configuration
interface that allows you to easily change runtime
parameters and also control execution schedule.
DEPLOY
The integrations we developare technically implemented as
JAVA programs. In the deployment step, these scripts
are deployed on the cloud hosted DataX platform.
MONITOR
Robust logging and error monitoring using web based
dashboard to help you quickly debug integration runs.