seattle bi meetup bi & etl @ big fish

SEATTLE BI MEETUPBI & ETL @ BIG FISH

April 2nd, 2014Emre Motan

About the Speaker Emre Motan BI Engineer, Big Fish Been in Seattle 1.5 years, previously in

Chicago Involved in BI community

Chicago SQL BI PASS chapter UW BI Certificate Program BI Over Beers TDWI

Random: Basketball, co-rec sports, greyhounds

About the Seattle BI Meetup Started in 2012, I took over after period

of inactivity Meetups will be monthly Primary goal is to educate

Topics will be wide-ranging but more technical

Networking is encouraged Speakers will be people who use

technologies

About BI Meetup (cont.) Looking to develop relationships with

willing speakers, venues, and sponsors Desire is to have meetings in different

venues each month As part of hosting, it would be nice to

“sponsor” with food & drink

About Big Fish World’s largest producer

of casual games Core business used to

be PC-based “Hidden Object” games

Now pushing deep into Mobile space with titles such as Big Fish Casino and Fairway Solitaire

Today’s Story BI/DW Implementation ETL Framework ETL Development BIML

BIDS Helper Mist BIML IDE w/ Hadron BIML Compiler

Summary

BI INFRASTRUCTURE

April 2nd, 2014Emre Motan

BI Infrastructure

What do we want from our ETL? Minimize manual coding errors Minimize time spent on boilerplate Support agile software development

practices Use software development best

practices Pair / collaborate Source control / diff changes Auto-generate / automate

Reduce effort spent on operations and support

ETL FRAMEWORK

April 2nd, 2014Emre Motan

ETL Framework Overview Cycles (subject areas, master package)

Jobs (restart-able units of work, package)

Steps (logic units, Data Flow / Execute SQL / …)

12

Job AStep 1Step 2Step 3

Job BStep 4Step 5Step 6

Job CStep 7Step 8Step 9

Job DStep 10Step 11Step 12

Job EStep 13Step 14Step 15

CycleLog Start

Log Success

Do Work Log Fail

13

1st RunJob AStep 1Step 2Step 3

Job BStep 4Step 5Step 6

Job CStep 7Step 8Step 9

Job DStep 10Step 11Step 12

Job EStep 13Step 14Step 15

SuccessFailureDidn’t Run

14

2nd RunJob AStep 1Step 2Step 3

Job BStep 4Step 5Step 6

Job CStep 7Step 8Step 9

Job DStep 10Step 11Step 12

Job EStep 13Step 14Step 15

Completed Successfullylast run; Skip Next Time

Rerun on Failure

Choose to Retry because of a logicaldependency

Choose to SkipRun because it didnot run last time

Example Job Flow

Data WarehouseStaging ODS Reporting

LayerStg_sales

Stg_customers

Stg_products

Stg_payment_methods

…

ods_sales

ods_customers

ods_products

ods_payment_methods

…

fact_sales

dim_customers

dim_products

dim_payment_methods

…

Source Systems

Ecommerce

ELT Server

Extract/Load

ETL Framework Summary Framework is injected at compile time

Uses SQL Server 2008 R2 via stored procedures Logical units automatically logged Event handlers automatically added Metadata based alterations for flow control (skip/restart)

Metadata based balances and validation scripts to detect warning/error conditions

Variables and values stored for use in ETL Metadata based run-time Alterations for flow control

(e.g. skip job, skip step) Custom tools to administer ETL infrastructure and

metadata

BIML ECOSYSTEM

April 2nd, 2014Emre Motan

Language and Compiler BIML (BI Markup Language)

Lightweight XML dialect Represents SQL Server BI Stack objects

(SSIS, SSAS, SQL Server) Works like ASP.NET / PHP (combines

declarative & imperative language) Hadron

Compiles BIML to SQL Server BI Stack artifacts

Called via MSBuild, Mist, …

Tools BIDS Helper

Free, open-source extension to BIDS Code in BIML, then generate SSIS Subset of functionality

Mist IDE Graphical & Text Based Editors Transformers Extensions

ETL DEVELOPMENT

April 2nd, 2014Emre Motan

Big Fish BI Engineering We integrate a wide variety of data

sources We don’t develop in BIDS/SSIS We code in BIML, compile in Mist or via

Hadron directly

BI Engineering ETL Development Flow

Develop BIML locally, committed to SVN Generate most of the code besides

business logic Run code validations before / during

compile Compile BIML during development or on

deployment to ELT boxes Produce SSIS packages Handle pushing to target environment

Kick off Cycles using Job Scheduler or DtExec

DEMOS

April 2nd, 2014Emre Motan

Demo: Simple SSIS Package Demo of BIMLScript1.biml

Show BIDS environment Show BIML Generate SSIS Run SSIS

Demo: Programmatic BIML Demo of BIMLScript2.biml

Introduce .NET addition to BIML script Describe what we’re doing with getting

tables from DB Describe how we’ll loop over each table,

and then each column of table, to generate insert commands

Generate SSIS Run SSIS

Demo: Mist Visual Designer Show audience visual designer of one

job Select elements to see visual designer We don’t use visual designer very often

since most code is auto-generated now and we have established patterns

Demo: Mist Project Show Mist environment with sample

cycle Show cycle file with one job Show job file Show metadata for sample table (source,

ODS) Show Extensions

Demo: Auto-generating ETL One substantial accelerator of our work is auto-

generating ETL for new extracts, loads, and processing

BIML representation of table Columns, business keys, primary keys, data types Annotations like ETL pattern required (full load,

incremental new, incremental new/updated) Only need to code transformation logic, all

boilerplate is auto-generated BimlScript to autogenerate boilerplate ETL code

Demo: NZ SSIS Console Cycle/Job/Step Status Alterations Variables Deploy / Execute Cycles

SUMMARY

April 2nd, 2014Emre Motan

Why did Big Fish choose BIML? Non-standard technology needs

(extensibility) Ease of developing and maintaining ETL to

leave more time for high business value work Plenty of people with SSIS experience in

Seattle Cost effective Organization already supported SQL Server

and SSIS Happy developers

Q & A

April 2nd, 2014Emre Motan

THANK YOU

April 2nd, 2014Emre Motan

Resources BIDSHelper.codeplex.com

Varigence.com

BimlScript.com

Biml Tutorials by Andy Leonard