biml for beginners - generating ssis packages with bimlscript (sqlsaturday gothenburg)

Biml for Beginners:

Generating SSIS packages with BimlScript

Cathrine Wilhelmsen

September 5th 2015

Session description

SSIS is a powerful tool for extracting, transforming and loading data, but creating the actual SSIS packages can be both tedious and time-consuming. Even if you use templates and follow best practices you often have to repeat the same steps over and over again. There are no easy ways to handle metadata and schema changes, and if there are new requirements you might have to go through all the packages one more time. It's time to bring the Don't Repeat Yourself principle to SSIS development.

In this session I will use the free BIDS Helper add-in to show you the basics of Biml and BimlScript, how to generate SSIS packages automatically from databases, how easy those packages can be changed, and how to move common code to separate files that can be included where needed. See why they say Biml allows you to complete in a day what once took more than a week!

@cathrinew

cathrinewilhelmsen.netData Warehouse Architect

Business Intelligence Developer

Cathrine Wilhelmsen

Who are you? (*)

SSIS and ETL Developer?

Easily bored?

Tired of repetitive work?

( * Probably not a cat )

Why are you here?

Long development time?

Many SSIS packages?

Slow GUI editor?

(Drag, drop, drag, drop, connect,

drag, drop, connect, resize, align,

drag, drop, resize, connect, align…)

job done!

new standards

yay

Have you ever experienced this?

Ready for a change?

Business Intelligence Markup Language

Easy to read and write XML dialect

Specifies business intelligence objects

Databases, schemas, tables, columns

SSIS packages

SSAS cubes, facts, dimensions (Mist only)

Highlights in Biml History

founded by Scott Currie, is born

Biml was extended with

Biml compiler added to

is launched

founded

is launched

2008:

2009:

2011:

2012:

2014:

2015:

How can Biml help you?

Timesaving: Many SSIS

Packages from one Biml file

Reusable: Write once and run

on any platform (2005 – 2014)

Flexible: Start simple, expand

as you learn

(Of course I can create 200 packages!

What do you need me to do after lunch?)

What do you need?

How does it work?

Generated packages are indistinguishable from manually created packages

Biml syntax

<Biml xmlns="http://schemas.varigence.com/biml.xsd">

<Packages>

<Package Name="EmptyPackage1"></Package>

<Package Name="EmptyPackage2"/>

</Packages>

</Biml>

Biml syntax: Root Element


<Packages>



</Packages>

</Biml>

Biml syntax: Collection of Elements


<Packages>



</Packages>

</Biml>

Biml syntax: Elements


<Packages>



</Packages>

</Biml>

Biml syntax: Attributes


<Packages>



</Packages>

</Biml>

Biml syntax: Full vs. Shorthand Syntax


<Packages>



</Packages>

</Biml>

Demo – Biml

Getting started with Biml

1. Download and install BIDS Helper (http://bidshelper.codeplex.com)

2. Right-click on SSIS project and click Add New Biml File

http://bidshelper.codeplex.com/

Intellisense

Intellisense while typing

CTRL+Space to AutoComplete or show Intellisense

Errors

Red squiggly line: Error

Blue squiggly line: Missing attribute or child element

Error spelling

Missing attribute: ConstraintMode

Errors

Hovering over errors will show descriptive text

Missing attribute: ConstraintMode

Error spelling

Right-click to Check Biml for Errors

Your first SSIS Package from Biml

Right-click on Biml file and click Generate SSIS Packages

Packages will appear under SSIS Packages

From Biml to SSIS

.biml vs .dtsx:

human-readable vs ALL THE CODE!

(150% zoom) (20% zoom)

I create SSIS packages faster than that

But wait!

The magic is in the

Extend Biml with C# or VB.NET code blocks

Import database structure and metadata

Loop over tables and columns

Add expressions to replace static values

Allows you to control and manipulate Biml code

BimlScript code blocks

<#@ … #> Directives (Instructions to the BimlCompiler)

<# … #> Control Blocks (Control logic)

<#= … #> Expression Control Blocks (Replace block with string value)

<#+ … #> Class Feature Control Blocks (Create helper methods)

BimlScript syntax


<Packages>

<# foreach (var table in RootNode.Tables) { #>

<Package Name="Load<#=table.Name#>"></Package>

<# } #>

</Packages>

</Biml>

BimlScript syntax: Control Blocks


<Packages>



<# } #>

</Packages>

</Biml>

BimlScript syntax: Expression Control Block


<Packages>



<# } #>

</Packages>

</Biml>

How does it work?

Yes, but how does it work?

37

Yes, but how does it actually work?


<Packages>



<# } #>

</Packages>

</Biml>


<Packages>

<Package Name="LoadCustomer"></Package>

<Package Name="LoadProduct"></Package>

<Package Name="LoadSales"></Package>

</Packages>

</Biml>

Demo – BimlScript

Basic for loop

<Packages>

<# for (int count = 1; count <= 5; count++) { #>

<Package Name="Load_Person_Person_<#=count#>">

</Package>

<# } #>

</Packages>

foreach (table in a database) loop

<#@ import namespace="Varigence.Hadron.CoreLowerer.SchemaManagement" #>

<# var conAW2014 = SchemaManager.CreateConnectionNode("AW2014", "Data Source...");#>

<# var AW2014DB = conAW2014.ImportDB("","", ImportOptions.ExcludeViews); #>

<Packages>

<# foreach (var table in AW2014DB.TableNodes) { #>

<Package Name="Load_<#=table.Schema#>_<#=table.Name#>">

</Package>

<# } #>

</Packages>

Don't Repeat Yourself

Move common code to separate files

Centralize and reuse in many projects

Update code once for all projects

1. Split and combine Biml files

2. Include files

3. CallBimlScript with parameters

Split and combine Biml files

Solve logical dependencies and simulate manual workflows by using tiers

Tiers instruct the BimlCompiler to compile files from lowest to highest tier

<#@ template tier="1" #>

Higher tiers can use and might depend on objects from lower tiers

Tier 1 - Create database connections

Example: Tier 2 - Create loading packages

Tier 3 - Create master package to execute loading packages

Split and combine Biml files

1. Create Biml files with specified tiers

2. Select all the tiered Biml files

3. Right-click and click Generate SSIS Packages

1

2

3

Behind the scenes: Split and combine Biml files

RootNode

<#@ template tier="0" #><Connections><Databases><Schemas>

<#@ template tier="1" #><Tables><Columns>

<#@ template tier="2" #><Packages>

<Connections><Databases><Schemas>RootNode




compile


<Connections><Databases><Schemas>RootNode




use


<Connections><Databases><Schemas><Tables><Columns>

RootNode




compile



RootNode



<#@ template tier="2" #><Packages> use



<Packages>

RootNode



<#@ template tier="2" #><Packages> compile



<Packages>

RootNode




generate


RootNode





Don't Repeat Yourself: Include files

Include common code in multiple files and projects

Can include many file types: .biml .txt .sql .cs

Use the include directive

<#@ include file="CommonCode.biml" #>

The include directive will be replaced by the content of the included file

Include pulls code from the included file into the main file

Don't Repeat Yourself: Include files

Don't Repeat Yourself: CallBimlScript with parameters

Works like a parameterized include

File to be called (callee) specifies the input parameters it accepts

<#@ property name="Table" type="AstTableNode" #>

File that calls (caller) passes input parameters

<#=CallBimlScript("CommonCode.biml", Table)#>

CallBimlScript pushes parameters from the caller to the callee, and the

callee returns code

Don't Repeat Yourself: CallBimlScript with parameters

De

mo

View compiled Biml

Credits: Marco Schreuder (@in2bi)

http://blog.in2bi.eu/biml/viewing-or-saving-the-

compiled-biml-file-s/

Helper file with high tier (tier="100")

Saves output of RootNode.GetBiml() to file

http://blog.in2bi.eu/biml/viewing-or-saving-the-compiled-biml-file-s/

What do you do next?

1. Download BIDS Helper

2. Identify your SSIS patterns

3. Rewrite one SSIS package to Biml to learn the basics

4. Expand with BimlScript

5. Get involved in the Biml community

Biml on Monday...

…BimlBreak the rest of the week

More Biml!

Don't miss Rasmus Reinholdt's session

Building a meta-driven near realtime

ETL solution with BIML and SSIS

at 14:55!

…and come chat with us in the breaks

Thank you!

@cathrinew

cathrinewilhelmsen.net

no.linkedin.com/in/cathrinewilhelmsen

[email protected]

cathrinewilhelmsen.net/biml

slideshare.net/cathrinewilhelmsenBiml resourcesover there!

biml for beginners - generating ssis packages with bimlscript (sqlsaturday gothenburg)

Data & Analytics