overcoming ssis deployment and configuration challenges erik veerman mentor, solid quality mentors...

Overcoming SSIS Deployment and Configuration Challenges

Erik VeermanMentor, Solid Quality MentorsSQL Server MVP

//ATLANTAMDF2

• SQL Server MVP (Most Valuable Professional)• Mentor with Solid Quality Mentors• ~15 years in the industry, 10 years SQL Server BI

experience (v1 SQL BI, beta SQL 7)… Telco, Manufacturing, Finance, Insurance, Retail, Real Estate…

• MS Worldwide BI Solution of the Year• SQL Server Magazine Innovator Cup• Microsoft Project REAL ETL Architect• Author:

• SQL Server 2008 Integration Services Professional • MS Press SQL Server BI Training Kit (70-445) (70-448 in progress!)• SQL Server 2005 Integration Services Expert• SQL Server 2005 Integration Services Professional

Speaker BIO – Erik Veerman

//ATLANTAMDF

Industry-Leading Technical Experts, Authors, Speakers

Solid Quality™ Mentors are more than 100 of the world’s top technical experts who specialize in assuring client success using integrated

Microsoft technologies.

“Catch the Solid Quality™ Advantage”

www.solidq.com

3

PRACTICE AREAS

Relational Database Management

Business Intelligence

Development Methodologies

SharePoint Collaboration

SERVICES

Advanced Public Training

Customized Private Training

Solution Delivery and Tuning

Enhanced Mentoring Services

//ATLANTAMDF4

• Who has had challenges with SSIS deployment or configurations?

• Has anyone punched a wall or other visible frustrations?

• Has anyone actually had to go to anger management?

Let’s be honest…

//ATLANTAMDF

Baby Steps: The Books-Online Basics–Security: Encryption and Connections–Configurations and Expressions–Deployment

The Problem: Bob Anger Management: The Solutions

Agenda

http://www.youtube.com/watch?v=C_gqZyUfjrk

http://www.youtube.com/watch?v=C_gqZyUfjrk

//ATLANTAMDF

BOL Basics: SSIS Package Security

Security is comprised of several layers to support both SQL and File System based scenarios–Packages can be encrypted–Packages can be digitally signed–Packages can be stored in SQL DB and protected with roles–Packages can be stored in the File System and protected with File

System Security

Package Definition:

Storage:

SSIS Package File (.DTSX):Encrypted, Digitally Signed

File System:File System Security

MSDB:MSDB SQL Server Roles

//ATLANTAMDF

BOL Basics: SSIS Package Security

Internal Package Security–Package encryption – Used to encrypting connection

passwords or the entire package–Package Password – Optional password applied to the

package to un-encrypt any encrypted contents External Package Security–MSDB database roles for packages deployed to SQL

Server. Reader and Writer roles included for execution and reading privileges–File system security for packages deployed to server file

systems or file shares

//ATLANTAMDF

BOL Basics: SSIS Package Encryption Levels

Don’t Save Sensitive Encrypt Sensitive with User Key Encrypt Sensitive with Password Encrypt All with User Key Encrypt All with Password Server Storage

//ATLANTAMDF

Package Encryption and MSDB

demo …

//ATLANTAMDF

BOL Basics: Securing Data and Connections

Data Security–Data is not stored as part of a package

Connection Schema Security–Connection schema information will be stored in

packages depending on the uses (Data Flow source/destinations and Execute SQL)–Lock down package access when appropriate

File system rights Package encryption/password Package roles and access

//ATLANTAMDF

Package Connections and Encryption

demo …

12

//ATLANTAMDF

SSIS Package Connection 1 = XYZ Variable 1 = XYZ

SSIS Package Connection 1 = XYZ Variable 1 = XYZ

SSIS Package Connection 1 = ABC Variable 1 = ABC

BOL Basics: SSIS Configurations

Update package properties at execution time based on external setting–Variable properties–Connection Properties–Package properties–Task and Container Properties

Data Flow Transformation properties are not configurable except at the Task level

Configuration• Connection 1 = ABC

• Variable 1 = ABC

//ATLANTAMDF

SSIS Configuration Types

XML File Configuration–Direct path or…–Environment Variable driven

SQL Server Configuration–Table Driven–Filter Column grouping configurations

Registry Configuration Parent Package Variable Configuration

–Inherit variables from master packages

Environment Variable Configuration

//ATLANTAMDF

Package Configurations

demo …

//ATLANTAMDF

BOL Basics: SSIS Property Expressions

Runtime updates of Control Flow components–Package level properties–Task and Container properties–Connections, too!

Expressions assignable through property windows and task editors

SSIS expression language based

SSIS PackageVariable1 = C:\FileA

Connection1 = Expression: @Variable1

• Step 1: Run Data Flow• Access Connection1• Extract from C:\FileA

• Step 2: Update Variable• Step 3: Run Data Flow

• Access Connection1• Extract from C:\FileB

C:\FileA

C:\FileB

C:\FileB

//ATLANTAMDF

SSIS Property Expressions

Common Property Expression usage–Dynamic SQL for non-parameterizable statements–Connections, Checkpoints, Disable flag–Loop Container enumeration functionality

Variable values can be updated by expressions through the EvaluateAsExpression and Expression property of the variable

//ATLANTAMDF

Property Expressions

demo …

//ATLANTAMDF

BOL Basics: Deployment

Deployment destinations–File System (file share or local drive)–MSDB Database (sysdtspackages90 or sysssispackages)

Deployment options–Manual windows command line script (Xcopy or Robocopy)–DTUtil command line script (file system or MSDB)–Deployment Wizard and Installer

//ATLANTAMDF

BOL Basics: Deployment

File System

MSDB

Development

Test orProduction

SSIS Package

SSIS Package

BIDS Project Folder File System or Share

File Copy, DTUtil, Installer Set

Development

Test orProduction

SSIS Package

SSIS Package

BIDS Project Folder MSDB Database

SSMS, DTUtil, Installer Set

//ATLANTAMDF

BOL Basics: Deployment Wizard

Designer can build a deployment set which includes a project's packages, configurations files and an installer file

You move the installer file set to another server/environment and execute to install packages to SQL or file system

Installer can "copy" misc files but will not install them, such as custom components

Deployment overwrites existing packages

//ATLANTAMDF

Package Deployment

demo …

//ATLANTAMDF



Agenda

24

• You want to live a normal IT life but the problem follows you around

• Just when you think you’ve solved the problem, there it is again

The Problem:

• You’re the expert, but the problem will not go away

• Just knowing the baby steps doesn’t get you far enough

//ATLANTAMDF

SSIS Deployment Challenges

Encryption causes your package to fail Connections and other properties aren’t updated Passwords are not available or too easily accessible Shared configurations require properties and connections exist SQL Configurations point to Dev database

Encryption Failure:

Server 1

Server 2

SSIS Package

Connection A

SSIS Packages MSDB or File System

SSIS Package

Connection APackage Deployed

EncryptionWith UserKey

EncryptionWith UserKey

Package Fails!

Can not decrypt

UserKey (New

Machine)

//ATLANTAMDF


Connections and other properties aren’t updated

Server 1

Server 2

SSIS Package

Dev Connection ADev Connection B

Dev Connection A

MSDB or File System

Package Error!

Connections Point to Wrong

Databases/Files

SSIS Package

Dev Connection ADev Connection B

Package Deployed

Dev Connection B

Prod Connection A

Prod Connection B

SSIS Packages

//ATLANTAMDF


SQL Configurations point to Dev database (or configuration database not accessible!)

Server 1

Server 2

SSIS PackageDev Connection ADev Connection BDev Config DB

Dev Connection A

MSDB or File System

Config DB Connection Not

Updated!


Package Deployed

Dev Connection B

Prod Connection A

Prod Connection B

Dev Config DB

1

2

1

Prod Config DB

2

SSIS Packages

//ATLANTAMDF

Pulling Your Hair Out!

demo …

//ATLANTAMDF



Agenda

Solution 1

Solution 1: Tie Bob to the Boat

• Pretend the problem is not there• Try and man-handle the problem• Make the problem worse

Solution 2Solution 2: Blow up the house

• Give up• Find a punching bag• Declare to Bill Gates that you

are not a PC anymore

//ATLANTAMDF

Dealing with Encryption

Situation: All connections allow Windows Authentication Use Windows Authentication only for all connections Turn off encryption (Set encryption to DontSaveSensitive)

Server 1

Server 2

SSIS Package

Connection A/B/C (Windows Auth)


SSIS Package

Connection A/B/C (Windows Auth)

Package Deployed

Encryption:DontSave-Sensitive


//ATLANTAMDF


Situation: One or more connections require password Option 1: EncryptSensitiveWithPassword

Pass in the package Password during execution Option 2: Set encryption to DontSaveSensitive

Use a Configuration to update the password Option 3: Deploy to MSDB, set encryption to ServerStorage

Option 1:

Server 1

Server 2

SSIS Package

Connection A,password = ABC


SSIS Package

Connection A,password = ABC

Package Deployed

Encryption:EncryptSensitive-WithPassword

Encryption:EncryptSensitive-WithPassword

Package Executed with DTExec and the /P switch

//ATLANTAMDF


Option 2: Set encryption to DontSaveSensitive Use a Configuration to update the password

Server 1

Server 2

SSIS Package

Connection A,password = [Blank]


SSIS Package

Connection A,password = [Blank]

Package Deployed



Config File

password = ABC

Config File

password = ABC

//ATLANTAMDF

Solving the Encryption Problem

demo …

//ATLANTAMDF

Dealing with Connections Option 1: Use a File Configuration (direct or indirect path) Option 2: Use a File and SQL Configuration Option 3: Use Property Expressions and Variables

Server 1

Server 2

SSIS PackageDev Connection ADev Connection B

Dev Connection A

MSDB or File System

SSIS PackageDev Connection ADev Connection B

Package Deployed

Dev Connection B

Prod Connection A

Prod Connection B

Dev Config File

1

2

1

Prod Config File

2

SSIS Packages

Option 1

//ATLANTAMDF

Dealing with Connections Option 2: Use a File and SQL Configuration

Server 1

Server 2


Dev Connection A

MSDB or File System


Package Deployed

Dev Connection B

Prod Connection A

Prod Connection B

Dev Config File

1

3

1

Prod Config File

SSIS Packages

SSIS Dev Config DB

2

SSIS Prod Config DB2

3

//ATLANTAMDF

Demo, Two-Pass Configuration Approach

demo …

//ATLANTAMDF

Dealing with Connections Option 3: Use a Package Variable and Expression

Server 1

Server 2

SSIS Package

Connection A VariableConnection B Variable

Connection A (Expression)Connection B (Expression)

Dev Connection A

MSDB or File System

Dev Connection B

Dev Config File

1

3

SSIS Packages

24

SSIS Package

Connection A VariableConnection B Variable

Connection A (Expression)Connection B (Expression)

Dev Connection A

Dev Connection B

Dev Config File

1 2

Package Deployed

Makes sharing configurations easier!

3

4

//ATLANTAMDF

Demo, Variable Based Configuration Approach

demo …

//ATLANTAMDF

BIDS Helper

CodePlex.com – BIDS Helper Tools

–Deployment SSIS Packages tool–dtsConfig File Formatter–Expression and Configuration Highlighter–Expression List–Non-default properties list–Variables Window Extensions–FixRelativePaths–…others…

41

//ATLANTAMDF42

Consider storing your packages on SQL Server

Consider storing your packages in

the file system

Will you have hundreds of SSIS

packages

YesDo you need

Pkg execution or read privileges for a

team

Package Storage Decision Flow

Do you Need a centralized

package backup and Security

model

Are you using SQL Server RDBMS?

Are you the DBA or do you have

DBA privileges?

Can your Source Control tool handle

deployment

No

Yes

No

Yes

No

No

Will you Have a lot of Master-

Child packages?

Yes

Yes

No

Other factors: cluster, multiple

servers, etc

No

Yes

No

Yes

//ATLANTAMDF

Use DTUtil and Command Line Scripting

Do you have a set of packages to be deployed per

deployment

Package Deployment Decision Flow

Do you Need to automate

deployment

Are you storing your packages in

SQL Server?

Can your Source Control handle

deployment

Do your Packages for

deployment always come from the same

BIDS project?

No

Yes

Yes

No

No

Yes

Yes

Manually deploy

package Files

Use the Source Control

Deployment Tools for file system

deployment

Use the SSIS Deployment

Wizard

Yes

No

No

//ATLANTAMDF

Package Encryption Decision Flow (version 1)

Are you Breathing?

Yes

EncryptSensitive-WithUserKey

EncryptAll-WithUserKey

DontSave-Sensitive

EncryptSensitive-WithPassword

EncryptAll-WithPassword

ServerStorage

//ATLANTAMDF


Are you Obsessive

Compulsive (like Bob)?

Yes

EncryptSensitive-WithUserKey


DontSave-Sensitive



No

ServerStorage

//ATLANTAMDF


Are you the only person that will ever edit/design

the package

YesEncryptSensitive-

WithUserKey


DontSave-Sensitive



No

Will you always

use the same Server?

Do you need a class in

anger management

No

No

Yes

ServerStorage

Yes

//ATLANTAMDF


Are you deploying

to SQL Server…

And are deploying using the SSIS Deployment Tool…

And don’t use configurations…

And your passwords don’t change…

Yes

No ServerStorage

See other options…

//ATLANTAMDF

Use an indirect (Env Var to

reference) file configuration

Do you or your org have

a standard or pref. for storing settings

in files or DBs?

Package Configuration Decision Flow

Do you have passwords or

sensitive properties in the package?

Will the file name or Path change

between servers?

Yes

Tables

No

Use a direct file configurationFiles

Is your server locked down (file and DB)?

Is it easier in your org to

secure files or tables?

No

No

Yes

Use SQL Config with a File Config

as a pointer

TablesYes

Files

Other factor: Sharing configs

Thank you

overcoming ssis deployment and configuration challenges erik veerman mentor, solid quality mentors...

Documents