tools and tips: from accidental to efficient data warehouse developer (sqlbits xiv)

87
Tools and Tips for Data Warehouse Developers Cathrine Wilhelmsen - SQLBits XIV

Upload: cathrine-wilhelmsen

Post on 14-Jul-2015

666 views

Category:

Data & Analytics


1 download

TRANSCRIPT

Page 1: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Tools and Tipsfor Data Warehouse Developers

Cathrine Wilhelmsen - SQLBits XIV

Page 2: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Session descriptionYou have probably heard about the Accidental DBA, but what about the Accidental Data Warehouse developer? We stumbled into the world of data warehousing, learned dimensional modeling and work with T-SQL and SSIS daily. We're masters of googling solutions to our problems and make sure our complex ETL processes run without errors. We deliver data to business users... but we don't deliver data as fast as we want.

You might not be able to rewrite your entire data warehouse or change your team's processes over night, but there are many things you can do to increase your own productivity and become a more efficient data warehouse developer.

In this session I will show you some of what I've learned and discovered that has made me burst out "Oh wow! Why did I not know this yesterday!?" including query improvements, free tools and scripts, SSMS features and even a couple of things I used to think were only useful for those scary DBAs.

Page 3: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Cathrine Wilhelmsen

@cathrinew

cathrinewilhelmsen.netData Warehouse Architect

Business Intelligence Developer

Page 4: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

you

DEsP

1-3 years?T-SQL?SSIS?

Page 5: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

once upon a time...

Page 6: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

V

Page 7: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

how I felt...

Page 8: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)
Page 9: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

how I want to be...

Page 10: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)
Page 11: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

what?SSMS

QueriesBiml for SSIS

DgRw

Page 12: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Tip #1: Visual Information

Page 13: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Connection Colors

Page 14: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Status Bar and Tab Text

Page 15: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Results in Separate Tab

Page 16: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Tab Groups - Vertical

Page 17: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Tab Groups - Horizontal

Page 18: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Split –one query in two windows

Page 19: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Tip #2: Shortcuts

Page 20: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Query Shortcuts

Page 21: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Keyboard Shortcuts

Assign shortcuts you frequently use

Remove shortcuts you accidentally click (no more "ooops")

msdn.microsoft.com/en-us/library/ms174205.aspx

Page 22: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Magic keys!

o

HOME END

PG UP PG DNCTRL ALT

SHIFT TAB

Page 23: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Show / Hide Query Results

CTRL R

Page 24: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Toggle Full Screen

ALTSHIFT ENTER

Page 25: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Cycle through windows

TABCTRL

Page 26: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Change database while writing query

CTRL U

Page 27: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Column / Multi-Line Editing

SHIFTALT

Page 28: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Comment / Uncomment

CTRL K CTRL C

Comment Line

CTRL K CTRL U

Uncomment Line

Page 29: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Tip #3: Search in SSMS

Page 30: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Free Tool: Redgate SQL Search

red-gate.com/products/sql-development/sql-search/

Page 31: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Free Tool: Redgate SQL Search

Page 32: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Tip #4: Templates and Snippets

Page 33: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Templates

Template Browser

Drag & Drop Templates

Create Templates

CTRL ALT T

Page 34: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Template Parameters

Replace Template Parameters with actual values

CTRL SHIFT M

Page 35: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Snippets

CTRL K CTRL X

Insert Snippet

CTRL K CTRL S

Surround With Snippet

Page 36: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Advanced Snippets and Formatting

Redgate SQL Prompt (Licensed)

ApexSQL Complete / Refactor

SSMS Tools Pack (Licensed)

SSMS Boost

Poor Man's T-SQL Formatter

dbForge SQL Complete (Licensed)

red-gate.com

apexsql.com

ssmstoolspack.com

ssmsboost.com

poorsql.com

devart.com/dbforge

Page 37: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Redgate SQL Prompt Demo

Page 38: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Tip #5: Registered Servers and Multiserver Queries

Page 39: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Registered Servers

Save and group servers

Is the server running?

Multiserver Queries

View Registered Servers

CTRL ALT G

Page 40: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Manage services from SSMS

Page 41: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Multiserver Queries

Page 42: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Multiserver Queries

Page 43: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Tip #6: SARGable Queries

Page 44: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

SARGable queries

"The query can efficiently seek using an index to find the correct rows searched for in WHERE or JOIN clauses"

Compare it to finding a person in a phone book

(We'll pretend we still use phone books)

Page 45: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Adama, LeeAdama, WilliamAgathon, KarlBaltar, GaiusDualla, AnastasiaGaeta, Felix

Henderson, CallyRoslin, LauraThrace, KaraTigh, SaulTyrol, GalenValerii, Sharon

SARGable queries

Find all rows where Name starts with "T"

Page 46: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Adama, LeeAdama, WilliamAgathon, KarlBaltar, GaiusDualla, AnastasiaGaeta, Felix

Henderson, CallyRoslin, LauraThrace, KaraTigh, SaulTyrol, GalenValerii, Sharon

SARGable queries

Find all rows where Name starts with "T"

Page 47: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Non-SARGable queries

"The query has to scan each row in the table to find the correct rows searched for in WHERE or JOIN clauses"

Compare it to finding a person in a phone book

(We'll keep pretending we still use phone books)

Page 48: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Adama, LeeAdama, WilliamAgathon, KarlBaltar, GaiusDualla, AnastasiaGaeta, Felix

Henderson, CallyRoslin, LauraThrace, KaraTigh, SaulTyrol, GalenValerii, Sharon

Non-SARGable queries

Find all rows where Name contains "al"

Page 49: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Adama, LeeAdama, WilliamAgathon, KarlBaltar, GaiusDualla, AnastasiaGaeta, Felix

Henderson, CallyRoslin, LauraThrace, KaraTigh, SaulTyrol, GalenValerii, Sharon

Non-SARGable queries

Find all rows where Name contains "al"

Page 50: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

WHERE Name LIKE '%al%'

WHERE Name LIKE 'T%'

WHERE LEFT(Name,1,1) = 'T'

SARGable or Non-SARGable?

Page 51: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

WHERE Name LIKE '%al%'

WHERE Name LIKE 'T%'

WHERE LEFT(Name,1,1) = 'T'

SARGable or Non-SARGable?

Page 52: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

WHERE CAST(EpisodeDate AS DATE) = '20050114'

WHERE CONVERT(CHAR(6), EpisodeDate, 112) = '200501'

WHERE YEAR(EpisodeDate) = 2005

WHERE EpisodeDate >= '20050101'AND EpisodeDate < '20060101'

SARGable or Non-SARGable?

Page 53: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

WHERE CAST(EpisodeDate AS DATE) = '20050114'

WHERE CONVERT(CHAR(6), EpisodeDate, 112) = '200501'

WHERE YEAR(EpisodeDate) = 2005

WHERE EpisodeDate >= '20050101'AND EpisodeDate < '20060101'

SARGable or Non-SARGable?

Page 54: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

WHERE Survivors < 40000

WHERE @Survivors BETWEEN Survivors-1000 AND Survivors+1000

WHERE Survivors BETWEEN @Survivors-1000 AND @Survivors+1000

SARGable or Non-SARGable?

Page 55: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

WHERE Survivors < 40000

WHERE @Survivors BETWEEN Survivors-1000 AND Survivors+1000

WHERE Survivors BETWEEN @Survivors-1000 AND @Survivors+1000

SARGable or Non-SARGable?

Page 56: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

sqlbits.com/Sessions/Event7/Understanding_SARGability_to_make_your_queries_run_faster

Page 57: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Tip #7: Query Analysis

Page 58: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Execution Plans

Display Estimated Execution Plan

CTRL L

Include Actual Execution Plan

CTRL M

Page 59: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Execution Plans

See how a query was or will be executed:

Page 60: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Details in Tooltips

Page 61: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Details in Properties

Page 62: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Free Tool: SQL Sentry Plan Explorer

sqlsentry.com/products/plan-explorer

Page 63: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Free Tool: SQL Sentry Plan Explorer

answers.sqlperformance.com

Page 64: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Free Book: SQL Server Execution Plansby Grant Fritchey

red-gate.com/community/books

Page 65: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Tip #8: Query Statistics

Page 66: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Statistics IO

SET STATISTICS IO OFF;

SET STATISTICS IO ON;

Page 67: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Statistics Time

SET STATISTICS TIME OFF;

SET STATISTICS TIME ON;

Page 68: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Free Tool: Statistics Parser

statisticsparser.com

Page 69: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Client Statistics

Include Client Statistics

SHIFT SALT

Page 70: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Client Statistics

Compare multiple query executions:

Page 71: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Tip #9: Activity Monitoring

Page 72: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Free Script: sp_WhoIsActive by Adam Machanic

sqlblog.com/blogs/adam_machanic

Page 73: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Free Script: sp_WhoIsActive by Adam Machanic

Page 74: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Tip #10: Automatically Generate SSIS Packages with Biml

Page 75: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Business Intelligence Markup Language

Easy to read and write XML dialect

Generate SSIS packages from metadata

Page 76: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

What do I need?

Free add-in for BIDS / SSDT-BIbidshelper.codeplex.com

Page 77: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

How does it work?

Create many SSIS packages from one Biml file

Page 78: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

…what do you need me to do after lunch?

Of course I can create 200 SSIS Packages!

Page 79: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Biml syntax<Biml xmlns="http://schemas.varigence.com/biml.xsd">

<Connections>

<OleDbConnection Name="Source" ConnectionString="…" />

</Connections>

<Packages>

<Package Name="EmptyPackage">

</Package>

</Packages>

</Biml>

Page 80: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

From Biml to SSIS

Page 81: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

From Biml to SSIS

Page 82: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

The magic is in the

Extend Biml with C# or VB.NET code blocks

Import database structure and metadata

Loop over tables and columns

Add expressions to replace static values

Page 83: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

BimlScript syntax

<#@ import namespace="Varigence.Hadron.CoreLowerer.SchemaManagement" #>

<# var conAW2014 = SchemaManager.CreateConnectionNode("AW2014", "..."); #>

<# var AW2014DB = conAW2014.ImportDB("","", ImportOptions.ExcludeViews); #>

<Packages>

<# foreach (var table in AW2014DB.TableNodes) { #>

<Package Name="Load_<#=table.Name#>">

</Package>

<# } #>

</Packages>

Page 84: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Biml for SSIS demo

Page 85: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

…BimlBreak the rest of the week

Biml on Monday…

Page 86: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

cathrinewilhelmsen.net/efficient

Not enough details? Too fast? Don't worry!Slide deck, links and resources:

Page 87: Tools and Tips: From Accidental to Efficient Data Warehouse Developer (SQLBits XIV)

Thank you!

@cathrinew

cathrinewilhelmsen.net

no.linkedin.com/in/cathrinewilhelmsen

[email protected]

cathrinewilhelmsen.net/efficient