tools and tips: from accidental to efficient data warehouse developer (sqlsaturday portugal)
TRANSCRIPT
Tools and Tips:From Accidental to EfficientData Warehouse Developer
Cathrine Wilhelmsen - SQLSaturday Lisbon 2015
Session descriptionYou have probably heard about the Accidental DBA, but what about the Accidental Data Warehouse developer? We stumbled into the world of data warehousing, learned dimensional modeling and work with T-SQL and SSIS daily. We're masters of googling solutions to our problems and make sure our complex ETL processes run without errors. We deliver data to business users... but we don't deliver data as fast as we want.
You might not be able to rewrite your entire data warehouse or change your team's processes over night, but there are many things you can do to increase your own productivity and become a more efficient data warehouse developer.
In this session I will show you some of what I've learned and discovered that has made me burst out "Oh wow! Why did I not know this yesterday!?" including query improvements, free tools and scripts, SSMS features and even a couple of things I used to think were only useful for those scary DBAs.
Thank you to the main sponsors!
Say thank you to organizers and volunteers!
They spend their FREE time to give you this event
Because they are crazy
Because they want YOU to learn from the BEST IN THE WORLD
Paulo Matos:
Pedro Simões:
André Batista:
Paulo Borges:
André Melancia:
Murilo Miranda:
Quilson Antunes:
Important Activities
Sponsor Sessions with Raffles
• 15:10 - Rumos, BI4ALL, Bold Int, CozyRoc, Pythian
WIT (Women in Technology)
• 15:10 at BizSpark Room (Ground Floor)
SQLClinic Challenges
• 17:00 BI (Steph Locke)
Cathrine Wilhelmsen
@cathrinew
cathrinewilhelmsen.netData Warehouse Architect
Business Intelligence Developer
you
DEsP
1-3 years?T-SQL?SSIS?
once upon a time...
V
how I felt...
how I want to be...
what?SSMS
QueriesBiml for SSIS
DgRw
Tip #1: Visual Information
Connection Colors
Status Bar and Tab Text
Results in Separate Tab
Tab Groups - Vertical
Tab Groups - Horizontal
Split –one query in two windows
Tip #2: Shortcuts
Query Shortcuts
Oh, by the way...
SELECT SUM(rows)
FROM sys.partitions
WHERE index_id IN (0, 1)
AND object_id =OBJECT_ID('Sales');
...querying sys.partitions can be better and faster than COUNT(*)
Keyboard Shortcuts
Assign shortcuts you frequently use
Remove shortcuts you accidentally click (no more "ooops")
msdn.microsoft.com/en-us/library/ms174205.aspx
Magic keys!
o
HOME END
PG UP PG DNCTRL ALT
SHIFT TAB
Show / Hide Query Results
CTRL R
Toggle Full Screen
ALTSHIFT ENTER
Cycle through windows
TABCTRL
Change database while writing query
CTRL U
Column / Multi-Line Editing
SHIFTALT
Comment / Uncomment
CTRL K CTRL C
Comment Line
CTRL K CTRL U
Uncomment Line
Tip #3: Search in SSMS
Free Tool: Redgate SQL Search
red-gate.com/products/sql-development/sql-search/
Free Tool: Redgate SQL Search
Tip #4: Templates and Snippets
Templates
Template Browser
Drag & Drop Templates
Create Templates
CTRL ALT T
Template Parameters
Replace Template Parameters with actual values
CTRL SHIFT M
Snippets
CTRL K CTRL X
Insert Snippet
CTRL K CTRL S
Surround With Snippet
Advanced Snippets and Formatting
Redgate SQL Prompt (Licensed)
ApexSQL Complete / Refactor
SSMS Tools Pack (Licensed)
SSMS Boost
Poor Man's T-SQL Formatter
dbForge SQL Complete (Licensed)
red-gate.com
apexsql.com
ssmstoolspack.com
ssmsboost.com
poorsql.com
devart.com/dbforge
Redgate SQL Prompt Demo
Tip #5: Registered Servers and Multiserver Queries
Registered Servers
Save and group servers
Is the server running?
Multiserver Queries
View Registered Servers
CTRL ALT G
Manage services from SSMS
Multiserver Queries
Multiserver Queries
Tip #6: SARGable Queries
SARGable queries
"The query can efficiently seek using an index to find the correct rows searched for in WHERE or JOIN clauses"
Compare it to finding a person in a phone book
(We'll pretend we still use phone books)
Adama, LeeAdama, WilliamAgathon, KarlBaltar, GaiusDualla, AnastasiaGaeta, Felix
Henderson, CallyRoslin, LauraThrace, KaraTigh, SaulTyrol, GalenValerii, Sharon
SARGable queries
Find all rows where Name starts with "T"
Adama, LeeAdama, WilliamAgathon, KarlBaltar, GaiusDualla, AnastasiaGaeta, Felix
Henderson, CallyRoslin, LauraThrace, KaraTigh, SaulTyrol, GalenValerii, Sharon
SARGable queries
Find all rows where Name starts with "T"
Non-SARGable queries
"The query has to scan each row in the table to find the correct rows searched for in WHERE or JOIN clauses"
Compare it to finding a person in a phone book
(We'll keep pretending we still use phone books)
Adama, LeeAdama, WilliamAgathon, KarlBaltar, GaiusDualla, AnastasiaGaeta, Felix
Henderson, CallyRoslin, LauraThrace, KaraTigh, SaulTyrol, GalenValerii, Sharon
Non-SARGable queries
Find all rows where Name contains "al"
Adama, LeeAdama, WilliamAgathon, KarlBaltar, GaiusDualla, AnastasiaGaeta, Felix
Henderson, CallyRoslin, LauraThrace, KaraTigh, SaulTyrol, GalenValerii, Sharon
Non-SARGable queries
Find all rows where Name contains "al"
WHERE Name LIKE '%al%'
WHERE Name LIKE 'T%'
WHERE LEFT(Name,1,1) = 'T'
SARGable or Non-SARGable?
WHERE Name LIKE '%al%'
WHERE Name LIKE 'T%'
WHERE LEFT(Name,1,1) = 'T'
SARGable or Non-SARGable?
WHERE CAST(EpisodeDate AS DATE) = '20050114'
WHERE CONVERT(CHAR(6), EpisodeDate, 112) = '200501'
WHERE YEAR(EpisodeDate) = 2005
WHERE EpisodeDate >= '20050101'AND EpisodeDate < '20060101'
SARGable or Non-SARGable?
WHERE CAST(EpisodeDate AS DATE) = '20050114'
WHERE CONVERT(CHAR(6), EpisodeDate, 112) = '200501'
WHERE YEAR(EpisodeDate) = 2005
WHERE EpisodeDate >= '20050101'AND EpisodeDate < '20060101'
SARGable or Non-SARGable?
WHERE Survivors < 40000
WHERE @Survivors BETWEEN Survivors-1000 AND Survivors+1000
WHERE Survivors BETWEEN @Survivors-1000 AND @Survivors+1000
SARGable or Non-SARGable?
WHERE Survivors < 40000
WHERE @Survivors BETWEEN Survivors-1000 AND Survivors+1000
WHERE Survivors BETWEEN @Survivors-1000 AND @Survivors+1000
SARGable or Non-SARGable?
sqlbits.com/Sessions/Event7/Understanding_SARGability_to_make_your_queries_run_faster
Tip #7: Query Analysis
Execution Plans
Display Estimated Execution Plan
CTRL L
Include Actual Execution Plan
CTRL M
Execution Plans
See how a query was or will be executed:
Details in Tooltips (ok)
Details in Properties (better)
Free Tool: SQL Sentry Plan Explorer
sqlsentry.com/products/plan-explorer
Free Book: SQL Server Execution Plansby Grant Fritchey
red-gate.com/community/books
Tip #8: Query Statistics
Statistics IO
SET STATISTICS IO OFF;
SET STATISTICS IO ON;
Statistics Time
SET STATISTICS TIME OFF;
SET STATISTICS TIME ON;
Client Statistics
Include Client Statistics
SHIFT SALT
Client Statistics
Compare multiple query executions:
Tip #9: Activity Monitoring
Free Script: sp_WhoIsActive by Adam Machanic
sqlblog.com/blogs/adam_machanic
Free Script: sp_WhoIsActive by Adam Machanic
Tip #10: Generate SSIS Packages with Biml
Business Intelligence Markup Language
Easy to read and write XML dialect
Generate SSIS packages from metadata
What do I need?
Free add-in for BIDS / SSDT-BIbidshelper.codeplex.com
How does it work?
Create many SSIS packages from one Biml file
…what do you need me to do after lunch?
Of course I can create 200 SSIS Packages!
Biml syntax<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Connections>
<OleDbConnection Name="Source" ConnectionString="…" />
</Connections>
<Packages>
<Package Name="EmptyPackage">
…
</Package>
</Packages>
</Biml>
From Biml to SSIS
From Biml to SSIS
The magic is in the
Extend Biml with C# or VB.NET code blocks
Import database structure and metadata
Loop over tables and columns
Add expressions to replace static values
BimlScript syntax
<#@ import namespace="Varigence.Hadron.CoreLowerer.SchemaManagement" #>
<# var conAW2014 = SchemaManager.CreateConnectionNode("AW2014", "..."); #>
<# var AW2014DB = conAW2014.ImportDB("","", ImportOptions.ExcludeViews); #>
<Packages>
<# foreach (var table in AW2014DB.TableNodes) { #>
<Package Name="Load_<#=table.Name#>">
…
</Package>
<# } #>
</Packages>
Biml for SSIS demo
…BimlBreak the rest of the week
Biml on Monday…
Bonus Tip: Be social in your community!
DEsP
#SQLHelp
cathrinewilhelmsen.net/efficient
Not enough details? Too fast? Don't worry!Slide deck, links and resources:
Go talk to the sponsors!
Thank you! Enjoy your lunch
@cathrinew
cathrinewilhelmsen.net
no.linkedin.com/in/cathrinewilhelmsen
cathrinewilhelmsen.net/efficient