configuration set values for parameters central connection manager configuration advanced property...
TRANSCRIPT
Deep Inside the Microsoft SQL Server Integration Services Server
Matt MassonMatthew RocheMicrosoft Corporation
DBI405
Best Practices
SQL Server 2012 Integration Services Server Best Practitioners
Matt MassonMatthew RocheMicrosoft Corporation
DBI405
Not actually a Session AgendaProject and Package DeploymentProject deployment model – what, how, and whyDeploying single packages – can it be done? Should it be done?
Logging and MonitoringSSIS Server built-in capabilitiesUsing the SSIS Server API3rd party and community options
Troubleshooting and DebuggingArchitectureTools and techniques
SSIS Server Lightning Review
• The SSIS Server is a set of components that which include a SQL Server user database (SSISDB), an execution host process (ISServerExec.exe) and the tools and APIs to manage and control them.
• The SSIS Server is the target of deployment for SSIS projects when the Project Deployment Mode is used.
• The SSIS Server (and the Project Deployment Mode) is an optional part of SSIS in SQL Server 2012, but key development capabilities (such as parameters and project-level shared connection managers) require its use.
What is the SSIS Server?
SSIS CatalogConfiguration
Set values for parameters
Central connection manager configuration
Advanced property override functionality
Security
Encryption of projects and parameter values
Row-level security to control access to packages
Management
Interactive package execution and SQL Agent integration
Dashboard and built in reports for troubleshooting
Diving into the SSIS Server
SQ
L S
erv
er
Insta
nce SSIS Server
SSIS Catalog
Deploy Manage Security Validate Execute
IS Objects
Security
Operation logs
State
Execution Process
Execution Control
Components
PowerShell SSMS Deployment Wizard ApplicationOM:
ManageOM:
ManageOM:
ManageOM:
ProjectOM:
ManageOM:
Project
SSIS Server Architecture
Runtime
The SSIS Catalog: SSISDB
SSISDB
• Provides a set of stored procedures and views for managing, configuring, executing and monitoring SSIS packages deployed to the SSIS Catalog
• The Catalog schema is designed for customer use, and is a supported API• Contains views, stored procedures, and functions
• The Internal schema is neither public nor supported• Contains base tables and lower-level objects
• All projects and sensitive values are encrypted • Protected by a pair of certificate and symmetric
key• Utilizes the built-in SQL Server encryption
infrastructure• http://
blogs.msdn.com/b/mattm/archive/2012/03/23/ssis-catalog-backup-and-restore.aspx
Important: Backup the database master key!
Whatever you dousing SSMS can be scriptedeither using T-SQL or PowerShell
Tables
Views
SSISDB
Stored Procedures
Stored Procedures
SQLCLR Assembly
ISServerExec.exe
Microsoft.SqlServer.IntegrationServices.Server. Shared.dll
Functions
• 42 store procedures• 25 views• 3 functions• 1 trigger• 1 SQL CLR assembly
Public objects:
Internal CatalogTriggers
SSISDB Overview
Client SQL Server instance
Entry point: T-SQL sp
InvokedCLRsp
Return success/failThrow if errors occur
External ProcessISServerExec
Client SQL Server instance
InvokedCLR SP
Return success/failThrow if errors
occur
Create process by impersonating
caller of the stored proc
Either Asynchronous or Synchronous
SSISDB: Usage Patterns
T-SQL sp (entry point)T-SQL sp invokes managed sp
T-SQL sp (entry point)T-SQL sp invokes managed spManaged sp creates external process by impersonating caller of sp
Client SQL Server instance
T-SQL sp
Return success/failThrow if errors occur
T-SQL spCASE A CASE B CASE C
Entry point: T-SQL sp
SSISDB - Protecting Sensitive DataGuiding Design Principle: No sensitive data should be stored in plain text in the server
Service Master Key
Database Master key
Project CertificateEnvironment Certificate
Execution Certificate
Project Symmetric key
Environment Symmetric key
Execution Symmetric key
Project Parameter
values
Environment values
Execution Parameter
values
Supported 'TRIPLE_DES_3KEY', 'AES_128', 'AES_192', 'AES_256'T-SQL functions: EncryptByKey; DecryptByKeyNote: We do not support sensitive data with length > 8000
SET @key_name = 'MS_Enckey_Proj_'+CONVERT(varchar,@project_id) SET @certificate_name = 'MS_Cert_Proj_'+CONVERT(varchar,@project_id)OPEN SYMMETRIC KEY key_name DECRYPTION BY CERTIFICATE certificate_nameSELECT parameter_name, DECRYPTBYKEY([sensitive_parameter_value]) FROM internal.[object_parameter_values]WHERE [project_id] = @project_idCLOSE SYMMETRIC KEY key_name
Note: The Catalog Views will automatically decrypt the values for you.
Projects and Deployment
Oh, the good old days……said nobody, ever.
Project and Package DeploymentPackage Deployment – The Bad Old DaysDesign time vs. deployment time – what is a project?Deployment locations and their design implications (production pain prevention predicated on palliative planning prior to package placement!)Does anyone remember the poison apple?
Project Deployment – The Glorious FutureDesign time and deployment time – what a project is!An ispac for your headacheA consistent experience between design time, deployment and run time
Projects & Parameters
Groups of packages (anywhere) Projects
SalesHR
HR DW project
SAP migration project
Configurations
<xml>x\y\MyTask\Server = “TestServer”
x y
Parameters
HR DW project
ServerName is String BatchNumber is Int32
CLR Cryptography
• When a project is deployed, the project .ispac file is stored as binary data in SSISDB
• The .ispac binary is also encrypted on the server, using SQLCLR and System.Security.Cryptography
SSISDB - Protecting Project Data
Service Master Key
Database Master key
Project Certificate
Project Symmetric key
key
internal. catalog_encryption_keys
TripleDESCryptoServiceProvider
AesCryptoServiceProvider(192)
AesCryptoServiceProvider(128)
AesCryptoServiceProvider(256)
Project binary
Project Deployment and the SSIS ServerWhat is an ispac file, anyway?The output of the project build processThe complete contents of the SSIS project: packages, parameters, and connections, oh my!The component that gets deployed when you deploy an SSIS project
What happens when I deploy a project?What DOESN’T happen?!But seriously folks… …let’s take a look at a demo!
Demo: Deploying an SSIS Project
Project Deployment and the SSIS ServerWhat is an ispac file, anyway?The complete contents of the SSIS project: packages, parameters, and connections, oh my!OpenXML (zip) document
What happens when I deploy a project?
Is There a Happy Medium?What about single-package deployment?
EXECUTE AS CALLEROPEN SYMMETRIC KEYWAITFOR DELAY
[internal].[encrypt_binarydata][internal].[deploy_project_internal][internal].[append_packages]
[internal].[projects][internal].[object_versions][internal].[packages]
Branching and Deployment
Development
Release
Integration
Multiple code branches.Regular integration from Development -> Integration -> Release.
All deployments come from a branch build.Never deploy “one off” changes.Deployment can be automated.
Test
Production
Server Execution and Logging
Oh, the good old days……wept everyone, always.
SSISDB and ISServerExec
SSISDB
• ISServerExec.exe – External host for SSIS package operations (deploy, validate and execute)
• ISServer Assembly• UNSAFE Assembly granted to
##MS_SQLEnableSystemAssemblyLoadingUser##
• Created from Microsoft.SqlServer.IntegrationServices.Server.dll
• SQLCLR stored procedures used for• Deploy, validate, and execute require impersonation when
starting external process• IPC communication with ISServerExecISServerExec
Processes
ISServerExec Overview
SSISDB
Logging EventsWrites back to SSISDB events that are produced during package execution
ISServerExec
SSIS Events
SqlConnection
Named Pipes
Named Pipe Server
IPC between ISServerExec and Stored Procedures• CLR stored procedure sends command to ISServerExec
Examples• Get me all the per-instance performance counters• Stop Operation• Create execution dump
• ISServerExec • Performs the operation• Sends back information via named pipes
SSIS Server Package Execution
SSISDB
EXEC [SSISDB].[catalog].[create_execution]…EXEC [SSISDB].[catalog].[set_execution_parameter_value] @execution_id, …
EXEC [SSISDB].[catalog].[set_execution_parameter_value] @execution_id, …
EXEC [SSISDB].[catalog].[start_execution] @execution_id
ISServerExec
Named Pipe Server
SSIS Events
catalog.executablescatalog.executable_statistics
Updated when the OnPostExecute eventfor each component if fired
Review: SSIS Server Execution Architecture
OM/Engine
ISServerExec
Named Pipe
IDTSEvents
ISServerExec.exe
Events Listener
ADO.Net
SqlServr.exe
Tables
Components
Views
Reports
TVFs
SSISDB
Log Provider
IDTSLogging
CreateProcessAsUser
API
Monitoring
• Logging is automatically performed by the serverNo specific design patterns must be followed by the package developerData is stored in the SSISDB catalog, and is available for reporting and analysis
• NoneLogging is turned off. Only the package execution status is logged.
• Basic(Default) All events are logged, except custom and diagnostic events.
• PerformanceOnly performance statistics, and OnError and OnWarning events, are logged.Enables use of Execution Performance report and catalog.execution_component_phases view
• VerboseAll events are logged, including custom and diagnostic events, including the DiagnosticEx event.Enables use of catalog.execution_data_statistics view.
SSIS Package Execution & Logging Levels
Built-In Reporting and LoggingBuilt-In Reports
Validate Pre Execute ProcessInput ProcessInput Post Execute
SELECT package_name, task_name, subcomponent_name, SUM(DATEDIFF(ms,start_time,end_time)) as active_time,DATEDIFF(ms,min(start_time),max(end_time)) as total_timeFROM catalog.execution_component_phasesWHERE execution_id = 1841GROUP BY package_name, task_name, subcomponent_name, execution_pathORDER BY package_name, task_name, subcomponent_name, execution_path
Component Timing & Row Counts
Custom Reporting and LoggingAll Catalog logging exposed in ViewsSome features (like real-time perf counters) only available while the package is running
Common pattern in previous versionsCapture events using Event Handlers or custom logging frameworkYou can link custom logging with Catalog logging with $User::ServerExecutionID system variable
Community Reporting Toolshttp://ssisreportingpack.codeplex.com and sp_ssiscataloghttp://www.mattmasson.com/2013/04/monitoring-ssis-package-executions/ (many links!)
3rd Party Commercial Reporting ToolsPragmatic Works - BI xPress Auditing Framework Wizard
Troubleshooting
Oh, the good old days……wept Matthew, over drinks.
• New in SSIS 2012 (for Verbose Logging Level)
• Captures diagnostic information whenever an Execute Package Task executes a child package
DiagnosticEx Event
Flattening the DiagnosticEx Event Data
WITH DiagnosticExTable(EventMessageID, EventName, MessageSourceName, XmlData)AS ( SELECT event_message_id,event_name,message_source_name,cast( message as xml) FROM catalog.event_messages m WHERE m.operation_id = 16 AND m.event_name = 'DiagnosticEx') SELECT EventMessageID,Eventname,MessageSourceName, parameter.value('declare namespace DTS=''www.microsoft.com/SqlServer/Dts''; (@DTS:ObjectName)[1]','nvarchar(260)') as ParameterName,parameter.value( 'declare namespace DTS=''www.microsoft.com/SqlServer/Dts''; (DTS:Property/text())[1]', 'nvarchar(256)') as ParameterValue FROM DiagnosticExTable CROSS APPLY XmlData.nodes('declare namespace DTS=''ww.microsoft.com/SqlServer/Dts''; (/DTS:ParameterValues/DTS:PackageParameter)') as PackageParameter(parameter)
Per-Instance Performance CountersThe SSISDB T-SQL API includes a function to return performance counters for SSIS package executions
SSISDB
SELECT *FROM catalog.dm_execution_performance_counters(<your execution ID value>)
ISServerExec
SSIS Events
Named Pipe Server
What are the values for the Perf Counters?
Perf Details
Results
ISServerExec
Creating Execution Dumps• Stored procedure to cause a running SSIS package to pause and create a
dump file• Dump file stored in …\Program Files\Microsoft SQL Server\110\Shared\
ErrorDumps• Similar to dtutil.exe /Dump
EXEC catalog.create_execution_dump @execution_id = 88
SSISDB
SSIS Events
Named Pipe Server
Create execution Dump DumpFile
Pause running packageDumpResume running package
Data Taps – Data Viewers on the Server
Data Tap Files
-- Create the data tap on a data flow path in the packageexec catalog.create_execution …
exec catalog.add_data_tap @execution_id, '\Package\DFT Load Dim Vendor','Paths[SRC DimDCVendor.OLE DB Source
Output]', 'DCVendorOutput.csv'
exec catalog.start_execution @execution_id …
• Data Taps are essentially server-side data viewers
• Created using one of two stored procedures• [catalog].[add_data_tap]: only for parent packages• [catalog].[add_data_tap_by_guid]: for both parent and child packages
• Data Taps create CSV outputs• Tap files are put under %DTS%\DataDumps folder• Tap files contain all data that passes through the specified data flow path
• Records are created in [catalog].[operation_messages]
• When a data tap file is created• If the specified data flow path is invalid• If the specified data flow task is never executed• If the data tap file creation fails
Creating Data Taps
Closing
Project Deployment ModelParameters. Reusable connection managers. Automatic logging, monitoring and reporting. Relative references for child execution. A real deployment utility. Parameters!
Remote ExecutionISServerExec.exe runs on the server where the package is deployed, not on the client where the execution is initiated. Boo yah!
T-SQL APIScript it from within SSMS. Save it to .SQL script files. Edit and customize. Store and version control with other system artifacts. Execute from any SQL-aware client. Smile, sit back, and enjoy a cold beverage.
SSIS Catalog DataExecution and operation data automatically logged. Built in reports. Community reports and stored procedures for ease of access. Opportunity for PowerPivot and Power View models and cross-catalog consolidation. Let’s overload the term “metadata” once and for all, because…
Matthew’s Favorite Bits of the SSIS Server
Related contentBreakout SessionsDBI-B210 BI Power Hour: Wednesday 1:30 PM, New Orleans Theater C
Related Certification ExamExam 70-463 - Implementing a Data Warehouse with Microsoft SQL Server 2012
Find Us Later At...The Data Platform booth in the Expo Hall
Track Resources
@sqlserver
mvaMicrosoft Virtual Academy
SQL Server Website
Get Certified!
Hands-On Labs
Download Data Explorer
Download Geoflow
Windows Azure
msdn
Resources for Developers
http://microsoft.com/msdn
Learning
Microsoft Certification & Training Resources
www.microsoft.com/learning
TechNet
Resources
Sessions on Demand
http://channel9.msdn.com/Events/TechEd
Resources for IT Professionals
http://microsoft.com/technet
Complete an evaluation on CommNet and enter to win!
Evaluate this session
Scan this QR code to evaluate this session and be automatically entered in a drawing to win a prize
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
Appendix
IPC between SSISDB and ISServerExec• Named Pipe: ISServerExec_{ExecutionGuid}
• Stopping execution needs, to issue a command to ISServerExec• Creating dynamic dump needs to issue a command and get file name back• Querying performance data needs to request a list of key-value pair
SSISDB(1) Named pipe
client
ISServerExec.exe
(2) Named pipe server
(3) pipe
(5) Stop executionCreate dump
Query perf data
Perf data
(4) Listen to the command(6) Perform actionYou can use
pipelist.exe to check the named pipe used
SSIS Package Execution Lifecycle
Created (1)
Pending(5)
Running(2)
Stopping(8)
Canceled (3)
Success(7)
Completed(9)
Failed(4)
Unexpected
Termination / Crash
(6)
catalog.start_execution
catalog.create_execution catalog.stop_operation
ISServerExec Unexpected Crash
• Asymmetric Key• MS_SQLEnableSystemAssemblyKey
• Logins• ##MS_SQLEnableSystemAssemblyLoadingUser##• ##MS_SSISServerCleanupJobLogin##
• SQL Server Agent Jobs• SSIS Server Maintenance Job – Cleans up execution log data
outside retention window and project versions beyond configured limit
• master Stored Procedure• dbo.sp_ssis_startup – Redirects to [SSISDB].[catalog].[startup]• Cleans up orphaned operation status for unexpected shutdown
SSIS Server Objects: What’s not in SSISDB?
SSISDB
• To debug ISServerExec.exe• Create a key named “ISServer” under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\110\SSIS\WaitOnStartup\ISServer• add a DWORD value with name "*" or the project id.• ISServerExec will pause at startup so you can attach a debugger
Debugging Server Package Execution
SSIS Server Security
SSIS Server Security Overview v1
• Provides row-level security for SSIS Securables (Folder, Project, Environment, Operations/Executions)
• Members of ssis_admin or sysadmin access all securables
• Views• catalog.explicit_object_permissions
Permissions explicitly assigned to the userhttp://msdn.microsoft.com/en-us/library/ff878037
• catalog.effective_object_permissions Effective permissions for the current principal for all objectshttp://msdn.microsoft.com/en-us/library/ff878149.aspx
SSIS Server Security Overview v2
SSISDB Securables and Permissions
Securable
Permissions
Read Modify Execute Manage Permission
Create New
Read Objects
Modify Objects
Execute Objects
Manage Objects
Permission
Folder ⦁ ⦁ ⦁ ⦁ ⦁ ⦁ ⦁ ⦁Project ⦁ ⦁ ⦁ ⦁
Environment ⦁ ⦁ ⦁
Operation / Execution ⦁ ⦁ ⦁
Troubleshooting Permissions• Check explicit (raw) permission entries from view [catalog].[explicit_object_permission]• Effective (valid) permission is a computation result from explicit permission items:
[catalog].[effective_object_permissions]• The server does not check permissions if a user is sysadmin/ ssis_admin
• Each view presents a filtered rowset from a base table• Each user can see only the rows for which he has
READ permission• sysadmin / ssis_admin can see all rows in all views
• Based on best practices• Implementation based on pattern documented in
“Implementing Row- and Cell-Level Security in Classified Databases” white paper
• http://technet.microsoft.com/en-us/library/cc966395.aspx
• Troubleshooting Row-Level Security• If a user can’t see some object
1. Logon as ssis_admin to see if it’s in base table2. Get the user sid from sys.database_principals3. Query view
catalog.effective_object_permissions to make sure the sid has READ permission on the record
SSISDB Views - Row-level Security