fy06 sql intro
Post on 25-Jan-2016
32 Views
Preview:
DESCRIPTION
TRANSCRIPT
Community inTouch | SQL Server 2005Community inTouch | SQL Server 2005Developing World Class Database Developing World Class Database
ApplicationsApplications
Risman AdnanRisman AdnanDeveloper and Platform EvangelistDeveloper and Platform EvangelistMicrosoft IndonesiaMicrosoft Indonesiarismana@microsoft.comrismana@microsoft.com
AgendaAgenda
InstallationInstallation
Optimal DatabaseOptimal Database
Database DesignDatabase Design
Developing on SQL Server 2005Developing on SQL Server 2005
CachingCaching
Tuning, Monitoring, and Diagnosing Tuning, Monitoring, and Diagnosing System ProblemsSystem Problems
InstallationInstallation
Which version has the features you Which version has the features you need?need?
Server InstallationServer Installation
Database InstallationDatabase Installation
XML supportXML support SELECT … FOR XMLSELECT … FOR XML OpenXML OpenXML XML ViewsXML Views XML UpdategramsXML Updategrams XML View MapperXML View Mapper XML Bulk LoadXML Bulk Load
URL and HTTP db accessURL and HTTP db access HTTP access to cubesHTTP access to cubes Multi-instance supportMulti-instance support Integrated Data MiningIntegrated Data Mining Full-Text Search in formatted Full-Text Search in formatted
docsdocs English Query for the WebEnglish Query for the Web C2 security rating (NSA)C2 security rating (NSA) Installation disk imagingInstallation disk imaging Active Directory integrationActive Directory integration Self-management and tuningSelf-management and tuning Distributed Partitioned ViewsDistributed Partitioned Views
Log ShippingLog Shipping Parallel CREATE INDEXParallel CREATE INDEX Parallel scanParallel scan Parallel DBCCParallel DBCC Failover clusteringFailover clustering Failover cluster managementFailover cluster management 32 CPU SMP system support32 CPU SMP system support 64 GB RAM support64 GB RAM support VIA SAN supportVIA SAN support Indexed viewsIndexed views ROLAP dimension storage ROLAP dimension storage Distributed Partitioned CubesDistributed Partitioned Cubes Online index reorganizationOnline index reorganization Differential backupDifferential backup User-defined functionsUser-defined functions Server-less snapshot backup Server-less snapshot backup SQL Query Analyzer debuggerSQL Query Analyzer debugger New data typesNew data types
Column-level Column-level collationscollations
Virtual Cube EditorVirtual Cube Editor Linked cubesLinked cubes MDX BuilderMDX Builder DimensionsDimensions Security in Analysis Security in Analysis
ServicesServices OLAP ActionsOLAP Actions Custom rollupsCustom rollups Cascading Cascading
referential referential integrity and actionsintegrity and actions
INSTEAD OF INSTEAD OF triggerstriggers
Indexes on Indexes on computed columnscomputed columns
Queued replicationQueued replication DTS enhancementsDTS enhancements Copy Database Copy Database
WizardWizard
SQL Server 2000SQL Server 2000A Major ReleaseA Major Release
SQL Server 2005SQL Server 2005.NET Framework.NET Framework
Common Language Runtime IntegrationCommon Language Runtime IntegrationUser-defined AggregatesUser-defined AggregatesUser-defined Data TypesUser-defined Data TypesUser-defined FunctionsUser-defined FunctionsSQL Server .NET Data ProviderSQL Server .NET Data ProviderExtended TriggersExtended Triggers
Data TypesData TypesNew XML DatatypeNew XML Datatypen/varchar(max)n/varchar(max)
SQL Server EngineSQL Server EngineNew Message Service BrokerNew Message Service BrokerHTTP Support (Native HTTP)HTTP Support (Native HTTP)Database Tuning Advisor Database Tuning Advisor Enhanced Read ahead & scanEnhanced Read ahead & scanOnline Index RebuildOnline Index RebuildMultiple Active Result Sets Multiple Active Result Sets Persisted Computed ColumnsPersisted Computed ColumnsQueuing SupportQueuing SupportSnapshot Isolation LevelSnapshot Isolation LevelNon-blocking Read Committed Non-blocking Read Committed Table and Index PartitioningTable and Index PartitioningNUMA supportNUMA support
Database Availability and RedundancyDatabase Availability and RedundancyFail-over Clustering (up to 8 node)Fail-over Clustering (up to 8 node)Enhanced Multi-instance SupportEnhanced Multi-instance SupportDatabase MirroringDatabase MirroringDatabase SnapshotsDatabase Snapshots
XMLXMLXQUERY Support (Server & Mid Tier)XQUERY Support (Server & Mid Tier)XML Data Manipulation Language XML Data Manipulation Language FOR XML EnhancementsFOR XML EnhancementsXML Schema (XSD) Support XML Schema (XSD) Support MSXML 6.0 (Native)MSXML 6.0 (Native) .Net XML Framework Support for .Net XML Framework Support for
XML and CLR integrationXML and CLR integrationNotification ServicesNotification Services
Management ToolsManagement ToolsSSMSSSMSMDX Query EditorMDX Query Editor
Version Control SupportVersion Control SupportSQLCMD Command Line ToolSQLCMD Command Line ToolSQL Service ManagerSQL Service Manager
Database MaintenanceDatabase MaintenanceBackup and Restore EnhancementsBackup and Restore EnhancementsChecksum Integrity ChecksChecksum Integrity ChecksDedicated Administrator ConnectionDedicated Administrator ConnectionDynamic AWEDynamic AWEFast RecoveryFast RecoveryHighly-available UpgradeHighly-available UpgradeOnline Index OperationsOnline Index OperationsOnline RestoreOnline RestoreParallel DBCC\Index OperationsParallel DBCC\Index Operations
Performance TuningPerformance Tuning Profiler EnhancementsProfiler EnhancementsProfiler/Perfmon IntegrationProfiler/Perfmon IntegrationProfiling Analysis ServicesProfiling Analysis ServicesExportable XML Showplan Exportable XML Showplan Exportable Deadlock TracesExportable Deadlock Traces
Full-text SearchFull-text Search Indexing of XML DatatypeIndexing of XML Datatype Index files integrated with SQL backupIndex files integrated with SQL backup
MDAC MDAC Side by Side installationSide by Side installationMicrosoft Installer base setupMicrosoft Installer base setupSupport for Active Directory DeploymentSupport for Active Directory Deployment
SQL Client .NET Data ProviderSQL Client .NET Data ProviderServer Cursor SupportServer Cursor SupportBulk load supportBulk load support
SecuritySecurityNo default metadata accessNo default metadata accessFine Grained Administration RightsFine Grained Administration RightsSeparation of Users and SchemaSeparation of Users and SchemaSSL Encryption at installationSSL Encryption at installationData EncryptionData EncryptionCode-signingCode-signing
ReplicationReplicationAuto-tuning Replication AgentsAuto-tuning Replication AgentsOracle PublicationOracle Publication Improved Blob Change TrackingImproved Blob Change Tracking
OLAP and Data MiningOLAP and Data MiningAnalysis Management Objects Analysis Management Objects Windows Integrated Backup and RestoreWindows Integrated Backup and RestoreUDM, multi-fact table supportUDM, multi-fact table supportDTS and DM IntegrationDTS and DM IntegrationEight new DM algorithmsEight new DM algorithmsAuto Packaging and DeploymentAuto Packaging and Deployment
SQL Integration Services (SQLIS)SQL Integration Services (SQLIS)New Architecture (DTR + DTP)New Architecture (DTR + DTP)Complex Control FlowsComplex Control FlowsControl Flow DebuggingControl Flow DebuggingFor Each EnumerationsFor Each EnumerationsProperty MappingsProperty MappingsFull Data Flow DesignerFull Data Flow DesignerFull DTS Control Flow DesignerFull DTS Control Flow DesignerGraphical Package ExecutionGraphical Package Execution Immediate Mode and Project ModeImmediate Mode and Project ModePackage (Advanced) Deployment ToolsPackage (Advanced) Deployment ToolsCustom Tasks and TransformationsCustom Tasks and Transformations
Reporting ServicesReporting ServicesMultiple Output Formats Multiple Output Formats Parameters (Static, Dynamic, Hierarchical)Parameters (Static, Dynamic, Hierarchical)Bulk Delivery of Personalized ContentBulk Delivery of Personalized ContentSupport Multiple Data Sources Support Multiple Data Sources STS (Web Parts, Doc Libraries)STS (Web Parts, Doc Libraries)Visual Design ToolVisual Design ToolCharting, Sorting, Filtering, Drill-ThroughCharting, Sorting, Filtering, Drill-ThroughScheduling, CachingScheduling, CachingComplete Scripting EngineComplete Scripting EngineScale Out architectureScale Out architectureOpen XML Report DefinitionOpen XML Report Definition
Dynamic Management ViewsDynamic Management Views
Developer and DBA ConvergenceDeveloper and DBA Convergence
Developers and DBA’s have traditionally Developers and DBA’s have traditionally worked to an interface, defined by:worked to an interface, defined by:
Table or View definitionTable or View definitionStored Procedure & result set definitionStored Procedure & result set definition
UIUIDevelopmentDevelopment
Web ServiceWeb ServiceDevelopmentDevelopment
DatabaseDatabaseDevelopmentDevelopment
Client/ServerClient/ServerDevelopmentDevelopment
DatabaseDatabaseDevelopmentDevelopment
CLR Development- Procedures- Functions- Triggers- Types- Aggregates
XML Development
InterfaceInterface
Communication BarriersCommunication Barriers
Pockets of information within disciplinesPockets of information within disciplines
Unclear delineation of responsibilitiesUnclear delineation of responsibilities
Conflicting best practices and architecturesConflicting best practices and architectures
Conflicting strategic goals and objectivesConflicting strategic goals and objectives
DatabaseDatabaseArchitectArchitectSolutionSolution
ArchitectArchitect
Database Administrationdoesn’t understand our development priorities
Development doesn’tunderstand good querydesign
Reduce complexityReduce complexitythrough clear releasethrough clear releasemanagement processesmanagement processes
Increase communication Increase communication and collaborationand collaboration
via product integrationvia product integration
Developer and DBA Developer and DBA ConvergenceConvergenceDevelopers:Developers:
Get productive with Get productive with the new toolsthe new tools
Closer integration of Closer integration of DBAs into the DBAs into the development processdevelopment process
Design application to Design application to be:be:
SupportableSupportable
Resilient to changeResilient to change
Ensure secure, robust Ensure secure, robust implementationimplementation
DBAs:DBAs:Maintain role as Maintain role as data stewarddata steward
Understand & Understand & advise on advise on appropriate appropriate technology choicestechnology choices
Factor XML & CLR Factor XML & CLR into schema designinto schema design
Ensure secure, Ensure secure, robust robust implementationimplementation
Bridge the gap!Bridge the gap!
OverviewOverview
Installing SQL Server 2005Installing SQL Server 2005Clean installClean install
ComponentsComponents
Security Settings during InstallSecurity Settings during InstallSetting the startup Service Account Setting the startup Service Account
Choosing AuthenticationChoosing Authentication
Verifying Initial ConfigurationVerifying Initial ConfigurationSQL Server Surface Area ConfigurationSQL Server Surface Area Configuration
Services Services
FeaturesFeatures
SQL Server Configuration ManagerSQL Server Configuration ManagerService Account ChangesService Account Changes
EncryptionEncryption
Installing SQL Server 2005Installing SQL Server 2005
Getting a current buildGetting a current buildInstallation through Windows InstallerInstallation through Windows InstallerPre-installation checklistPre-installation checklist
Operating System ChoicesOperating System ChoicesVisual Studio – Beta?Visual Studio – Beta?
Versions from which to chooseVersions from which to choose32-bit v 64-bit (Itanium and x64)32-bit v 64-bit (Itanium and x64)Developer, Enterprise, Express…Developer, Enterprise, Express…
Licensing and featuresLicensing and featuresSCC (System Configuration Checker)SCC (System Configuration Checker)InstallationInstallation
Build Naming ConventionsBuild Naming Conventions
Stability ImprovesStability ImprovesInterim builds/Internal ReleasesInterim builds/Internal ReleasesIDW – Internal Developer WorkstationIDW – Internal Developer WorkstationCTP – Community Tech PreviewCTP – Community Tech PreviewBeta – Wider ReleaseBeta – Wider ReleaseRC – Release CandidateRC – Release CandidateRTM – Release to Manufacturing RTM – Release to Manufacturing (“golden”)(“golden”)RTM + Hotfix – Only to specific RTM + Hotfix – Only to specific customerscustomersRTM + SP Beta – Service Pack BetaRTM + SP Beta – Service Pack BetaRTM + SP – Service Pack ReleaseRTM + SP – Service Pack Release
Choosing a Product to InstallChoosing a Product to Install
SQL Server 2005 Developer EditionSQL Server 2005 Developer Edition32-bit and 64-bit32-bit and 64-bit
64-bit (Itanium and x64 for AMD Athlon and 64-bit (Itanium and x64 for AMD Athlon and Intel EMT64)Intel EMT64)
SQL Server 2005 Standard EditionSQL Server 2005 Standard Edition32-bit and 64-bit32-bit and 64-bit
SQL Server 2005 Workgroup EditionSQL Server 2005 Workgroup Edition32-bit only32-bit only
SQL Server 2005 Express EditionSQL Server 2005 Express Edition32-bit only32-bit onlyTools in separate MSI (XM Download)Tools in separate MSI (XM Download)
Can run multiple versions side-by-Can run multiple versions side-by-sideside
Microsoft Express Product Microsoft Express Product RangeRange
Brian’s article: Brian’s article: Get a Lean, Mean Dev Machine with the Express Editions of Visual Basic and SQL Server 2005Get a Lean, Mean Dev Machine with the Express Editions of Visual Basic and SQL Server 2005 http://msdn.microsoft.com/msdnmag/issues/04/09/ExpressEditions/toc.asp?frame=true
Installation as an “Application”Installation as an “Application”
Not using Install ShieldNot using Install ShieldThird party productThird party productHard to “control”Hard to “control”Only stopped installation before file copy: Only stopped installation before file copy: “Installation has enough information to start “Installation has enough information to start copying files…”copying files…”Limited rollback capabilities, uninstalling partial Limited rollback capabilities, uninstalling partial and failed install was messyand failed install was messy
Uses Windows InstallerUses Windows InstallerMicrosoft productMicrosoft productCan download/review Windows Installer SDKCan download/review Windows Installer SDK
MSDN, Win32 and COM Development, MSDN, Win32 and COM Development, Administration and Management, Setup, Windows Administration and Management, Setup, Windows InstallerInstaller
Components have an MSI fileComponents have an MSI fileInstallation can be rolled back until the very Installation can be rolled back until the very endend
Options for InstallationOptions for Installation
Local Installation – using GUILocal Installation – using GUI
Local “unattended” and/or “silent” Local “unattended” and/or “silent” installationinstallation
Even for clusters!Even for clusters!
Remote installationRemote installation
Component UpdateComponent Update
.NET .NET FrameworFramework 2.0k 2.0
SQL SQL Native Native Client Client (SNAC)(SNAC)
Support Support FilesFiles
.NET Framework 2.0.NET Framework 2.0
Required for:Required for:SQL Server Management StudioSQL Server Management StudioExpress Manager (used for SQL Express)Express Manager (used for SQL Express)Engine (used by SQLCLR)Engine (used by SQLCLR)
How to Identify .NET Framework How to Identify .NET Framework Version on your MachineVersion on your Machine
%WINDIR%\Microsoft.NET\Framework\%WINDIR%\Microsoft.NET\Framework\version Right-click mscorlib.dll, click version Right-click mscorlib.dll, click Properties, and then click Version.Properties, and then click Version.Administrative Tools, .NET Framework 2.0 Administrative Tools, .NET Framework 2.0 Configuration. At the top of the right Configuration. At the top of the right pane, .NET Framework version displays.pane, .NET Framework version displays.
SQL Native ClientSQL Native Client
Why SNAC?Why SNAC?Separates “infrastructure” from “drivers”Separates “infrastructure” from “drivers”SNAC is owned by SQL and only updated SNAC is owned by SQL and only updated by SQL (during install/service packs)by SQL (during install/service packs)SNAC is on the “drivers” sideSNAC is on the “drivers” sideInfrastructure owned by OS and only Infrastructure owned by OS and only updated by OSupdated by OSEnables side-by-side driver installation Enables side-by-side driver installation with minimal downtimewith minimal downtimeMDAC doesn’t allow side-by-side MDAC doesn’t allow side-by-side therefore requires stopping the apptherefore requires stopping the appMDAC to be removed on 64-bit as wellMDAC to be removed on 64-bit as well
System System Consistency Consistency
CheckerChecker
Directly Directly impacts the impacts the likelihood of likelihood of successful successful installinstall
Doesn’t Doesn’t mean you’ll mean you’ll have have success success with with upgrade…upgrade…
Hardware RecommendationsHardware Recommendations
File System—NTFS, no compression for installation File System—NTFS, no compression for installation drive or primary database file or log filedrive or primary database file or log file
More smaller/faster disks over fewer/larger disksMore smaller/faster disks over fewer/larger disks
Be sure to update all drivers and firmware!Be sure to update all drivers and firmware!
OS should be on redundant disksOS should be on redundant disks
SQL installation should be on redundant diskSQL installation should be on redundant disk
Transaction logs should be on fast, redundant and Transaction logs should be on fast, redundant and preferably isolated diskspreferably isolated disks
TempDB should be isolated and if on multi-proc TempDB should be isolated and if on multi-proc use multiple files (Q328551)use multiple files (Q328551)
Databases should be “moved” to support better Databases should be “moved” to support better performance (Q224071)performance (Q224071)
KB Articles KB Articles Q224071 and Q328551 both on SQL Q224071 and Q328551 both on SQL Server 2000; best practices still applyServer 2000; best practices still apply
Components to InstallComponents to Install
Separate Separate “package“packages” (.MSI s” (.MSI files)files)High High Level is Level is for for “typical” “typical” set of set of featuresfeaturesClick Click Advanced Advanced for for feature feature treetree
Components to InstallComponents to Install
SQL Server Database SQL Server Database ServicesServices
Database Engine, Replication, Full-TextDatabase Engine, Replication, Full-Text
Analysis ServicesAnalysis Services Analysis Services server components Analysis Services server components for Business Intelligencefor Business Intelligence
Reporting Services Reporting Services Reporting Services and Report ManagerReporting Services and Report Manager
Notification Services Notification Services Notification Services (Engine and Notification Services (Engine and Client), Bulk Event for XMLClient), Bulk Event for XML
Data Transformation Data Transformation Services (Integration Services (Integration Services)Services)
SQL IS replaces SQL DTSSQL IS replaces SQL DTS
Workstation components, Workstation components, Books Online and Books Online and development toolsdevelopment tools
Management Tools, Command Prompt Management Tools, Command Prompt Tools, Reporting Services Tools, Tools, Reporting Services Tools, Connectivity Components, Connectivity Components, Programming Models, Management Programming Models, Management Studio, Computer Manager, Profiler, Studio, Computer Manager, Profiler, and Replication Monitorand Replication Monitor
Business Intelligence Development Business Intelligence Development Studio Studio
Documentation and samples - Books Documentation and samples - Books Online, code samplesOnline, code samples
Default and Named InstancesDefault and Named Instances
Default InstanceDefault InstanceAccessed by “computername”Accessed by “computername”
SQL Server Service = mssqlserverSQL Server Service = mssqlserver
SQL Server Agent Service = sqlserveragentSQL Server Agent Service = sqlserveragent
Full-Text = MSFTESQLFull-Text = MSFTESQL
Analysis Server = MSSQLServerOLAPService Analysis Server = MSSQLServerOLAPService
Named InstanceNamed InstanceAccessed by computername\instancenameAccessed by computername\instancename
SQL Server Service = mssql$instancenameSQL Server Service = mssql$instancename
SQL Server Agent Service = SQL Server Agent Service = sqlagent$instancenamesqlagent$instancename
Full-Text = msftesql$instancenameFull-Text = msftesql$instancename
Analysis Server = MSOLAP$instancenameAnalysis Server = MSOLAP$instancename
Directory StructureDirectory Structure
No longer dependent on instance No longer dependent on instance namename
All directories MSSQL.n even for other All directories MSSQL.n even for other components such as Analysis components such as Analysis ServicesServices
How can you detect?How can you detect?Registry: HKEY_LOCAL_MACHINE\Registry: HKEY_LOCAL_MACHINE\SOFTWARESOFTWARE\Microsoft\Microsoft SQL Server\Microsoft\Microsoft SQL Server\Instance Names\SQL\Instance Names\SQL
Name = Instance name (without MSSQL)Name = Instance name (without MSSQL)
Data = Install Directory (mssql.n)Data = Install Directory (mssql.n)
Installation Directories Installation Directories
2000
2005
Programmatically Detecting DirProgrammatically Detecting DirDECLARE @InstanceName sql_variant,
@InstanceDir sql_variant,@SQLDataRoot nvarchar(512),@ExecStr nvarchar(max)
SELECT @InstanceName = ISNULL(SERVERPROPERTY('InstanceName'), 'MSSQLServer')
EXECUTE master.dbo.xp_regread 'HKEY_LOCAL_MACHINE', 'SOFTWARE\Microsoft\Microsoft SQL Server\Instance Names\
SQL', @InstanceName, @InstanceDir OUTPUT
SELECT @ExecStr = 'EXECUTE master.dbo.xp_regread '+ '''HKEY_LOCAL_MACHINE'', ' + '''SOFTWARE\Microsoft\Microsoft SQL Server\' + convert(varchar, @InstanceDir) + '\Setup'', ''SQLDataRoot'', @SQLDataRoot OUTPUT'
EXEC master.dbo.sp_executesql @ExecStr, N'@SQLDataRoot nvarchar(512) OUTPUT', @SQLDataRoot OUTPUT
DECLARE @InstanceName sql_variant,@InstanceDir sql_variant,@SQLDataRoot nvarchar(512),@ExecStr nvarchar(max)
SELECT @InstanceName = ISNULL(SERVERPROPERTY('InstanceName'), 'MSSQLServer')
EXECUTE master.dbo.xp_regread 'HKEY_LOCAL_MACHINE', 'SOFTWARE\Microsoft\Microsoft SQL Server\Instance Names\
SQL', @InstanceName, @InstanceDir OUTPUT
SELECT @ExecStr = 'EXECUTE master.dbo.xp_regread '+ '''HKEY_LOCAL_MACHINE'', ' + '''SOFTWARE\Microsoft\Microsoft SQL Server\' + convert(varchar, @InstanceDir) + '\Setup'', ''SQLDataRoot'', @SQLDataRoot OUTPUT'
EXEC master.dbo.sp_executesql @ExecStr, N'@SQLDataRoot nvarchar(512) OUTPUT', @SQLDataRoot OUTPUT
Startup Service Startup Service
CustomizCustomize allows e allows you to you to set each set each individualindividually and to ly and to different different accountsaccounts
Local Local System System oror
Domain Domain AccountAccount……
Service Account ChoicesService Account Choices
Starting with most secure/controlledStarting with most secure/controlledDedicated Local account* (not local Dedicated Local account* (not local administrator)administrator)Local service (do other services use Local service (do other services use this?)this?)Dedicated Domain account (not local Dedicated Domain account (not local administrator)administrator)Network service (do other services use Network service (do other services use this?)this?)Local account* which is a local Local account* which is a local administratoradministratorDomain account which is a local Domain account which is a local administratoradministrator
NEVER use:NEVER use:Local System Local System Domain AdministratorDomain Administrator
* Local account information is stored in the local SAM. Domain accounts are likely to be far more secure froma physical perspective.
AuthenticationAuthentication
WindowsWindowsMixedMixedPassword Password only only specified specified for Mixed for Mixed (change (change as of as of IDW13) IDW13) due to due to unattendeunattended d installatioinstallation n requiremerequirementnt
Windows AuthenticationWindows Authentication
More SecureMore SecureStrong Password requirementsStrong Password requirementsAccount Lockout policiesAccount Lockout policiesPassword expirationPassword expiration
Can be changed laterCan be changed laterAfter installation be sure to manually After installation be sure to manually set the password of the sa account set the password of the sa account otherwise internally set to a otherwise internally set to a randomly generated password—if you randomly generated password—if you change to mixed mode you will change to mixed mode you will “enable” your sa account but not “enable” your sa account but not know the password. However, you know the password. However, you can easily change it.can easily change it.
Mixed Mode AuthenticationMixed Mode Authentication
Windows Authentication plusWindows Authentication plusSQL Server AuthenticationSQL Server Authentication
Uses Windows 2003 Server password Uses Windows 2003 Server password policies, if installed on Windows 2003 policies, if installed on Windows 2003 ServerServerRequires sa password to be set during Requires sa password to be set during installation if Mixed mode is choseninstallation if Mixed mode is chosenStronger password requirements in 2005Stronger password requirements in 2005
All passwords must be at least 6 characters All passwords must be at least 6 characters long and satisfy at least three of the four long and satisfy at least three of the four criteria:criteria:
It must contain uppercase letters (A-Z)It must contain uppercase letters (A-Z)It must contain lowercase letters (a-z)It must contain lowercase letters (a-z)It must contain numbers (0-9)It must contain numbers (0-9)It must contain non-alphanumeric characters (!, #, It must contain non-alphanumeric characters (!, #, %, or $)%, or $)
See BOL for complete list of requirementsSee BOL for complete list of requirements
Collation SettingCollation Setting
Affects Affects searches, searches, lookups and lookups and ORDER BYORDER BYCharacter Character data is data is affected:affected:
ASCiiASCiicharcharvarcharvarchartexttext
UnicodeUnicodeNcharNcharNvarcharNvarcharNtextNtext
Collation ConceptsCollation Concepts
May be changed at any point in chainMay be changed at any point in chain
Different options offer different gains – most in Different options offer different gains – most in language sorting, some in performancelanguage sorting, some in performance
Can assign on the fly for just a single query – or Can assign on the fly for just a single query – or within a view for subsequent usewithin a view for subsequent use
Computed Columns and Indexed Views can aid in Computed Columns and Indexed Views can aid in performance for lookups with changed collations performance for lookups with changed collations (there are likely to be version specific restrictions)(there are likely to be version specific restrictions)
Windows Install
SQL Server Install
Database Creation
Table Creation
Query Usage
I n h e r i t e d
} Use COLLATE option to change
Collation OptionsCollation OptionsTo see the list of collationsTo see the list of collationsSELECT * FROM ::fn_helpcollations()SELECT * FROM ::fn_helpcollations()
To see the server's settingTo see the server's settingsp_helpsortsp_helpsort
To see the database's settingTo see the database's settingsp_helpdb dbnamesp_helpdb dbname
To see the table's setting (for each To see the table's setting (for each column)column)sp_help tnamesp_help tname
For more information, check out these For more information, check out these BOL topics:BOL topics:
COLLATECOLLATE
"Specifying Collations""Specifying Collations"
Initial Configuration Initial Configuration
Off by defaultOff by default
SSL Encrypted CommunicationsSSL Encrypted Communications
Using SQL Server Configuration Using SQL Server Configuration ManagerManager
Running SQL Server 2000 and SQL Running SQL Server 2000 and SQL Server 2005 side-by-sideServer 2005 side-by-side
Special Feature: ImagingSpecial Feature: ImagingRenaming a Windows computername – Renaming a Windows computername – without ever having started the SQL without ever having started the SQL Server instance, special caseServer instance, special case
Database Servers/InstancesDatabase Servers/Instances
Analysis ServicesAnalysis Services
Off by DefaultOff by Default
Principle of least privilegePrinciple of least privilege
Minimal Configuration to run – many Minimal Configuration to run – many installed services (even when chosen installed services (even when chosen during installation) are “off by default”during installation) are “off by default”
Secure by DesignSecure by Design
Secure by DefaultSecure by Default
Secure by DeploymentSecure by Deployment
Where to enable, where to view?Where to enable, where to view?sp_configure, Computer sp_configure, Computer
Management, catalog views, no single Management, catalog views, no single central waycentral way
Enter SQL Surface Area ConfigurationEnter SQL Surface Area Configuration
Surface Area ConfigurationSurface Area Configuration
SQL Server Program Group, SQL Server Program Group, Configuration ToolsConfiguration Tools
Review SQL Server Setup Help from Review SQL Server Setup Help from link on main dialoglink on main dialog
Configure Services and ProtocolsConfigure Services and ProtocolsIn this first release more options in In this first release more options in Configuration ManagerConfiguration Manager
Surface Area Configuration for Surface Area Configuration for FeaturesFeatures
““Lockdown” most common/vulnerable Lockdown” most common/vulnerable featuresfeatures
sp_configure settingssp_configure settings
Catalog View queries to view (endpoints)Catalog View queries to view (endpoints)
Services – Local Only?Services – Local Only?
SQLSAC – HTTP EndpointsSQLSAC – HTTP Endpoints
Encrypted CommunicationsEncrypted Communications
SSL is used even if not present at SSL is used even if not present at installationinstallation
Checked at installation, if present usedChecked at installation, if present used
If not present, SQL Server generates a If not present, SQL Server generates a 1024 bit certificate1024 bit certificate
Logins/passwords encrypted alwaysLogins/passwords encrypted always
Uses certificate created at installation Uses certificate created at installation if SSL not presentif SSL not present
Use “Certificate Picker” in SQL Server Use “Certificate Picker” in SQL Server Configuration Manager to explicitly Configuration Manager to explicitly choose SSL certificatechoose SSL certificate
SSL Encryption/Certificate SSL Encryption/Certificate PickerPickerProtocol Protocol PropertiPropertieses
Login/Login/pwd pwd encrypteencryptedd
ALL ALL traffic traffic encrypteencrypted = YESd = YES
CertificaCertificate Picker te Picker tabtab
Configuration ManagerConfiguration ManagerStart and Stop ServicesStart and Stop ServicesChange auto start and service Change auto start and service propertiespropertiesDoes not require your services Does not require your services to be started to change to be started to change propertiespropertiesService Account password Service Account password changes don’t require restart of changes don’t require restart of serviceserviceControl Supported Network Control Supported Network libraries and SSL encryptionlibraries and SSL encryptionCreate aliasesCreate aliasesLaunch from:Launch from:
SSMS, Right click Configure SSMS, Right click Configure ServicesServicesSQL Configuration ManagerSQL Configuration ManagerComputer ManagementComputer Management
ServiceServiceControl ManagerControl Manager
Server NetworkServer NetworkUtilityUtility
SQL ServerSQL Server“Properties”“Properties”
SQL ConfigurationSQL ConfigurationManagerManager
Computer Manager – Computer Manager – 2000/20052000/2005
Running 2000/2005 “Side-by-Running 2000/2005 “Side-by-side”side”
Possible combination:Possible combination:SQL Server 2000 default/named SQL Server 2000 default/named instance(s)instance(s)SQL Server 2005 default/named SQL Server 2005 default/named instance(s)instance(s)
SQL Server 2005 can support SQL Server 2005 can support multiple instances of multiple versions multiple instances of multiple versions (Developer and Express)(Developer and Express)
SQL Server 2000SQL Server 2000Directory structure was related to Directory structure was related to instance type and nameinstance type and name
SQL Server 2005SQL Server 2005Directory structure NOT related to Directory structure NOT related to instance type or nameinstance type or name
Re-register COM Components?Re-register COM Components?
Renaming A Server After Renaming A Server After StartupStartup
Server name is NOT detected by SQL ServerServer name is NOT detected by SQL Server
Review the list of servernamesReview the list of servernamessp_helpserversp_helpserver
Drop the old servernameDrop the old servernamesp_dropserver 'oldservername'sp_dropserver 'oldservername'
Add the new servername – make sure you Add the new servername – make sure you define it as the “local” instance of SQL define it as the “local” instance of SQL ServerServer
sp_addserver 'newservername', localsp_addserver 'newservername', local
May have problems with servername related May have problems with servername related objects/jobs – i.e. msdb.dbo.sysjobsobjects/jobs – i.e. msdb.dbo.sysjobs
update sysjobs update sysjobs set originating_server = ' set originating_server = 'newnewservername'servername'where originating_server = 'where originating_server = 'oldoldservername'servername'
Optimal Database StructuresOptimal Database Structures
Anatomy of Data ModificationsAnatomy of Data Modifications
Optimizing Data FilesOptimizing Data Files
Logging and RecoveryLogging and Recovery
TransactionsTransactions
Database StructureDatabase Structure
Up to 32,767 Databases per Instance
Up to 32,767 Files per Database (Total is for both Data and Log files)
Minimum of 2 files – one for data, one for log
MDF = Primary (MAIN) Data File
NDF = Non-Primary Data File
LDF = Log Data File
Should be only 1 log file
Data FileData File*.mdf (1)*.mdf (1)
*.ndf (0-*.ndf (0-nn))
Log FileLog File*.ldf (1-*.ldf (1-nn))
DatabaseDatabase
1.1. User sends UPDATEUser sends UPDATE Update is highly selective (only 5 rows)Update is highly selective (only 5 rows) Indexes exist to aid in finding these rows Indexes exist to aid in finding these rows
efficientlyefficiently The update is a SINGLE statement batch The update is a SINGLE statement batch
NOT enclosed in BEGIN TRAN…COMMIT NOT enclosed in BEGIN TRAN…COMMIT TRAN block therefore this is IMPLICIT TRAN block therefore this is IMPLICIT transactiontransaction
The Anatomy of a Data The Anatomy of a Data ModificationModification
2.2. Server receives the request and Server receives the request and locates the data in cache OR reads locates the data in cache OR reads the data from disk into cachethe data from disk into cache
Since this is highly selective only the Since this is highly selective only the necessary pages are read into cache necessary pages are read into cache (maybe a few extra but that’s not (maybe a few extra but that’s not important here)important here)
Let’s use an example where the 5 rows Let’s use an example where the 5 rows being modified are located on 3 different being modified are located on 3 different data pagesdata pages
The Anatomy of a Data The Anatomy of a Data ModificationModificationThe Anatomy of a Data The Anatomy of a Data ModificationModification
What it looks like:What it looks like:DataData
UPDATE…
Data
Log
Server…
Cache
3.3. SQL Server proceeds to lock the SQL Server proceeds to lock the necessary datanecessary data
Locks are necessary to give us a Locks are necessary to give us a consistent point FOR ALL rows from which consistent point FOR ALL rows from which to startto start
If any other transaction(s) have ANY of If any other transaction(s) have ANY of these rows locked we will wait until ALL these rows locked we will wait until ALL locks have been acquired before we can locks have been acquired before we can proceed.proceed.
In the case of this update In the case of this update (because it’s (because it’s highly selective and because indexes highly selective and because indexes exist to make this possible) exist to make this possible) SQL Server SQL Server will use row level locking.will use row level locking.
The Anatomy of a Data The Anatomy of a Data ModificationModificationThe Anatomy of a Data The Anatomy of a Data ModificationModification
What it looks like: What it looks like: LocksLocks
Cache
Page
Page
Page
Row
Row
Row
Row
Row
Update Lock
Update Lock
Update Lock
Update Lock
Update Lock
4.4. SQL Server can now begin to make the SQL Server can now begin to make the modifications – for EVERY row the process modifications – for EVERY row the process will include:will include:1.1. Change to a stricter lock (eXclusive lock) Change to a stricter lock (eXclusive lock)
An update lock helps to allow better concurrency An update lock helps to allow better concurrency by being compatible with other shared locks by being compatible with other shared locks (readers). Readers can read the pre-modified data (readers). Readers can read the pre-modified data as it is transactionally consistentas it is transactionally consistent
The eXclusive lock is required to make the change The eXclusive lock is required to make the change because once modified no other reads should be because once modified no other reads should be able to see this un-committed changeable to see this un-committed change
2.2. Make the modification (in cache)Make the modification (in cache)3.3. Log the modification to the transaction log pages Log the modification to the transaction log pages
(also in cache)(also in cache)
The Anatomy of a Data The Anatomy of a Data ModificationModificationThe Anatomy of a Data The Anatomy of a Data ModificationModification
What it looks like:What it looks like:ModificationsModifications
Cache
Page
Page
Page
Row
Row
Row
Row
Row
Update LockExclusive Lockxx
Update LockExclusive Lockxx
Update LockExclusive Lockxx
Update LockExclusive Lockxx
Update LockExclusive Lockx
x
L
5.5. Finally, the transaction is complete – this is Finally, the transaction is complete – this is the MOST critical stepthe MOST critical step
All rows have been modifiedAll rows have been modified There are no other statements in this transaction There are no other statements in this transaction
– i.e. Implicit transaction– i.e. Implicit transaction Steps are:Steps are:
1.1. Write all log pages to transaction log ON Write all log pages to transaction log ON DISKDISK
2.2. Release the locksRelease the locks
3.3. Send a message to the user:Send a message to the user:
(5 Rows Affected)(5 Rows Affected)
The Anatomy of a Data The Anatomy of a Data ModificationModificationThe Anatomy of a Data The Anatomy of a Data ModificationModification
What it looks like:What it looks like:Write-Ahead LoggingWrite-Ahead Logging
LData
Log
Server…
Cache~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sequential writesChangeChangeChangeChange…
Log
5 Rows Affected
After the log entries are made and the locks are
released…
So now what?So now what?
The transaction log ON DISK – is up to The transaction log ON DISK – is up to datedate
The data in CACHE – is up to dateThe data in CACHE – is up to date
But when does the data get written But when does the data get written from cache to disk?from cache to disk?
CheckpointCheckpoint
CheckpointCheckpoint
It’s important to realize that a checkpoint It’s important to realize that a checkpoint does NOT just write committed pages… does NOT just write committed pages… Instead a checkpoint writes Instead a checkpoint writes ALLALL pages pages which have changed since they were which have changed since they were brought into cache – brought into cache – regardlessregardless of the state of the state of the transaction which changed them!of the transaction which changed them!
Why?Why?To reduce roll-forward recovery time To reduce roll-forward recovery time during restart recoveryduring restart recovery
To batch I/Os to disk and reduce disk To batch I/Os to disk and reduce disk thrashing for data writesthrashing for data writes
Transaction Recovery and Transaction Recovery and Checkpoints Checkpoints
Transactions… Action Requiredif restart recovery
None
Checkpoint System Failure
11
22
33
44
55
Roll forward
Roll back
Roll forward
Roll back
L D
L/D
L/D
L
L
Time
Restart RecoveryRestart Recovery
Automatically on system restartAutomatically on system restartSQL Server 2000SQL Server 2000
RedoRedoUndoUndoUsers allowed access to databaseUsers allowed access to database
SQL Server 2005 (Enterprise Edition Only)SQL Server 2005 (Enterprise Edition Only)RedoRedoUsers allowed access to databaseUsers allowed access to databaseUndoUndo
Faster access to database – including faster Faster access to database – including faster failover on Clusterfailover on ClusterRecords are protected (i.e. Locked)Records are protected (i.e. Locked)
Key PointsKey Points
Data Portion mostly random reads – Data Portion mostly random reads – except at checkpointexcept at checkpoint
Log Portion mostly sequential writesLog Portion mostly sequential writes
Separate physical disks minimizes Separate physical disks minimizes contention at the drive level – first contention at the drive level – first choice in tuningchoice in tuning
Log is critical in recoveryLog is critical in recovery
Protect the logProtect the log
Minimize impact to logMinimize impact to log
Be aware of environmental changesBe aware of environmental changes
Optimizing Data FilesOptimizing Data Files
Defrag the physical disksDefrag the physical disks
Effective RAID array configurationEffective RAID array configuration
Pre-allocate to a reasonable initial sizePre-allocate to a reasonable initial size
Don’t let auto-growth get out of controlDon’t let auto-growth get out of controlConsider allowing Fast File Initialization Consider allowing Fast File Initialization (Enterprise Edition Only—Perform Volume (Enterprise Edition Only—Perform Volume Maintenance Tasks to SQL Server service account)Maintenance Tasks to SQL Server service account)
While a lot of these will help for the log While a lot of these will help for the log (and I’ll explain why in a moment) there (and I’ll explain why in a moment) there are more important things to be aware of are more important things to be aware of IN the data portion – tables and indexes IN the data portion – tables and indexes (rebuilding)(rebuilding)
Growing Files AutomaticallyGrowing Files Automatically
Enable Data Files and Log File to Grow Enable Data Files and Log File to Grow AutomaticallyAutomatically
Allocate Reasonable Initial Sizes Allocate Reasonable Initial Sizes
Set Maximum Size for Data Files so you Set Maximum Size for Data Files so you don’t run out of space when you have no don’t run out of space when you have no disk space!disk space!
Especially important for server consolidation Especially important for server consolidation and/or shared disks re: multiple databasesand/or shared disks re: multiple databases
Set File Growth Increment to Reasonable Set File Growth Increment to Reasonable SizeSize
Preference – fixed increment less than or equal Preference – fixed increment less than or equal to 1GBto 1GB
Helped by fast file initialization (SQL Server Helped by fast file initialization (SQL Server 2005 only)2005 only)
Optimizing Log Files Optimizing Log Files
Isolate Isolate thethe Transaction Log (only one!) Transaction Log (only one!)Defrag the physical disksDefrag the physical disksEffective RAID array configurationEffective RAID array configurationPre-allocate to a reasonable initial sizePre-allocate to a reasonable initial sizeDon’t let auto-growth get out of controlDon’t let auto-growth get out of controlCheck and fix your internal Check and fix your internal fragmentationfragmentationDon’t be caught up in nothing but Don’t be caught up in nothing but speed!speed!See Kim’s blog entry: See Kim’s blog entry: 8 Steps to Better Transaction Log Throu8 Steps to Better Transaction Log Throughputghput, June 25, 2005, June 25, 2005
Log filesLog files
Limited read activityLimited read activityReplicationReplication
RollbackRollback
Optimize for write activityOptimize for write activityRAID 0+1 w/hot spareRAID 0+1 w/hot spare
RAID 1RAID 1
RAID 1+0 better redundancyRAID 1+0 better redundancy
preferably NOT RAID 5 preferably NOT RAID 5
Optimize for SQL Server “logging” activityOptimize for SQL Server “logging” activityCapacity Planning (initial and reasonable size)Capacity Planning (initial and reasonable size)
Minimize excessive autogrowths (clean up Minimize excessive autogrowths (clean up excessive VLFs)excessive VLFs)
How the Transaction Log WorksHow the Transaction Log Works
On commit, activity is written to the log – On commit, activity is written to the log – active and likely sequentialactive and likely sequentialActivity moves through log sequentially, Activity moves through log sequentially, fills and then goes to a second file or fills and then goes to a second file or autogrowsautogrowsExcessive autogrowth causes:Excessive autogrowth causes:
Slight pause on autogrowSlight pause on autogrowWindows call to extend a file (may cause file Windows call to extend a file (may cause file fragmentation)fragmentation)Adds “VLFs” to the file on each autogrowthAdds “VLFs” to the file on each autogrowth
Virtual Log File Virtual Log File 44
Virtual Log File Virtual Log File 55
Virtual Log File Virtual Log File 11
Virtual Log File Virtual Log File 22
Inactive Virtual Log File
Min LSN Min LSN (first open(first open
tran)tran)
Active Virtual Log File
Inactive/UnusedLog File
StartStartLogical LogLogical Log
EndEndLogical LogLogical Log
Virtual Log File Virtual Log File 33
Inactive Virtual Log File
Inactive Virtual Log File
Transaction Log OptimizationTransaction Log Optimization
Pre-allocate the log to appropriate Pre-allocate the log to appropriate sizesizeMinimize autogrowthsMinimize autogrowthsLog might NOT show 0 Percent Log Log might NOT show 0 Percent Log Used after backup for example - the Used after backup for example - the % Log Used will be ~6%% Log Used will be ~6%Disk Bound Systems might Disk Bound Systems might experience some performance experience some performance degradation at log backupdegradation at log backupConsider RAID 1+0 instead of RAID 1, Consider RAID 1+0 instead of RAID 1, RAID 5 generally is not recommendedRAID 5 generally is not recommendedOptimize existing file – minimize VLFsOptimize existing file – minimize VLFs
The Effects of LoggingThe Effects of Logging
Log is written AHEAD of the data Log is written AHEAD of the data portionportionLog is the ONLY place where Log is the ONLY place where transactional consistency is known transactional consistency is known (i.e. guaranteed)(i.e. guaranteed)Once a checkpoint occurs SQL Server Once a checkpoint occurs SQL Server doesn’t need the information in the doesn’t need the information in the log – for committed (or inactive) log – for committed (or inactive) transactions (the log could even be transactions (the log could even be cleared however…)cleared however…)Without the transaction log a Without the transaction log a database cannot function (i.e. database cannot function (i.e. marked suspect)marked suspect)Need to make sure this is redundant Need to make sure this is redundant AND optimal…AND optimal…What can effect the logging?What can effect the logging?
Minimize impact of loggingMinimize impact of logging
Cannot clear the log unless the Cannot clear the log unless the transactional information is inactive – if a transactional information is inactive – if a transaction is pending then it stays in the transaction is pending then it stays in the log until it’s completed…log until it’s completed…Restart recovery will take longer if there Restart recovery will take longer if there were a lot of long running transactions were a lot of long running transactions pending at the time of failurepending at the time of failureAlways try to avoid:Always try to avoid:
Long running transactionsLong running transactionsTransactions which span more than one batchTransactions which span more than one batchTransactions that require user interactionTransactions that require user interactionNested Transactions (complex and usually longer Nested Transactions (complex and usually longer running)running)
Consider changing recovery modelConsider changing recovery model
What can impact your ability to What can impact your ability to recover?recover?Recovery ModelsRecovery Models
Full Full Everything is Fully LoggedEverything is Fully LoggedAll operations allowed and FULLY loggedAll operations allowed and FULLY loggedOperations like creating or rebuilding an Operations like creating or rebuilding an INDEX takes as large a log as the size of INDEX takes as large a log as the size of the operationthe operation
Bulk_Logged Bulk_Logged Minimal Logging for Minimal Logging for SOMESOME Operations (not NON-LOGGED)Operations (not NON-LOGGED)
Operations whose performance is Operations whose performance is affected, see BOL Topic affected, see BOL Topic Minimally Logged OperationsMinimally Logged Operations ALL other operations (i.e. updates, ALL other operations (i.e. updates, inserts, etc. take the same log space and inserts, etc. take the same log space and time as the FULL recovery modeltime as the FULL recovery model
Simple Simple Log Truncation on Log Truncation on CheckpointCheckpoint
Logging, Recovery Models, Logging, Recovery Models, Recovery and PerformanceRecovery and Performance
Acceptable to Switch?Acceptable to Switch?
How, Why and Why Not?How, Why and Why Not?
Batch Processing ScenarioBatch Processing Scenario
Batch Processing ScriptBatch Processing Script
Switching Recovery Models – Switching Recovery Models – SummarySummary
Acceptable to Switch?Acceptable to Switch?
YesYesIf users are not allowed in database If users are not allowed in database during batch processingduring batch processingNo modifications are made during batch No modifications are made during batch process that are not recoverable through process that are not recoverable through some other means (some other means (i.e. re-running the i.e. re-running the batch processbatch process))Data loss during the batch process is OKData loss during the batch process is OK
NoNoModifications are made by Modifications are made by users/processes that are not recoverable users/processes that are not recoverable OLTP System requiring 24x7x365OLTP System requiring 24x7x365Data loss is NOT acceptableData loss is NOT acceptableDatabase Mirroring Database Mirroring
BATCH OPERATIONBATCH OPERATION
BULK_LOGGEDBULK_LOGGED
Advantages:Advantages:Minimal loggingMinimal loggingFaster Bulk OperationFaster Bulk Operation
Disadvantages:Disadvantages:No No STOPATSTOPATData loss possible if media Data loss possible if media failure…failure…If database becomes If database becomes suspect, cannot back up suspect, cannot back up tail of log.tail of log.
Advantages:Advantages:Up-to-the-Up-to-the-minute minute recoveryrecovery
Point-in-time Point-in-time recoveryrecovery
Access to Access to “tail” of the “tail” of the loglog
LOG BACKUP
FULLFULL
ALTER DATABASE <name>SET RECOVERY BULK_LOGGED
ALTER DATABASE <name>SET RECOVERY FULL
LOG BACKUP11
22
33
44
Switching Recovery ModelsSwitching Recovery Models
FULLFULL
-- Check the status before changing it!IF DATABASEPROPERTYEX('dbname', 'Updateability') = 'READ_ONLY'
ALTER DATABASE dbnameSET RESTRICTED_USER,READ_WRITE WITH ROLLBACK AFTER 60
go-- Perform Batch Operations…goALTER DATABASE dbname
SET READ_ONLY, MULTI_USERWITH ROLLBACK AFTER 30
go
-- Check the status before changing it!IF DATABASEPROPERTYEX('dbname', 'Updateability') = 'READ_ONLY'
ALTER DATABASE dbnameSET RESTRICTED_USER,READ_WRITE WITH ROLLBACK AFTER 60
go-- Perform Batch Operations…goALTER DATABASE dbname
SET READ_ONLY, MULTI_USERWITH ROLLBACK AFTER 30
go
Make sure to review the
sample script with tons of comments!
Switching Recovery ModelsSwitching Recovery ModelsUsing ALTER DATABASEUsing ALTER DATABASE
Availability, What About Availability, What About Locking?Locking?
ACID Transaction Design ACID Transaction Design RequirementsRequirements
Atomicity Consistency Atomicity Consistency IsolationIsolation DurabilityDurability
Isolation LevelsIsolation LevelsLevel 0 – Read UncommittedLevel 0 – Read Uncommitted
Level 1 – Read CommittedLevel 1 – Read Committed
Level 2 – Repeatable ReadsLevel 2 – Repeatable Reads
Level 3 – SerializableLevel 3 – Serializable
DefaultDefault Isolation Level in BOTH Isolation Level in BOTH 2000/2005 is ANSI/ISO Level 1, Read 2000/2005 is ANSI/ISO Level 1, Read CommittedCommitted
In implementation this default level In implementation this default level uses lockinguses locking
ACID PropertiesACID Properties
AtomicityAtomicityA transaction must be an atomic unit of work; A transaction must be an atomic unit of work; either all of its modifications are performed, or either all of its modifications are performed, or nonenone
ConsistencyConsistencyWhen completed, a transaction must leave all When completed, a transaction must leave all data and all related structures in a consistent data and all related structures in a consistent state state
IsolationIsolationA transaction either sees data in the state it A transaction either sees data in the state it was in before another concurrent transaction was in before another concurrent transaction modified it, or it sees the data after the second modified it, or it sees the data after the second transaction has completed, but it does not see transaction has completed, but it does not see an intermediate statean intermediate state
DurabilityDurabilityTransaction should persist despite system Transaction should persist despite system failurefailure
Isolation Level 0 – Read Isolation Level 0 – Read UncommittedUncommittedPhenomenon: Dirty readsPhenomenon: Dirty readsA read transaction can read another A read transaction can read another
transaction’s uncommitted (or in-flight) transaction’s uncommitted (or in-flight) changes – resulting in “dirty reads”changes – resulting in “dirty reads”
DML statements always use exclusive DML statements always use exclusive lockinglocking
In Implementation:In Implementation: row locks are not used row locks are not used (SCH_S locks are used) for the dirty read (SCH_S locks are used) for the dirty read transaction and locks against data being transaction and locks against data being accessed are not honoredaccessed are not honored
Resulting Phenomenon:Resulting Phenomenon: statements statements execute with the possibility of inaccurate execute with the possibility of inaccurate data since the “in-flight” data read may data since the “in-flight” data read may continue to change or even be invalidated continue to change or even be invalidated (rolled back)(rolled back)
Isolation Level 1 – Read Isolation Level 1 – Read CommittedCommittedPhenomenon: Inconsistent analysisPhenomenon: Inconsistent analysisThe default behavior in ALL releasesThe default behavior in ALL releases
In-flight transaction’s data cannot be read In-flight transaction’s data cannot be read by a read committed transaction – only by a read committed transaction – only committed changes are visiblecommitted changes are visible
DML statements always use exclusive DML statements always use exclusive lockinglocking
In ImplementationIn Implementation: locks are released (for : locks are released (for readers – not modifiers) as resources are readers – not modifiers) as resources are read, a row may be read more than once in read, a row may be read more than once in some scenariossome scenarios
Resulting PhenomenonResulting Phenomenon: Reads are not : Reads are not repeatable through the life of a transaction repeatable through the life of a transaction – as a result a row may not be read – as a result a row may not be read consistently during the life of a transactionconsistently during the life of a transaction
Isolation Level 2 – Repeatable ReadIsolation Level 2 – Repeatable ReadPhenomenon: PhantomsPhenomenon: Phantoms
In-flight transaction’s data cannot be read In-flight transaction’s data cannot be read and data modified is accessible only to the and data modified is accessible only to the repeatable read transactionrepeatable read transaction
Data read, but not modified is accessible Data read, but not modified is accessible to other transactions for reads, but not to other transactions for reads, but not DMLDML
In Implementation:In Implementation: locks are held for the locks are held for the life of a transaction, rows which are read life of a transaction, rows which are read are locked and can be repeatably read are locked and can be repeatably read during the life of a transactionduring the life of a transaction
Resulting Phenomena:Resulting Phenomena: rows which were rows which were not present at the beginning of the not present at the beginning of the transaction can appear – in the resulttransaction can appear – in the result
Isolation Level 3 – SerializableIsolation Level 3 – SerializablePhenomenon: NonePhenomenon: None
In-flight transaction’s data cannot be read In-flight transaction’s data cannot be read and data modified is accessible only to the and data modified is accessible only to the serializable transactionserializable transactionData read, but not modified is accessible to Data read, but not modified is accessible to other transactions for reads, but not DMLother transactions for reads, but not DML In Implementation:In Implementation: locks are held for the life locks are held for the life of a transaction and held at higher levels of a transaction and held at higher levels within indexes to prevent rows from entering within indexes to prevent rows from entering the “set”the “set”Implementation side effect:Implementation side effect: to prevent rows to prevent rows from entering the “set” of data, the “set” of from entering the “set” of data, the “set” of data needs to be locked. If appropriate data needs to be locked. If appropriate indexes do not exist then higher levels of indexes do not exist then higher levels of locking might be necessary – i.e., table-level locking might be necessary – i.e., table-level locking.locking.
Key range Key range and index and index reminderreminder
Isolation in ImplementationIsolation in Implementation
All versions of SQL Server – including All versions of SQL Server – including SQL Server 2005 – use a locking-SQL Server 2005 – use a locking-based implementation based implementation by defaultby default
In the default environment and to In the default environment and to reduce the possibility of various reduce the possibility of various anomalies, transactional duration and anomalies, transactional duration and possibly higher levels of locking are possibly higher levels of locking are necessary – resulting in blockingnecessary – resulting in blocking
There are benefits of locking, if true There are benefits of locking, if true “isolation” is desired“isolation” is desired
Isolation in Implementation Isolation in Implementation (cont’d)(cont’d)
SQL Server 2005 combines an SQL Server 2005 combines an optional row versioning-based optional row versioning-based implementation with this locking-implementation with this locking-based implementation based implementation
Two end results (and methods) for Two end results (and methods) for implementing row-level versioningimplementing row-level versioning
Read Committed using Statement-level Read Committed using Statement-level SnapshotSnapshotAlso known as: Also known as:
Statement-level Read ConsistencyStatement-level Read Consistency
Transaction-level Read ConsistencyTransaction-level Read ConsistencySnapshot IsolationSnapshot Isolation
SQL Server 2005 Isolation SQL Server 2005 Isolation OptionsOptions
DefaultDefault – Uses Locking, no options set – Uses Locking, no options setRead Committed with Statement-level SnapshotRead Committed with Statement-level Snapshot
DB Option: DB Option: READ_COMMITTED_SNAPSHOTREAD_COMMITTED_SNAPSHOT enabled enabledUses locking for writes, versioning for reads for statement-Uses locking for writes, versioning for reads for statement-level consistencylevel consistency
Transaction-level Snapshot IsolationTransaction-level Snapshot IsolationDB Option: DB Option: ALLOW_SNAPSHOT_ISOLATIONALLOW_SNAPSHOT_ISOLATION enabled enabledUses default locking for everything, ALLOWS snapshot for Uses default locking for everything, ALLOWS snapshot for transactions when requested through transactions when requested through SET TRANSACTION ISOLATION LEVEL SNAPSHOTSET TRANSACTION ISOLATION LEVEL SNAPSHOTIf requested, transaction-level consistency through row If requested, transaction-level consistency through row versioningversioning
Both Options ConfiguredBoth Options ConfiguredStatement-level consistency through versioning Statement-level consistency through versioning automaticallyautomaticallyTransaction-level consistency where requestedTransaction-level consistency where requested
Setting Isolation BehaviorSetting Isolation Behavior
Default BehaviorDefault Behavior
Read Committed using Statement-Level Read Committed using Statement-Level SnapshotSnapshot
Snapshot IsolationSnapshot Isolation
Both RCSI and Snapshot IsolationBoth RCSI and Snapshot Isolation
ALTER DATABASE <database_name> SET READ_COMMITTED_SNAPSHOT ONWITH ROLLBACK AFTER 5
ALTER DATABASE <database_name>SET ALLOW_SNAPSHOT_ISOLATION ON
ALTER DATABASE <database_name> SET RCSLS...ALTER DATABASE <database_name> SET Snapshot...
Read CommittedRead CommittedDefault BehaviorDefault Behavior
All phenomena – except dirty reads All phenomena – except dirty reads are possible, even in the bounds of a are possible, even in the bounds of a single select querysingle select query
In volatile databases, a long running In volatile databases, a long running query may produce inconsistent query may produce inconsistent resultsresults
Can increase isolation to remove Can increase isolation to remove phenomenaphenomena
Increasing isolation requires locks to be Increasing isolation requires locks to be held longerheld longer
This can create blockingThis can create blocking
Tips to Minimize BlockingTips to Minimize Blocking
Write efficient transactions – keep Write efficient transactions – keep them short and in one batchthem short and in one batchDo not allow interaction in the midst Do not allow interaction in the midst of the batchof the batchUse indexes to help SQL Server find – Use indexes to help SQL Server find – and lock – only the necessary dataand lock – only the necessary dataConsider estimates for long running Consider estimates for long running queries and/or migrating data to a queries and/or migrating data to a secondary analysis serversecondary analysis serverProblems with locking becoming Problems with locking becoming blocking are often when long running blocking are often when long running (and conflicting) transactions execute(and conflicting) transactions execute
Locking Prevents ConflictsLocking Prevents ConflictsAt the expense of blockingAt the expense of blocking
No other transactions can invalidate No other transactions can invalidate the data set through modificationsthe data set through modificationsIs this always what you want or Is this always what you want or need?need?
Queuing applications typically want Queuing applications typically want locks, if someone is modifying that row – locks, if someone is modifying that row – we want to go to another not see last we want to go to another not see last transactional statetransactional stateVery volatile prices – don’t want to give Very volatile prices – don’t want to give them the last price if it’s currently being them the last price if it’s currently being updated, so wait… (updated, so wait… (but the update’s but the update’s fastfast))
What about long running transactions What about long running transactions or cases where you don’t need or cases where you don’t need absolute current…absolute current…
Read Committed Snapshot IsolationRead Committed Snapshot IsolationDatabase Changed to READ_COMMITTED_SNAPSHOTDatabase Changed to READ_COMMITTED_SNAPSHOT
No phenomena are possible in the bounds No phenomena are possible in the bounds of a single statementof a single statement
In volatile databases, a multi-statement In volatile databases, a multi-statement transaction may yield different results for transaction may yield different results for different statements which access the different statements which access the same datasame data
Each statement is consistent but only for Each statement is consistent but only for the execution of that statement, not for the execution of that statement, not for the life of the transaction (the life of the transaction (if the if the transaction has multiple statementstransaction has multiple statements))
Each time data is read by a new statement Each time data is read by a new statement the latest version is usedthe latest version is used
Snapshot IsolationSnapshot IsolationDatabase Changed to ALLOW_SNAPSHOT_ISOLATIONDatabase Changed to ALLOW_SNAPSHOT_ISOLATION
Setting ALLOWS users to ask for Snapshot Setting ALLOWS users to ask for Snapshot Isolation – NOT on by defaultIsolation – NOT on by defaultALL phenomena and ALL default locking ALL phenomena and ALL default locking behavior is EXACTLY the same unless you behavior is EXACTLY the same unless you explicitly ask for Snapshot Isolationexplicitly ask for Snapshot IsolationOnce requests, no phenomena are possible Once requests, no phenomena are possible in the bounds of a transaction running in the bounds of a transaction running under snapshot isolationunder snapshot isolationIn volatile databases, a multi-statement In volatile databases, a multi-statement transaction will always see the transaction will always see the transactionally accurate version which transactionally accurate version which existed when the transaction startedexisted when the transaction startedVersions must stick around longer, multi-Versions must stick around longer, multi-statement transactions may have conflictsstatement transactions may have conflicts
Isolation Levels: In SummaryIsolation Levels: In Summary
READ UNCOMMITTED (Level 0)READ UNCOMMITTED (Level 0)““Dirty Reads” – An option ONLY for readersDirty Reads” – An option ONLY for readersAny data (even that which is Any data (even that which is in-flight/locked) can be viewedin-flight/locked) can be viewed
READ COMMITTED (Level 1 – Default)READ COMMITTED (Level 1 – Default)Only committed changes are visibleOnly committed changes are visibleData in an intermediate state cannot be Data in an intermediate state cannot be accessedaccessed
Read Committed w/statement-level Read Committed w/statement-level SnapshotSnapshot
Statement-level read consistencyStatement-level read consistencyNew non-blocking, non-locking (ex. New non-blocking, non-locking (ex. SCH_S), SCH_S), version-based Level 1version-based Level 1
Isolation Levels: In Summary Isolation Levels: In Summary (cont’d)(cont’d)
REPEATABLE READS (Level 2)REPEATABLE READS (Level 2)All reads are consistent for the life of a All reads are consistent for the life of a transactiontransactionShared locks are NOT released after the data is Shared locks are NOT released after the data is processed – does not allow writers (does allow processed – does not allow writers (does allow other readers)other readers)Does not protect entire set (phantoms may Does not protect entire set (phantoms may occur)occur)
SERIALIZEABLE (Level 3)SERIALIZEABLE (Level 3)All reads are consistent for the life of a All reads are consistent for the life of a transactiontransactionAvoids phantoms – no new recordsAvoids phantoms – no new records
Snapshot Isolation – 2005Snapshot Isolation – 2005 Transaction-Level consistency using snapshot Transaction-Level consistency using snapshot New non-blocking, non-locking, version-based New non-blocking, non-locking, version-based transactionstransactions
Controlling Isolation LevelsControlling Isolation LevelsTable-level changesTable-level changes
From Clause, per table (no spaces)From Clause, per table (no spaces)Level 0 – READUNCOMMITTED, NOLOCKLevel 0 – READUNCOMMITTED, NOLOCKLevel 1 – READCOMMITTED (locking)Level 1 – READCOMMITTED (locking)Level 1 – READCOMMITTED (versioning)Level 1 – READCOMMITTED (versioning)
Only in 2005 and Only in 2005 and onlyonly if the database option to if the database option to READ_COMMITTED_SNAPSHOT is onREAD_COMMITTED_SNAPSHOT is onCan be overridden with READCOMMITTEDLOCKCan be overridden with READCOMMITTEDLOCK
Level 2 – REPEATABLEREAD Level 2 – REPEATABLEREAD Level 3 – SERIALIZABLE, HOLDLOCKLevel 3 – SERIALIZABLE, HOLDLOCK
FROM dbo.titles WITH(READUNCOMMITTED)FROM dbo.titles WITH(READUNCOMMITTED) JOIN dbo.publishers WITH(SERIALIZABLE)JOIN dbo.publishers WITH(SERIALIZABLE)
Controlling Isolation Levels Controlling Isolation Levels Session-level changes (cont’d)Session-level changes (cont’d)
Session level settings impact entire session Session level settings impact entire session but can be overridden with table-level but can be overridden with table-level settingssettings
Level 0 – READ UNCOMMITTEDLevel 0 – READ UNCOMMITTEDLevel 1 – READ COMMITTEDLevel 1 – READ COMMITTEDLevel 2 – REPEATABLE READLevel 2 – REPEATABLE READLevel 3 – SERIALIZABLELevel 3 – SERIALIZABLE
SET TRANSACTION ISOLATION LEVEL SET TRANSACTION ISOLATION LEVEL optoptREAD UNCOMMITTEDREAD UNCOMMITTEDREAD COMMITTEDREAD COMMITTEDREPEATABLE READREPEATABLE READSERIALIZABLESERIALIZABLE
SNAPSHOTSNAPSHOTOnly in 2005 and Only in 2005 and onlyonly if the database option is on if the database option is on
Allowing Read Committed using Allowing Read Committed using Statement-level SnapshotStatement-level Snapshot
Database optionDatabase optionALTER DATABASE <database_name> ALTER DATABASE <database_name>
SET READ_COMMITTED_SNAPSHOT ONSET READ_COMMITTED_SNAPSHOT ONWITH ROLLBACK AFTER 5WITH ROLLBACK AFTER 5
No other changes necessary…No other changes necessary…No changes to your queries or your applications – No changes to your queries or your applications – they automatically return with statement-level they automatically return with statement-level consistency consistency ((NOTE: You may need to modify your applications if you NOTE: You may need to modify your applications if you depend on locking – re: queues, readers to wait for changedepend on locking – re: queues, readers to wait for change))
Changes to blocking…Changes to blocking…However, if this is NOT your performance problem However, if this is NOT your performance problem (meaning concurrency isn’t your bottleneck) then (meaning concurrency isn’t your bottleneck) then you may hinder performance not improve you may hinder performance not improve Expect this change in behavior at a costExpect this change in behavior at a cost
Allowing Snapshot IsolationAllowing Snapshot Isolation
Database optionDatabase optionALTER DATABASE <database_name>ALTER DATABASE <database_name>SET ALLOW_SNAPSHOT_ISOLATION ONSET ALLOW_SNAPSHOT_ISOLATION ON
Session settingSession settingSET TRANSACTION ISOLATION LEVEL SET TRANSACTION ISOLATION LEVEL SNAPSHOTSNAPSHOT
Changes to applications:Changes to applications:Request snapshot isolationRequest snapshot isolation
Test for conflict detectionTest for conflict detection
Expect this change in behavior at a Expect this change in behavior at a higherhigher cost cost
Potential IssuesPotential Issues
Cost in row overhead – when Cost in row overhead – when enabled, enabled, 14 bytes added to row14 bytes added to row
If snapshot, do you depend on If snapshot, do you depend on locking?locking?
OK, most of you will say no but what OK, most of you will say no but what about about status queues…status queues…
Tip: Use READCOMMITTEDLOCK hintTip: Use READCOMMITTEDLOCK hint
If Snapshot Isolation, could you have If Snapshot Isolation, could you have conflicts?conflicts?
Be sure to have proper conflict detection Be sure to have proper conflict detection and error handling, see Kim’s Snapshot and error handling, see Kim’s Snapshot Isolation whitepaper for details and Isolation whitepaper for details and examplesexamples
Management/MonitoringManagement/Monitoring
Version Store in TempDBVersion Store in TempDB
Versions removed when no longer neededVersions removed when no longer needed
Read committed using statement-level Read committed using statement-level snapshot won’t hold versions as long – in snapshot won’t hold versions as long – in theory because only statement-leveltheory because only statement-level
Snapshot isolation may have more impact Snapshot isolation may have more impact on TempDB as versions held for life of on TempDB as versions held for life of transactiontransaction
Lots of long running transactions may Lots of long running transactions may stress TempDBstress TempDB
See Kim’s Snapshot Isolation whitepaper See Kim’s Snapshot Isolation whitepaper for monitoring with DMVs (Dynamic for monitoring with DMVs (Dynamic Management Views)Management Views)
Database DesignDatabase Design
Data TypesData Types
The Data RowThe Data Row
IndexesIndexes
PartitioningPartitioning
Security in DesignSecurity in Design
EntitiesEntities
Basic principle behind a table – the Basic principle behind a table – the way we describe some specific way we describe some specific person, place or thing as a base person, place or thing as a base object within the databaseobject within the databaseFocus on a single element - Person, Focus on a single element - Person, place, thing…place, thing…
Authors – Each author is an Entity Authors – Each author is an Entity described by a row in the authors table.described by a row in the authors table.Titles – Each title, separate from the Titles – Each title, separate from the author(s) that wrote the title is described author(s) that wrote the title is described by a row in the titles table.by a row in the titles table.Publishers – Each publisher, separate Publishers – Each publisher, separate from any title(s) they may have from any title(s) they may have published is described by a row in the published is described by a row in the publishers table.publishers table.
AttributesAttributes
Basic principle behind a column – the Basic principle behind a column – the way we describe some specific way we describe some specific element about the entityelement about the entity
Each column of a table should Each column of a table should describe ONLY that entity…describe ONLY that entity…
Core elements of an attributeCore elements of an attributeAttribute NameAttribute Namepriceprice
Data typeData type moneymoney
NullabilityNullability NULLNULL
Every attribute should describe the primary key, the whole primary key, and nothing but the primary key.
Data TypesData Types
System Supplied Data TypesSystem Supplied Data TypesCharacterCharacter
Unicode CharacterUnicode Character
NumericNumeric
Date and TimeDate and Time
Exact NumericExact Numeric
UniqueidentifierUniqueidentifier
BinaryBinary
MiscellaneousMiscellaneous
Find the right data type for the job:Find the right data type for the job:
Use Unicode over ASCiiUse Unicode over ASCii
If the datatype varies:If the datatype varies: < 5 chars should be nchar< 5 chars should be nchar
5-20 chars – questionable5-20 chars – questionable
> 20 char – lean more towards> 20 char – lean more towardsnvarcharnvarchar
For numeric data:For numeric data: Find the right rangeFind the right range Standardize on decimal or Standardize on decimal or
numericnumericUse uniqueidentifier sparinglyUse uniqueidentifier sparinglyUnderstand precision and rangeUnderstand precision and range
Data TypeData Type Max/Exact Max/Exact StorageStorage NotesNotes
char(n)char(n)varchar(n)varchar(n)
8000 8000 bytesbytes
1 byte per char - 8000 Characters1 byte per char - 8000 Characters
nchar(n)nchar(n)nvarchar(n)nvarchar(n)
8000 8000 bytesbytes
2 bytes per char - 4000 Characters2 bytes per char - 4000 Characters
tinyinttinyint 1 byte1 byte 0-2550-255
smallintsmallint 2 bytes2 bytes -32768 to 32767-32768 to 32767
intint 4 bytes4 bytes -2-23131 to 2 to 23131-1-1
bigintbigint 8 bytes8 bytes -2-26363 to 2 to 26363-1-1
uniqueidentifuniqueidentifierier
16 bytes16 bytes 98E94963-F193-4E69-9262-98E94963-F193-4E69-9262-7B692125557F7B692125557F
smalldatetimesmalldatetime 4 bytes4 bytes Jan 1, 1900 with precision to the Jan 1, 1900 with precision to the minuteminute
datetimedatetime 8 bytes8 bytes Jan 1, 1753 with precision to a Jan 1, 1753 with precision to a timetick (3.33 ms)timetick (3.33 ms)
smallmoneysmallmoney 4 bytes4 bytes - - 214,748.3648 through 214,748.3648 through +214,748.3647+214,748.3647
moneymoney 8 bytes8 bytes -922,337,203,685,477.5808 through -922,337,203,685,477.5808 through +922,337,203,685,477.5807+922,337,203,685,477.5807
SQL Server 2005 Data TypesSQL Server 2005 Data Types
Large Value Data Types, Large Value Data Types, n/varchar(max) and varbinary(max)n/varchar(max) and varbinary(max)
In row storage limit 8000 bytesIn row storage limit 8000 bytes
Work like typical varchar but support Work like typical varchar but support LOB values (231 bytes)LOB values (231 bytes)
Custom Types – CLR basedCustom Types – CLR based
Be aware that LOB types added to a Be aware that LOB types added to a table prevent online index rebuildstable prevent online index rebuilds
May want to consider vertical May want to consider vertical partitioning for special typespartitioning for special types
Character DataCharacter Data
Unicode – nchar, nvarchar, ntextUnicode – nchar, nvarchar, ntextApplication (web data) is stored nativelyApplication (web data) is stored natively
Supports wide variety of client environmentsSupports wide variety of client environments
Make sure to ALWAYS use 'N'Make sure to ALWAYS use 'N'
Examples:Examples:SELECT * FROM tname WHERE col = N'value'SELECT * FROM tname WHERE col = N'value'DECLARE @variable nvarchar(100)DECLARE @variable nvarchar(100)SELECT @variable = N'value'SELECT @variable = N'value'SELECT @Str = N'string' + N'string' + N'string'SELECT @Str = N'string' + N'string' + N'string'
CLR Extensibility requires unicode!CLR Extensibility requires unicode!
ASCii – char, varchar, textASCii – char, varchar, textThink before using if new data/tables use Unicode Think before using if new data/tables use Unicode insteadinstead
NullabilityNullability
No specific value required at INSERTNo specific value required at INSERTNULL values DO NOT mean empty NULL values DO NOT mean empty space (stored separately from the space (stored separately from the column data)column data)Working with NULLsWorking with NULLs
Accessing columns which allow NULL Accessing columns which allow NULL values can cause inconsistencies when values can cause inconsistencies when developers/users are not aware of themdevelopers/users are not aware of themMath with NULL values can produce Math with NULL values can produce interesting results (? – NULL = NULL)interesting results (? – NULL = NULL)ANSI session settings can affect results ANSI session settings can affect results sets when accessing columns that allow sets when accessing columns that allow nullsnulls
NULL values v. DEFAULT valueNULL values v. DEFAULT value
Creating TablesCreating Tables
Creating and Dropping a basic TableCreating and Dropping a basic Table
Determining Which Type of Determining Which Type of Constraint to UseConstraint to Use
Enforcing Data IntegrityEnforcing Data Integrity
How SQL Server Organizes Data in How SQL Server Organizes Data in RowsRows
How SQL Server Organizes Rows on a How SQL Server Organizes Rows on a PagePage
Initial Table Structure with Clustered Initial Table Structure with Clustered IndexIndex
Intro to Vertical PartitioningIntro to Vertical Partitioning
Generating ScriptsGenerating Scripts
Creating a TableCreating a Table
Specifying NULL or NOT NULLSpecifying NULL or NOT NULLNot requiredNot required
Defaults to session settingDefaults to session settingSET ANSI_NULL_DFLT_ON ONSET ANSI_NULL_DFLT_ON ON
Creating and Dropping a TableCreating and Dropping a Table
Column nameColumn name Data typeData typeNULL or NULL or NOT NULLNOT NULL
CREATE TABLE dbo.publishers ( pub_id
pub_namecitystatecountry
)
char(4) varchar(40) varchar(20)char(2)varchar(30)
NOT NULL,NULL,NULL,NULL,NULL
The Data RowThe Data Row
Header Fixed Data NB VB Variable Data
NullBlock
VariableBlock
4 bytes
Data
Rows on a PageRows on a Page
Page HeaderPage Header
Row ARow ARow ARow A
Row CRow CRow CRow C
Row BRow BRow BRow B
ABC
Data rows
Row Offset Table2 bytes each
96bytes
8,096bytes
Optimal Row WidthOptimal Row Width
Consider Table Usage above all elseConsider Table Usage above all elseEstimate Average Row LengthEstimate Average Row Length
OverheadOverheadFixed Width ColumnsFixed Width ColumnsEstimate Average from realistic sample Estimate Average from realistic sample datadataSELECT avg(datalength(columnname)) FROM SELECT avg(datalength(columnname)) FROM
tnametname
Calculate Rows/PageCalculate Rows/Page8096 Bytes/Page8096 Bytes/Page
??? Bytes/Row??? Bytes/Row
Calculate Wasted Bytes—on Disk Calculate Wasted Bytes—on Disk in Memoryin Memory
= Rows/Page
Row Size/Table StructureRow Size/Table Structure
Rows CAN span pages in SQL Server Rows CAN span pages in SQL Server 20052005Is that always the best choice?Is that always the best choice?Table access/usage patterns should Table access/usage patterns should dictate structuredictate structureKeep active/OLTP columns in primary Keep active/OLTP columns in primary tabletableSeparate LOB types to secondary Separate LOB types to secondary table for online table/index rebuildstable for online table/index rebuildsCreate initial table structure to Create initial table structure to minimize the need for rebuilds minimize the need for rebuilds ((Designing for Performance – more Designing for Performance – more info coming up!info coming up!))
Design Techniques SummaryDesign Techniques Summary
Rows CAN span pages however, Rows CAN span pages however, record size impacts disk space, cache record size impacts disk space, cache utilization, concurrency and utilization, concurrency and maintenance restrictionsmaintenance restrictionsThe lowest granularity of locking The lowest granularity of locking holds an entire row – partitioning a holds an entire row – partitioning a table vertically or horizontally can table vertically or horizontally can improve concurrency, minimize improve concurrency, minimize restrictions on online rebuilds and restrictions on online rebuilds and improve cache utilizationimprove cache utilizationOptimizer is aware of constraints – Optimizer is aware of constraints – efficient horizontal partitioning efficient horizontal partitioning requires constraintsrequires constraintsDuplicating Keys which define Duplicating Keys which define relationships can improve join relationships can improve join performance by giving the optimizer performance by giving the optimizer more options for performing a joinmore options for performing a join
Key PointsKey Points
Minimize data redundancy by using Minimize data redundancy by using attributes specific only to the entity table attributes specific only to the entity table definitiondefinitionSpecify Appropriate Data Types and Data Specify Appropriate Data Types and Data Type Sizes – try to keep row sizes compactType Sizes – try to keep row sizes compactAlways Specify Column Characteristics in Always Specify Column Characteristics in CREATE TABLE – especially column CREATE TABLE – especially column NullabilityNullabilityGenerate Scripts as backup scripts and for Generate Scripts as backup scripts and for learninglearningTake into account the usage of the data Take into account the usage of the data when defining tables, consider partitioning when defining tables, consider partitioning tables based on usagetables based on usageDefine Constraints on your tables – for Define Constraints on your tables – for performance as well as centralizationperformance as well as centralization
Why partition?Why partition?
Resource Contention LimitationsResource Contention Limitations
Varying Access patternsVarying Access patterns
Maintenance RestrictionsMaintenance Restrictions
Availability RequirementsAvailability Requirements
To remove resource blocking or To remove resource blocking or minimize maintenance costsminimize maintenance costs
But:But:Partitioning does not automatically mean Partitioning does not automatically mean Distributed Partitioned Views (DPVs)Distributed Partitioned Views (DPVs)
DPVs are a form of scale-out partitioningDPVs are a form of scale-out partitioning
Table’s Shared Data/IndexesTable’s Shared Data/Indexes
A Single Large Table…A Single Large Table… Presents different types of problemsPresents different types of problems
ManagementManagement
Index create/build or rebuildsIndex create/build or rebuilds
Backup/Restore and RecoveryBackup/Restore and Recovery
Escalation Escalation
May have different access patternsMay have different access patterns Some data new – OLTP (inserts/updates for Some data new – OLTP (inserts/updates for new and current sales)new and current sales)
Some data old for historical lookups – small Some data old for historical lookups – small singleton for OTLP lookups or larger for singleton for OTLP lookups or larger for analysisanalysis
Does NOT have to be one table!Does NOT have to be one table!
Indexes – Smaller TablesIndexes – Smaller Tables
Index Creation/BuildingIndex Creation/BuildingSmaller Tables – take less timeSmaller Tables – take less time
Different index decisionsDifferent index decisionsOLTP – less indexesOLTP – less indexes
DSS – more indexesDSS – more indexes
Index Maintenance Index Maintenance Smaller Tables – takes less time to Smaller Tables – takes less time to rebuild/defragrebuild/defrag
Different Access patterns – require less Different Access patterns – require less frequent maintenance or NONEfrequent maintenance or NONE
OLTP – rebuild/defrag more frequentlyOLTP – rebuild/defrag more frequently
DSS – build once!DSS – build once!
Backup Strategies VaryBackup Strategies Vary
Transaction ProcessingTransaction ProcessingBackups (should be) more frequentBackups (should be) more frequent
More critical dataMore critical data
Should be highly availableShould be highly available
Usually smaller portion of VLDB Usually smaller portion of VLDB
Decision SupportDecision SupportBackups needed less frequentlyBackups needed less frequently
Less critical dataLess critical data
Should be available but not “as” criticalShould be available but not “as” critical
Usually a larger portion of VLDBUsually a larger portion of VLDB
AvailabilityAvailability
SQL Server 2000SQL Server 2000Database online or notDatabase online or notEntire database offline, if damaged…Entire database offline, if damaged…
SQL Server 2005SQL Server 2005Database has an associated “state”Database has an associated “state”
Online, restoring, recovering, …Online, restoring, recovering, …Database in only one state at any given timeDatabase in only one state at any given time
Files/Filegroups also have statesFiles/Filegroups also have statesSecondary filegroups have independent stateSecondary filegroups have independent stateFilegroups can be offline and/or recovering while Filegroups can be offline and/or recovering while remainder of database is availableremainder of database is available
State of the database State of the database State of the primary filegroupState of the primary filegroupState of the transaction logState of the transaction log
Access PatternsAccess Patterns
Transaction ProcessingTransaction ProcessingSmaller INSERT/UPDATE/DELETE activitySmaller INSERT/UPDATE/DELETE activity
Speed/durability of highest importanceSpeed/durability of highest importance
Decision SupportDecision SupportLarger Range QueriesLarger Range Queries
Less critical dataLess critical data
More indexesMore indexes
Query access patterns may Query access patterns may conflict with write activityconflict with write activity
Partitioning StrategiesPartitioning Strategies
Vertical PartitioningVertical PartitioningMultiple tables – “subsets” of columnsMultiple tables – “subsets” of columns
Different column setsDifferent column setsSeparate sets based on usage patternsSeparate sets based on usage patterns
Primary key is redundant, possibly other Primary key is redundant, possibly other columnscolumns
Horizontal Range Partitioning Horizontal Range Partitioning Multiple tables – all columnsMultiple tables – all columns
Different row setsDifferent row setsSeparate sets based on date rangeSeparate sets based on date range
Separate sets based on lists – also “range” Separate sets based on lists – also “range” partitionspartitions
Consider Customer Table Consider Customer Table with 1,600,000 Rowswith 1,600,000 Rows
14 Columns1000 Bytes/Row8 Rows/Page200,000 Pages1.6GB Table
CustomerPersonal
18 Columns*1600 Bytes/Row5 Rows/Page320,000 Pages2.5GB Table
CustomerProfessional
17 Columns*2000 Bytes/Row4 Rows/Page400,000 Pages3.2 GB Table
CustomerMisc
47 Columns4600 Bytes/RowOnly 1 Row/Page
3400+ Bytes Wasted1.6 Million Pages12.8 GB Table
Customer
*The Primary key column must be made redundant for these two additional tables. 47 Columns in Employee. 49 Columns total between 3 tables.
Customer= 12.8 GB
Partitioned Tables= 7.3 GB
•Savings in Overall Disk Space (5.5 GB Saved)•Not reading data into cache when not necessary•Locks are table specific therefore less contention at the row level
Vertical PartitioningVertical Partitioning
Optimizing row size for…Optimizing row size for…CachingCaching
More rows on a page, more rows in memoryMore rows on a page, more rows in memory
LockingLockingOnly locking the columns that are of interest. Only locking the columns that are of interest. Minimize row based conflictsMinimize row based conflicts
Usage Defines Vertical PartitionsUsage Defines Vertical PartitionsLogically Group Columns to minimize Logically Group Columns to minimize joinsjoins
Consider Read Only vs. OLTP ColumnsConsider Read Only vs. OLTP Columns
Consider Columns often used togetherConsider Columns often used together
Sales2002_01
Sales2002_02
Sales2002_03
Sales2002_04...
Table or View
Sales2000
Table or View
Sales2001
Table or View
Sales2002
Horizontal PartitioningHorizontal Partitioning
874 Million RowsAll Sales Since January 2000•Current Sales (INSERTs) often
blocked by DSS queries. •Index Rebuilds take quite a
bit of time•Weekly FULL Database Backups are backing up the same sales
(2000 & 2001) repeatedly
Sales
Sales2000_01
Sales2000_02
Sales2000_03
Sales2000_04...
Sales2000_12
Sales2001_01
Sales2001_02
Sales2001_03
Sales2001_04...
Sales2001_12
Using CHECK ConstraintsCHECK (SalesDate >= '20010101'
AND SalesDate < '20010201')Sales2002_12
Horizontal PartitioningHorizontal Partitioning
More manageable chunksMore manageable chunksMinimize the impact of maintenance Minimize the impact of maintenance
Improve loading options for sliding range Improve loading options for sliding range scenarioscenarioIndex Maintenance (Smaller/Less Index Maintenance (Smaller/Less Frequent)Frequent)Backup/Restore (File/Filegroup strategies Backup/Restore (File/Filegroup strategies to minimize backup frequency of to minimize backup frequency of predominantly ReadOnly tables – more predominantly ReadOnly tables – more options for restore when isolated options for restore when isolated disk/RAID array corruption)disk/RAID array corruption)Lock Escalation (Modifications to one Lock Escalation (Modifications to one partition do not cause escalation against partition do not cause escalation against the others – as they likely would if the others – as they likely would if everything were in one table!)everything were in one table!)
““Partitioning” StrategiesPartitioning” Strategies
Proportional Fill using FilegroupsProportional Fill using Filegroups
One table on one filegroup with four One table on one filegroup with four filesfiles
Easy to administerEasy to administer
Data is proportionally filled among four Data is proportionally filled among four filesfiles
Data on each file is not “similar” in the Data on each file is not “similar” in the sense that it’s not 1st Q, 2nd Q, etc.sense that it’s not 1st Q, 2nd Q, etc.
All Objects and all dates are interleavedAll Objects and all dates are interleaved
First Glance…First Glance…
Proportionally filledProportionally filled
Single Single Table Table
Single Single FGFG
Four Four Tables Tables
Four Four FGsFGs
Functional Partitioning Functional Partitioning StrategiesStrategiesData-based partitioningData-based partitioning
SQL Server 2000 – Partitioned Views Four tables created on four separate Four tables created on four separate filegroupsfilegroupsMore complex to administer/createMore complex to administer/createData is NOT proportionally filledData is NOT proportionally filledData on each file IS “similar” as the data Data on each file IS “similar” as the data is filled based on date – i.e. 1is filled based on date – i.e. 1stst Q, 2 Q, 2ndnd Q, Q, etc.etc.The same quarter’s dates are on the The same quarter’s dates are on the same file (index and table are storage same file (index and table are storage aligned)aligned)
SQL Server 2005 – Partitioned TablesOne table created on a partition schemeOne table created on a partition schemeData placement and breakdown is same Data placement and breakdown is same as PVsas PVsEasier to setup!Easier to setup!
Functional PartitioningFunctional PartitioningMore granular controlMore granular control – both 2000/2005 – both 2000/2005
Varying access patternsVarying access patterns
Varying index defrag patternsVarying index defrag patterns
Very fast sliding window control (Very fast sliding window control (more details more details next!next!))
More backup/restore choicesMore backup/restore choices – in 2000 – in 2000 but many restrictions removed in 2005but many restrictions removed in 2005
File/filegroup backupsFile/filegroup backups
Even support for file/filegroup in Simple Even support for file/filegroup in Simple Recovery Model (backups separated between Recovery Model (backups separated between RW and RO backups and RO backups can be RW and RO backups and RO backups can be performed at any frequency!)performed at any frequency!)
Higher AvailabilityHigher Availability – in 2005 because of – in 2005 because of partial database availability featurespartial database availability features
Active Sales in OLTP systemActive Sales in OLTP system
Secondary VLDB used for reportingSecondary VLDB used for reporting
Range of two year’s dataRange of two year’s data
Monthly the active OLTP data is copied Monthly the active OLTP data is copied over and the oldest month is backed over and the oldest month is backed up and archivedup and archived
SepSep20022002
OctOct20022002
NovNov20022002
DecDec20022002 ……
AprApr20032003
MayMay20032003 ……
JunJun20042004
JulJul20042004
AugAug20042004
SepSep20042004
SepSep20022002
OctOct20022002
NovNov20022002
DecDec20022002 ……
AprApr20032003
MayMay20032003 ……
JunJun20042004
JulJul20042004
AugAug20042004
SepSep20042004
On October 1, 2004On October 1, 2004
OctOct20042004
VLDB – Sales ReportingVLDB – Sales Reporting
Rolling Range (or Sliding Rolling Range (or Sliding Window)Window)Key ComponentsKey ComponentsData LoadData Load
Single TableSingle TableActive Table impactedActive Table impacted
Indexes need to be updatedIndexes need to be updated
Partitioned Object (PV in 2000/PT in 2005)Partitioned Object (PV in 2000/PT in 2005)Table outside of active view manipulatedTable outside of active view manipulated
Indexes can be built separately of active tablesIndexes can be built separately of active tables
Data RemovalData RemovalSingle Table – same problemSingle Table – same problem
Active Table impactedActive Table impacted
Indexes need to be updatedIndexes need to be updated
Partitioned Object (PV in 2000/PT in 2005)Partitioned Object (PV in 2000/PT in 2005)Table can be removed from PO and then droppedTable can be removed from PO and then dropped
"Single" table process:"Single" table process:1.1. Alter the table's check constraint to support new Alter the table's check constraint to support new
quarter (drop the old constraint)quarter (drop the old constraint)2.2. Load the data using BULK INSERTLoad the data using BULK INSERT3.3. Delete the first quarter of F04Delete the first quarter of F044.4. Alter the table's check constraint to show new Alter the table's check constraint to show new
year range (drop the old constraint)year range (drop the old constraint)
Partitioned table process:Partitioned table process:1.1. Create New table - with constraintsCreate New table - with constraints2.2. Load the data using BULK INSERTLoad the data using BULK INSERT3.3. Build Indexes – CL + 3 NC on FYQ1 FilegroupBuild Indexes – CL + 3 NC on FYQ1 Filegroup4.4. Change the view to use FY05Q1 and remove Change the view to use FY05Q1 and remove
FY04Q1FY04Q15.5. Drop the table which represents FY04Q1 (could Drop the table which represents FY04Q1 (could
archived/backup)archived/backup)
1 min 36 sec1 min 36 sec
40+ minutes40+ minutes
Total TimesTotal Times
Range Partitioned TablesRange Partitioned Tables
Scripts from Whitepaper on Partitioned Scripts from Whitepaper on Partitioned TablesTables
Step 1: Create FilegroupsStep 1: Create Filegroups
Step 2: Create Files in FilegroupsStep 2: Create Files in Filegroups
Step 3: Create Partition FunctionStep 3: Create Partition Function
Step 4: Create Partition SchemeStep 4: Create Partition Scheme
Step 5: Create Table(s) on SchemeStep 5: Create Table(s) on Scheme
Step 6: Verify Data using system table Step 6: Verify Data using system table (optional)(optional)
Step 7: Add data to tables – SQL Server Step 7: Add data to tables – SQL Server redirects data and queries to appropriate redirects data and queries to appropriate partitionpartition
New Data must come inNew Data must come in
Old data must be removedOld data must be removed
The table should only change it’s overall The table should only change it’s overall data set… with simple metadata changesdata set… with simple metadata changes
To create metadata ONLY changesTo create metadata ONLY changesCreate a new table on the same filegroup where Create a new table on the same filegroup where the partition residesthe partition resides
Structure it IDENTICALLY to that of the Structure it IDENTICALLY to that of the partitioned tablepartitioned table
SWITCH the partition to live in the new table and SWITCH the partition to live in the new table and the table to become the partition… this either the table to become the partition… this either brings data IN or OUT of the tablebrings data IN or OUT of the table
The Sliding Window ScenarioThe Sliding Window Scenario
Developing on SQL Server 2005Developing on SQL Server 2005
Choices, Choices and More ChoicesChoices, Choices and More Choices
Secure DevelopmentSecure Development
Performance and ScalabilityPerformance and Scalability
Development DilemmaDevelopment Dilemma
T-SQLT-SQL
XMLXML
CLRCLR
Computation Computation & Framework & Framework
accessaccess
Relational Relational data accessdata access
Semi-structuredSemi-structureddata accessdata access
SQL Server 2005 ToolkitSQL Server 2005 Toolkit
SQLSQL
ServiceServiceBrokerBroker
TransactTransactSQLSQL
CLRCLRPrograms & TypesPrograms & Types
Relational DataRelational Data
SQLSQL XMLXML FTSFTS
SQL Server 2005 ToolkitSQL Server 2005 Toolkit
SQLSQL XMLXML FTSFTSXMLXML
XMLXMLtypetype
X/QueryX/Query XML IndexesXML Indexes
Semi-Structured Semi-Structured DataData
XML XML SchemaSchema
SQL Server 2005 ToolkitSQL Server 2005 Toolkit
SQLSQL XMLXML FTSFTSFTSFTS
MicrosoftMicrosoftSearchSearch
OfficeOfficeDocumentsDocuments
Integrated with Integrated with database backupdatabase backup
Unstructured Unstructured DataData
Good Scenario for CLR UsageGood Scenario for CLR Usage
Data validation & network traffic reductionData validation & network traffic reduction
Writing general purpose functions:Writing general purpose functions:Data passed as argumentsData passed as arguments
Little/no additional data accessLittle/no additional data access
Complex computation applied on a row by row basis to Complex computation applied on a row by row basis to the datathe data
Scalar types & custom aggregationsScalar types & custom aggregations
Leveraging the power of the .NET FrameworkLeveraging the power of the .NET FrameworkAccess to a rich set of pre-built functionalityAccess to a rich set of pre-built functionality
Replacing Extended Stored Procedures (XP)Replacing Extended Stored Procedures (XP)The CLR is safer: The CLR is safer:
No access violations making SQL Server crash No access violations making SQL Server crash
No leaks making SQL Server slow down & crashNo leaks making SQL Server slow down & crash
Better performance & scalability (managed memory model)Better performance & scalability (managed memory model)
No security issues…No security issues…
Bad Scenario for CLR UsageBad Scenario for CLR Usage
Heavy data access – Transact-SQL set Heavy data access – Transact-SQL set based access will be fasterbased access will be faster
Don’t write SELECT statements as CLR Don’t write SELECT statements as CLR procedures!procedures!
Complex typesComplex types8K size limitation8K size limitation
All data is read/re-written when updatedAll data is read/re-written when updated
Pre-Aggregation for ReportsPre-Aggregation for ReportsCLR Aggregates cannot be used in Indexed CLR Aggregates cannot be used in Indexed ViewsViews
Your application must support previous Your application must support previous versions of SQL Serverversions of SQL Server
Technology for technology’s sake…Technology for technology’s sake…
XML DevelopmentXML Development<xs:complexType name ="myelement" <xs:complexType name ="myelement"
mixed="true">mixed="true"> <xs:sequence minOccurs="0"> <xs:sequence minOccurs="0"> <xs:element name="child1" <xs:element name="child1" type="xs:integer" />type="xs:integer" /> <xs:element name="child2" <xs:element name="child2" type="xs:integer" />type="xs:integer" /> </xs:sequence> </xs:sequence></xs:complexType></xs:complexType>
<xs:element name="myelement" <xs:element name="myelement" type="myelement" />type="myelement" />
<myelement><myelement> Some text Some text <child1>3</child1> <child1>3</child1> Some more text Some more text <child2>4</child2> <child2>4</child2> Some more text Some more text</myelement></myelement>
<xs:complexType name <xs:complexType name ="myelementWithChildren">="myelementWithChildren"> <xs:sequence> <xs:sequence> <xs:element name="child1" <xs:element name="child1" type="xs:integer" />type="xs:integer" /> <xs:element name="child2" <xs:element name="child2" type="xs:integer" />type="xs:integer" /> </xs:sequence> </xs:sequence></xs:complexType></xs:complexType>
<xs:element name="myelement" <xs:element name="myelement" type="xs:anyType" />type="xs:anyType" />
Native XML datatypeNative XML datatypeOptimized on-disk Optimized on-disk structure (not “just a structure (not “just a blob”)blob”)
XML IndexesXML IndexesEntities & AttributesEntities & Attributes
X/PATH Query Engine X/PATH Query Engine Integrated with core Integrated with core relational operatorsrelational operators
Schema SupportSchema SupportStrongly typed XMLStrongly typed XML
CLR/XML IntegrationCLR/XML IntegrationXMLTextReaderXMLTextReaderAccess to XSLTAccess to XSLT
Full Text SearchFull Text Search
XML IndexesXML Indexes
Primary Index:Primary Index:Requires a clustered primary key on the base Requires a clustered primary key on the base tabletable
B-Tree of element and attribute names, node B-Tree of element and attribute names, node values, and node types, retains document order values, and node types, retains document order and structure, as well as the path from the root and structure, as well as the path from the root of the XML instance to each node for efficient of the XML instance to each node for efficient evaluation of path expressionsevaluation of path expressions
Secondary Indexes off of the primary XML Secondary Indexes off of the primary XML index index
PATH PATH columns (path, value) columns (path, value)
PROPERTY PROPERTY columns (PK, path, value) columns (PK, path, value)
VALUE VALUE columns (value, path) columns (value, path)
Scenario for XML DevelopmentScenario for XML Development
Good Scenario:Good Scenario:Data is semi-structured, Data is semi-structured, small core of fixed data small core of fixed data with many, sparsely with many, sparsely populated extended populated extended attributes attributes
Multi-value Property Multi-value Property bagsbags
Complex Property bagsComplex Property bags
““WordXML”WordXML”
Documents are large but Documents are large but rarely updatedrarely updated Indexing will pay off Indexing will pay off
Data is hierarchicalData is hierarchical regular expressions regular expressions are well suited for finding are well suited for finding datadata
Bad Scenario:Bad Scenario:““Database in a Cell”Database in a Cell”
Documents are large and Documents are large and updated frequentlyupdated frequently
Document update Document update contention is likelycontention is likely
Data is fully structured & Data is fully structured & populated populated candidate candidate for conversion to for conversion to relational schemarelational schema
Data contains large Data contains large binary objects (2GB binary objects (2GB limitation)limitation)
Transact-SQL EnhancementsTransact-SQL Enhancements
ROW_NUMBERROW_NUMBER
RANK, DENSE_RANK RANK, DENSE_RANK
Common Table ExpressionsCommon Table Expressions
PIVOT/UNPIVOTPIVOT/UNPIVOT
CROSS APPLY and OUTER APPLYCROSS APPLY and OUTER APPLY
Error handling – TRY/CATCH Error handling – TRY/CATCH
DDL Triggers (synchronous)DDL Triggers (synchronous)
Event Notifications (asynchronous)Event Notifications (asynchronous)
Parameterized TOPParameterized TOP
Whose Fault was SQL Injection?Whose Fault was SQL Injection?
OK. We got your attention. It was not OK. We got your attention. It was not you anyway, of course.you anyway, of course.
Security for a database developer Security for a database developer was a very rude wake-up call.was a very rude wake-up call.
We We feel feel your pain.your pain.
We can help you – in this session!We can help you – in this session!
Defense in DepthDefense in DepthUsing a layered approach:Using a layered approach:
Increases an attacker’s risk of detection Increases an attacker’s risk of detection
Reduces an attacker’s probability of Reduces an attacker’s probability of successsuccess
Policies, Procedures, & Awareness
Policies, Procedures, & Awareness
OS hardening, SQL Server patchingOS hardening, SQL Server patching
Firewalls, packet filtersFirewalls, packet filters
Guards, locks, tracking devices, Guards, locks, tracking devices, HSM, tamper-evident labelsHSM, tamper-evident labels
SSL, IPSec, No problemsSSL, IPSec, No problems
Application hardening, authorizationApplication hardening, authorization
Permissions, encryptionPermissions, encryption
DBA education against social DBA education against social engineeringengineering
Physical SecurityPhysical Security
PerimeterPerimeter
Internal NetworkInternal Network
HostHost
ApplicationApplication
DataData
Application ProtectionApplication Protection
Protection against attacks against the Protection against attacks against the applicationapplication
Techniques and practices used by the Techniques and practices used by the database code (modules, stored database code (modules, stored procedures) to avoid being exploited procedures) to avoid being exploited by callersby callers
Permissions and ACLs on the data are Permissions and ACLs on the data are also part of application protection from also part of application protection from the perspective of the calling applicationthe perspective of the calling application
Secure DevelopmentSecure Development
How things used to work?How things used to work?
User-schema separationUser-schema separation
Granular permissionsGranular permissions
EXECUTE ASEXECUTE AS
SQL Injection (Classic)SQL Injection (Classic)
SQL Injection (Future)SQL Injection (Future)
Security through EncapsulationSecurity through Encapsulation
Granting access to a process or a Granting access to a process or a method without direct access to base method without direct access to base objectsobjects
How?How?Grant access to the stored procedures, Grant access to the stored procedures, views and/or functions without granting views and/or functions without granting access to the base objectaccess to the base object
RequiresRequiresThe objects in the dependency chain The objects in the dependency chain cannot be broken (object ownership cannot be broken (object ownership chaining)chaining)
The user has to have direct base object The user has to have direct base object access (which is what you’re trying to access (which is what you’re trying to avoid)avoid)
Do NOT Give Base Object Do NOT Give Base Object AccessAccessThe SQL Server 2000 WayThe SQL Server 2000 Way
Only works when the Only works when the object dependency object dependency chain is NOT brokenchain is NOT brokenOnly works when the Only works when the commands inside of commands inside of the stored procedure the stored procedure are not built/executed are not built/executed dynamicallydynamically
dbo.Salesdbo.Salesdbo.Salesdbo.Sales
UserUserUserUser
dbo.DeleteASalesdbo.DeleteASalesdbo.DeleteASalesdbo.DeleteASales
Granted execute accessto procedure
has NO direct permissionsmay even be DENIED permissions to Sales
User-Schema SeparationUser-Schema Separation
Database can contain Database can contain multiple schemasmultiple schemasEach schema has an Each schema has an owning principal – user owning principal – user or roleor roleEach user has a default Each user has a default schema schema for name resolutionfor name resolutionDatabase objects live Database objects live in schemasin schemasObject creation inside Object creation inside schema requires schema requires CREATE permission CREATE permission andand ALTER or CONTROL ALTER or CONTROL permission permission on the schemaon the schemaOwnership chaining still Ownership chaining still based on owners not based on owners not schemas schemas
Role1Role1
OwnsOwns has default has default schema2schema2
OwnsOwns
OwnsOwns
Schema1Schema1 Schema2Schema2
Schema3Schema3
SP1SP1Fn1Fn1
Tab1Tab1
DatabaseDatabase
User1User1 Approle1Approle1
Default SchemaDefault Schema
Used for name resolution purposesUsed for name resolution purposes
Common name resolution across multiple Common name resolution across multiple usersusers
No need to rely on DBO schemaNo need to rely on DBO schema
Using DBO schema may result in security Using DBO schema may result in security issuesissues
Object Creation requires higher privilegesObject Creation requires higher privileges
Mitigates concerns resulting from ownership Mitigates concerns resulting from ownership chainingchaining
Instead create “buckets” of objects Instead create “buckets” of objects through “schemas” where schemas have through “schemas” where schemas have owners and developers have default owners and developers have default schemas and/or control on needed schemas and/or control on needed schemasschemas
Do NOT Give Base Object Do NOT Give Base Object AccessAccessThe SQL Server 2005 WayCreate a schema – for example one Create a schema – for example one
which contains procedures (and which contains procedures (and possibly even base tables) and then possibly even base tables) and then GRANT EXECUTE at the schema levelGRANT EXECUTE at the schema level
Add objects to appropriate schemaAdd objects to appropriate schema
Grant access to the schema or the Grant access to the schema or the individual objects – if chaining is individual objects – if chaining is required, it is still based on the required, it is still based on the owner… the owner of the schemaowner… the owner of the schema
What if Dynamic String What if Dynamic String ExecutionExecution
By default – and for better security – By default – and for better security – if the stored procedure has a if the stored procedure has a statement which is built dynamically statement which is built dynamically (using EXEC('string') or (using EXEC('string') or EXEC(@variable)) then the context EXEC(@variable)) then the context under which the dynamically under which the dynamically constructed string executes is constructed string executes is ALWAYS the callerALWAYS the caller
Which is what helps to prevent some Which is what helps to prevent some forms of SQL Injectionforms of SQL Injection
This is really a This is really a goodgood thing BUT… thing BUT…
Can be limitingCan be limitingEnter: Enter: EXECUTE ASEXECUTE AS
Module Execution ContextModule Execution Context
Execute AS Execute AS CALLERCALLERDefault behavior, same as SQL Server 2000Default behavior, same as SQL Server 2000Use when caller’s permission needs to be checked – or Use when caller’s permission needs to be checked – or when ownership chaining will sufficewhen ownership chaining will suffice
Execute AS Execute AS 'UserName''UserName'Statements execute as the username specifiedStatements execute as the username specifiedImpersonate permission required on user specified Impersonate permission required on user specified
Execute AS Execute AS OWNEROWNERStatements execute as the current owner of the moduleStatements execute as the current owner of the moduleImpersonate privileges on owner required, at setting Impersonate privileges on owner required, at setting timetimeOn ownership change, context is new ownerOn ownership change, context is new owner
Execute AS Execute AS SELFSELFStatements execute as the person specifying the execute Statements execute as the person specifying the execute as clause for the module – even if the ownership changesas clause for the module – even if the ownership changesMay be useful in application scenarios where calling May be useful in application scenarios where calling context may changecontext may change
EXECUTE AS…EXECUTE AS…
Solves problemsSolves problemsAllows permissions to be granted where Allows permissions to be granted where never possible (e.g. granting truncate never possible (e.g. granting truncate table)table)
Wrap ANYTHING inside a stored Wrap ANYTHING inside a stored procedure and set the context to run as procedure and set the context to run as someone who has permissions – even someone who has permissions – even dynamic string execution – then give dynamic string execution – then give execute permissionexecute permission
Creates potential for further SQL Creates potential for further SQL InjectionInjection
What if you’re code is not well tested What if you’re code is not well tested and uses dynamically executed stringsand uses dynamically executed strings
SQL InjectionSQL Injection
SQL statement(s) or clause(s) SQL statement(s) or clause(s) injected into an existing SQL injected into an existing SQL commandcommand
Injected string appended to Injected string appended to application inputapplication input
Text boxesText boxes
Query stringsQuery strings
Manipulated values in HTML Manipulated values in HTML
Why SQL injection works?Why SQL injection works?Application accepts arbitrary user inputApplication accepts arbitrary user input
Connection made in context of higher Connection made in context of higher privileged accountprivileged account
SQL Injection MitigationSQL Injection Mitigation
Do not trust the user’s input!Do not trust the user’s input!Look for valid input and reject everything Look for valid input and reject everything elseelse
Protect identifiers with QUOTENAME()Protect identifiers with QUOTENAME()
Regular expressions are your friend!Regular expressions are your friend!
Do not use string concatenation Do not use string concatenation Use parameterized queries to build Use parameterized queries to build queriesqueries
Restrict information in error Restrict information in error messagesmessages
Use a low-privileged accountUse a low-privileged accountDO NOT use sa or sysadmin role memberDO NOT use sa or sysadmin role member
Data ProtectionData Protection
Use of cryptography to protect the Use of cryptography to protect the data layer from theft or manipulationdata layer from theft or manipulation
Particularly important if offline data is in Particularly important if offline data is in transittransitImportant for regulatory reasonsImportant for regulatory reasons
Prevent admin from accessPrevent admin from accessSarbanes OxleySarbanes Oxley
Manipulation (integrity) protection Manipulation (integrity) protection uses digital signaturesuses digital signatures
Not implemented for data in SQL Server Not implemented for data in SQL Server 20052005
Can overcome this with judicious use of Can overcome this with judicious use of encryption aloneencryption alone
Implemented for code signingImplemented for code signing
Check Your OSCheck Your OS
Verify what algorithms Verify what algorithms and key lengths are and key lengths are supported by your OS, supported by your OS, as this depends on as this depends on your CSP your CSP (Cryptographic (Cryptographic Services Provider)Services Provider)
Generate keys for all Generate keys for all algorithms then look in algorithms then look in the sys.symmetric_keysthe sys.symmetric_keys
For example, we For example, we found the following found the following during testing – this during testing – this may change in the may change in the futurefuture
At present, there is no At present, there is no way to change the way to change the CSP…CSP…
XP SP2XP SP2 WS200WS20033
DESDES 56 (64)56 (64) 56 (64)56 (64)3DES3DES 128128 128128AES12AES1288
-- 128128
AES19AES1922
-- 192192
AES25AES2566
-- 256256
RC2RC2 128128 128128RC4RC4 4040 4040
Before You BeginBefore You Begin
Your DBA must ensure that the Your DBA must ensure that the server has a Service Master Key server has a Service Master Key (SMK) that agrees with your disaster (SMK) that agrees with your disaster recovery strategy and a Database recovery strategy and a Database Master Key (DMK) has been created Master Key (DMK) has been created for each database that will use for each database that will use encryptionencryption
There are serious implications on There are serious implications on security (use strong password for SQL security (use strong password for SQL service account) and for disaster service account) and for disaster recoveryrecovery
ALTER SERVICE MASTER KEY can be used to ALTER SERVICE MASTER KEY can be used to change recovery optionschange recovery options
CREATE MASTER KEY ENCRYPTION BY CREATE MASTER KEY ENCRYPTION BY PASSWORD =PASSWORD =
Use a strong password, write it down, keep it Use a strong password, write it down, keep it safesafe
Key GenerationKey Generation
The key should be impossible to The key should be impossible to guess. Preferably random.guess. Preferably random.
CREATE SYMMETRIC… will generate a CREATE SYMMETRIC… will generate a fairly random key for you – good!fairly random key for you – good!
You can base the key on data supplied You can base the key on data supplied by you, use KEY_SOURCE* clause – good by you, use KEY_SOURCE* clause – good for generating identical keys from a for generating identical keys from a high-quality passwordhigh-quality password
Note: KEY_SOURCE may be renamed to Note: KEY_SOURCE may be renamed to DERIVED_FROM in the released versionDERIVED_FROM in the released version
PasswordsPasswords
Make sure passwords used to protect or create Make sure passwords used to protect or create keys are very strongkeys are very strongIn our opinion, it is better to create a very long In our opinion, it is better to create a very long and complex password that you will have to write and complex password that you will have to write down and store in a well-protected safe in your down and store in a well-protected safe in your companycompany
You can divide it into two halves and store in separate You can divide it into two halves and store in separate envelopes in different safesenvelopes in different safes
E.g.: *87(HyfdlkRM?_764#{(**%GRtj*(NS£”_+^$(E.g.: *87(HyfdlkRM?_764#{(**%GRtj*(NS£”_+^$(No dictionary words, more than 20 characters, many No dictionary words, more than 20 characters, many non-printing charactersnon-printing charactersChallenge: usability!Challenge: usability!
Consider also:Consider also:Not keeping passwords as text in your code, but store Not keeping passwords as text in your code, but store and retrieve them through DPAPI and a .NET componentand retrieve them through DPAPI and a .NET componentUsing good quality password generatorsUsing good quality password generators
Key ProtectionKey Protection
SQL Server 2005 insists that the key you SQL Server 2005 insists that the key you create is further encrypted for its create is further encrypted for its protectionprotection
Yes, that’s double-encryption, but that is not Yes, that’s double-encryption, but that is not double-security. Actually, it reduces security a double-security. Actually, it reduces security a little in some caseslittle in some cases
CREATE SYMMETRIC…ENCRYPT BYCREATE SYMMETRIC…ENCRYPT BYPASSWORDPASSWORD
Your (v. good) password generates a key to 3DES Your (v. good) password generates a key to 3DES encrypt the key you are protectingencrypt the key you are protectingNote, 3DES is less secure than AES, so this Note, 3DES is less secure than AES, so this
CERTIFICATECERTIFICATEYour key is encrypted using the public key of a Your key is encrypted using the public key of a certificatecertificateThis, in essence, is hybrid encryptionThis, in essence, is hybrid encryptionIf private key is kept secure (and offline), this is a very If private key is kept secure (and offline), this is a very good way to protect a symmetric keygood way to protect a symmetric key
Or another SYMMETRIC or ASYMMETRIC key – Or another SYMMETRIC or ASYMMETRIC key – less useful but interestingless useful but interesting
EncryptionEncryption
1.1. Create or retrieve the keyCreate or retrieve the key
2.2. Open the key – this means decrypt it Open the key – this means decrypt it with its (secondary) password or with its (secondary) password or certificate or other keycertificate or other key
3.3. Use function ENCRYPTBYKEY inside Use function ENCRYPTBYKEY inside DMLDML
DecryptionDecryption
1.1. Create or retrieve the keyCreate or retrieve the key
2.2. Open the keyOpen the key
3.3. Use function DECRYPTBYKEY inside Use function DECRYPTBYKEY inside SELECT and all other DMLSELECT and all other DML
Performance IssuesPerformance Issues
As you know, security is expensiveAs you know, security is expensive
AsymmetricAsymmetric encryption alone is not viable for encryption alone is not viable for large amounts of datalarge amounts of data
SymmetricSymmetric cryptography can be both strong and cryptography can be both strong and fast: AESfast: AES
HybridHybrid encryption is fast enough and solves the encryption is fast enough and solves the problem of symmetric key distribution logisticsproblem of symmetric key distribution logistics
Don’t encrypt everything, this rarely makes sense Don’t encrypt everything, this rarely makes sense and would be very costlyand would be very costly
But, our (suspect) performance tests showed that But, our (suspect) performance tests showed that all algorithms were as fast as each other. This is all algorithms were as fast as each other. This is not what we expected, and either signifies other not what we expected, and either signifies other overheads which cancel out the differences, or we overheads which cancel out the differences, or we need to re-test on a released product (RTM).need to re-test on a released product (RTM).
Top 10 Best Practices in Top 10 Best Practices in SecuritySecurity1.1. Risk assessment and threat analysisRisk assessment and threat analysis2.2. Trap “developer quality” error Trap “developer quality” error
messages – avoid indecent disclosuremessages – avoid indecent disclosure3.3. Avoid using dynamic string execution Avoid using dynamic string execution 4.4. Think encryption for sensitive dataThink encryption for sensitive data5.5. Do not create objects in DBO schemaDo not create objects in DBO schema6.6. Hide underlying application schemaHide underlying application schema7.7. Use Roles for managing permissionsUse Roles for managing permissions8.8. Account for catalog securityAccount for catalog security9.9. Avoid encryption as it precludes Avoid encryption as it precludes
indexing and performanceindexing and performance10.10.Know your CSP. Check allowed key Know your CSP. Check allowed key
lengths if needed. Chose carefully!lengths if needed. Chose carefully!
Performance and ScalabilityPerformance and Scalability
Using IndexesUsing Indexes
Index ConceptsIndex Concepts
Table and Index InternalsTable and Index Internals
Initial Table DesignInitial Table DesignThe Clustered Index DebateThe Clustered Index Debate
Secondary Non-clustered IndexesSecondary Non-clustered Indexes
Letting ITW/DTA Help!Letting ITW/DTA Help!
Index UsageIndex UsageConceptualConceptual
Allows faster access to data, improving:Allows faster access to data, improving:Lookup/query timeLookup/query time
Insert/Update time – record location definedInsert/Update time – record location defined
Delete time – record location also definedDelete time – record location also defined
Con: overhead for modificationsCon: overhead for modifications
Index Strategy ConceptsIndex Strategy ConceptsFewer indexes are better than lots of indexesFewer indexes are better than lots of indexes
Wider indexes have more usesWider indexes have more usesCan be used for point queriesCan be used for point queries
Can be used by NARROW low selectivity queriesCan be used by NARROW low selectivity queries
Write Effective Queries to better take Write Effective Queries to better take advantage of indexes…advantage of indexes…
Accessing Data: Limit DataAccessing Data: Limit Data
Subset of columns = ProjectionSubset of columns = ProjectionDo not use * (unless against view)Do not use * (unless against view)Optimizer has more chances for optimizing Optimizer has more chances for optimizing query when result set is NARROW (only the query when result set is NARROW (only the required columns)required columns)
Subset of rows = SelectionSubset of rows = SelectionUse positive search argumentsUse positive search argumentsIsolate the column to one side of the expressionIsolate the column to one side of the expression
USEUSE: MonthlySalary > value/12 (constant, seekable): MonthlySalary > value/12 (constant, seekable)DO NOT USEDO NOT USE: MonthlySalary * 12 > value (must scan): MonthlySalary * 12 > value (must scan)
Be cautious with LEADING wildcardsBe cautious with LEADING wildcardsUSEUSE: LastName LIKE 'S%': LastName LIKE 'S%'Try not to just append %val% to every applicationTry not to just append %val% to every application
Consider using Views, Stored Procedures Consider using Views, Stored Procedures and Functions to limit the columns/rowsand Functions to limit the columns/rows
Index ConceptsIndex ConceptsBook analogyBook analogy
Think of a book—with indexes in the backThink of a book—with indexes in the back
The book has one form of logical orderingThe book has one form of logical ordering
For references, you use the indexes in the back…For references, you use the indexes in the back…to find the data in which you are interested you to find the data in which you are interested you look up the keylook up the key
When you find the key, you must lookup the data When you find the key, you must lookup the data based on its location… i.e., a “bookmark” lookupbased on its location… i.e., a “bookmark” lookup
The bookmark always depends on the (book) The bookmark always depends on the (book) content ordercontent order
IndexIndex – Species Common Name
IndexIndex – Species Scientific Name
IndexIndex – Animal by Country, Name
IndexIndex – Animal by Continent, Country, Name
IndexIndex – Animal by Type, Name Bird, Mammal, Reptile,
etc…
IndexIndex – Animals by Habitat, Name Air, Land, Water
Index ConceptsIndex ConceptsTree AnalogyTree Analogy
If a tree were data and you were looking for If a tree were data and you were looking for leaves with a certain property, you would leaves with a certain property, you would have two options to find that data:have two options to find that data:
1) Touch every leaf – interrogating 1) Touch every leaf – interrogating each one to determine if they each one to determine if they held that property…held that property…SCANSCAN
2) If those leaves (which 2) If those leaves (which had that property) were had that property) were grouped such that you grouped such that you could start at the root, move could start at the root, move to the branch and then directly to the branch and then directly to those leaves…to those leaves…SEEKSEEK
Table StructureTable Structure
HEAP:HEAP: A table without a clustered A table without a clustered index index
Clustered Table:Clustered Table: A table with a A table with a clustered indexclustered index
Non-clustered Indexes Non-clustered Indexes do notdo not affect affect the base table’s structurethe base table’s structureHowever, Non-clustered Indexes are However, Non-clustered Indexes are affected by whether or not the table affected by whether or not the table is Clustered…is Clustered…
Hint: The non-clustered index dependency on the clustered Hint: The non-clustered index dependency on the clustered index should impact your choice for the clustering key!index should impact your choice for the clustering key!
Heap: ProsHeap: Pros
Effective for Staging DataEffective for Staging Data
Indexes can be created after load and Indexes can be created after load and index creation can be parallelizedindex creation can be parallelized
Efficient for SCANs ONLY – when no Efficient for SCANs ONLY – when no UPDATEs (otherwise, forwarding pointers – UPDATEs (otherwise, forwarding pointers – scans become scans become significantlysignificantly less efficient) less efficient)
Space Efficient – Space from Deletes is Space Efficient – Space from Deletes is reused on subsequent Inserts (at the cost reused on subsequent Inserts (at the cost of performance)of performance)
Best used when table is not typical OLTP or Best used when table is not typical OLTP or combo OLTP/DSS tablecombo OLTP/DSS table
Forward only “logging” tableForward only “logging” table
Table StructureTable StructureClustered tableClustered table
Clustered Index defines order – applied at Clustered Index defines order – applied at CREATIONCREATION
Expensive—in time and space—to build Expensive—in time and space—to build (an additional 1-2 times the index size for (an additional 1-2 times the index size for (re)build)(re)build)
Table is a doubly-linked list – order Table is a doubly-linked list – order maintained LOGICALLYmaintained LOGICALLY
MightMight be expensive to maintain be expensive to maintain
…4000 Pages of Employees in Clustering Key Order
1, Griffith, …
2, Ulaska, …
3, Johnson, …
…
20, Morrisson, …
21, Ambers, …
22, Johany, …
23, Smith, …
…
40, Griffen, …
41, Shen, …
42, Alberts, …
43, Landon, …
…
60, Lynne, …
79981, Geller, …
79982, Smith, …
79983, Jones, …
…
80000, Kirkert, …
79961, Morris, …
79962, Simon, …
79963, Jones, …
…
79980, Debry, …
79941, Baker, …
79942, Shehy, …
79943, Laws, …
…
79960, Miller, …
File1, Page 5982 File1, Page 5983 File1, Page 5984 File1, Page 11231 File1, Page 11232 File1, Page 11233
Clustered EmployeeIDClustered EmployeeIDEmployee TableEmployee Table
Consider a clustered index on an identity column (even Consider a clustered index on an identity column (even better if this is the table’s Primary Key)better if this is the table’s Primary Key)
…1, Griffith, …
2, Ulaska, …
3, Johnson, …
…
20, Morrisson, …
21, Ambers, …
22, Johany, …
23, Smith, …
…
40, Griffen, …
41, Shen, …
42, Alberts, …
43, Landon, …
…
60, Lynne, …
79981, Geller, …
79982, Smith, …
79983, Jones, …
…
80000, Kirkert, …
79961, Kiesow, …
79962, Simon, …
79963, Gellock, …
…
79980, Debry, …
79941, Baker, …
79942, Shehy, …
79943, Laws, …
…
79960, Miller, …
File1, Page 5982 File1, Page 5983 File1, Page 5984 File1, Page 11231 File1, Page 11232 File1, Page 11233
…1, 1, 5982
21, 1, 5983
41, 1, 5984
…
~16000
~64001
…
79941, 1, 11231
79961, 1, 11232
79981, 1, 11233
File1, Page 12982 File1, Page 12986
1, 1, 12982
16001, 1, 12983
32001, 1, 12984
48001, 1, 12985
64001, 1, 12986
File1, Page 12987
Root = 1 Page
Intermediate Level = 5 Pages
Balanced-TreeTotal Overhead in terms of Disk
Space
= 6 Pages or < 1%
Leaf Level4000
Pages
Non-Clustered IndexesNon-Clustered IndexesPhysicalPhysical
Depend on whether the table is a Heap or Depend on whether the table is a Heap or ClusteredClustered
Clustered TableClustered TableRows use the Clustering KeyRows use the Clustering Key
No additional OH to add this column – may use No additional OH to add this column – may use actual data to define the clustering keyactual data to define the clustering key
Minimizes NC Index manipulation if rows move Minimizes NC Index manipulation if rows move (the NC still points to the CL Key)(the NC still points to the CL Key)
Clustering key should be static and narrowClustering key should be static and narrow
HeapHeapGenerally, not recommendedGenerally, not recommended
…000-00-0001, 497
000-00-0002, 349
000-00-0003, 5643
…
000-00-0539, 12
…
…
…
…
…
…
…
…
…
…
…
…
…
…
999-99-9999, 42
…
…
…
…
…
…
…
…
…
…
File1, Page 16897 File1, Page 16898 File1, Page 16899 File1, Page 11810 File1, Page 18111 File1, Page 18112
539 entries
File1, Page 19197
539 entries
539 entries
539 entries
228 entries
149 entries
Root = 1 Page
Leaf Level149
Pages
Total Overhead in terms of Disk Space
= 150 Pages or < 4%
Non-Clustered IndexNon-Clustered IndexUnique Key SSNUnique Key SSN
Leaf level contains the non-clustered key(s) to Leaf level contains the non-clustered key(s) to define the orderdefine the order
Also includes either the Heap’s Fixed RID or the Also includes either the Heap’s Fixed RID or the Table’s Clustering KeyTable’s Clustering Key
Clustered Index CriteriaClustered Index Criteria
UniqueUniqueYes – No overhead, data takes care of this criteria Yes – No overhead, data takes care of this criteria
NO – SQL Server must “uniquify” the rows on INSERT. This NO – SQL Server must “uniquify” the rows on INSERT. This costs time and space. Each duplicate has a 4-byte costs time and space. Each duplicate has a 4-byte uniquifier. In SQL Server 2000, all uniquifiers are uniquifier. In SQL Server 2000, all uniquifiers are regenerated when the CL index is rebuild. This is not true regenerated when the CL index is rebuild. This is not true in SQL Server 2005.in SQL Server 2005.
NarrowNarrowYes – Keeps the NC indexes narrowYes – Keeps the NC indexes narrow
NO – Possibly wastes spaceNO – Possibly wastes space
StaticStaticYes – Improves PerformanceYes – Improves Performance
NO – Costly to maintain during updates to the key NO – Costly to maintain during updates to the key especially if row movement and/or splitsespecially if row movement and/or splits
In fact, an identity column that’s ever-increasing is ideal…In fact, an identity column that’s ever-increasing is ideal…
Clustering on an IdentityClustering on an Identity
Naturally UniqueNaturally Unique (although not guaranteed – should combine with key (although not guaranteed – should combine with key constraint)constraint)
Naturally StaticNaturally Static (although should be enforced through permissions (although should be enforced through permissions and/or trigger)and/or trigger)
Naturally NarrowNaturally Narrow (only numeric values possible, whole numbers scale (only numeric values possible, whole numbers scale = 0)= 0)
Naturally creates a hot spot…Naturally creates a hot spot…Needed pages for INSERT already in cacheNeeded pages for INSERT already in cacheMinimizes cache requirementsMinimizes cache requirementsHelps reduce fragmentation due to INSERTsHelps reduce fragmentation due to INSERTsHelps improve availability by naturally needing Helps improve availability by naturally needing less defragless defrag
Clustering Key ChoicesClustering Key Choices
IdentityIdentity column columnDate, identity Date, identity
Not date alone as not uniqueNot date alone as not uniqueGUIDGUID
Populated by client-side call to .NET client Populated by client-side call to .NET client GUID generator (NOT a good CL key but GUID generator (NOT a good CL key but ok as NC Primary Key)ok as NC Primary Key)Populated by server-side newid() function Populated by server-side newid() function (no pattern, not ever-increasing)(no pattern, not ever-increasing)Populated by NEW server-side Populated by NEW server-side newsequentialid() function (Creates an newsequentialid() function (Creates an ever-increasing GUID)ever-increasing GUID)
Key points: Narrow, static, unique, ever-increasingKey points: Narrow, static, unique, ever-increasing
Key Constraints Create IndexesKey Constraints Create Indexes
Primary Key ConstraintPrimary Key ConstraintDefaults to Unique ClusteredDefaults to Unique Clustered
Only One Per TableOnly One Per Table
ALTER TABLE EmployeeALTER TABLE Employee ADD CONSTRAINT EmployeePK ADD CONSTRAINT EmployeePK PRIMARY KEY CLUSTERED (EmployeeID) PRIMARY KEY CLUSTERED (EmployeeID)
Unique Key ConstraintsUnique Key ConstraintsDefault to Unique Non-ClusteredDefault to Unique Non-Clustered
Up to 249 Per TableUp to 249 Per Table
ALTER TABLE EmployeeALTER TABLE Employee ADD CONSTRAINT EmployeeSSNUK ADD CONSTRAINT EmployeeSSNUK UNIQUE NONCLUSTERED (SSN) UNIQUE NONCLUSTERED (SSN)
Optimal Table Structures Optimal Table Structures Poor clustered index choicePoor clustered index choice
LastName or non-sequential GUIDLastName or non-sequential GUIDFragmentation Fragmentation poor performance poor performance
Non-unique Non-unique poor performance poor performanceUniquifier wastes space and timeUniquifier wastes space and time
SQL 2000 only, rebuilding a clustered index on a non-SQL 2000 only, rebuilding a clustered index on a non-unique unique clustered index forces non-clustered indexes to be clustered index forces non-clustered indexes to be rebuiltrebuilt
Volatile Volatile poor performance poor performanceMost-duplicated valueMost-duplicated value
Could cause record relocation and therefore Could cause record relocation and therefore fragmentationfragmentation
Wide Wide poor performance poor performanceWastes space/time – wide keys take longer to Wastes space/time – wide keys take longer to maintain/insert.maintain/insert.
The wider the key the wider the non-clustered The wider the key the wider the non-clustered indexesindexes
TIP: Don’t do this! TIP: Don’t do this!
Optimal Table Structures Optimal Table Structures (cont’d)(cont’d)Heap is betterHeap is betterNo clustered IndexNo clustered Index
No data row Fragmentation No data row Fragmentation forwarding rows forwarding rows
Could have LOB fragmentation (new Could have LOB fragmentation (new LOB_Compaction can help this type of LOB_Compaction can help this type of fragmentation)fragmentation)
Wastes space in non-clustered indexes by using fixed-Wastes space in non-clustered indexes by using fixed-RIDRID
Scans (due to locking/consistency requirements) can Scans (due to locking/consistency requirements) can be more expensive!be more expensive!
Optimized for space over speedOptimized for space over speed
TIP: Create HEAPs for staging tables as an TIP: Create HEAPs for staging tables as an interim table for high performance data interim table for high performance data loading!loading!
See presentation “High Performance Data See presentation “High Performance Data Loading” Loading” on on www.SQLDev.Netwww.SQLDev.Net for more tips! for more tips!
Optimal Table Structures Optimal Table Structures (cont’d)(cont’d)The The RIGHTRIGHT Clustered Index Clustered IndexThe RIGHT Clustered Index The RIGHT Clustered Index
Cluster on identity column or possibly add one for Cluster on identity column or possibly add one for clusteringclustering
Cluster on composite key (date, identity) for tables Cluster on composite key (date, identity) for tables that that also have this patternalso have this pattern
Key Criteria: Static, Narrow and UniqueKey Criteria: Static, Narrow and Unique
Effective Hot Spot(s) using Identity (or date, id) or Effective Hot Spot(s) using Identity (or date, id) or composite key to create multiple (but not random) composite key to create multiple (but not random) hot spots – possibly consider partitioning for large hot spots – possibly consider partitioning for large tables/high volume OLTPtables/high volume OLTP
TIP: If this doesn’t exist, add a surrogate column solely to cluster TIP: If this doesn’t exist, add a surrogate column solely to cluster on it!on it!
TIP: Make sure to find the RIGHT Clustered index for TIP: Make sure to find the RIGHT Clustered index for youryour environmentenvironment
TIP: Make sure to AUTOMATE defragmentation if/when TIP: Make sure to AUTOMATE defragmentation if/when necessarynecessary
Clustering for PerformanceClustering for Performance
Cluster for internal benefits which Cluster for internal benefits which lead to:lead to:
Better insert/update performance re: Better insert/update performance re: fewer splitsfewer splits
Better Availability as full table Better Availability as full table maintenance may not be neededmaintenance may not be needed
Reliance on Nonclustered indexes is Reliance on Nonclustered indexes is greater:greater:
Faster access to narrow/low selectivity Faster access to narrow/low selectivity range queriesrange queries
NC Indexes are used in numerous non-NC Indexes are used in numerous non-obvious ways (multiple NC Indexes can obvious ways (multiple NC Indexes can be joined to cover a query)be joined to cover a query)
Reliance on Nonclustered Reliance on Nonclustered Indexes…Indexes…
Faster access to narrow/low selectivity range Faster access to narrow/low selectivity range queriesqueriesNC Indexes are used in numerous non-NC Indexes are used in numerous non-obvious ways (multiple NC Indexes can be obvious ways (multiple NC Indexes can be joined to cover a query)joined to cover a query)More flexible in the definition (i.e., Indexed More flexible in the definition (i.e., Indexed Views can include computations, substrings, Views can include computations, substrings, etc.) etc.) New INCLUDE clause can allow wider New INCLUDE clause can allow wider covering indexes!covering indexes!Non-clustered indexes are easier/faster to Non-clustered indexes are easier/faster to rebuild rebuild (re: smaller) when they become fragmented(re: smaller) when they become fragmentedNon-clustered indexes are easier to keep less Non-clustered indexes are easier to keep less fragmented – i.e., rebuilds with FILLFACTOR fragmented – i.e., rebuilds with FILLFACTOR has greater benefit since the index row is has greater benefit since the index row is narrowernarrower
Non-Clustered IndexesNon-Clustered Indexes
Internally, non-clustered indexes always Internally, non-clustered indexes always store an entry for EVERY row of the table store an entry for EVERY row of the table If the columns requested in the query are IN If the columns requested in the query are IN the index (regardless of order) then the the index (regardless of order) then the index COVERs the queryindex COVERs the queryIf queries are narrow this is more likely to If queries are narrow this is more likely to happenhappenReal-world queries…for example, accessing Real-world queries…for example, accessing a large number of tables in a join are a large number of tables in a join are typically narrow (the join condition, maybe a typically narrow (the join condition, maybe a search argument and possibly one or two search argument and possibly one or two columns in the select list)columns in the select list)Is it likely that you would ever execute:Is it likely that you would ever execute:
SELECT * FROM 8-table-joinSELECT * FROM 8-table-join
…000-00-0001, 497
000-00-0002, 349
000-00-0003, 5643
…
000-00-0539, 12
…
…
…
…
…
…
…
…
…
…
…
…
…
…
999-99-9999, 42
…
…
…
…
…
…
…
…
…
…
File1, Page 16897 File1, Page 16898 File1, Page 16899 File1, Page 11810 File1, Page 18111 File1, Page 18112
539 entries
File1, Page 19197
539 entries
539 entries
539 entries
228 entries
149 entries
Root = 1 Page
Leaf Level149
Pages
What kind of structure is this?What kind of structure is this?
A non-clustered index on SSN on a table that has a A non-clustered index on SSN on a table that has a clustered index on EmpID (as we’ve seen)clustered index on EmpID (as we’ve seen)A clustered in on SSN n a table that has ONLY two A clustered in on SSN n a table that has ONLY two columns, SSN and EmpIDcolumns, SSN and EmpID
NOTE:NOTE: A non-clustered index is EXACTLY the same as a clustered A non-clustered index is EXACTLY the same as a clustered index – but only on the columns that make up the NC index index – but only on the columns that make up the NC index (with the RID or CL Key).(with the RID or CL Key).
Seems Like…Seems Like…
Non-clustered indexes are only useful Non-clustered indexes are only useful for relatively selective queries…for relatively selective queries…
What about the following queries?What about the following queries?
oror
SELECT EmpID, SSN SELECT EmpID, SSN FROM Employee FROM Employee WHERE SSN between x and yWHERE SSN between x and y
SELECT EmpID, SSN SELECT EmpID, SSN FROM Employee FROM Employee WHERE SSN between x and yWHERE SSN between x and y
SELECT EmpID, SSN SELECT EmpID, SSN FROM Employee FROM Employee WHERE EmpID < 10000WHERE EmpID < 10000
SELECT EmpID, SSN SELECT EmpID, SSN FROM Employee FROM Employee WHERE EmpID < 10000WHERE EmpID < 10000
Think about the access patterns:• Table Scan (always an option)• Index Seek w/partial scan The nc index on SSN “covers” the query
Think about the access patterns:• SARG is on clustering key
The nc index on SSN “covers” the query
Finding the Right BalanceFinding the Right Balance
Start with a minimal number of Start with a minimal number of indexesindexes
Clustered IndexClustered IndexPrimary Key (nonclustered, if not Primary Key (nonclustered, if not clustered)clustered)Unique KeysUnique Keys
ManuallyManually index foreign keys index foreign keysNon-unique indexesNon-unique indexesSpeed up join performanceSpeed up join performance
Use Database Tuning AdvisorUse Database Tuning AdvisorManually index based on either:Manually index based on either:
Specific query tuning where DTA didn’t Specific query tuning where DTA didn’t helphelpQuery frequencyQuery frequency
Non-Clustered IndexesNon-Clustered IndexesFor optimal query performanceFor optimal query performance
Non-Clustered Indexes are BETTER for low Non-Clustered Indexes are BETTER for low selectivity/range queries because:selectivity/range queries because:
Non-clustered indexes ARE EXACTLY the same as a Non-clustered indexes ARE EXACTLY the same as a clustered index—but narrowerclustered index—but narrower
If your query only needs columns IN an index then the If your query only needs columns IN an index then the index is said to “cover” your queryindex is said to “cover” your query
No – this does No – this does notnot mean that you need to cover every mean that you need to cover every query!!!query!!!
What it does mean: You can make one or two existing What it does mean: You can make one or two existing indexes slightly wider and you probably will not hurt indexes slightly wider and you probably will not hurt INSERT, UPDATE, and DELETE performance and may INSERT, UPDATE, and DELETE performance and may RADICALLY improve SELECT performanceRADICALLY improve SELECT performance
Non-Clustered Indexes are narrower and easier to Non-Clustered Indexes are narrower and easier to maintain (let ITW/DTA help…)maintain (let ITW/DTA help…)
Use INCLUDE to create narrower trees, more Use INCLUDE to create narrower trees, more relevant leaf levels for covering!relevant leaf levels for covering!
Covering an AggregateCovering an Aggregate
Increasing Gains with different types Increasing Gains with different types of indexes can always be achievedof indexes can always be achieved
Each query can be isolated and tuned Each query can be isolated and tuned as if it were in a sandbox – not the as if it were in a sandbox – not the goalgoal
Finding the right balance starts with:Finding the right balance starts with:Base Indexes (CL, PK, UK, FKs)Base Indexes (CL, PK, UK, FKs)
Capturing a workload, tuning with DTACapturing a workload, tuning with DTA
Prioritizing the queries that are still Prioritizing the queries that are still performing poorly and then choose the performing poorly and then choose the optimal level of gain by understanding optimal level of gain by understanding the trade-offsthe trade-offs
Indexing for AggregationsIndexing for Aggregations
Two types of Aggregates: Two types of Aggregates: Stream and HashStream and Hash
Try to Achieve Stream to Minimize Try to Achieve Stream to Minimize Overhead in temp table creationOverhead in temp table creation
Computation of the Aggregate Still Computation of the Aggregate Still RequiredRequired
Lots of Users, Contention and/or Lots of Users, Contention and/or Minimal Cache can Aggravate the Minimal Cache can Aggravate the problem!problem!
Aggregate QueryAggregate Query
Member has 10,000 RowsMember has 10,000 Rows
Charge has 1,600,000 RowsCharge has 1,600,000 Rows
SELECT c.member_no AS MemberNo, SELECT c.member_no AS MemberNo, sum(c.charge_amt) AS TotalSalessum(c.charge_amt) AS TotalSales
FROM dbo.charge AS cFROM dbo.charge AS cGROUP BY c.member_noGROUP BY c.member_no
SELECT c.member_no AS MemberNo, SELECT c.member_no AS MemberNo, sum(c.charge_amt) AS TotalSalessum(c.charge_amt) AS TotalSales
FROM dbo.charge AS cFROM dbo.charge AS cGROUP BY c.member_noGROUP BY c.member_no
Aggregate Query (cont’d)Aggregate Query (cont’d)Table scan + hash aggregateTable scan + hash aggregate
Table Scan of Charge TableTable Scan of Charge TableLargest structure to evaluateLargest structure to evaluateWorst case scenarioWorst case scenario
Worktable created to store intermediate Worktable created to store intermediate aggregated results – OUT OF ORDER aggregated results – OUT OF ORDER (HASH)(HASH)Data Returned OUT OF ORDER – unless Data Returned OUT OF ORDER – unless ORDER BY addedORDER BY addedAdditional ORDER BY causes another stepAdditional ORDER BY causes another stepfor SORT – sorting can be expensive!for SORT – sorting can be expensive!
SELECT c.member_no AS MemberNo, SELECT c.member_no AS MemberNo, sum(c.charge_amt) AS TotalSalessum(c.charge_amt) AS TotalSales
FROM dbo.charge AS cFROM dbo.charge AS cGROUP BY c.member_noGROUP BY c.member_no
SELECT c.member_no AS MemberNo, SELECT c.member_no AS MemberNo, sum(c.charge_amt) AS TotalSalessum(c.charge_amt) AS TotalSales
FROM dbo.charge AS cFROM dbo.charge AS cGROUP BY c.member_noGROUP BY c.member_no
Worst CaseWorst Case
Clustered Index Clustered Index ScanScan(table scan)(table scan)1,600,000 rows1,600,000 rowsHash AggregateHash Aggregateyields 9,114 yields 9,114 rows out of rows out of orderorderSortSortonly has to sort only has to sort 9,114 rows 9,114 rows instead of instead of 1,600,000 rows1,600,000 rowsReturn DataReturn Data
Table 'charge'. Table 'charge'. Logical reads 9335Logical reads 9335
Aggregate QueryAggregate QueryIndex scan + hash aggregateIndex scan + hash aggregate
Out of Order Covering Index on Charge Out of Order Covering Index on Charge TableTable
Index Exists which is narrower than base tableIndex Exists which is narrower than base tableUsed instead of table – to cover the queryUsed instead of table – to cover the query
Worktable still created to store Worktable still created to store intermediate aggregated results – OUT OF intermediate aggregated results – OUT OF ORDER (HASH)ORDER (HASH)Data Returned OUT OF ORDER – unless Data Returned OUT OF ORDER – unless ORDER BY addedORDER BY addedAdditional ORDER BY causes another step Additional ORDER BY causes another step for SORT – sorting can be expensive!for SORT – sorting can be expensive!
SELECT c.member_no AS MemberNo, SELECT c.member_no AS MemberNo, sum(c.charge_amt) AS TotalSalessum(c.charge_amt) AS TotalSales
FROM dbo.charge AS cFROM dbo.charge AS cGROUP BY c.member_noGROUP BY c.member_no
SELECT c.member_no AS MemberNo, SELECT c.member_no AS MemberNo, sum(c.charge_amt) AS TotalSalessum(c.charge_amt) AS TotalSales
FROM dbo.charge AS cFROM dbo.charge AS cGROUP BY c.member_noGROUP BY c.member_no
Not as BadNot as Bad
COVERING COVERING Index ScanIndex Scan1,600,000 1,600,000 narrower rowsnarrower rows
Hash AggregateHash Aggregateyields 9,114 yields 9,114 rows out of rows out of orderorder
SortSortonly has to sort only has to sort 9,114 rows 9,114 rows instead of instead of 1,600,000 rows1,600,000 rows
Return DataReturn Data
Table 'charge'. Table 'charge'. Logical reads 3770Logical reads 3770
Aggregate QueryAggregate QueryIndex scan + stream aggregateIndex scan + stream aggregate
Covering Index on Charge Table (GROUP BY Covering Index on Charge Table (GROUP BY first)first)
Index Exists which is narrower than base tableIndex Exists which is narrower than base tableUsed instead of table – to cover the queryUsed instead of table – to cover the queryCovers the GROUP BY so data is groupedCovers the GROUP BY so data is grouped
Less work to aggregate results IN ORDERLess work to aggregate results IN ORDERData Returned IN ORDER – unless ORDER BY Data Returned IN ORDER – unless ORDER BY or other joins addedor other joins addedAdding an ORDER BY identical to the GROUP Adding an ORDER BY identical to the GROUP BY does NOT cause any additional step for BY does NOT cause any additional step for sorting!sorting!
SELECT c.member_no AS MemberNo, SELECT c.member_no AS MemberNo, sum(c.charge_amt) AS TotalSalessum(c.charge_amt) AS TotalSales
FROM dbo.charge AS cFROM dbo.charge AS cGROUP BY c.member_noGROUP BY c.member_no
SELECT c.member_no AS MemberNo, SELECT c.member_no AS MemberNo, sum(c.charge_amt) AS TotalSalessum(c.charge_amt) AS TotalSales
FROM dbo.charge AS cFROM dbo.charge AS cGROUP BY c.member_noGROUP BY c.member_no
Much Better!Much Better!
COVERING COVERING Index ScanIndex Scan1,600,000 1,600,000 narrower narrower rowsrows
Stream Stream AggregateAggregatealso yields also yields 9,114 rows 9,114 rows IN ORDERIN ORDERNO SORT NO SORT REQUIREDREQUIRED
Return DataReturn Data
Table 'charge'. Table 'charge'. Logical reads 3770Logical reads 3770
See the See the DifferenceDifference
??
ConcernsConcerns
More temp tablesMore temp tablesMore contention in tempdbMore contention in tempdbLarger tempdb requiredLarger tempdb requiredPerformance Varies on each Performance Varies on each executionexecutionAggregate needs to be computedAggregate needs to be computed
Is there a better way?Is there a better way?Indexed Views!Indexed Views!
Table 'charge'. Table 'charge'. Scan count 1, Scan count 1,
logical reads 34logical reads 34
Table Table 'SumOfAllCharges'. 'SumOfAllCharges'.
Logical reads 35Logical reads 35
Aggregate Aggregate QueryQuery
Indexed Indexed viewview
Query with NO Query with NO useful indexesuseful indexesQuery with NO Query with NO useful indexesuseful indexes
Query with covering Query with covering Index in wrong orderIndex in wrong orderQuery with covering Query with covering Index in wrong orderIndex in wrong order
Query with covering Query with covering index in correct orderindex in correct orderQuery with covering Query with covering index in correct orderindex in correct order
Query w/Indexed View Query w/Indexed View
and no computations!and no computations!
Query w/Indexed View Query w/Indexed View
and no computations!and no computations!
See the See the DifferenceDifference
??
Indexed View for AggregatesIndexed View for AggregatesKey pointsKey points
TempDB access not necessaryTempDB access not necessary
NO worktables are necessaryNO worktables are necessary
Aggregated set should be small but not too Aggregated set should be small but not too small as to create a hot ROW spot of small as to create a hot ROW spot of activity (which can create excessive activity (which can create excessive blocking)blocking)
GROUP BY member_no – probably OKGROUP BY member_no – probably OK
GROUP BY state – too few rows in aggregateGROUP BY state – too few rows in aggregate
GROUP BY country – GROUP BY country – avoidavoid like the plague! like the plague!
Performance of data modification Performance of data modification statements should be testedstatements should be tested
INCLUDE Non-key ColumnsINCLUDE Non-key ColumnsCompared to Indexed ViewsCompared to Indexed Views
Indexed Views allow aggregates – adding Indexed Views allow aggregates – adding interesting columns in the leaf level of an interesting columns in the leaf level of an index offers “creative covering”index offers “creative covering”
With new INCLUDE clause, leaf level of With new INCLUDE clause, leaf level of index can include non-key columnsindex can include non-key columns
Index key [has been since 7.0] limited to Index key [has been since 7.0] limited to 900 bytes/16 columns – this is to keep tree 900 bytes/16 columns – this is to keep tree structure and non-leaf levels optimal/smallstructure and non-leaf levels optimal/small
Allows more covering indexesAllows more covering indexes
Indexed View v. Include – depends on what Indexed View v. Include – depends on what needs to be in the leaf levelneeds to be in the leaf level
Finding the Right BalanceFinding the Right BalanceIndex strategies: SummaryIndex strategies: Summary
Determine Primary Usage of Table – OLTP Determine Primary Usage of Table – OLTP vs. OLAP vs. Combo? This determines vs. OLAP vs. Combo? This determines Clustered IndexClustered IndexCreate Constraints – Primary Key and Create Constraints – Primary Key and Alternate/Candidate KeysAlternate/Candidate KeysManually Add Indexes to Foreign Key Manually Add Indexes to Foreign Key ConstraintsConstraintsCapture a Workload(s) and Run through Capture a Workload(s) and Run through Index Tuning WizardIndex Tuning WizardAdd additional indexes to help improve Add additional indexes to help improve SARGs, Joins, AggregationsSARGs, Joins, Aggregations
Are you done?Are you done? NO!NO!NO!NO!
Keeping Performance OptimalKeeping Performance Optimal
Write effective queries to help Write effective queries to help optimizer use indexesoptimizer use indexes
Make sure statistics are accurateMake sure statistics are accurateKeep auto update statistics ONKeep auto update statistics ON
Keep auto create statistics ONKeep auto create statistics ON
Make sure indexes don’t become Make sure indexes don’t become overly fragmentedoverly fragmented
Rebuild Table/Index StructuresRebuild Table/Index Structures
Defrag Table/Index StructuresDefrag Table/Index Structures
Indexing WebcastsIndexing Webcasts
MSDN Webcast:MSDN Webcast: Indexing for Performance - Indexing for Performance - Finding the Right Balance (SQL Server 2000)Finding the Right Balance (SQL Server 2000)
http://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032254503&Culture=en-Uhttp://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032254503&Culture=en-USS
MSDN Webcast:MSDN Webcast: Indexing for Performance - Indexing for Performance - Index Maintenance Best Practices (SQL Index Maintenance Best Practices (SQL Server 2000)Server 2000)http://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032256511&Culture=en-Uhttp://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032256511&Culture=en-USS
TechNet It’s Sh0wtime Webcast:TechNet It’s Sh0wtime Webcast: Index Creation Index Creation Best Practices with SQL Server 2005Best Practices with SQL Server 2005http://www.microsoft.com/uk/technet/itsshowtime/sessionh.aspx?videoid=29http://www.microsoft.com/uk/technet/itsshowtime/sessionh.aspx?videoid=29
TechNet It’s Sh0wtime Webcast:TechNet It’s Sh0wtime Webcast: Index Index Defragmentation Best Practices with SQL Defragmentation Best Practices with SQL Server 2005Server 2005http://www.microsoft.com/uk/technet/itsshowtime/sessionh.aspx?videoid=30http://www.microsoft.com/uk/technet/itsshowtime/sessionh.aspx?videoid=30
MSDN Webcast Series:MSDN Webcast Series: Effectively Designing a Effectively Designing a Scalable and Reliable DatabaseScalable and Reliable Database
See Kimberly’s blog See Kimberly’s blog for PDC for PDC
Resources & LinksResources & Links
CachingCaching
The Zen of CachingThe Zen of Caching
Caching at the Middle TierCaching at the Middle Tier
Caching on the ClientCaching on the Client
Caching ZenCaching Zen
Moving information to a location Moving information to a location where it can be accessed very quicklywhere it can be accessed very quickly
From main memory to high-speed From main memory to high-speed memory on the processormemory on the processor
From the hard drive to main memoryFrom the hard drive to main memory
From main memory on one machine to From main memory on one machine to another machineanother machine
Caching is framework for managing Caching is framework for managing statestate
State being data + statusState being data + status
Caching Zen: Why?Caching Zen: Why?
PerformancePerformanceStore the data as close as possible to Store the data as close as possible to consumerconsumer
ScalabilityScalabilityCommon data is often processed the Common data is often processed the same way for multiple userssame way for multiple users
Process it once and store the resultsProcess it once and store the results
AvailabilityAvailabilityProvide access to data even when the Provide access to data even when the primary data store or service is primary data store or service is unavailableunavailable
Caching Zen: When?Caching Zen: When?
On-demand: cache the data after it On-demand: cache the data after it has been requested and formattedhas been requested and formatted
A priori: preload the cache with both A priori: preload the cache with both raw and formatted dataraw and formatted data
Combine the twoCombine the two
Caching Zen: Where?Caching Zen: Where?
Generally as close to the data Generally as close to the data consumer as possible—wherever consumer as possible—wherever accessing the cache can help withaccessing the cache can help with
PerformancePerformance
ScalabilityScalability
AvailabilityAvailability
Actual cache storage can beActual cache storage can beMemory resident—often on serversMemory resident—often on servers
Disk resident—often on clientDisk resident—often on client
Caching Zen: IssuesCaching Zen: Issues
StalenessStaleness
CoherencyCoherency
SecuritySecurity
StalenessStaleness
Data that is cached becomes Data that is cached becomes obsoleteobsolete
Staleness is the difference between Staleness is the difference between the master data’s current value and the master data’s current value and the current version that is cachedthe current version that is cached
Staleness is relative to how often the Staleness is relative to how often the master copy of the data changes and master copy of the data changes and how tolerant an application is to stale how tolerant an application is to stale datadata
CoherencyCoherency
Large systems often have multiple Large systems often have multiple cachescaches
Often seen with web server farmsOften seen with web server farms
Caches need to kept in sync Caches need to kept in sync otherwise clients can get inconsistent otherwise clients can get inconsistent resultsresults
SecuritySecurity
Data stored in cache needs to Data stored in cache needs to protectedprotected
Application logic can enforce access Application logic can enforce access rules to in-memory cache on serversrules to in-memory cache on servers
Client-side cache might require both Client-side cache might require both in-memory and disk level protectionin-memory and disk level protection
SpywareSpyware
Unsafe networksUnsafe networks
UsersUsers
Staleness SolutionStaleness Solution
Cache expiration settings based upon Cache expiration settings based upon the application’s tolerance to stale the application’s tolerance to stale datadata
Can be time driven where cache is Can be time driven where cache is reloaded based upon an expiration reloaded based upon an expiration settingsetting
Can be data driven where cache is Can be data driven where cache is updated when master data is updated when master data is updatedupdated
Coherency SolutionCoherency Solution
Depending upon the number of Depending upon the number of distinct cache stores, updates can distinct cache stores, updates can occuroccur
At the same timeAt the same timeWorks when cache is small and speed of Works when cache is small and speed of update is quickupdate is quick
At defined intervalsAt defined intervalsWorks when cache is large and cache is load-Works when cache is large and cache is load-balancedbalanced
Security SolutionSecurity Solution
Store cache data encryptedStore cache data encrypted
Ensure communications between Ensure communications between cache and data source are securecache and data source are secure
Cache SyncCache Sync
New feature of SQL Server 2005New feature of SQL Server 2005
Cache source creates data based Cache source creates data based subscription that server will monitorsubscription that server will monitor
When data changes, server notifies When data changes, server notifies subscriber of change and reasonsubscriber of change and reason
Uses new Service Broker infrastructureUses new Service Broker infrastructure
Exposed in SQLClient 2.0 and Exposed in SQLClient 2.0 and ASP.NET 2.0ASP.NET 2.0
High-level and low-level implementations High-level and low-level implementations availableavailable
State Transformation PipelineState Transformation Pipeline
State can be presented in many State can be presented in many formatsformats
Raw data object to business object to Raw data object to business object to presentation objectpresentation object
Caching possible at any and all points Caching possible at any and all points in the pipelinein the pipeline
Caching at the Middle TierCaching at the Middle Tier
Prefer existing host processes and Prefer existing host processes and servicesservices
COM+ components and object poolingCOM+ components and object pooling
IIS with ASP.NETIIS with ASP.NET
Use custom services as neededUse custom services as neededWindows Services with custom endpoint Windows Services with custom endpoint and cache enginesand cache engines
Caching at the Client TierCaching at the Client Tier
Local servicesLocal servicesSQL Server ExpressSQL Server Express
Custom Windows ServicesCustom Windows Services
In-memoryIn-memorySystem.Web.CachingSystem.Web.Caching
DataSetsDataSets
Custom objectsCustom objects
Tuning, Monitoring, and Diagnosing System Tuning, Monitoring, and Diagnosing System ProblemsProblems
SQL Server ProfilerSQL Server Profiler
System MonitorSystem Monitor
Database Tuning AdvisorDatabase Tuning Advisor
SQL Server ProfilerSQL Server Profiler
Can analyze the SQL Server Database Can analyze the SQL Server Database Engine and Analysis ServicesEngine and Analysis ServicesSignificantly easier to setup (Events, Data Significantly easier to setup (Events, Data Columns and Filter dialogs combined)Columns and Filter dialogs combined)
Special Events: Service Broker, Notification Special Events: Service Broker, Notification Services, etc…Services, etc…Special Event types: Showplan XML and Special Event types: Showplan XML and Deadlock Graph – can be saved to one (each Deadlock Graph – can be saved to one (each type) or individual files for later review and type) or individual files for later review and interaction!interaction!
Events that do/do not produce values for Events that do/do not produce values for given data columns are obvious at given data columns are obvious at selection (check boxes) and viewing selection (check boxes) and viewing (yellowed)(yellowed)Can Profile SQL Server 2000 and 2005Can Profile SQL Server 2000 and 2005Permissions to profile are grantablePermissions to profile are grantable
Profiler EnhancementsProfiler Enhancements
SQL Server 2000 profiler scripts run SQL Server 2000 profiler scripts run with out change on 2005with out change on 2005Uses XML based definitionsUses XML based definitionsSave showplan in XML formatSave showplan in XML formatSave results in XML formatSave results in XML formatCan replay one or more rollover filesCan replay one or more rollover filesAllows profiling of Allows profiling of
Analysis ServicesAnalysis ServicesData Transformation ServicesData Transformation Services
Correlation of Events to Performance Correlation of Events to Performance Monitor dataMonitor data
Profiler UI CondensedProfiler UI Condensed
System Monitor (PerfMon) System Monitor (PerfMon) Integration Integration Performance Counter LogsPerformance Counter Logs
Create a Profiler TraceCreate a Profiler Trace
Create a Performance Monitor LogCreate a Performance Monitor Log
Works solely based on time – make Works solely based on time – make sure the two clients (if different) are sure the two clients (if different) are time correlatedtime correlated
Open Trace (complete load), Use File Open Trace (complete load), Use File – Import Performance Data, Select – Import Performance Data, Select Objects/Counters…Objects/Counters…
Works with SQL Server 2000 and SQL Works with SQL Server 2000 and SQL Server 2005 as it’s client side!Server 2005 as it’s client side!
Performance Monitor LogPerformance Monitor Log
DefineDefineObjects Objects (complete (complete groups)groups)
CountersCounters
Sample IntervalSample Interval
File LocationFile Location
Start log (time)Start log (time)
Stop log (time)Stop log (time)
Profiler with PerfMon Profiler with PerfMon IntegrationIntegration
Database Tuning AdvisorDatabase Tuning Advisor
Index Tuning Wizard replaced by DTAIndex Tuning Wizard replaced by DTA
Adds:Adds:Partitioning recommendationsPartitioning recommendations
Time-bound tuningTime-bound tuning
Indexes with Included columns -> more Indexes with Included columns -> more efficient coveringefficient covering
XML Input/Output XML Input/Output
Drop ONLY modeDrop ONLY mode
Parameterized command line executionParameterized command line execution
Database Tuning AdvisorDatabase Tuning Advisor
Can be launched from-Can be launched from-SQL Server management StudioSQL Server management Studio
ProfilerProfiler
Import previously saved Session Import previously saved Session DefinitionDefinition
Stored in XML formatStored in XML format
Workload optionsWorkload optionsCan be a *.trc, *.sql or *.xml formatCan be a *.trc, *.sql or *.xml format
Can be a SQL Server TableCan be a SQL Server TableFor table saved in 2000 need to launch DTA For table saved in 2000 need to launch DTA while connected to 2000 serverwhile connected to 2000 server
Database Tuning AdvisorDatabase Tuning Advisor
Workload optionsWorkload optionsClone SessionClone Session
Actions Actions clone session clone sessionLaunches duplicate DTA window while Launches duplicate DTA window while maintaining chosen settingsmaintaining chosen settings
Tuning optionsTuning optionsPhysical Design Structures (PDS)Physical Design Structures (PDS)
To use in databaseTo use in databaseTo keep in databaseTo keep in database
Partitioning StrategyPartitioning StrategyOnline index recommendationsOnline index recommendations
Default is OFFDefault is OFFChange in Advanced Tuning OptionsChange in Advanced Tuning Options
Database Tuning AdvisorDatabase Tuning Advisor
Optimization ProcessOptimization ProcessConfiguration infoConfiguration infoReading workloadReading workloadPerforming analysisPerforming analysisReportsReports
Tuning summary reportTuning summary reportEvent frequencyEvent frequencyQuery detailQuery detailQuery-index relationsQuery-index relationsQuery cost rangeQuery cost rangeIndex usageIndex usage
RecommendationsRecommendationsIndex, partition schemesIndex, partition schemes
INF: How to Create a SQL Server 2000 TraceINF: How to Create a SQL Server 2000 Trace ((283790283790))HOW TO: Programmatically Load Trace Files into HOW TO: Programmatically Load Trace Files into TablesTables ( (270599270599))How To: Stop a Server-Side Trace in SQL Server How To: Stop a Server-Side Trace in SQL Server 20002000 ( (822853822853))INF: How to Monitor SQL Server 2000 TracesINF: How to Monitor SQL Server 2000 Traces (283786)(283786)INF: Stored Procedure to Create a SQL Server INF: Stored Procedure to Create a SQL Server 2000 Blackbox Trace2000 Blackbox Trace (281671) (281671)BUG: BOL Incorrectly States That Users Do Not BUG: BOL Incorrectly States That Users Do Not Need to Be Sysadmin to Use Profiler or SQL Need to Be Sysadmin to Use Profiler or SQL Profiler SPs Profiler SPs (this is a SQL Server 2000 limitation – (this is a SQL Server 2000 limitation – 310175310175))INF: Job to Monitor SQL Server 2000 Performance INF: Job to Monitor SQL Server 2000 Performance and Activityand Activity ( (283696283696))
Minimizing the Impact of Minimizing the Impact of ProfilingProfilingServer-side Trace QueuesServer-side Trace Queues
Session SummarySession Summary
Building a scalable, reliable, and Building a scalable, reliable, and performant real-world system takes performant real-world system takes MANY steps…MANY steps…There’s no magic bullet!There’s no magic bullet!But, it is possible…But, it is possible…
using these techniques!using these techniques!
Join Indonesia SQL Server CommunityJoin Indonesia SQL Server CommunitySubscribeSubscribe
dotnet-subscribe@netindonesia.netdotnet-subscribe@netindonesia.net
Un-SubscribeUn-Subscribedotnet-unsubscribe@netindonesia.netdotnet-unsubscribe@netindonesia.net
© 2005 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
top related