SQL Server Waits Types

Sunil Kumar Anna

2

3

Terminology in SQL Server in terms of CPU =schedulers

4

5

6

7

8

9

More CPUs = more schedulers = more work can get done 

10

Worker Thread life cycle

Most of your seen these status in the output of select * from sys.sysprocesses

11

Compilation Time

Wait time

CPU Time

Is your query running slow ?

Kind of SchedulersSQL server is like a operating system like windows. It has its own mechanism 

to handle the memory management, IO and thread scheduling. SQL server might be using the windows API to do these task.

SQLOS has it's own CPU Scheduler, called the SOS Scheduler and it does not follow OS scheduling (looks strange as it runs on the preemptive OS) .Its called as UMS (User Mode Scheduling) in 2000 and SOS (SQL OS) in 2005 and above.

SQL Server to schedule it's own processor workload, using a cooperative scheduling algorithm. This means that instead of forcing a thread to yield, it allows to threads to yield voluntarily

12

13

Preemptive scheduling:-

Operating system can interrupt the thread execution any time. OS can halt the thread execution and schedule another thread to run at any time

14

Examples for Preemptive wait types

15

One Scheduler = 1 logical CPU1. Schedulers are responsible for allocating system resources (such as CPU time) to

SQL Server processes.2. Schedulers with IDs above 255 are called “hidden "schedulers; these are used by

system processes.  

3. Each scheduler /master (one per logical CPU) accepts new tasks and hands them off to workers. SOS Scheduler allows one worker at a time to be exposed to the

CPU.

16

How and why a worker thread come into the picture ??????????

scheduler _tasks.txt

17

Note :-

If a scheduler is Visible, it means that it is mapping directly to a CPU Core and will accept user requests. If it is set to hidden, it means it is associated with a Resource Monitor. The scheduler associated with the DAC is easy to identify, as it ends with (DAC) ! ;-)

Schedulers =CPUS

sys.dm_os_schedulers

Each scheduler will accept Task

sys.dm_os_tasks

Workers are the humble servants who carry out the task assigned to them by the Master (scheduler)

sys.dm_os_workers

18

Task  :-Task represents the work that needs to be performed

Select * from sys.dm_os_tasks

19

6 processor 32 bit server will have :

256 (Fixed number for 4 processor) + 2 (remaining Processors [6-4] ) * 8 = 256 + 16 = 272 Worker Threads

8 processor 32 bit server will have

256 (Fixed number for 4 processor) + 4 (remaining Processors [8-4] ) * 8 = 256 + 32 = 288 Worker Threads

16 processor 64 bit server will have:

512 (Fixed number for 4 processor) + 12 (remaining Processors [16-4] ) * 16 = 512+ 32 = 704 Worker Threads

32 processor 64 bit server will have:

512 (Fixed number for 4 processor) + 28 (remaining Processors [32-4] ) * 16 = 512+ 448 = 960 Worker Threads

Example calculation

For 32 bit operating system:Total available logical CPU’s <= 4 :     max worker threads = 256 Total available logical CPU’s > 4 :  max worker threads = 256 + ((logical CPUS’s - 4) * 8)

For 64 bit operating system:Total available logical CPU’s <= 4 :     max worker threads = 512 

Total available logical CPU’s > 4 : max worker threads = 512 + ((logical CPUS’s - 4) * 16)

Worker (worker thread) 

Workers are the humble servants who carry out the task assigned to them by the Master (scheduler).

Worker threads are the agents of SQL Server which are scheduled in CPU and they carry out the tasks

Select * from sys.dm_os_workers If you leave 'Max. Worker threads' to 0 then SQL Server will decide the worker thread count based on formula below:

20

How to Sets the maximum number of Workers in SQL Server 

21

22

Boost SQL Server priorityIf you enable this option, SQL Server will run the sqlservr.exe process and threads as High Priority instead of its usual Normal priority. Hence, when SQL Server service will request CPU, other processes in need of CPU time won’t be prioritized. In some scenarios it can lead to problems and most of the time it won’t bring any benefit. Microsoft do not recommend to enable this feature

SQL Server runs at priority base of 13 in windows instead 7

windows supports thread priority level ranging from 0 (Lowest) to 31 (Highest). If all the threads have same priority they are scheduled in  round robin basis, but in reality threads running on OS will have different priorities. Among all the threads which can be run windows scheduler picks thread with highest priority to run first. Priority of a thread can be changed using WINAPI SetThreadPriority, Similarly priority of a process can be set while creating the process  (WINAPI CreateProcess  dwCreationFlags ) ,using WINAPI SetPriorityClass after the process is created and by using tools like task manager.

Note:- 1. when a Scheduler idle more than 15 minutes , or Windows facing memory pressure . SQL Server will

attempt to Trim the Worker pool to release the memory occupied by Worker2. General recommended to leave Max. worker threads to 03. Upgrading an instance of the SQL Server 2000 Database Engine to a newer version retains the

configuration value as 255 for max worker threads. You have to change this value to 0 to take advantage of SQL Server automatic worker thread calculations

4. When a request is received, SQL Server will try to locate a idle thread to schedule the request. If an idle worker was not located, new worker will be created. If all workers have been created and none is idle, then the request is queued which will be picked up by any worker thread from the pool of worker threads created. Query below gives the amount of worker threads created at the moment in your SQL Server

Max. worker threads

23

SQL Worker Thread Stack Sizes

SQL Server Build OS Build Stack Size

SQL Server x86 (32-bit) x86 (32-bit) 512 KB

SQL Server x86 (32-bit) x64 (64-bit) 768 KB (see notes 1 and 2)

SQL Server x64 (64-bit) x64 (64-bit) 2048 KB

SQL Server IA64 (64-bit Itanium) IA64 (64-bit Itanium) 4096 KB (see note 3)

We can use the above values for calculating MemToLeave

((MaxWorkerThreads * StackSize) + DefautReservationSize) 

 In fact, on 64bit SQL Server systems, the engine doesn’t reserve a large amount of memory for the buffer pool as with 32bit. So there is no need to leave any VAS space around since we don’t reserve most of it

24

SQL Worker threads management ---Dbcc sqlperf(umsstats)--

Scheduler ID .0 num users 13.0 num runnable .0 num workers 12.0 idle workers 5.0 work queued .0 cntxt switches 25488.0 <-sql cntxt switches(idle) 18344.0 <-kernelScheduler Switches .0Total Work 14296.0

Item Definition

num users Number of SQL threads on the scheduler

num runnable Number of SQL threads that are “runnable” (Scheduler queue length)

num workers Number of actual workers to process threads (thread pool)

idle workers Number of idle workers

cntxt switches Context switches between runnable threads

cntxt switches(idle) Context switches to the “idle” thread

25

Understanding how SQL Server executes a query

26

27

28

29

30

Common SQL Server Waits/Bottlenecks

31

32

It’s like several chefs are making soup. If one chef’s only job is to add the cilantro, there’s going to be some waiting involved. So is the extra help worth the trouble?

Hints:- For servers that have NUMA configured, MAXDOP should not exceed the number of CPUs that are assigned to each NUMA node. Meaning that if a server has 4 NUMA nodes then MAXDOP will be 4 or less. This will reduce threading overhead that occurs which will then be utilizing more NUMA nodes simultaneously.

33

MaxDOP

The option, max degree of parallelism controls the number of processors that can be used to run a single SQL Server statement using a parallel execution plan. The default value for this configuration is 0, indicating that all available processors can be used.

Generally to set MAXDOP value in SQL Server we follow below considerations.

Guidelines to configure MAXDOP value in SQL Server:-

1. For servers that use more than eight processors, use this configuration: MAXDOP=8 

2. For servers that have eight or less processors, use this configuration where N equals

the number of processors: MAXDOP=0 to N.  3. For servers that have NUMA configured, MAXDOP should not exceed the number

of CPUs cores) that are assigned to each NUMA node. 4. For servers that have hyper-threading enabled, MAXDOP should not be 0 and

should not be greater than half the number of visible schedulers(24) 5. Maximum Degree of Parallelism (MAXDOP) to 1 for SQL instances which host

SharePoint database 6. if the first step will not work then probably we have to alert the MAXDOP

value as 2 to core CPU’s allocated to NUMA Node and test the application

Note:- There is no need to restart SQL Server after changing MAXDOP setting in SQL Server.

34

Here you can see CXPACKET wait time variation for each Worker thread

Hyper-Threading :-

Note: If we see CXPACKET wait type is more, recommend to check to find any outdated updates stats or Fragmented/missing indexes in relevant database then go for MDOP settings.

35

Symptoms and detecting Parallelism Issue:

Usually CXPACKET wait type can be used to monitor parallel query execution behavior in OLTP systems. But keep in mind that CXPACKET does not necessarily indicate that parallelism is an issue. This wait means that the parent thread is waiting to synchronize all output from the child threads. However, if you see SQL blocking on CXPACKET, it indicates that the SQL Server is facing resource contention such as lack of indexes, out-dated statistics, I/O and CPU bottleneck, parameter sniffing issue, excessive sort operation and so on. 

Generally the combined waits from EXCHANGE, EXECSYNC and CXPACKET can be used to measure and identify whether parallelism is an issue or not.By increasing MAXDOP, if you see that the CPU usages goes high and the number of waiting tasks increases, this generally indicates that there is a parallelism issue. “Avg waiting tasks” in “Activity Monitor” can be used to observe the behavior quickly

Maxdop.txt

If we configure MDOP is 1 that mean telling sql server that not to use Parallelism. That is not recommended until it is valid for the environemnt

36

execution plan.txt

37

SQLOS worker (thread) that has voluntarily yielded the CPU to another worker (SQL aims to prevent starvation)

Yield= Give away

SQLOS worker (thread) that has voluntarily Give Away the CPU to another worker (SQL aims to prevent starvation of CPU)

To keep a single executing thread from taking 100% CPU, our code automatically relinquishes control back to the Scheduler.  The Scheduler looks to see if there is another thread/query ready to run(runnable). If yes, it will start that thread/query, run it for a short while, then switch back.  If no one else is ready to run, we switch back to the first thread. While the first thread is WAITING for the Scheduler to check for other runnable threads and while those threads are running, we clock SOS_SCHEDULER_YIELD wait

When a thread moves out of CPU it is called as yielding

Context Switching:When a thread is yielded from CPU windows stores the CONTEXT (current state such as CPU registers, program counters Etc) information of the current thread and loads the CONTEXT of the new thread which will run in the CPU.   Why? A thread is yielded when executing an instruction, How the thread will resume from same point when it is rescheduled in CPU. Context information is stored while yielding and loaded when thread in rerun. 

Quantum end : All the threads which is executed in operating system will get a time slice called as quantum. When a thread completes its quantum it is yielded and next thread is run. If there is no other thread is ready to run than thread run for another quantum

38

39

1. When you see ASYNC_NETWORK_IO waits, the issue is outside the database

2. SQL Server is waiting on a client somewhere on the network to acknowledge that it has some data. The client is often processing one row at a time rather than caching it and processing it. So, SQL spins up ASYNC_NETWORK_IO waiters as it waits. This has nothing to do with the network because for this test there isn’t a network

3. It typically indicates a network-related or client application issue 4. If the server process waits, this fills network buffers5. If network buffers fill, SQL Server can’t send more data to the client6.  If you see a query waiting on things like ASYNC_NETWORK_IO or

PREEMPTIVE_OS_WAITFORSINGLEOBJECT, then that query is waiting for a lazy client application to read its data.

7. PREEMPTIVE_OS_WAITFORSINGLEOBJECT – This is SQL calling out to the OS asking for more data to be processed.

40

41

42

43

Reading not final data- Exam

44

a transaction re-reads a single record and finds that it has been changed or deleted

45

either the new records you see have been inserted or data has changed in existing records such that these records now meet the filter criteria. Either way these records are new to you

Ghostly appearing figure

46

47

48

Latch in Memory- Page latch

51

NULL latch (NL): Not usedKEEP latch (KP): Have two purposes: to keep a page in the buffer cache while another latch is placed upon it, and the second is to maintain reference counts.SHARED latch (SH): Taken out when a request to read the data page is received.UPDATE latch (UP): Milder than an EX latch, this allows reads but no writes on the page while being updated.EXCLUSIVE latch (EX): Severe latch that allows no access to the page while being written. Only one per page can be held.DESTROY latch (DT): Used to destroy a buffer and evict it from the cache (returning the page to the free list).

52

Locks are used for TRANSACTIONAL CONSISTENCY 

Latches are used for MEMORY CONSISTENCY

Latches are lightweight – they are held for the duration of the physical operation then released. (locks are held for the duration of transaction)

53

SQL Server uses buffer latches to protect pages in the buffer pool and I/O latches to protect pages not yet loaded into the buffer pool.

54

58

To increase concurrency and performance, buffer latches are held only for the duration of the physical operation on the page, unlike locks which are held for the duration of the logical transaction.

60

Latch contention 

competition by users of a system for the facility at the same timeAt the beginning of application deployment, contention

does not show up, but as the client-base (concurrency) increases, application response time will gradually suffer. Once in a while, users start receiving time-out messages and observe infrequent poor application response.

Solution :- Creating a new tempdb files and enabling Trace Flag 1118 will give a solution

61

Trace Flag 1118 (Uniform allocation)

This trace flag is available starting with SQL 2000 SP3 to reduce tempdb contention by forcing uniform extent allocations as opposed to mixed extent allocations. This trace flag is only to be used if you seeing contention (wait_Stats) on the PFS/GAM pages like 2:1:1 etc.. 

Temp DB file:-Increasing the number of data files based on number of coresIncrease the number of TEMPDB data files and size them equally

tempdb count suggestion.txt

62

SQL Server is looking for the page which is not exit in the buffer cache, Then Asynchronous IO processed it to load the page from harddisk to buffer chache, until SQL Server has to wait to respond I/O subsystem. Hence the wait type as PAGEIOLATCH_EX

Workaround

Most of the cases it’s not because of IO subsystem.

if you see CXPACKET value is high along with PAGEIOLATCH_EX is wait type then we may need to create proper non clustered indexes or need to be updated index statistics. It also one reason due to Memory pressure on the Server to under Memory pressure by checking below. That is the case we need to increase Buffer memory for SQL Server.

66

67

68

RESOURCE_SEMAPHORE_QUERY_COMPILE

Assume SQL Server Max server memory is set to 1000MB and Currently data /index pages is consuming 800MB with in Max server memory (bpool) and 3 queries are reaching SQL Server for compilation, each of them requiring 300 MB for compilation.If all three queries are compiled simultaneously total compilation might take 900MB of memory causing all the data and index pages to be dropped from BPOOL causing consequent increased physical IO and poor performance of query during execution(to bring data pages back to memory). On the other hand let us assume each of this 3 queries need only 2 MB of compilation memory, There is no reason for SQL Server to throttle the number of compilation.To overcome above challenges SQL 2005+ throttles the number of concurrent compiles that can happen at any time based on memory usage during the compile. SQL Server memory grant is controlled by a object called “Resource Semaphore”.Compile memory: When a query is compiled in SQL Server, the compilation process needs memory (for parsing, algeberaization and optimization) called compile memory. This memory doesn’t include the memory required to execute the query.Challenges with Compile memory: This memory used for Query compilations are usually expected to fit into SQL Server main-memory and to be relatively short-lived. Like any other consumers of Bpool, this is implemented by “stealing” pages from the Buffer Pool and hence it blocks other memory consumers from using that memory until a query compilation completes.

Even if the SQL Server has enough memory to service multiple simultaneous query compilation, allowing all of them to occur at the same time might lead to stealing a significant number of pages from the buffer pool, with consequent increased physical IO , poor performance of query execution and causing memory pressure within SQL Server. However on the other side, a throttling mechanism that is too restrictive could lead to a non-optimal usage of the system resources and decreased throughput for compile intensive workloads, So SQL Server came with more dynamic approach to solve the problem which is to better manage system resources, memory in particular. Such management should hence be based on and driven by the amount of memory used.

When a query starts, there is no limit on how many queries can be compiled. As the memory consumption increases and reaches a threshold, the query must pass a gateway to continue. There is a progressively decreasing limit of simultaneously compiled queries after each gateway. The size of each gateway depends on the platform and the load. Gateway sizes are chosen to maximize scalability and throughput.

If the query cannot pass a gateway, the query will wait until memory is available. Or, the query will return a time-out error (Error 8628). Additionally, the query may not acquire a gateway if the user cancels the query or if a deadlock is detected. If a query passes several gateways, the query does not release the smaller gateways until the compilation process has completed. This behavior lets only a few memory-intensive compilations occur at the same time. Additionally, this behavior maximizes throughput for smaller queries.

Purpose :The whole gateway architecture is a way for SQL Server to keep memory usage for the compilation of incoming queries under control. This includes· avoiding out-of-memory conditions· Preventing a few very complex queries from eating up all memory and impact the whole server across the board because they compile at the same time· Making sure that compiling complex queries does not prevent SQL Server from compiling small ‘standard’ queries.· Serializing very large consumers to make sure each one succeeds in turn as opposed as all of them starting together and failing because of lack of memory to complete.You can think of it as a “Quality of Service” for compilation.

Implementation :· The resource that will drive the compilation throttling is the memory.o But since we may ultimately limit how many complex queries compile at a given time, this will also in turn limit the usage of other resource, mainly CPU.o As the KB mentions, we’re talking compile-time memory here, not execution-time memory.· Based on how much memory is used by the query for its ongoing compilation, the query will need to gain access to gateways :o There are 3 gateways : small, medium and large.o Each gateway is associated to a memory threshold.o Each gateway only allows a given number of queries to compile at a given time (see table below).

A query asking for a gateway will attempt to grab a 'RESOURCE_SEMAPHORE_QUERY_COMPILE' semaphore.§ If it asks for a gateway but gateway is full, it will therefore wait for that semaphore, and this reflects in DMVs.§ There’s only one waittype for all 3 gatewayso The memory consumed by compilation is tracked and updated throughout the compilation duration and gateway requests subsequently occur when applicable.§ When requesting a larger gateway, lower gateways slots (semaphores) are kept.o The memory threshold is static based on architecture for the small gateway, and is fully dynamic for medium and large (ie. value depends on current system state and change over time).o Specific timeouts will prevent queries from waiting forever for a 'RESOURCE_SEMAPHORE_QUERY_COMPILE' semaphore.o Obviously when a query compilation finishes the gateway semaphores it owns are released.

Gateway Type Small Medium Large

Number of simultaneous query

allowed.

4 x Number of logical CPUs allocated to SQL

Server.

Number of logical CPUs allocated to SQL

Server.

1

Because the medium and large thresholds are dynamic (in a nutshell, they get lower when server load increases), that means that there will be little throttling on a lightly loaded system, therefore we maximize resource usage (no “hardcoded” thresholds). That also means that the performance of a given set of queries (with big compilation memory footprint) might vary based on concurrent workload (the switch to large gateway means full serialization of selected queries, which can certainly have some impact...).

Each gateway only allows a given number of queries to compile at a given timeA query asking for a gateway will attempt to grab a 'RESOURCE_SEMAPHORE_QUERY_COMPILE' semaphore If it asks for a gateway but gateway is full, it will therefore wait for that semaphore, and this reflects in DMVs

There’s only one waittype for all 3 gateways .The memory threshold is static based on architecture for the small gateway, and is fully dynamic for medium and large (ie. value depends on current system state and change over time).

How to identify RESOURCE_SEMAPHORE_QUERY_COMPILE waits? To get an idea look at the sys.sysprocesses table for sessions waiting on RESOURCE_SEMAPHORE_QUERY_COMPILEselect sp.*, st.text from sys.sysprocesses sp CROSS APPLY sys.dm_exec_sql_text(sp.sql_handle) AS stWHERE sp.lastwaittype LIKE ‘RESOURCE_SEMAPHORE_QUERY_COMPILE%’ ORDER BY sp.waittime DESC;

73

ExampleThere could be two possible reasons for RESOURCE_SEMAPHORE_QUERY_COMPILE waits1. Memory pressure within SQL Server caused by others using lot of stolen memory or OS memory pressure

74

 

In this case you will see thresholds for medium and big gateways very low. In this situation you have to identify who is consuming most of stolen memory and tune them to increase the available memory or add additional memory that can used by SQL server. When the available memory decrease, threshold for medium and big gateways would decrease significantly and increase the number of queries which have to enter medium /big gateways, So the number of parallel compiles will decrease increasing the overall wait time. This DBCC memory status output is from system which has 48 processor and has excessive RESOURCE_SEMAPHORE_QUERY_COMPILE waits because of memory pressure.

Small Gateway (default) Value—————————————- ———–Configured Units 192 // 190 number of units . 48 CPU’s * 4=192Available Units 109Acquires 83Waiters 0Threshold Factor 380000Threshold 380000(6 row(s) affected)Medium Gateway (default) Value—————————————- ———–Configured Units 48 //48 number of units. SO 48 CPU’s *1=48Available Units 0 //There is no available slots. All slots are busy.Acquires 48

Waiters 34 //34 Queries are waitingThreshold Factor 12Threshold 2204603 //Threshold is very low 2 MB (This value is in bytes)(6 row(s) affected)Big Gateway (default) Value—————————————- ———–Configured Units 1 //1 per instanceAvailable Units 0 //There is no available slots. All slots are busy.Acquires 1Waiters 47 // 47 Queries are waitingThreshold Factor 8Threshold 3306905 //Threshold is very low 3 MB (This value is in bytes)

There is huge amount of available memory but the available units in gateway is exhausted. This situation normally occurs when we have many queries that have high compile time and use lot of memory for compilation.Compile time and Compile memory can be captures using the show plan XML for query compile event in profiler.Below is extract from show plan XML for query compile event in profiler. This query has used approximately 150 MB of compile memory and the threshold for Big gateway is around 143 MB .

So only one query which needs more than 143 MB can compile at a time , This can cause contention when there are multiple queries waiting for compile/recompile. Also an important factor to notice in this XML plan is compile time is ~139 times the CPU. So likely this query waited for most of the time for resource semaphore

Big Gateway (default) Value—————————————- ———–Configured Units 1Available Units 0Acquires 1Waiters 47Threshold Factor 8Threshold 149640500 <QueryPlan CachedPlanSize="312" CompileTime="139847" CompileCPU="1002" CompileMemory="152320">

How to fix RESOURCE_SEMAPHORE_QUERY_COMPILE waits?

1. Add additional memory to SQL Server sys.dm_exec_query_resource_semaphoressys.dm_exec_query_memory_grants2. Reduce the number of compile and recompiles happening in SQL Server.3. Threshold for the gateways are dynamic (except for small gateway) and therefore memory pressure occurs from other sources (Internal to SQL Server or system wide) reduces the amount of memory available for compiles and Queries are throttled to higher gateways sooner. Make sure SQL Server is not starving for memory.4. Identify the queries which are consuming large compile memory and tune them (CompileMemory in show plan XML query compile can be used).5. RESOURCE_SEMAPHORE_QUERY_COMPILE waits are very common in 32-bit SQL Server because of virtual address space limitation, so migrate to 64-Bit SQL Server.

77

78

79

BAD_PAGE_PROCESS

BAD_PAGE_PROCESS Occurs when the background suspect page logger is trying to avoid running more than every five seconds. Excessive suspect pages cause the logger to run frequently

select * from msdb.dbo.suspect_pages

DBCC CHECKDB (<dbname>) WITH ALL_ERRORMSGS, NO_INFOMSGS

dbcc traceon(3604) go dbcc page(<dbname>,1,1002854,0)

81