mutex vs semaphores

28
Mutex vs. Semaphores – Part 1: Semaphores It never ceases to amaze me how often I see postings in newsgroups, etc. asking the difference between a semaphore and a mutex. Probably what baffles me more is that over 90% of the time the responses given are either incorrect or missing the key differences. The most often quoted response is that of the “The Toilet Example (c) Copyright 2005, Niclas Winquist ” . This summarises the differences as: A mutex is really a semaphore with value 1 No, no and no again. Unfortunately this kind of talk leads to all sorts of confusion and misunderstanding (not to mention companies like Wind River Systems redefining a mutex as a “Mutual-Exclusion Semaphore” – now where is that wall to bang my head against?). Firstly we need to clarify some terms and this is best done by revisiting the roots of the semaphore. Back in 1965, Edsger Dijkstra, a Dutch computer scientist, introduced the concept of a binary semaphore into modern programming to address possible race conditions in concurrent programs. His very simple idea was to use a pair of function calls to the operating system to indicate entering and leaving a critical region. This was achieved through the acquisition and release of an operating system resource called a semaphore. In his original work, Dijkstra used the notation of P & V, from the Dutch words Prolagen (P), a neologism coming from To try and lower, and Verhogen (V) To raise, To increase. With this model the first task arriving at the P(S) [where S is the semaphore] call gains access to the critical region. If a context switch happens while that task is in the critical region, and another task also calls on P(S), then that second task (and any subsequent tasks) will be blocked from entering the critical region by being put in a waiting state by the operating system. At a later point the first task is rescheduled and calls V(S) to indicate it has left the critical region. The second task will now be allowed access to the critical region.

Upload: varungupta

Post on 27-Oct-2014

277 views

Category:

Documents


4 download

TRANSCRIPT

Page 1: Mutex vs Semaphores

Mutex vs. Semaphores – Part 1: Semaphores

It never ceases to amaze me how often I see postings in newsgroups, etc. asking the difference

between a semaphore and a mutex. Probably what baffles me more is that over 90% of the time the

responses given are either incorrect or missing the key differences. The most often quoted response is

that of the “The Toilet Example (c) Copyright 2005, Niclas Winquist” . This summarises the differences

as:

A mutex is really a semaphore with value 1

No, no and no again. Unfortunately this kind of talk leads to all sorts of confusion and

misunderstanding  (not to mention companies like Wind River Systems redefining a mutex as a

“Mutual-Exclusion Semaphore” – now where is that wall to bang my head against?).

Firstly we need to clarify some terms and this is best done by revisiting the roots of the semaphore.

Back in 1965, Edsger Dijkstra, a Dutch computer scientist, introduced the concept of a binary

semaphore into modern programming to address possible race conditions in concurrent programs. His

very simple idea was to use a pair of function calls to the operating system to indicate entering and

leaving a critical region. This was achieved through the acquisition and release of an operating system

resource called a semaphore. In his original work, Dijkstra used the notation of P & V, from the Dutch

words Prolagen (P), a neologism coming from To try and lower, and Verhogen (V) To raise, To increase.

With this model the first task arriving at the P(S) [where S is the semaphore] call gains access to the

critical region. If a context switch happens while that task is in the critical region, and another task also

calls on P(S), then that second task (and any subsequent tasks) will be blocked from entering the

critical region by being put in a waiting state by the operating system. At a later point the first task is

rescheduled and calls V(S) to indicate it has left the critical region. The second task will now be

allowed access to the critical region.

A variant of Dijkstra’s semaphore was put forward by another Dutchman, Dr. Carel S. Scholten. In his

proposal the semaphore can have an initial value (or count) greater than one. This enables building

programs where more than one resource is being managed in a given critical region. For example, a

counting semaphore could be used to manage the parking spaces in a robotic parking system. The

initial count would be set to the initial free parking places. Each time a place is used the count is

Page 2: Mutex vs Semaphores

decremented. If the count reaches zero then the next task trying to acquire the semaphore would be

blocked (i.e. it must wait until a parking space is available). Upon releasing the semaphore (A car

leaving the parking system) the count is incremented by one.

Scholten’s semaphore is referred to as the General or Counting Semaphore, Dijkstra’s being known

as the Binary Semaphore.

Pretty much all modern Real-Time Operating Systems (RTOS) support the semaphore. For the majority,

the actual implementation is based around the counting semaphore concept. Programmers using

these RTOSs may use an initial count of 1 (one) to approximate to the binary semaphore. One of the

most notable exceptions is probably the leading commercial RTOS VxWorks from Wind River Systems.

This has two separate APIs for semaphore creation, one for the Binary semaphore (semBCreate) and

another for the Counting semaphore (semCCreate).

Hopefully we now have a clear understanding of the difference between the binary semaphore and the

counting semaphore. Before moving onto the mutex we need to understand the inherent dangers

associated with using the semaphore. These include:

Accidental release

Recursive deadlock

Task-Death deadlock

Priority inversion

Semaphore as a signal

All these problems occur at run-time and can be very difficult to reproduce; making technical support

very difficult.

Accidental release

This problem arises mainly due to a bug fix, product enhancement or cut-and-paste mistake. In this

case, through a simple programming mistake, a semaphore isn’t correctly acquired but is then

released.

When the counting semaphore is being used as a binary semaphore (initial count of 1 – the most

common case) this then allows two tasks into the critical region. Each time the buggy code is executed

Page 3: Mutex vs Semaphores

the count is increment and yet another task can enter. This is an inherent weakness of using the

counting semaphore as a binary semaphore.

Deadlock

Deadlock occurs when tasks are blocked waiting on some condition that can never become true, e.g.

waiting to acquire a semaphore that never becomes free. There are three possible deadlock situations

associated with the semaphore:

Recursive Deadlock

Deadlock through Death

Cyclic Deadlock (Deadly Embrace)

Here we shall address the first two, but shall return to the cyclic deadlock in a later posting.

Recursive Deadlock

Recursive deadlock can occur if a task tries to lock a semaphore it has already locked. This can

typically occur in libraries or recursive functions; for example, the simple locking of malloc being called

twice within the framework of a library. An example of this appeared in the MySQL database bug

reporting system: Bug #24745 InnoDB semaphore wait timeout/crash – deadlock waiting for itself

Deadlock through Task Death

What if a task that is holding a semaphore dies or is terminated? If you can’t detect this condition then

all tasks waiting (or may wait in the future) will never acquire the semaphore and deadlock. To

partially address this, it is common for the function call that acquires the semaphore to specify an

optional timeout value.

Priority Inversion

The majority of RTOSs use a priority-driven pre-emptive scheduling scheme. In this scheme each task

has its own assigned priority. The pre-emptive scheme ensures that a higher priority task will force a

lower priority task to release the processor so it can run. This is a core concept to building real-time

systems using an RTOS. Priority inversion is the case where a high priority task becomes blocked for

an indefinite period by a low priority task. As an example:

An embedded system contains an “information bus”

Sequential access to the bus is protected with a semaphore.

Page 4: Mutex vs Semaphores

A bus management task runs frequently with a high priority to move certain kinds of data in

and out of the information bus.

A meteorological data gathering task runs as an infrequent, low priority task, using the

information bus to publish its data. When publishing its data, it acquires the semaphore, writes

to the bus, and release the semaphore.

The system also contains a communications task which runs with medium priority.

Very infrequently it is possible for an interrupt to occur that causes the (medium priority)

communications task to be scheduled while the (high priority) information bus task is blocked

waiting for the (low priority) meteorological data task.

In this case, the long-running communications task, having higher priority than the

meteorological task, prevents it from running, consequently preventing the blocked

information bus task from running.

After some time has passed, a watchdog timer goes off, notices that the data bus task has

not been executed for some time, concludes that something has gone drastically wrong, and

initiates a total system reset.

This well reported event actual sequence of events happened on NASA JPL’s Mars Pathfinder

spacecraft.

Semaphore as a Signal

Unfortunately, the term synchronization is often misused in the context of mutual exclusion.

Synchronization is, by definition “To occur at the same time; be simultaneous”. Synchronization

between tasks is where, typically, one task waits to be notified by another task before it can continue

execution (unilateral rendezvous). A variant of this is either task may wait, called the bidirectional

rendezvous. This is quite different to mutual exclusion, which is a protection mechanism. However, this

misuse has arisen as the counting semaphore can be used for unidirectional synchronization. For this

to work, the semaphore is created with a count of 0 (zero).

Note that the P and V calls are not used as a pair in the same task. In the example, assuming Task1

calls the P(S) it will block. When Task 2 later calls the V(S) then the unilateral synchronization takes

place and both task are ready to run (with the higher priority task actually running). Unfortunately

“misusing” the semaphore as synchronization primitive can be problematic in that it makes debugging

harder and increase the potential to miss “accidental release” type problems, as a V(S) on its own (i.e.

not paired with a P(S)) is now considered legal code.

Page 5: Mutex vs Semaphores

In the next posting I shall look at how the mutex address most of the weaknesses of the semaphore.

14 Comments a “Mutex vs. Semaphores – Part 1: Semaphores”

1. Michael Barr says: September 8th, 2009 at 6:38 pm

Niall,

Welcome to the blogosphere! As I would expect from you, you've done a nice job above explaining some rather complex (and often misunderstood) issues.

However, I have a problem with one of the examples. The robotic parking garage implementation suffers from implementation by a counting semaphore. Only a mutex is truly needed. Here's why…

Each of the parking spaces is an individually identifiable object. In a computer, it is analogous to a fixed-sized memory buffer. If the buffer contains data, some part of the application code has a pointer to it. If the buffer is empty, it must be tracked in a "free list" data structure. One suitable data structure is a linked list maintained within the empty buffers plus a head pointer. (But the important general point is there is always at least one piece of metadata.)

The free list data structure must be protected via a mutual exclusion primitive (preferably a mutex). Gaining access to that data structure will tell the caller if there are any free parking spots.

The suggested use of a counting semaphore neither (a) gets you deep enough into the implementation to pick a specific parking spot nor (b) is an alternative to the mutex, which must always be used. Thus the counting semaphore is a waste of space in the solution to that problem.

Generalizing, I recommend avoiding use of the term "counting semaphore" altogether. Only mutexes and semaphores are of practical use, for data protection and signaling respectively.

I've blogged a bunch on mutexes and semaphores at http://www.embeddedgurus.net/barr-code and, on the points you're discussing, highly recommend the article Mutexes and Semaphores Demystified at http://www.netrino.com/Embedded-Systems/How-To/RTOS-Mutex-Semaphore

Cheers,Michael

2. Rennie Allen says: September 8th, 2009 at 9:39 pm

While I think I agree on where you are going with this, the argument is difficult to grasp because you seem to be comparing an implementation with an abstract functional concept.

Page 6: Mutex vs Semaphores

From the functional concept perspective, a conceptual mutex (rather than a specific implementation such as pthreads) could be considered to be a semaphore with a count of one. Because the generic concept of the function of a mutex is ill-defined, I am convinced that I could write a set of cover functions for a mutex, implemented in terms of a semaphore with a count of one; and that you'd be hard pressed to show me why that doesn't provide all the attributes of a conceptual mutex function (i.e. conceptually, they are the same).

Certainly, if we consider this question within the constraints of the SysV behavioral model of the semaphore and the pthreads behavioral model of a mutex, then I think we can agree that there are many behavioral details that make a SysV semaphore with a count of 1, and a pthread mutex significantly different.

I think it is important to make clear though, that you are talking about SysV semaphores and pthread mutexes (you are, aren't you?) rather than the conceptual model of semaphores and mutexes.

3. Dan says: September 8th, 2009 at 11:01 pm

Michael Barr (former editor of Embedded Systems Programming, now president of Netrino) has a good article about the differences between mutexes & semaphores at the following location:

http://www.netrino.com/node/202

Also another article discusses the "Perils of Preemption" (including issues you covered such as deadlock & priority inversion) at this link:

http://www.netrino.com/Embedded-Systems/How-To/Preemption-Perils

Keep up the good work. More people need to learn about such topics.

4. Bill Dittmann says: September 9th, 2009 at 3:16 pm

Niall,Good article. In the RTXC Quadros RTOS mutex implementation, nesting locks on the same mutex are allowed, supported, and safe.

We do 2 things to make it safe. First, we remember who locked (owns) the mutex and keep a counter as it is locked and locked. The counter scheme properly tracks the scope of the mutex lock.

We also make sure that only the owner can release (unlock) the mutex.

Page 7: Mutex vs Semaphores

In my experiences, during development phases, it is not uncommon for a task to inadvertently release when it is doesn’t even own the lock. RTXC Quadros detects this lock underflow or release by non-owner condition, e.g., a program design flaw, during runtime.

5. Solti says: February 1st, 2010 at 3:11 am

Hi Niall,

Can I say when we are dealing with mutual exclusion problems; a binary semaphore is the same to a mutex?

Suppose we use them correctly, and there is a resource which allows only one user at every time instant. There must be a get/return pair in 'user' if we use mutex, and there must also be a wait/signal pair if we use binary semaphore. So in this case, can I say they are totally the same?

Thank you very much, and I really appreciate your articles.

Solti

6. Shreshtha says: May 20th, 2010 at 7:18 am

Hi All,With such a great quality of information and discussion from industry’s eminent people, this page should come in top every time mutex semaphore is searched. Wonderful discussion!

From Linux perspective here is my understanding about difference of Mutex and Sem – (please correct me if I am wrong anywhere)

Note – I find it’s very important that we are clear about the part of Linux we are discussing this topic – Kernel Space or User Space.

i) Scope – The scope of mutex is within a process address space which has created it and is used for synchronization of common resource access. Whereas semaphore can be used across processes space and hence it can be used for interprocess synchronization/signaling. Hence mutex must be released by same thread which is taking it.

ii) Mutex is lightweight and faster than semaphore

(only valid for userspace)iii) Mutex can be acquired by same thread successfully multiple times with condition that it should

Page 8: Mutex vs Semaphores

release it same number of times. Other thread trying to acquire will block. Whereas in case of semaphore if same process tries to acquire it again it blocks as it can be acquired only once.

I found this tread very informative in this context – http://alinux.tv/Kernel-2.6.29/mutex-design.txt

Cheers,@Shreshtha19

7. Shreshtha says: July 12th, 2010 at 8:06 am

above link is not working. here is working one –http://www.kernel.org/doc/Documentation/mutex-design.txt

Mutex vs. Semaphores – Part 2: The Mutex

In Part 1 of this series we looked at the history of the binary and counting semaphore, and then went

on to discuss some of the associated problem areas. In this posting I aim to show how a different RTOS

construct, the mutex, may overcome some, if not all, of these weaknesses.

To address the problems associated with semaphore, a new concept was developed during the late

1980’s. I have struggled to find its first clear definition, but the major use of the term mutex (another

neologism based around MUTual EXclusion) appears to have been driven through the development of

the common programming specification for UNIX based systems. In 1990 this was formalised by the

IEEE as standard IEEE Std 1003.1 commonly known as POSIX.

The mutex is similar to the principles of the binary semaphore with one significant difference: the

principle of ownership. Ownership is the simple concept that when a task locks (acquires) a mutex

only it can unlock (release) it. If a task tries to unlock a mutex it hasn’t locked (thus doesn’t own) then

an error condition is encountered and, most importantly, the mutex is not unlocked. If the mutual

exclusion object doesn’t have ownership then, irrelevant of what it is called, it is not a mutex.

Page 9: Mutex vs Semaphores

The concept of ownership enables mutex implementations to address the problems discussed in part

1:

1. Accidental release

2. Recursive deadlock

3. Task-Death deadlock

4. Priority inversion

5. Semaphore as a signal

Accidental Release

As already stated, ownership stops accidental release of a mutex as a check is made on the release

and an error is raised if current task is not owner.

Recursive Deadlock

Due to ownership, a mutex can support relocking of the same mutex by the owning task as long as it is

released the same number of times.

Priority Inversion

With ownership this problem can be addressed using one of the following priority inheritance

protocols:

[Basic] Priority Inheritance Protocol

Priority Ceiling Protocol

The Basic Priority Inheritance Protocol enables a low-priority task to inherit a higher-priorities task’s

priority if this higher-priority task becomes blocked waiting on a mutex currently owned by the low-

priority task. The low priority task can now run and unlock the mutex – at this point it is returned back

to its original priority.

The details of the Priority Inheritance Protocol and Priority Ceiling Protocol (a slight variant) will be

covered in part 3 of this series.

Death Detection

If a task terminates for any reason; the RTOS can detect if that task current owns a mutex and signal

waiting tasks of this condition. In terms of what happens to the waiting tasks, there are various

models, but two dominate:

All tasks readied with error condition;

Only one task readied; this task is responsible for ensuring integrity of critical region.

Page 10: Mutex vs Semaphores

When all tasks are readied, these tasks must then assume critical region is in an undefined state. In

this model no task currently has ownership of the mutex. The mutex is in an undefined state (and

cannot be locked) and must be reinitialized.

When only one task is readied, ownership of the mutex is passed from the terminated task to the

readied task. This task is now responsible for ensuring integrity of critical region, and can unlock the

mutex as normal.

Mutual Exclusion / Synchronization

Due to ownership a mutex cannot be used for synchronization due to lock/unlock pairing. This makes

the code cleaner by not confusing the issues of mutual exclusion with synchronization.

Caveat

A specific Operating Systems mutex implementation may or may not support the following:

Recursion

Priority Inheritance

Death Detection

Review of some APIs

It should be noted that many Real-Time Operating Systems (or more correctly Real-Time Kernels) do

not support the concept of the mutex, only supporting the Counting Semaphore (e.g. MicroC/OS-II).

[CORRECTION: The later versions of uC/OS-II do support the mutex, only the original version did not].

In this section we shall briefly examine three different implementations. I have chosen these as they

represent the broad spectrum of APIs offered (Footnote 1):

VxWorks Version 5.4

POSIX Threads (pThreads) – IEEE Std 1003.1, 2004 Edition

Microsoft Windows Win32 – Not .NET

VxWorks from Wind River Systems is among the leading commercial Real-Time Operating System used

in embedded systems today. POSIX Threads is a widely supported standard, but has become more

widely used due to the growth of the use of Embedded Linux. Finally Microsoft Window’s common

programming API, Win32 is examined. Windows CE, targeted at embedded development, supports this

API.

However, before addressing the APIs in detail we need to introduce the concept of a Release Order

Policy. In Dijkstra’s original work the concept of task priorities was not part of the problem domain.

Therefore it was assumed that if more than one task was waiting on a held semaphore, when released

Page 11: Mutex vs Semaphores

the next task to acquire the semaphore would be chosen on a First-Come-First-Serve (First-In-First-Out;

FIFO) policy. However once tasks have priorities, the policy may be:

FIFO            – waiting tasks ordered by arrival time

Priority        – waiting tasks ordered by priority

Undefined    - implementation doesn’t specify

VxWorks v5.4

VxWorks supports the Binary Semaphore, the Counting Semaphore and the Mutex (called the Mutual-

Exclusion Semaphore in VxWorks terminology). They all support a common API for acquiring (semTake)

and releasing (semGive) the particular semaphore. For all semaphore types, waiting tasks can be

queued by priority or FIFO and can have a timeout specified.

The Binary Semaphore has, as expected, no support for recursion or inheritance and the taker and

giver do not have to be same task. Some additional points of interest are  that there is no effect of

releasing the semaphore again; It can be used as a signal (thus can be created empty); and supports

the idea of a broadcast release (wake up all waiting tasks rather than just the first). The Counting

Semaphore, as expected, is the same as the Binary Semaphore with ability to define an initial count.

The Mutual-Exclusion Semaphore is the VxWorks mutex. Only the owning task may successfully call

semGive. The VxWorks mutex also has the ability to support both priority inheritance (basic priority

inheritance protocol) and deletion safety.

POSIX 

POSIX is an acronym for Portable Operating System Interface (the X has no meaning). The current

POSIX standard is formally defined by IEEE Std 1003.1, 2004 Edition. The mutex is part of the core

POSIX Threads (pThreads) specification (historically referred to as IEEE Std 1003.1c-1995).

POSIX also supports both semaphores and priority-inheritance mutexes as part of what are called

Feature Groups. Support for these Feature Groups is optional, but when an implementation claims that

a feature is provided, all of its constituent parts must be provided

and must comply with this specification. There are two main Feature Groups of interest, the Realtime

Group and Realtime Threads Groups.

The semaphore is not part of the core standard but is supported as part of the Realtime Feature Group.

The Realtime Semaphore is an implementation of the Counting semaphore.

The default POSIX mutex is non-recursive , has no priority inheritance support or death detection.

However, the Pthreads standard allows for non-portable extensions (as long as they are tagged with “-

np”).  A high proportion of programmers using POSIX threads are programming for Linux. Linux

supports four different mutex types through non-portable extensions:

Page 12: Mutex vs Semaphores

Fast mutex                  – non-recursive and will deadlock [default]

Error checking mutex – non-recursive but will report error

Recursive mutex        – as the name implies

Adaptive mutex         - extra fast for mutli-processor systems

These are extremely well covered by Chris Simmonds in his posting Mutex mutandis: understanding

mutex types and attributes.

Finally the Realtime Threads Feature Group adds mutex support for both priority inheritance and

priority ceiling protocols.

Win32 API

Microsoft Window’s common API is referred to as Win32. This API supports three different primitives:

Semaphore            – The counting semaphore

Critical Section     - Mutex between threads in the same process; Recursive, no timeout,

queuing order undefined

Mutex                    – As per critical sections, but can be used by threads in different processes;

Recursive, timeout, queuing order undefined

The XP/Win32 mutex API does not support priority inheritance in application code, however the

WinCE/Win32 API does!

Win32 mutexes do have built-in death detection; if a thread terminates when holding a mutex, then

that mutex is said to be abandoned. The mutex is released (with WAIT_ABANDONED error code) and a

waiting thread will take ownership. Note that Critical sections do not have any form of death detection.

Critical Sections have no timeout ability, whereas mutexes do. However Critical Sections support a

separate function call TryEnterCriticalSection. A major weakness of the Win32 API is that the queuing

model is undefined (i.e. neither Priority nor FIFO). According to Microsoft this is done to improve

performance.

So, what can we gather from this? First and foremost the term mutex is less well defined than the

semaphore. Secondly,the actual implementations from RTOS to RTOS vary massively. I urge you to go

back and look at your faviourite RTOS and work out what support, if any, you have for the mutex. I’d

love to hear from people regarding mutual exclusion support (both semaphores and mutexes) for their

RTOS of choice. If you’d like to contact me do so at nsc(at)acm.org.

Page 13: Mutex vs Semaphores

Finally, Part 3 will look at a couple of problems the mutex doesn’t solve, and how these can be

overcome. As part of that it will review the Basic Priority Inheritance Protcol and the Prority Ceiling

Protocol.

At a later date I will also address the use of, and problems associted with, the semaphore being used

for task synchronisation.

ENDNOTES

1. Please I do not want to get into the “that’s not a real-time OS” debate here – let’s save that for

another day!

2. A number of people pointed out that Michael Barr (former editor of Embedded Systems

Programming, now president of Netrino) has a good article about the differences between

mutexes & semaphores at the following location: http://www.netrino.com/node/202. I urge you

to read his posting as well.

3. Apologies about not having the atom feed sorted – this should all be working now

6 Comments a “Mutex vs. Semaphores – Part 2: The Mutex”

1. Randell J says: September 19th, 2009 at 6:54 am

Back In The Day…

Semaphores in the Amiga Exec (which was modeled on Xinu) were by your/modern terminology Mutexes with support for recursion. You can think of them (and recursive Mutexes in general) as Counting Semaphores with the counts going in the opposite direction: acquiring a "Semaphore" increases the count (if unowned (count == 0) or if you already own it).

On top of that, we implemented "Read/Write Semaphores", where any number of people could acquire the "semaphore" for reading protected data, but only one could for writing to it. This was effectively the equivalent of a "read-permission" Mutex and a Write-permission mutex (though higher performance than a decomposed implementation).

Roughly (pseudo-code):–acquire(read):—-if (I own a lock) // if recursion is allowed——count++ // note: others may also own it—-else if (writer_count != 0)——add self to queue—-else——count++ //increase count of (read) owners

Page 14: Mutex vs Semaphores

–acquire(write):—-if (I own a lock)——count++——writer_count++—-else if (count != 0)——add self to queue——writer_count++—-else——count++——writer_count++ // makes sure readers won't read while we're writing

The operations for release are obvious (make sure writer_count is dealt with correctly). It can be tricky to implement recursion with this; you can't just store a handle of the 'owner' since you have multiple owners.

You can extend this with queuing mods, like "a write request jumps ahead of all read requests" (note that you can only have a queue if a writer either owns the resource or is in the queue also, waiting for all the readers to release). I think we had that as an option (and I think I used an internal implementation of that in the filesystem code, which was all coded as coroutines – now there's a paradigm that virtually disappeared, at least formally).

2. Sergey Oboguev says: November 9th, 2011 at 10:05 pm

The term was not used at user level until PPL (see below), but mutex primitive is used very extensively in VMS kernel. See “VMS Internals and Data Structures”, any edition (still easily findable at amazon) or VMS documentation on driver writing (there are PDFs easily findable online) and/or System Dump Analyzer documentation or, better yet, kernel sources/listings (a little harder to get,

but obtainable for motivated people   ).

The relevant functions are SCH$LOCKR, SCH$LOCKW, SCH$UNLOCK, SCH$LOCKRNOWAIT, SCH$LOCKWNOWAIT, SCH$LOCKWEXEC, SCH$LOCKREXEC located in module [SYS.SRC]MUTEX.MAR. Many mutex objects throughout the kernel also bear a word “mutex” in their name, for example I/O database mutex IOC$GL_MUTEX and so on, there are dozens of various mutexes in the system.

In terms of functionality VMS kernel mutexes are actually closer to modern RWLocks since they allow both exclusive locking (“writer lock”) and shared locking (“reader lock”). They also elevate process IPL to ASTDEL (2), to prevent process deletion while a mutex is held — the process can still be preempted, but control flow cannot be interrupted by ASTs and, in particular, cannot be deleted while any kernel mutex is held.

At user level, principal VMS mechanism for exclusion were ENQ/DEQ locks, which had a richer semantics than most existing locking primitives, and in addition were cluster-wide, but of course had greater overhead than grandma’s futex.

Besides VMS kernel mutexes, mutexes (of different design than kernel ones) were also used in user space, internally in run-time libraries for ADA and at some point CRTL, and eventually were released

Page 15: Mutex vs Semaphores

for developers use as a part of Parallel Processing Library which, according to the headers in the sources, dates back to 1986.

3. Sergey Oboguev says: November 17th, 2011 at 5:49 am

On a somewhat related curious matter, one of the big “discoveries” in the field of synchronization was 1990 discovery that “test-and-set” hogs the interconnect bus with intense traffic and that “test-and-test-and-set” performs much better, since it spins on the cached value and does not issue interlocked instruction until the cache coherence indicates the lock may be available.

The most quoted article on the issue nowadays is Anderson’s 1990 article “The Performance of Spin Lock Alternatives for Shared—Memory Multiprocessors”, though it was actually preceded by other articles on the matter published in 1990.

I am sure there are thousands of references to Anderson’s article, indeed google alone finds 16600 references on the web and 800 references in google books. Web also contains 236.000 references to “test and test and set” idea allegedly pioneered by the Anderson’s article.

Yet, VMS SMP spinlock code (module [SYS.SRC]SPINLOCK.MAR) written 5 years earlier, in 1985, does just that — spins on local cached value, and performs interlocked operation only after the cache is updated, with the code section prepended by the comment: “The busy wait loop below assumes that a cache is present and that cache coherency is maintained on all available processors. Note that if the cache is not present and working, the busy wait is likely to impact the system performance overall by making many references to memory.”

I would expect many other examples like this could be found, and thus invention and use of mutex in the industry many years before it made a splash in academic publications is by no means anything too unique.

4. Sergey Oboguev says: November 17th, 2011 at 6:14 am

Back to the origins of the term “mutex”, lookup in google books suggests that the term originated in 1967, originally as a designation for binary semaphore (indeed, the very first use findable in google books appears to be as a name of semaphore variable) and then gradually acquiring a separate meaning.

The word was introduced apparently not without some resistance to the reader’s ears. One reviewer of J.L. Jolley’s “Data Study” (published in 1968) writing in “New Scientist” (1968) complains:

“The basic idea is a simple one and many will be irritated by the author’s use of such words as mutex, ambisubterm, homeostasis and idempotency to provide names for many quite ordinary phenomena”.

Page 16: Mutex vs Semaphores

Mutex vs. Semaphores – Part 3 (final part): Mutual Exclusion Problems

As hopefully you can see from the previous posting, the mutex is a significantly safer mechanism to

use for implementing mutual exclusion around shared resources. Nevertheless, there are still a couple

of problems that use of the mutex (in preference to the semaphore) will not solve. These are:

Circular deadlock

Non-cooperation

Circular Deadlock

Circular deadlock, often referred to as the “deadly embrace” problem is a condition where two or more

tasks develop a circular dependency of mutual exclusion. Simply put, one task is blocked waiting on a

mutex owned by another task. That other task is also block waiting on a mutex held by the first task.

So how can this happen? Take as an example a small control system. The system is made up of three

tasks, a low priority Control task, a medium priority System Identification (SI) task and a high priority

Page 17: Mutex vs Semaphores

Alarm task. There is an analogue input shared by the Control and the SI tasks, which is protected by a

mutex. There is also an analogue output protected by a different mutex.

The Control task waits for mutexes ADC and DAC:

mutex_lock (ADC);

mutex_lock (DAC);

/* critical section */

mutex_unlock (ADC);

mutex_unlock (DAC);

The SI Task waits for mutexes DAC and ADC:

mutex_lock (DAC); 

mutex_lock (ADC); 

/* critical section */ 

mutex_unlock (DAC); 

mutex_unlock (ADC);

Unfortunately, under certain timing conditions, this can lead to deadlock. In this example the Control

task has locked the ADC, but before locking the DAC has been pre-empted by the higher priory SI task.

The SI task then locks the DAC and tries to lock the ADC. The SI task is now blocked as the ADC is

already owned by the Control task. The Control task now runs and tries to lock the DAC. It is blocked as

the DAC is held by the SI task. Neither task can continue until the mutex is unlocked and neither

mutex can be unlocked until either task runs – classic deadlock.

Page 18: Mutex vs Semaphores

For circular deadlock to occur the following conditions must all be true:

A thread has exclusive use of resources (Mutual exclusion)

A thread can hold on to a resource(s) whilst waiting for another resource (Hold and wait)

A circular dependency of thread and resources is set up (Circular waiting)

A thread never releases a resource until it is completely finished with it (No resource

preemption)

These conditions can be addressed in a number of ways. For example, a design policy may stipulate

that if a task needs to lock more than one mutex it must either lock all or none.

Priority Ceiling Protocol

With the Priority Ceiling Protocol (PCP) method each mutex has a defined priority ceiling, set to that of

the highest priority task which uses the mutex. Any task using a mutex executes at its own priority –

until a second task attempts to acquire the mutex.  At this point it has its priority raised to the ceiling

value, preventing suspension and thus eliminating the “hold and wait” condition.

In the deadlock example shown before, the significant point is when the SI task tries to lock the DAC.

Before that succeeded and lead to cyclic deadlock. However with a PCP mutex, both the ADC and DAC

mutex will have a ceiling priority equal to the SI’s task priority. When the SI task tries to lock the DAC,

then the run-time system will detect that the SI’s task priority is not higher than the priority of the

locked mutex ADC. The run-time system suspends the SI task without locking the DAC mutex. The

control task now inherits the priority of the SI task and resumes execution.

Page 19: Mutex vs Semaphores

Non-cooperation

The last, but most important aspect of mutual exclusion covered in these ramblings relies on one

founding principle: we have to rely on all tasks to access critical regions using mutual exclusion

primitives. Unfortunately this is dependent on the design of the software and cannot be detected by

the run-time system. This final problem was addressed by Tony Hoare, called the Monitor.

The Monitor

The monitor is a mechanism  not typically supplied by the RTOS, but something the programmer tends

to build (a notable exception is Ada95′s protected object mechanism). A monitor simply encapsulates

the shared resource and the locking mechanism into a single construct (e.g. a C++ Object that

encapsulates the mutex mechanism). Access to the shared resource, then, is through a controlled

interface which cannot be bypassed (i.e. the application never explicitly calls the mutex, but calls upon

access functions).

Finishing Off…

This goal of these initial postings is to demonstrate that common terms used in the real-time

programming community are open to ambiguity and interpretation. Hopefully you should now be clear

about the core differences between the Binary Semaphore, General (counting) Semaphore and the

Mutex.

The underlying difference between the Semaphores and the Mutex is the Principle of Ownership.

Given the principle of ownership a particular implementation of a mutex may

support Recursion, Priority inheritanceand Death Detection.

ENDNOTE 

An aspect of the mutex I haven’t covered here is that many operating systems support the concept of

acondition variable. A condition variable allows a task to wait on a synchronization primitive within a

Page 20: Mutex vs Semaphores

critical region. The whole aspect Synchronization Patterns (e.g. semaphore as a signal) within the

context of RTOSs will be the subject of my next posting.

Posted on October 5th, 2009» Feed to this thread

» Trackback

1 Comments a “Mutex vs. Semaphores – Part 3 (final part): Mutual Exclusion Problems”

1. Pete says: November 11th, 2009 at 3:51 pm

Your description of the Priority Ceiling Protocol is incorrect. What you describe is more similar to PCP *Emulation* (also called the Priority Protect Protocol in POSIX).

Real PCP doesn't raise the task's priority to the blocked task's priority (not the ceiling). It uses the ceiling to block a task on a wait/lock operation of a mutex if the locking task has a lower priority than the current priority ceiling (highest ceiling of any currently locked mutex), even if the mutex was available.

It's complex, which is why the version you state is more commonly seen.

Page 21: Mutex vs Semaphores

Mutex mutandis: understanding mutex types and attributes

The mutex is a simple little thing with one job to do: ensure mutual exclusion between threads. Yet

Linux has up to 16 variations of the mutex, depending on which versions of the kernel and C library

you have (see the table at the end). In this article I will try to explain why there are so many and how

they affect your programs.

Basics

A mutex has two states: locked and unlocked. Once it has been locked by one thread any others are

forced to wait until the first unlocks it again. Only the thread that locked the mutex may unlock it. With

these three facts you can use a mutex to ensure exclusive access to a shared resource, such as a data

structure. In the example below I have included the mutex in the structure being protected, which is

good design practice

struct some_data {

pthread_mutex_t data_lock;

// other data items

} my_data;

...

pthread_mutex_lock (&my_data.data_lock);

// write some values

pthread_mutex_unlock (&my_data.data_lock);

Only one thread can hold data_lock at a time, ensuring that the data values are always consistent.

Complications

Mutexes are so important to the correct behavior of an application that small details about their

implementation can make a big difference overall. Here are some things to consider

what is most important: speed or correct behavior?

what happens if you try to lock the same mutex twice?

if several threads are waiting for a mutex to be unlocked, which one should get the mutex next?

is it acceptable for a high priority thread to be blocked indefinitely by a lower priority thread (leading

to priority inversion)?

what happens if the thread that has locked the mutex terminates without unlocking?

Your response to these questions will determine the sort of mutex you need.

Page 22: Mutex vs Semaphores

Types of mutex: fast, error checking, recursive and adaptive

Linux has four types of mutex. The code snippets below show how to declare and initialise the default

type, which is fast. Sounds good, but what does that mean? It means that speed is preferred over

correctness: there is no check that you are the owner in pthread_mutex_unlock() so any thread can

unlock a fast mutex. Also it doesn't check if you have already locked the mutex, so you can deadlock

yourself, and there are no checks anywhere that the mutex has been intialised correctly.

You declare a mutex of default (fast) type at run-time like this

pthread_mutex_t mutex;

...

pthread_mutex_init (&mutex, NULL);

or statically at compile-time like this

pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;

If you prefer correctness over speed, you need to set the type to be error checking. Error checking

mutexes return EDEADLK if you try to lock the same one twice and EPERM if you unlock a mutex that

isn't yours. To create such a mutex you need to initialise a muter_attr and pass it to

pthread_mutex_init() like so:

pthread_mutex_t mutex;

pthread_mutexattr_t attr;

pthread_mutexattr_init (&attr);

pthread_mutexattr_settype (&attr, PTHREAD_MUTEX_ERRORCHECK_NP);

pthread_mutex_init (&mutex, &attr);

or, you can do it statically at compile-time in one line like this:

pthread_mutex_t mutex = PTHREAD_ERRORCHECK_MUTEX_INITIALIZER_NP;

Note: the _NP suffix indicates a non-portable extension to the POSIX specification. In fact the latest

specification, 1003.1-2008 [2], includes most of the Linux additions so you can leave the _NP ending

off. I have chosen not to because some older versions of the header files only have the _NP variants.

Next is the recursive mutex, which does everything that the error checking mutex does except that

you can lock the same mutex multiple times. It keeps a count of the number of times it has been

locked and you must unlock it the same number of times before it becomes truly unlocked. As with the

other types, you can declare and initialise one like this:

Page 23: Mutex vs Semaphores

pthread_mutex_t mutex;

pthread_mutexattr_t attr;

pthread_mutexattr_init (&attr);

pthread_mutexattr_settype (&attr, PTHREAD_MUTEX_RECURSIVE_NP);

pthread_mutex_init (&mutex, &attr);

or like this:

pthread_mutex_t mutex = PTHREAD_RECURSIVE_MUTEX_INITIALIZER_NP;

The last type is adaptive which is an extra fast mutex for multi processor systems. It combines a

spinlock with an ordinary mutex: instead of blocking straight away on a locked mutex it spins for a

short while re-trying the lock and then blocks in the normal way. On a single processor it doesn't spin

and so is identical to a fast mutex.

Handling errors

There is not much point using error checking mutexes if you ignore the return value! Except that in

reality the main reason that you will get errors from pthread_mutex_lock() and

pthread_mutex_unlock() is because of logic errors in your code. I use this macro to detect errors during

development and then compile it out for production.

#ifdef DEBUG

#define pthread_mutex_lock_check(mutex) \

({ \

int __ret = pthread_mutex_lock (mutex); \

if (__ret != 0) \

printf ("pthread_mutex_lock_check in %s line %u: error %d -

%s\n", \

__FILE__, __LINE__, __ret, strerror (__ret)); \

__ret; \

})

#else

#define pthread_mutex_lock_check pthread_mutex_lock

#endif

Wake-up order

when a mutex is unlocked and there are several threads blocked waiting for it the system has to

decide which gets the mutex next. Until recently the choice was simply the thread that had been

waiting longest, but with Linux kernels from 2.6.22 onwards the thread chosen will be the highest

priority real-time thread. If there are no real-time threads then it will be the longest waiter as before.

Page 24: Mutex vs Semaphores

Sharing a mutex between processes

Most often the shared resource being protected by a mutex is a global variable in a process address

space and so the threads using the mutex are all local to that process. Sometimes you will have data

in a shared memory segment, for example using the POSIX or SYSV IPC shared memory functions, and

so the mutex needs to be locked and unlocked by threads from different processes. In such a case, the

mutex must be initialsed with the shared attribute

pthread_mutex_t mutex;

pthread_mutexattr_t attr;

pthread_mutexattr_init (&attr);

pthread_mutexattr_setpshared (&attr, PTHREAD_PROCESS_SHARED);

pthread_mutex_init (&mutex, &attr);

Apart from being shared the behavior of the mutex is the same as local ones.

Problems with real-time threads: priority inversion

If (and only if) you have threads with real-time scheduling policies SCHED_FIFO or SCHED_RR you may

experience priority inversion[3] which results in a high priority thread that is waiting to lock a mutex

being blocked by a lower priority thread. One way to resolve the problem is to set the priority protocol

of the mutex to priority inheritance, with the result that the thread holding the mutex inherits the

priority of the highest priority thread waiting for the mutex and so it cannot be preempted by

intermediate priority threads. Here is how to do it

#define __USE_UNIX98 /* Needed for PTHREAD_PRIO_INHERIT */

#include <pthread.h>

pthread_mutex_t mutex;

pthread_mutexattr_t attr;

pthread_mutexattr_init (&attr);

pthread_mutexattr_setprotocol (&attr, PTHREAD_PRIO_INHERIT);

pthread_mutex_init (&mutex, &attr);

Priority inheritance can be combined with any of the four types. However, it adds a large overhead to

the implementation and so it does not make sense to combine it with the fast or adaptive types.

Unexpected termination: the robust mutex

Supposing a thread has locked a mutex and then terminates, what then? In the normal run of things

the mutex will remain locked for ever (OK, until the next reboot), causing any threads trying to lock it

Page 25: Mutex vs Semaphores

to deadlock. This is particularly a problem if you are sharing a mutex between processes and one of

them segfaults or is killed.

This is where the robust attribute comes in. The first stage is to initialise the mutex with the robust

option. It can be combined with any of the four types and with the priority inheritance attribute. Here

we go:

#define __USE_GNU /* Needed for PTHREAD_MUTEX_ROBUST_NP */

#include <pthread.h>

pthread_mutex_t mutex;

pthread_mutexattr_t attr;

pthread_mutexattr_init (&attr);

pthread_mutexattr_setrobust_np (&attr, PTHREAD_MUTEX_ROBUST_NP);

pthread_mutex_init (&mutex, &attr);

Now, if the thread owning the mutex terminates with it locked, any other thread that is trying to lock it

will unblock with error code EOWNERDEAD. In other words, this mutex no longer functions as a mutex.

If you want to repair the situation you must validate the data that the mutex was protecting, maybe

remove some inconsistent state, and then call pthread_mutex_consistent_np(). Then you must lock it,

in the same thread that marked it as consistent. Now it is a fully functional mutex again.

Finally, if the mutex is unlocked without being made consistent, it will be in a permanently unusable

state and all attempts to lockit will fail with the error ENOTRECOVERABLE. You have blown it: the only

thing you can do with such a mutex is to destroy it.

All the above adds quite a lot of complexity to the implementation of a mutex, so robust mutexes are

NOT going to be fast.

Summary

We have four types of mutex each of which may be robust and may have the priority inheritance

protocol, which gives us 4 x 2 x 2 = 16 different possibilities. Here are my suggestions on which to use.

During development, use error checking - the extra overhead is very small

Use recursive mutexes if you have library code where you cannot be sure that a mutex has already

been locked elsewhere

In high performance production code, use fast or (if you have more than one CPU) adaptive types

If you have real time threads, look carefully at the dependencies between the threads and use

priority inheritance where necessary

Page 26: Mutex vs Semaphores

If you share mutexes between processes (or if your threads terminate in odd places) use robust

mutexes

Features vs libc version

Some features are dependent on the version of the 'C' library you use. Here I am considering the two

most often used in embedded devices: GNU libc (glibc) and the microcontroller C library, uClibc [4].

Fast, error checking and recursive mutex types are present in all versions of glibc and uClibc

The adaptive type and the robust and priority inheritance protocol are ONLY implemented in glibc 2.5

onwards

Main-line uClibc DOES NOT implement adaptive, robust and priority inheritance (*)

(*) Because uClibc uses the older LinuxThreads library, not the Native POSIX Threads (nptl) library as

glibc does. There is an on-going project to port nptl to uClibc but it is not yet functional on all

architectures.

Features vs kernel version

If you are using glibc (or uClibc with nptl) you also need to keep a watch on the kernel version. Here

are the minimum versions for the various features

Norma

l

Robus

tPriority Inheritance Priority wake-up queue

2.6.0 2.6.17 2.6.18 2.6.22