Reader/Reader blocking in reader/writer locks(blog.nelhage.com)
blog.nelhage.com
Reader/Reader blocking in reader/writer locks
https://blog.nelhage.com/post/rwlock-contention/
7 comments
This is a decent reminder blog post about something that is obvious on the surface, but subtle when you forget to think about it and make a mistake.
I've found that thinking of a lock as something you don't want to hold goes a long way. The point of locks is to release them.
Jeff Preshing has a good series on locks and concurrency. This[0] post (potentially with an extra dozen I read in the same sitting) is the one that really changed how I thought about locks: before I read it, I was very into lock freedom at every opportunity.
0: https://preshing.com/20111118/locks-arent-slow-lock-contenti...
I've found that thinking of a lock as something you don't want to hold goes a long way. The point of locks is to release them.
Jeff Preshing has a good series on locks and concurrency. This[0] post (potentially with an extra dozen I read in the same sitting) is the one that really changed how I thought about locks: before I read it, I was very into lock freedom at every opportunity.
0: https://preshing.com/20111118/locks-arent-slow-lock-contenti...
> So reader/writer locks tend to rely on the additional concurrency, and on readers not blocking other readers.
> software has a constant tendency to get slower, absent deliberate and careful efforts to hold the line, as developers add features and complexity. And if slower processes don’t immediately impact performance (because they only hold read locks), they’re more likely to go unnoticed. Thus, it’s not unlikely that over time read lock durations will creep upwards, mostly without effect, until one happens to coincide with an attempt to grab a write lock.
For me, the takeaway here is to maintain discipline in software regarding locking behavior. It's almost always a bad idea to hold a lock (rw or not) for an extended period of time.
Notably, this is how the Linux kernel developers responded to the case linked in the discussion of mmap_sem - the operation quadratic in number of threads was thrown out to avoid holding the rwsem for extended periods of time.
Also, re "and the lock is held by readers for a significant length of time": I would imagine that Rusty Russell would consider a few microseconds a "significant length of time" and wonder if you were insane if your software held a lock for 60 seconds.
> software has a constant tendency to get slower, absent deliberate and careful efforts to hold the line, as developers add features and complexity. And if slower processes don’t immediately impact performance (because they only hold read locks), they’re more likely to go unnoticed. Thus, it’s not unlikely that over time read lock durations will creep upwards, mostly without effect, until one happens to coincide with an attempt to grab a write lock.
For me, the takeaway here is to maintain discipline in software regarding locking behavior. It's almost always a bad idea to hold a lock (rw or not) for an extended period of time.
Notably, this is how the Linux kernel developers responded to the case linked in the discussion of mmap_sem - the operation quadratic in number of threads was thrown out to avoid holding the rwsem for extended periods of time.
Also, re "and the lock is held by readers for a significant length of time": I would imagine that Rusty Russell would consider a few microseconds a "significant length of time" and wonder if you were insane if your software held a lock for 60 seconds.
In a perfect world "maintaining discipline" is indeed the correct solution, but my experience tells me that humans cannot be relied upon to do that. Some individual humans can, but not the collective whole.
The discipline has to come from elsewhere. Either from the management (reducing the number of people you have to rely on from N to log(N)), or from tooling (reducing the number to log(1)).
As always, education is the silver bullet. Instill proper values, enable the skills to follow through, and in a mere 40-60 years the industry will be in a better place. As long as there is no incentive to mass-produce junior software developers on the cheap, we're golden.
The discipline has to come from elsewhere. Either from the management (reducing the number of people you have to rely on from N to log(N)), or from tooling (reducing the number to log(1)).
As always, education is the silver bullet. Instill proper values, enable the skills to follow through, and in a mere 40-60 years the industry will be in a better place. As long as there is no incentive to mass-produce junior software developers on the cheap, we're golden.
> What to do about it?
Design with RCU, if you can.
> Time out writer acquisitions.
> If potentially starving writers is acceptable, adding a timeout to write-side acquisitions will bound how long readers can be forced to pause. GoCardless’ library for online Rails migrations implements this option for Postgres, and also discusses the challenges and implications a bit.
This is nearly the same as simply not having writer priority. If "potentially starving writers is acceptable", that implies we can consider reader-priority locks, which do exactly that: potentially starve writers.
Under writer priority, we hold back new readers so that the old readers can leave and let the waiting writer in. If we balk when the new readers are blocked for too long, and bump the writer, then it's simply not writer priority any more.
Design with RCU, if you can.
> Time out writer acquisitions.
> If potentially starving writers is acceptable, adding a timeout to write-side acquisitions will bound how long readers can be forced to pause. GoCardless’ library for online Rails migrations implements this option for Postgres, and also discusses the challenges and implications a bit.
This is nearly the same as simply not having writer priority. If "potentially starving writers is acceptable", that implies we can consider reader-priority locks, which do exactly that: potentially starve writers.
Under writer priority, we hold back new readers so that the old readers can leave and let the waiting writer in. If we balk when the new readers are blocked for too long, and bump the writer, then it's simply not writer priority any more.
Please make the second "reader" in the title lowercase, otherwise it's very confusing to parse (looks like three things separated by slashes, instead of a single sentence with two words that have slashes in them).
For the database example, could you change your isolation level for the reading queries?
The problem has been fixed twice: by moving the top-level locking for that table to epoch (FreeBSD RCU equivalent), and also by using per-cpu counters to track the number of TCP connections in every state. So now the user-space collector just needs to fetch a small array of tcp states from the kernel.