Nobody cares about our concurrency control research [pdf](cs.cmu.edu)
cs.cmu.edu
Nobody cares about our concurrency control research [pdf]
https://www.cs.cmu.edu/~pavlo/slides/pavlo-keynote-sigmod2017.pdf
26 comments
> What application and system developers care about in modern architectures is application level correctness
Meme template: You guys care about correctness?
Also, please don't lump together application and system developers. In terms of software quality these are worlds apart. It's a lot more common in system world to care about correctness than it is in the application world. Of course, there are always exceptions, but a typical example of either camp goes about quality control very differently.
In my experience, outside of a very small sector of application development, where correctness helps sales (eg. medicine, or other fields where software mistakes come with serious repercussions) the goal is to make the "good path" work in favorable conditions. It is also permissible to have "excusing circumstances" (a.k.a. documented defects) that render software inoperable.
To contrast this, system developers typically have to deal with systems that are expected to work hell or high water. Well, again, outside of cases when failures are forgivable, like, management software / UI to the system rather than system itself. Same reason how you'd expect library code to be better tested and more robust in the face of edge cases than any individual application using the library.
---
So, as you mentioned "as soon as the application spans multiple stores", application developers will give up all hope because there are too many layers, too many components, too much opaque third-party code. Whereas in systems, this is typically still all under your control. I.e. think about products like Ceph or Etcd -- they are designed to span multiple stores and still be correct.
I had to go back and forth between application development and systems. And, for me, the world of application development is very unappealing specifically because of the attitude to quality. From my key-hole perspective, I wish people made fewer applications and concentrated on making systems more robust, because I feel like this would make computers more reliable, faster and cheaper... but there's a strong financial incentive to make applications, even though they are less reliable, slower and more expensive in the long term, because initial development is faster and cheaper.
Meme template: You guys care about correctness?
Also, please don't lump together application and system developers. In terms of software quality these are worlds apart. It's a lot more common in system world to care about correctness than it is in the application world. Of course, there are always exceptions, but a typical example of either camp goes about quality control very differently.
In my experience, outside of a very small sector of application development, where correctness helps sales (eg. medicine, or other fields where software mistakes come with serious repercussions) the goal is to make the "good path" work in favorable conditions. It is also permissible to have "excusing circumstances" (a.k.a. documented defects) that render software inoperable.
To contrast this, system developers typically have to deal with systems that are expected to work hell or high water. Well, again, outside of cases when failures are forgivable, like, management software / UI to the system rather than system itself. Same reason how you'd expect library code to be better tested and more robust in the face of edge cases than any individual application using the library.
---
So, as you mentioned "as soon as the application spans multiple stores", application developers will give up all hope because there are too many layers, too many components, too much opaque third-party code. Whereas in systems, this is typically still all under your control. I.e. think about products like Ceph or Etcd -- they are designed to span multiple stores and still be correct.
I had to go back and forth between application development and systems. And, for me, the world of application development is very unappealing specifically because of the attitude to quality. From my key-hole perspective, I wish people made fewer applications and concentrated on making systems more robust, because I feel like this would make computers more reliable, faster and cheaper... but there's a strong financial incentive to make applications, even though they are less reliable, slower and more expensive in the long term, because initial development is faster and cheaper.
This resonates with a dry-run of my architecture ( https://www.youtube.com/watch?v=bWYgChA_aYA ). A central thesis is that that "sure, you can have all these infinitely scalable services" but then the composition of all of them is very hard to reason about.
The interesting thing as I play with my platform as I build a product right now under the thesis is that I'm focused almost 100% on the domain modelling problem for the business rather than thinking about architecture, databases, etc. I'm just writing code, and I have a reliable product from the get-go.
Honestly, I think we have been solving the wrong problems collectively for many years. Granted, I'm years away from a platform for the masses, but it is a super power that my partners and I have to go storm various industries to beat the average.
The interesting thing as I play with my platform as I build a product right now under the thesis is that I'm focused almost 100% on the domain modelling problem for the business rather than thinking about architecture, databases, etc. I'm just writing code, and I have a reliable product from the get-go.
Honestly, I think we have been solving the wrong problems collectively for many years. Granted, I'm years away from a platform for the masses, but it is a super power that my partners and I have to go storm various industries to beat the average.
There is a good article [1] discussing this state of affairs where application developers are straight jacked because they end up integrating too many systems and there is no distributed transaction covering them all.
[1] https://queue.acm.org/detail.cfm?id=3025012
[1] https://queue.acm.org/detail.cfm?id=3025012
"Maybe READ COMMITTED is good
enough or maybe people don’t know
how dirty their data actually is…"
I'm going to go with the latter option. And further posit that on the whole most people working as "full stack" devs in our industry don't really get databases. In particular they don't get the relational model and its advantages at a fundamental level and are perfectly willing to paper over it with ORMs, statically structured "model" objects, and break their (logically connected) data up into disjoint microservices that end up doing inefficient server-side software joins by web service call and...
Putting it another way, they want the DB to play second fiddle to frameworks, architectures, and techniques that operate with different approaches (usually OO & SOA) than the logical/relational modelling approach that an RDBMS is best for. And chaos and misuse is the result. That's my usual rant anyways...
And, so yeah, it's also not a surprise that people aren't really "getting" transactions and concurrency. Certainly the wave of "NoSQL" stores (thankfully waning) showed this. People were willing to toss coherence in favour of scalability (often even when they didn't need it or hadn't proved that they did.) But often the "NoSQL" thing was simply a product of a mentality that went something like "I just want a data store" and "my object doesn't fit in a table", not getting what a table really is... And so we got Redis and Mongo, systems that didn't offer any performance advantages really, but looked on the surface like they'd remove complexity ... but ...
And then I'd say that this general lack of enthusiasm around the relational model and ACID means that people haven't demanded the right things from their database tooling.
Ok this rant is going to go on too long ... I just had my morning coffee...
I'm going to go with the latter option. And further posit that on the whole most people working as "full stack" devs in our industry don't really get databases. In particular they don't get the relational model and its advantages at a fundamental level and are perfectly willing to paper over it with ORMs, statically structured "model" objects, and break their (logically connected) data up into disjoint microservices that end up doing inefficient server-side software joins by web service call and...
Putting it another way, they want the DB to play second fiddle to frameworks, architectures, and techniques that operate with different approaches (usually OO & SOA) than the logical/relational modelling approach that an RDBMS is best for. And chaos and misuse is the result. That's my usual rant anyways...
And, so yeah, it's also not a surprise that people aren't really "getting" transactions and concurrency. Certainly the wave of "NoSQL" stores (thankfully waning) showed this. People were willing to toss coherence in favour of scalability (often even when they didn't need it or hadn't proved that they did.) But often the "NoSQL" thing was simply a product of a mentality that went something like "I just want a data store" and "my object doesn't fit in a table", not getting what a table really is... And so we got Redis and Mongo, systems that didn't offer any performance advantages really, but looked on the surface like they'd remove complexity ... but ...
And then I'd say that this general lack of enthusiasm around the relational model and ACID means that people haven't demanded the right things from their database tooling.
Ok this rant is going to go on too long ... I just had my morning coffee...
I also agree with the later option.
I've been using databases for a very long time. For the past 2 decades there has been a drive by framework providers to transparently handle such things and relieve developers of any difficult decisions. Often there isn't an option or it is deeply hidden. I've arrived on jobs to find Spring applications where database transactions have been unwittingly disabled (or not enabled) and no one has realised.
There is also a lack of upfront thinking about types of data and the required consistency. Some items really should be consistent and some don't need to be.
In one of Martin Fowler's earlier books he describes the difference between knowledge and operational data. Knowledge data is almost like configuration - perhaps the description of a currency and number of decimal places it has. Operational data might be an invoice generated using that currency. The description of a currency changes rarely (unlike a currency rate), it could be cached a long time for use by many processes without issue. However, the invoice must be consistent, but only one process will be involved in creating it and nothing else is held up waiting for it, and once it is created - it will never change.
In practice most data items lie somewhere along this spectrum. However, you choose your tool and then everthing is either ACID or NoSQL.
I've been using databases for a very long time. For the past 2 decades there has been a drive by framework providers to transparently handle such things and relieve developers of any difficult decisions. Often there isn't an option or it is deeply hidden. I've arrived on jobs to find Spring applications where database transactions have been unwittingly disabled (or not enabled) and no one has realised.
There is also a lack of upfront thinking about types of data and the required consistency. Some items really should be consistent and some don't need to be.
In one of Martin Fowler's earlier books he describes the difference between knowledge and operational data. Knowledge data is almost like configuration - perhaps the description of a currency and number of decimal places it has. Operational data might be an invoice generated using that currency. The description of a currency changes rarely (unlike a currency rate), it could be cached a long time for use by many processes without issue. However, the invoice must be consistent, but only one process will be involved in creating it and nothing else is held up waiting for it, and once it is created - it will never change.
In practice most data items lie somewhere along this spectrum. However, you choose your tool and then everthing is either ACID or NoSQL.
Well, it really doesn't help that from the top-5 engines on his slides, only Postgres doesn't have a completely horrible performance profile for serializable transactions. So, people that try to learn about it run away quite quickly.
But I'm not sure if most systems need it anyway. Personally, I remember very few times where I had to change from the default transaction model.
As a rule, when people need full serializability, they create it with some other, more flexible mechanism.
But I'm not sure if most systems need it anyway. Personally, I remember very few times where I had to change from the default transaction model.
As a rule, when people need full serializability, they create it with some other, more flexible mechanism.
Perhaps fair, but I think though this ties back to one point I was making:
"And then I'd say that this general lack of enthusiasm around the relational model and ACID means that people haven't demanded the right things from their database tooling."
In general there are plenty of solutions in database research and implementation. But because the broad community of developers doesn't put data first, and have a low expectation and knowledge of what a DBMS can and should do...
It took us years to get to the place where Postgres is common place instead of e.g. MySQL; and back in the day "serious" folks spent tens of thousands on Oracle licenses... I guess that's progress, but there's so much more that can be done in this sphere.
"And then I'd say that this general lack of enthusiasm around the relational model and ACID means that people haven't demanded the right things from their database tooling."
In general there are plenty of solutions in database research and implementation. But because the broad community of developers doesn't put data first, and have a low expectation and knowledge of what a DBMS can and should do...
It took us years to get to the place where Postgres is common place instead of e.g. MySQL; and back in the day "serious" folks spent tens of thousands on Oracle licenses... I guess that's progress, but there's so much more that can be done in this sphere.
You make some ok points but it's not hard to see why Redis and Mongo took off. During their hype cycle, they could cluster/shard more easily than RDBs. You could just throw hardware at a problem.
I think it's an unfortunate history that the boom in popular "high scalability" type applications happened before SSDs became commonplace. Feeling I/O bound and having to "go horizontal" to increase IOPS was something everybody felt they had to do.
I remember reaching for Cassandra to solve a problem for a startup that needed to do what seemed like massive concurrent writes; the workload simply saturated what a single Postgres instance could be expected to do.
But since then, SSD & nVME happened, and, well it's mind boggling.
Just rough numbers.
7200 RPM HD IOPS: ~100 nVME IOPS: ~500,000
Now, one doesn't always have a properly designed storage engine & DB that can take advantage of this huge differential. But what a yawning gap. Solid state storage changed everything.
Honestly, horizontal scaling is often reached for too quickly. For redundancy, I get it (but that's more a replication story.) For performance reasons? For most regular OLTP workloads for most average sized website type applications, I am not convinced these days.
I remember reaching for Cassandra to solve a problem for a startup that needed to do what seemed like massive concurrent writes; the workload simply saturated what a single Postgres instance could be expected to do.
But since then, SSD & nVME happened, and, well it's mind boggling.
Just rough numbers.
7200 RPM HD IOPS: ~100 nVME IOPS: ~500,000
Now, one doesn't always have a properly designed storage engine & DB that can take advantage of this huge differential. But what a yawning gap. Solid state storage changed everything.
Honestly, horizontal scaling is often reached for too quickly. For redundancy, I get it (but that's more a replication story.) For performance reasons? For most regular OLTP workloads for most average sized website type applications, I am not convinced these days.
Would this be a possible alternative?
Careful, deliberate consideration of a handrolled data model, with plenty of prototyping up-front with Excel and the customer, playing out various scenarios.
Careful, deliberate consideration of a handrolled data model, with plenty of prototyping up-front with Excel and the customer, playing out various scenarios.
im so glad I got taught database normal forms in high school because I sure as anything didnt learn it at university.
That, in the long and short, is probably approximately where the problem lies.
That, in the long and short, is probably approximately where the problem lies.
> its advantages at a fundamental level
I used to be a dev and I wonder what you have in mind here ???
For me the two important points with a DB are:
- entities connections (foreign keys), data types and a few other things are specified declaratively (so you don't have to think about enforcing them, the DB does it for you, so you never, ever forget). So I liked to have very strong DB models (no denormalization :-)) as the ultimate line of defense of my data quality. In my business, that was worth millions of euros (and politics).
- transactions: Read commited was good enough in most cases (clients didn't access the same data) adn sometimes serializable was necessary (for example, toi generate ID's unique acorss the database and across all the clients)
I used to be a dev and I wonder what you have in mind here ???
For me the two important points with a DB are:
- entities connections (foreign keys), data types and a few other things are specified declaratively (so you don't have to think about enforcing them, the DB does it for you, so you never, ever forget). So I liked to have very strong DB models (no denormalization :-)) as the ultimate line of defense of my data quality. In my business, that was worth millions of euros (and politics).
- transactions: Read commited was good enough in most cases (clients didn't access the same data) adn sometimes serializable was necessary (for example, toi generate ID's unique acorss the database and across all the clients)
Should have a (2017) in the title?
(I was doing student assistant jobs for a database research group in 2015. This was old news even in 2017 (though I suppose nobody put it quite so obviously). I'd be curious if anything changed since then.)
(I was doing student assistant jobs for a database research group in 2015. This was old news even in 2017 (though I suppose nobody put it quite so obviously). I'd be curious if anything changed since then.)
Content aside, I just wanted to take a moment to admire the visual design of the presentation. The icons on the top left of each slide; the two-line slide titles and the page numbers; the black, white, and red of the “some papers are lit up” slides; the circles containing rectangles containing portraits; the ‘emphasis’ slides containing just a short sentence in white on a black background (which is both higher contrast and inverted compared to the dark-gray-on-light-gray used for normal slides)… it’s all very fancy without being too obtrusive. Especially for a presentation that’s just an individual presenting at an academic conference, as opposed to some company’s glitzy keynote trying to sell something to a mass audience. I’m impressed.
> Assumption #1: All transactions execute as stored procedures.
This kind of surprised me. Is that really a widespread assumption?
As an application developer I'm used to starting+committing transactions myself, outside the context of a stored procedure; I've done this with SQL Server and SQLite.
This kind of surprised me. Is that really a widespread assumption?
As an application developer I'm used to starting+committing transactions myself, outside the context of a stored procedure; I've done this with SQL Server and SQLite.
"Maybe READ COMMITTED is good enough or maybe people don’t know how dirty their data actually is…"
Look, a deadlock or slowness may technically be correct, but it can mean a lost sale. In practice, a worse problem than data inconsistency. SERIALIZABLE costs too much. I usually use snapshot, with a does of repeatable read where absolutely necessary.
Look, a deadlock or slowness may technically be correct, but it can mean a lost sale. In practice, a worse problem than data inconsistency. SERIALIZABLE costs too much. I usually use snapshot, with a does of repeatable read where absolutely necessary.
On slide 22, what does his fake name say in Greek? Catastropheaz Dedomeviv?
I have no idea what it's supposed to say. I think I just put "Andy Pavlopolis" into Google Translate. It's meant to be a joke. Before I started at CMU, all of the database professors in the city of Pittsburgh (Carnegie Mellon + University of Pittsburgh) were Greek. So the joke is that the only reason that I got hired at CMU was because I pretended to be Greek with a fake Greek name.
It's in the video here:
https://www.youtube.com/watch?v=M2MEcvMHzkY&t=4507s
It's in the video here:
https://www.youtube.com/watch?v=M2MEcvMHzkY&t=4507s
Katastropheas dedomenon
Means, I think, "given over to overthrows".
Means, I think, "given over to overthrows".
google translate gives a different translation: data shredder or data destructor
https://translate.google.com/?sl=el&tl=en&text=%CE%BA%CE%B1%...
https://translate.google.com/?sl=el&tl=en&text=%CE%BA%CE%B1%...
This presentation is (IMHO) worthless. There's no real data or information.
BTW, Spanner runs on paxos.
BTW, Spanner runs on paxos.
Don't care...
I'd diagnose the core problem differently, however. What application and system developers care about in modern architectures is application level correctness from the perspective of the client (customer, tester, etc). What a lot of the research is interested in is isolation within the context of a single database. That's a useful building block, but doesn't compose into end-to-end correctness in ways that most application architects and developers can reason about as soon as the application spans multiple stores (because SoA, or because microservices, or because it interacts with a SaaS, or whatever).
So application builders are left with the problem of not being able to reason about how to build correct applications, and not having the tools to build those applications correctly, and so end up settling for some level of weird behavior under concurrency. Turning up the database isolation level may hurt performance, but does little to rein in the overall incorrectness, so there is little motivation to do it.
If the database research community wants application developers and operators to care about isolation more, I suspect that we need to care more about atomicity, isolation, and consistency "in the large". Think more broadly, across whole microservice or SoA architectures, and provide ways to give customers ACID properties end-to-end, and ways to compose databases, services, and systems that preserve these properties.
That doesn't mean that per-database or per-component isolation isn't important. It's practically important, and there's good research on the effect it has[1]. The problem is that its not sufficient, and so folks feel less motivated to pay the (perceived) performance costs.
For folks who'd like to understand isolation levels better, I'd recommend starting with "Seeing is Believing" (https://www.cs.cornell.edu/lorenzo/papers/Crooks17Seeing.pdf) and "A critique of ANSI SQL isolation levels" (https://www.microsoft.com/en-us/research/wp-content/uploads/...) both of which do a good job explaining the differences between levels in a way that isn't tied to implementation.
[1] One good paper there is "ACIDRain: Concurrency-Related Attacks on Database-Backed Web Applications" (http://www.bailis.org/papers/acidrain-sigmod2017.pdf).