CouchDB 3.4.1 Released(blog.couchdb.org)
blog.couchdb.org
CouchDB 3.4.1 Released
https://blog.couchdb.org/2024/09/27/3-4-1/
46 comments
Curious to hear who is using it still. I experimented with it 2010/2011 IIRC but never build a real project with it.
We tried to use it alongside pouchdb to provide offline first experience. We ran into a lot of issues, mainly with pouchdb bugs. Then after reading this list [1] of issues in the couchdb architecture we decided to ditch it and stop trying to make pouchdb work.
[1] https://news.ycombinator.com/item?id=17115649
[1] https://news.ycombinator.com/item?id=17115649
Out of curiosity, if you would like to start a new, offline-first project that needs syncing, which database would be the best candidate today?
I'm biased since I work on it, but https://ditto.live/ provides an SDK that allows P2P and cloud sync. You interact with it like a database: write your queries against your data, and it will move between devices automatically
I would still recommend you give CouchDB and PouchDB a fair shot. They are used successfully by many folks.
One interesting idea I’ve seen is none[1]. That’s definitely more on the exploration side than the shipping one, though.
[1] https://tonsky.me/blog/crdt-filesync/
[1] https://tonsky.me/blog/crdt-filesync/
I can think of two complex pieces of software which intentionally shunned databases thinking they can just do its thing in "flat files" - git and DokuWiki.
What ended happening is that both developed their own custom shi*ty database engines because it turns out that things like indexes are just generally pretty useful, but doing them right is pretty damn difficult. I'm pretty sure that if Linus/git chose e.g. SQLite instead of "flat files", we'd have way fewer data corruption problems, and a more capable/extensible git.
What ended happening is that both developed their own custom shi*ty database engines because it turns out that things like indexes are just generally pretty useful, but doing them right is pretty damn difficult. I'm pretty sure that if Linus/git chose e.g. SQLite instead of "flat files", we'd have way fewer data corruption problems, and a more capable/extensible git.
I don’t know if Git’s storage is deserving of this level of scorn—I can’t lay claim to any in-depth knowledge, but if it were indeed a big problem I’d expect it to come up frequently in comparisons with Fossil, which stores things using SQLite. (As a counterpoint, I use git-annex quite a bit, and it almost certainly couldn’t integrate as neatly into Fossil’s storage approach as it does into Git’s.) So I’d appreciate any details here.
All that is beside the point, though: the article above is not about using or not using flat files as a storage primitive, it’s about using files of whatever nature as a replication and version reconciliation mechanism, in view of the fact that concurrent editing is inevitably application-specific, so we might as well lean into it instead of leaving it to a database. In that sentence, “a database” is not just any database, it’s one of a very short list of multimaster databases with relatively loose schemas, which includes CouchDB and—among legitimately FOSS projects—I’d struggle to name more.
This is not a decision about data storage at all, in other words. It is a decision about protocols. Experience shows that the alternative does not end up being an off-the-shelf database (even CouchDB, which does seem like a major road not taken looking back at Canonical’s efforts a decade ago), the alternative is usually a central synchronization server speaking a custom protocol. (CalDAV, CardDAV, Bitwarden, etc.)
And if you want to do your CRDT or OT or whatnot over per-client SQLite databases instead of per-client text files, all the more power to you.
Finally, I tried to phrase my comment above in a way that makes it clear that it’s a suggestion of a direction to have fun in, not of a principle to architect your production app around. So the sneering in your comment is... honestly disheartening to read. Like, do people even hack anymore? I know they do, but every time I read something like this I become a little bit less sure of it.
All that is beside the point, though: the article above is not about using or not using flat files as a storage primitive, it’s about using files of whatever nature as a replication and version reconciliation mechanism, in view of the fact that concurrent editing is inevitably application-specific, so we might as well lean into it instead of leaving it to a database. In that sentence, “a database” is not just any database, it’s one of a very short list of multimaster databases with relatively loose schemas, which includes CouchDB and—among legitimately FOSS projects—I’d struggle to name more.
This is not a decision about data storage at all, in other words. It is a decision about protocols. Experience shows that the alternative does not end up being an off-the-shelf database (even CouchDB, which does seem like a major road not taken looking back at Canonical’s efforts a decade ago), the alternative is usually a central synchronization server speaking a custom protocol. (CalDAV, CardDAV, Bitwarden, etc.)
And if you want to do your CRDT or OT or whatnot over per-client SQLite databases instead of per-client text files, all the more power to you.
Finally, I tried to phrase my comment above in a way that makes it clear that it’s a suggestion of a direction to have fun in, not of a principle to architect your production app around. So the sneering in your comment is... honestly disheartening to read. Like, do people even hack anymore? I know they do, but every time I read something like this I become a little bit less sure of it.
hum, allow me to toot my own horn here. I've written something to that effect in 2021. https://raphael.lullis.net/thinking-heads-are-not-in-the-clo...
I am (very) biased but Couchbase has a pretty solid Mobile offering for native apps. I have worked on the Sync Gateway component responsible for replication for the last six years.
Sync Gateway still maintains a CouchDB-compatible REST API, and PouchDB _mostly_ works thanks to that, but there are some corner cases and features that PouchDB does not support so YMMV with it. Our native app libraries have used a more performant websocket-based replication protocol for many, many years now, and I'd really love to have the time investigating a PouchDB adapter using this WS protocol instead.
Sync Gateway still maintains a CouchDB-compatible REST API, and PouchDB _mostly_ works thanks to that, but there are some corner cases and features that PouchDB does not support so YMMV with it. Our native app libraries have used a more performant websocket-based replication protocol for many, many years now, and I'd really love to have the time investigating a PouchDB adapter using this WS protocol instead.
I build an internal project using CouchDB2 as a backend in 2017 and it's still used today. CouchDB definitely surpassed my expectations in what has been possible to do with it. Its biggest advantages are that data sync is effortless and that it uses HTTP as a protocol, so you can communicate directly with it from a web application.
Used it this school year for a research project that needs solid replication of content and a firehose of updates. There are dozens of such projects but nothing beats the simplicity and versatility of basic HTTP calls with JSON documents and the robustness of a 15-years old project that Just Works.
Was some time ago (2016), but we built and released Typeform LITE (form creator via text-entry-like interface for smartphones) with CouchDB + PouchDB which was a pleasure as far as I can remember. Was really neat to get local-first functionality almost out of the box with web tech on smartphones.
We are using it internally as just a document store (with some erlang views but that is it). It scales very nicely, but for smaller documents postgres does the job for us so we are (slowly) moving away to simplify the tech stack.
I tried to build some hobby stuff with it (mostly worked), planning to use it instead of mongodb @work but I got hit by the limitations of the JS engine version used at the time and it got frustrating too fast for a small hobbyist project. The mongodb/couchdb thing at work was cancelled so I didn't spend more time with it.
People behind it are really, really nice though.
One of the top (the top I think ?) obsidian sync plugin uses it.
People behind it are really, really nice though.
One of the top (the top I think ?) obsidian sync plugin uses it.
We still use couchdb & pouchdb only for syncing data for our salesmen app.
It's an offline first PWA (Salesman need to make an order even if offline).
And pouchdb make it easier because we don't have to care about indexeddb, leveldb or whatever of users browser's db.
So its a single codebase for serving web, ios and android.
Its great for sync but less then good for other uses
It's an offline first PWA (Salesman need to make an order even if offline).
And pouchdb make it easier because we don't have to care about indexeddb, leveldb or whatever of users browser's db.
So its a single codebase for serving web, ios and android.
Its great for sync but less then good for other uses
I'm using it as a database for my application, SolarThing: http://github.com/wildmountainfarms/solarthing
I was naive when I chose it (I was just starting to program then), but it has served me well. It's replication feature is simple to setup and is robust. It uses a REST API for any calls to the database, and it is a well designed API. I don't use all its features, but I really like it for the ones I use.
I was naive when I chose it (I was just starting to program then), but it has served me well. It's replication feature is simple to setup and is robust. It uses a REST API for any calls to the database, and it is a well designed API. I don't use all its features, but I really like it for the ones I use.
I have built advertisement bidders and ad servers. I avoid querying the database on every request (and caches) by creating app in-memory materialized views of the database.
See https://www.confluent.io/blog/leveraging-power-database-unbu... for more about this.
Another cool feature of the replication protocol is that I can tell if all app servers are on the same version of the data.
We are still using couchdb for user profile synchronisations. Basically user details and data from mobile synched to the cloud. In our application users do not interact between one another. So basically we use db per user model. This gives us synchronisation between mobile/web clients. Pity that there is no up to date mobile library for couchDB and we depend on an old one.
We used 1.6 for a large project in 2014 or so, the 2.x release removed key features for us and we were never able to upgrade. The direction they went after that just never really worked for us so we only have that legacy project still running.
CouchDB dev here. 2.x only added new features, not sure what you’re referring to that was removed. The only on incompatibility was the changes feed ids moving from numbers to strings, which is hardly a huge migration :)
It is still being used by a cool open-source project, Kazoo, which is essentially a multi-tenant PBX.
I was wondering the same thing
CouchDB was always a curious product that asked the question, "what would happen if your database and your application server were the same product?"
Unfortunately I think the answer was nobody would use it.
Unfortunately I think the answer was nobody would use it.
CouchDB dev here. While that was our idea early on for a while, we have loooong (10+ years) moved away from this narrative.
CouchDB is a document database that can be essentially indefinitely clustered to grow and shrink with your application / traffic demands.
It also comes with a unique replication protocol that allows you to synchronise casually connected instances (say offices around the world, or mobile devices and a cloud) much more like git uses push & pull. No other database really puts that into an open source / open government project.
There’s tons more cool stuff that CouchDB does and its being used in mission critical infrastructure that you’re relying on every day :)
CouchDB is a document database that can be essentially indefinitely clustered to grow and shrink with your application / traffic demands.
It also comes with a unique replication protocol that allows you to synchronise casually connected instances (say offices around the world, or mobile devices and a cloud) much more like git uses push & pull. No other database really puts that into an open source / open government project.
There’s tons more cool stuff that CouchDB does and its being used in mission critical infrastructure that you’re relying on every day :)
> There’s tons more cool stuff that CouchDB does and its being used in mission critical infrastructure that you’re relying on every day :)
If you could share a bit on those I think the community would be extremely interested. There was that CERN thing a decade ago, if it's still true or there are other actors with similar amounts of data, there would be fewer comments about not finding CouchDB useful
If you could share a bit on those I think the community would be extremely interested. There was that CERN thing a decade ago, if it's still true or there are other actors with similar amounts of data, there would be fewer comments about not finding CouchDB useful
Yeah we are working on that. Sometimes it is hard to get public info out of private projects, but there’s going to be a bit more on that in the not so distant future :)
> ...replication protocol that allows you to synchronise casually connected instances...
That single featurte right there was why i first had started playing with couchDB several years ago. It really had been quite awesome! I think other features that help it really help with offline-first approaches also are really cool! But, ultimately, after playing with it (and really liking it), i just didn't have a need for too many personal use-cases....since sqlite was "good enough". I thought - and still believe - that for wider (maybe corporate/enterprise?) uses, it can still fit the need....but i don;t hear much buzzz about it in general...so in my case, i stopped using it, and forgot about it. Apologies to any devs behind it, no offnese meant...i just got my personal use-cases covered by sqlite...and on enterprise side, too many internal politics to convince stakeholders that couchdb would be better than other solutions in some use cases.
Also, the documentation back in the day to get started was ok-ish, but there didn't seem to be much around....i guess it could be one of those things where it is/was a great thing...but if not enough attention and use exists, then it won;t trigger a wider audience of devs who will also document more stuff and further triugger a virtuous cycle?
Anyway, good to see the devs are still at it! :-)
That single featurte right there was why i first had started playing with couchDB several years ago. It really had been quite awesome! I think other features that help it really help with offline-first approaches also are really cool! But, ultimately, after playing with it (and really liking it), i just didn't have a need for too many personal use-cases....since sqlite was "good enough". I thought - and still believe - that for wider (maybe corporate/enterprise?) uses, it can still fit the need....but i don;t hear much buzzz about it in general...so in my case, i stopped using it, and forgot about it. Apologies to any devs behind it, no offnese meant...i just got my personal use-cases covered by sqlite...and on enterprise side, too many internal politics to convince stakeholders that couchdb would be better than other solutions in some use cases.
Also, the documentation back in the day to get started was ok-ish, but there didn't seem to be much around....i guess it could be one of those things where it is/was a great thing...but if not enough attention and use exists, then it won;t trigger a wider audience of devs who will also document more stuff and further triugger a virtuous cycle?
Anyway, good to see the devs are still at it! :-)
No offence taken, SQLite is great technology :) — My company makes a (commercial) CouchDB add-on based on SQLite to add SQL queries to CouchDB (link provided for informational purposes): https://neighbourhood.ie/products-and-services/structured-qu...
How interesting! Thanks for sharing! :-)
Map/Reduce was a hype of the days it was created, but you're perfectly fine not using it. CouchDB has been integrating MongoDB-style queries for a few releases now, and this very release introduces a full-text search engine. Map/Reduce is still there because it just works.
The real shame is that Map/Reduce is simply a toy, easy-to-add bonus when looking at the data structure, but that's not what CouchDB is or ever was about. The one reason it was created is replication and simplistic conflict handling, and it does it perfectly.
The reason CouchDB never took off is that it targets offline-first, something you'll see associated with peer-to-peer systems, but does that only for servers: you're supposed to install that thing through packages, configure a text file, run it with your preferred daemon manager, ... If CouchDB had a simple desktop version you could start in one click and forget about, it would have been way more interesting. Alas.
The real shame is that Map/Reduce is simply a toy, easy-to-add bonus when looking at the data structure, but that's not what CouchDB is or ever was about. The one reason it was created is replication and simplistic conflict handling, and it does it perfectly.
The reason CouchDB never took off is that it targets offline-first, something you'll see associated with peer-to-peer systems, but does that only for servers: you're supposed to install that thing through packages, configure a text file, run it with your preferred daemon manager, ... If CouchDB had a simple desktop version you could start in one click and forget about, it would have been way more interesting. Alas.
There are client implementations, such as PouchDB for the browser and Couchbase Lite (community edition) for mobile platforms.
The Mac version is probably closest to what you mean, but yeah, an “Access/FileMaker“ type of app, most easily realised by bundling CouchDB and Electron probably, would go a long way. If anyone needs a project ;)
I think that better describes zodb/hope. Or gemstone glass.
https://zodb.org/en/latest/tutorial.html
https://zodb.org/en/latest/tutorial.html
Well, npm did... https://github.com/npm/npm-registry-couchapp
still does (in a limited capacity :)
In the release notes highlights:
> Optionally replace md5 with xxHash for data integrity checksums. 30% speed up on larger (128k) docs, no difference for smaller docs.
Happy to see xxHash getting more and more popular!
> Optionally replace md5 with xxHash for data integrity checksums. 30% speed up on larger (128k) docs, no difference for smaller docs.
Happy to see xxHash getting more and more popular!
I think there’s a small oversight in the release notes:
> In that case, upgrade to 3.4.0 without changing the default and enable xxHash in 3.5.0 or later so you can roll back to 3.4.0 then.
This should probably say 3.4.1, not 3.4.0, in the two places in that sentence where it says 3.4.0. Right?
> In that case, upgrade to 3.4.0 without changing the default and enable xxHash in 3.5.0 or later so you can roll back to 3.4.0 then.
This should probably say 3.4.1, not 3.4.0, in the two places in that sentence where it says 3.4.0. Right?
fixed, thanks