You can do big data in SQLite. Concurrent writes, sure, I'd recommend something else.
If you think the majority of systems require massively concurrent writes, I think you need to look a bit harder. SQLite is, after all, the most widely deployed database system, ever.
It's a reference to Eternal September, the name applied to the cultural shift of the internet as the general public started gaining access to it en masse in the 90s.
Sloptember is clearly a reference to this - the similarity being that masses of AI generated content, from social media posts to open source contributions are replacing the human internet. In a way this is related to the "dead internet theory", an idea I previously found hard to believe, but these days could easily be true.
If the history of the internet interests you, both these are worth looking up.
Each commit should build and pass tests, yes. When I say "partial commits", I don't mean that the commits are arbitrary - each commit should be as small as possible to implement a specific fix/feature. I've also heard it described as the smallest unit that you may want to revert.
For example, if you are working on something, but it requires adding an API to some module, then the first commit 1 is to add the new API (+ tests), and the second commit is the new code that uses that API.
Unfortunately many developers I have worked with would just combine these (and more) into a single commit (because they are part of the same work task). However this makes review, bisect, blame and revert harder (if you need to revert commit 2, you don't want to also revert the API you added if that was tested and bug-free).
A good way of thinking about it is that every commit is itself version-controlled, allowing unlimited edits. This even allows two people in an evolve-enabled repo to make changes to history at the same time, and Mercurial will resolve any conflicts. It makes it trivial to commit (and even share) a "WIP" commit which you can later amend/split/whatever. It's different from git where you basically can't edit history after pushing (in Mercurial this only becomes true if you push to a non-evolvution or "publishing" repo, where everything then gets squashed for public consumption).
To be clear, Mercurial does not have a staging area, but it does have allow selective commits (and selective uncommits) via prompt-based or interactive UI selection of hunks. Disagreeing with the need for a staging area is not the same as saying selective commits are unnecessary (I use Mercurial more than git and I rarely commit everything in my working directory in a single commit - I like small commits).
We see this in our open-source community. We've had a community channel for over two decades, where community members help newcomers and each other solve problems and answer questions.
Increasingly we have people join who tell us they've been struggling with a problem "for days". Per routine, we ask for their configuration, and it turns out they've been asking ChatGPT, Claude or some other LLM for assistance and their configuration is a total mess.
Something about this feels really broken, when a channel full of domain experts are willing to lend a hand (within reason) for free. But instead, people increasingly turn to the machines which are well-known to hallucinate. They just don't think it will hallucinate for them.
In fact I see this pattern a lot. People use LLMs for stuff within their domain of expertise, or just ask them questions about washing cars, and they laugh at how incompetent and illogical they are. Then, hours later, they will happily query ChatGPT for mortgage advice, or whatever. If they don't have the knowledge to verify it themselves then they seem more willing to believe it is accurate, where in fact they should be even more careful.
URL parsing/normalisation/escaping/unescaping is a minefield. There are many edge cases where every implementation does things differently. This is a perfect example.
It gets worse if you are mapping URLs to a filesystem (e.g. for serving files). Even though they look similar, URL paths have different capabilities and rules than filesystems, and different filesystems also vary. This is also an example of that (I don't think most filesystems support empty directory names).
Yes, it is. The worst offenders hammer us (and others) with thousands upon thousands of requests, and each request uses unique IP addresses making all per-IP limits useless.
We implemented an anti-bot challenge and it helped for a while. Then our server collapsed again recently. The perf command showed that the actual TLS handshakes inside nginx were using over 50% of our server's CPU, starving other stuff on the machine.
Yes, definitely. To me, the idea of a chat server that doesn't federate is as absurd as setting up an email server that doesn't federate. I understand that today people know more contacts with email addresses than XMPP addresses, but if we ever want to free ourselves from the current walled gardens, we need to stop treating chat as something that only happens in walled gardens.
Some people get worried about the idea of "federation", thinking that it somehow means their server is less private, and their data is being spread across a mesh of servers, and stuff like that. That's true in some decentralized/distributed chat protocols, but not in XMPP. Connections between servers only happen on-demand, similar how when you send email between different email providers, they will connect to each other to deliver the messages.
However we do have a feature which allows disabling federation access for specific accounts, for example to prevent kids from communicating with anyone outside their own Snikket server. This is a feature I want to expand on, so that you can permit communication with a limited number of approved contacts on other servers.
I'm still experimenting with the messaging on the Snikket website. However my general approach with the site was to pitch Snikket to people who don't know what XMPP is, which is, frankly, the majority of people. Instead, I wanted to focus on explaining features it enables rather than protocol details. But I'm aware it has caused a lot of head-scratching among people who already know Snikket uses XMPP :)
I see Snikket as kind of a gateway into the XMPP ecosystem for people who are unfamiliar with it. After all, if you're already familiar with XMPP then the chances are you'll probably be happier with Prosody or ejabberd, and you'll already have opinions about which clients you want to use (e.g. the upstreams of Snikket).
Yeah, iOS is definitely a weaker spot and we're aware of it. Monal is currently working on an overhaul of their UI (they have a grant allocated for it: https://nlnet.nl/project/Monal-IM-UI/ ).
There is also a new app in the works between Cheogram and Snikket. There is a beta available, but it's still young (and we won't apply any Snikket branding until E2EE is complete).
Be aware that this post has known issues that the author is not interested in fixing. In their own words (in response to clarifications by one of the OMEMO folk):
"I'll make an edit later about the protocol version thing, but I'm not interested in having questions answered. My entire horse in this race is for evangelists to f** off and leave me alone. That's it. That's all I want." [censorship of profanity mine]
You won't find this quote in the article with Ctrl+F, it's in the screenshot, where they omitted the original constructive comment by one of the OMEMO contributors that they chose to moderate, which you can find here: https://www.moparisthebest.com/tim-henkes-omemo-response.txt
So, by all means, read the blog post. But just be aware that its ultimate goal was not to be an unbiased accurate technical article.
I'm the founder of both the Prosody and Snikket projects. Sorry about triggering alarms :) I can try to explain...
Prosody is a popular choice of XMPP server software. It's used for all kinds of stuff, from self-hosted chat servers to powering Jitsi Meet, to Internet-of-Things applications.
Prosody is extremely flexible, and has a bunch of configuration options that allow you to adapt it and extend it however you want. For some people, this is ideal. Those people should continue using Prosody.
Snikket has a different scope. It is specifically an answer to a question like "How can I easily make a self-hosted WhatsApp/Signal for my family/friends using open-source software?"
- Snikket contains Prosody, for the core chat part. But it's Prosody with a very specific configuration, and the configuration is part of the project, it's not intended to be modified by the person deploying Snikket. They only need to provide the domain name.
- Snikket also includes additional components that a modern chat service needs. For example, it includes a STUN/TURN server to ensure that audio/video calls work reliably (again, preconfigured).
- Snikket provides its own apps, which are tested and developed in sync with each other and with the server. This avoids the common problem of incompatibilities that occur when you have an open ecosystem such as XMPP, where different open-source project developers may develop features at different paces, leaving users to figure out which ones support which feature. It also solves the discoverability and decision fatigue for users (searching "Snikket" on an app store will get you an app that you know is compatible with your Snikket server, you don't have to go through a list of XMPP clients and figure out which one is suitable).
- Snikket servers are not designed to be open public servers (these are an administrative nightmare). Instead, your server is closed and private by default. As the admin, you choose who signs up to your server by sending invitation links. The invitations also serve to simplify the account setup process - no need to prompt users to "choose a server", etc. They just need to provide a username.
Projects such as Conversations differ by running a single public server (conversations.im) and guiding people to sign up on that server, or choose one of a long list of free public XMPP providers. In some cases that's all what you want. But onboarding a group of people that way is not fun (for example, they all have to share their addresses with the group add each other to their contact lists one-by-one - Snikket makes discovery of contacts within the same server automatic).
Beyond these things, Snikket is all open-source and XMPP. But there is a focus on making a good polished and secure "product", if you like, rather than supporting the entire diverse XMPP ecosystem which includes a range of software of varying quality (weekend projects and more recently, 100% vibe-coded clients). For example, Snikket servers require certain security and authentication features which some older codebases that have fallen far behind modern XMPP standards (think Pidgin, etc.) simply don't support today.
> it’s actually all based on prosody and conversations?
As mentioned, I develop Prosody. I also collaborate with the Conversations developer and other XMPP projects. There's nothing shady here. The goal is just to make a best-in-class XMPP project that solves one particular use case (and it was primarily my own use case to begin with of course - I wanted to move my family off WhatsApp).
As the founder of both projects, explaining the difference between the two projects is roughly 20% of my working day (okay, not quite 20% but sometimes it feels that way).
You're not wrong. PKI has better protections against MITM, dialback has better protections against certificate leaks/misissuance.
I think the ideal approach would be combining both (as mentioned, there have been some experiments with that), except when e.g. DANE can be used ( https://prosody.im/doc/modules/mod_s2s_auth_dane_in ). But if DANE can be used, the whole CA thing is irrelevant anyway :)
There is a lot of confusion caused by overlapping terminology in this issue.
By "client certificates" I mean (and generally take most others in this thread to mean) certificates which have been issues with the clientAuth key purpose defined in RFC 5280. This is the key purpose that Let's Encrypt will no longer be including in their certificates, and what this whole change is about.
However when one server connects to another server, all of TCP, TLS and the application code see the initiating party as a "client", which is distinct from say, an "XMPP client" which is an end-user application running on e.g. some laptop or phone.
The comment I was responding to clearly specified " I don't see how TLS-with-client-EKU [...]" which was more specific, however I used the more vague term "client certificates" to refer to the same thing in my response for brevity (thinking it would be clear from the context). Hope that clarifies things!
Yeah, the resistance is outside the XMPP community. However we have a long history of working with internet standards, and it's disappointing to now be in an era where "the internet" has become just a synonym for "the web", and so many interesting protocols and ideas get pushed aside because of the focus on browsers, the web and HTTPS.
They weren't "HTTPS certificates" originally, just certificates. They may be "HTTPS certificates" today if you listen to some people. However there was never a line drawn where one day they weren't "HTTPS certificates" and the next day they were. The ecosystem was just gradually pushed in that direction because of the dominance of the browser vendors and the popularity of the web.
I put "HTTPS certificates" in quotes in this comment because it is not a technical term defined anywhere, just a concept that "these certificates should only be used for HTTPS". The core specifications talk about "TLS servers" and "TLS clients".
Firstly, nobody is actually calling for authentication using client certificates. We use "normal" server certificates and validate the usual way, the only difference is that such a certificate may be presented on the "client" side of a connection when the connection is between two servers.
The statement that dialback is generally more susceptible to MITM is based on the premise that it is easier to MITM a single victim XMPP server (e.g. hijack its DNS queries or install an intercepting proxy somewhere on the path between the two servers) than it is to do the same attack to Let's Encrypt, which has various additional protections such as performing verification from multiple vantage points, always using DNSSEC, etc.
Of these, (1) and (2) are already implemented in XMPP.
(1) just isn't that widely deployed due to low DNSSEC adoption and setup complexity, but there is a push to get server operators to use it if they can.
(2) is defined in RFC 7711: https://www.rfc-editor.org/rfc/rfc7711 however it has more latency and complexity compared to just using a valid certificate directly in the XMPP connection's TLS handshake. Its main use is for XMPP hosting providers that don't have access to a domain's HTTPS.
If you think the majority of systems require massively concurrent writes, I think you need to look a bit harder. SQLite is, after all, the most widely deployed database system, ever.