I don't know about bounties, because I'm not personally in favour of vigilantism, but I do take your point.
Honestly, I think the ease in which people can be anonymous is major problem here. Anyone with an internet connection can buy botnet time with Bitcoin and accept a ransom in the same way. It makes it incredibly difficult to follow the path back to the attacker.
At this point pretty much the only thing you can do is collect data and share it with CERT and other relevant law enforcement. I don't have a good sense of how effective they can be, but it makes sense that the more data they have, the better chance they have at identifying specific botnets and follow the path back to the owners.
Generally, we talk about what we're doing because we're all excited about what we do. It's always been that way for us, and our customers really appreciate the honesty and transparency.
On this particular one, we really did learn a lot and we were keen to share some of that. It's really difficult to run an internet service and we feel we have a duty to try and make this easier, or at less better-understood, where we can.
Our business can't exist with a large diverse network that anyone can get involved on, and we couldn't have got to this point without the knowledge of others, whether that's embedded in the open-source software we run or in the blog posts and emails of other people that figured out hard stuff. It wouldn't be right for us to take and not give something back in return.
There's also an element of defiance in this post too. We got punched in the face. We're not going to respond by hiding in a corner. We're going to say "you know what, fuck you" and we're going to help (and have helped) others to do the same in whatever way we can.
Indeed. As I note in the post, this wasn't something we _needed_ to do at that time, we were just curious.
Replacing a function with a faster implementation is trivial; its just another deployment and we do several of those each week. Changing data format adds operational complexity - two codepaths need to be run and maintained and we increase system load and lower response times for the duration. (The data format is versioned, so that's easy - no architectural problem there).
Its totally worth doing if you need to, and we're not afraid of that, but you don't do it on a whim - it takes proper planning and testing and needs multiple people involved.
The results in the tests are all with optimisations (-O3 -march=sandybridge -mtune=intel), as mentioned in the post.
The exception is the Debian packaged version of zlib, because we don't control that. That's the reason I include stock zlib in the tests - if it had been wildly different from the system zlib, I would have looked into recompiling the Debian package with more optimisations. There were no major differences and indeed, inspecting the package further shows that it is compiled with optimisations.
On the Cyrus side, we used to link to the Debian zlib, so we get those optimisations. For the new code bundled in Cyrus itself, we have to enable optimisations ourselves.
I think that mostly, your benchmarks have to match your workloads. Most of the CRC32 benchmarks I've seen are looking at larger buffers. The xxhash function mentioned elsewhere in this thread was claimed to be "an order of magnitude" faster, but again, large buffers - the gain over CRC32 on the same tests were rather more modest (though not at all insignificant).
In this case, I think (but am curious, will investigate further at some point) our Cyrus servers are doing enough checksumming work to keep any tables hot in the cache. So the tests are hopefully a useful indicator of where improvements can be made.
My quick tests here (using the same methods outlined in the post) suggests its around 30% faster on 64-byte buffers. If we're ever shopping for a new hash function entirely, I'll make sure its considered. Right now its not worth it because as noted in the post, we're not actually under any particular stress and changing hash functions is a big job.
In what way do you consider the privacy policy cavalier? What would you prefer it said?
I don't believe we made any claims that Australia is a "privacy Eden" or that we're "paragon of privacy". Indeed, we frequently say we're _not_ a privacy service, just an email service that cares about privacy among other things.
More specifics on both these points would help us discuss them properly.
Maybe. They'd have to issue that warrant to our datacentre operators though, not us, because there's nowhere to send the documentation. And then they can compel our datacentre not to talk about it, if they like, but they can't stop us talking about.
Really though, the point of all this isn't to say they can't take our servers - of course they can, via legal and illegal means. The point is more to say that they can't do it _quietly_, which greatly raises the bar, because now you've got a PR shitstorm to deal with.
But really, it's not going to happen, because we have good legal processes in place. There are proper channels from most countries in the world to the appropriate Australian authorities, and from there to us, and once that request comes in we service it and that's that.
If you want reasonably secure and private email, and you're not doing really dodgy shit, we're probably a safer choice that many. But we're not selling a privacy service, just an email service. If privacy is 100% non-negotiable for you, then you'll need to look elsewhere.
Dumb storage is fine if you only have a single actor that can maintain a copy of the state at all times. As soon as you've got multiple actors on the data (another client on your desktop or phone, or even something doing mail delivery), then you need a way to determine what changed on the server so you can update your state.
If the server has no ability to tell you what changed, then you're left with having to download and check everything. On a large data set, that's pretty much impossible to do quickly and without a lot of network traffic.
Of course, no server is actually that dumb - even a file listing with file sizes can get you part of the way there. But if you've got 10000 files in a file store, that list can still get pretty heavy. If you're willing to make the server smarter, eventually you can get to the point where the server can give you only what changed since the last time you checked. JMAP isn't unique in this; IMAP has MODSEQs, *DAV has collection synchronisation, etc.
JMAP specifically doesn't really care much about the actual format of the data it works with. The only thing it really needs is an immutable ID, so you could use the same model to store all sorts of things (and at FastMail we do, with things like client settings).
> JMAP server implementations probably will require a good database, not just a dumb FS like in remoteStorage.
To do JMAP well you need to be able to calculate change sets, which does mean a database, though a fairly light one. I think you could do a server without delta updates by always returning a cannotCalculateChanges error in response to getUpdates calls, but it would be very inefficient on both the client and the wire.
> However, I would like to collaborate wherever possible.
Sure! Best place to start is probably the jmap-discuss list:
If you want a laundry list of SSRF methods you should protect against, a great place to start is this slide deck from a talk at ONsec a couple of years ago:
There's not much around yet. The only things that are in any kind of usable/complete state are the proxy, the demo client and the JS client library, all mentioned in the original post.
Cyrus has partial support (the basic data model and contacts) and is running in production at FastMail. The aim is for full support this year.
On the client side, Roundcube has committed to adding support in the next version (after their recent successful Indiegogo campaign) and Thunderbird is serious investigating it (there's a GSoC student looking at it).
I'm aware of two companies that will be shipping products (client and server) based on JMAP this year. I'm not going to mention names because that's not cool.
Finally, I'm personally building a JMAP data model library in Rust, a standalone (non-IMAP) server on top of it, and I'm part way through adding support to fetchmail and mutt.
I'm not aware of a sync service but it would be pretty straightforward to build. I'll have a look at it sometime, probably taking code from the FastMail app for Android, which has a partial JMAP client in it already.
I need to pull this info together into a "software" page on jmap.io - I'll try to do that soon. In the meantime, if you're working on something that uses JMAP, let me know :)
Honestly, I think the ease in which people can be anonymous is major problem here. Anyone with an internet connection can buy botnet time with Bitcoin and accept a ransom in the same way. It makes it incredibly difficult to follow the path back to the attacker.
At this point pretty much the only thing you can do is collect data and share it with CERT and other relevant law enforcement. I don't have a good sense of how effective they can be, but it makes sense that the more data they have, the better chance they have at identifying specific botnets and follow the path back to the owners.