Every time this topic comes up, it seems people invariably divide into one of two groups. In the first group are those who trust in the odds and play the probability game. In the other, those who demand absolute guarantees. Neither group seems fully capable of understanding the other's standpoint.
Let me add my own perspective: Distributed locks are a fallacy. They can be beneficial in decreasing contention, under the assumption that "most of the time, only one actor will be active". However, by themselves, they offer no solid guarantees. The blog post addresses this point by introducing the concept of fencing tokens. These tokens have the potential to provide concrete guarantees, but they require the cooperation of downstream systems for enforcement, which isn't always possible.
I was really surprised to see Antirez argue for the probability approach.
A use case that I wish authorization service providers would talk more about is support for "list" queries e.g. What resources of type foo can the user read?
In really simple cases you may model this as a one-shot check on a logical collection resource e.g. If I have an organization, and an organization owns many repositories, I may check for action:read on resource:/organizations/:id/repositories. It's very limiting though. What if I want to list all repositories across all organizations I have access to? What if I have multiple levels I want to cut across? Do I need to do all the sub-queries and aggregations myself? Do I need to do a one-shot check on every potential resource? What if the answer is "no" for every one, and I end up doing a table scan of the entire DB just to produce an empty result set? etc.
You provide a distributed lease, not a lock. A distributed lease by itself doesn’t provide mutual exclusion. Distributed leases are typically accompanied with a fencing token (which your service cannot provide out of band) or an optimistic lock on the underlying "exclusive" resources (which could be implemented by the consumer of your service). I think of distributed leases as an optimization e.g. they provide soft exclusion in the happy path, which may reduce thrashing on the underlying real mutual exclusion mechanism (like an optimistic lock) under general use.
Your docs and terminology guide users into building incorrect systems e.g “ensure only a single instance of a process runs at any given time” is simply not true.
edit: I had a quick look at your Python library, and while Python isn't my forte, it looks like it can be trivially induced to break the mutual exclusion mirage: the "requests" lib's default request timeout is None e.g infinite, so what happens when `client.try_heartbeat` goes out to lunch indefinitely? Looks like `locks._run_heartbeat_loop` will hang, the lease will never be renewed, and within 60 seconds a competing lease holder will claim it. Boom. So you fix the timeout issue and problem solved right? Still no dice... what happens when you hit a pathological hang in your non-real-time runtime or OS? Boom.
There's so much more to CI/CD than the build definitions (e.g. dashboarding, log viewing/retention, access control, manual interventions, secret management, test reporting, etc.) and while some of your points resonate very strongly with me (e.g. local builds), I can't help but wonder what the endgame is here?
You've raised $27m to what? Give away a free tool for engineers to define their builds in? While they continue to use and pay for another service to actually run the builds? I assume you intend to replace the CI service at some point and move up the stack to monetize?
Without more transparency its easy to imagine something like...
Step 1. Drive uptake of your tool by selling people on the pitfalls of "CI lock-in"
Step 2. Introduce your own CI solution, which people can now easily switch to
Step 3. Lock people in
I am yet to work on a team with a simple, reliable and repeatable build process.
Most build processes (from basic scripts or Makefiles all the way up to CI/CD product-specific pipeline configurations) are complex, brittle, opaque and poorly understood by most of the team. I've seen time and time again one or two people emerge as the "build" people because most team members want to run as far away as they can, and it's easier to let someone else gatekeep the mysterious black box that sometimes does something useful. These processes typically grow organically on an as-needed basis and are rarely designed under a unifying vision.
I think it's a combination of lack of experience on the engineers part, bad off the shelf tooling, lack of recognition of the cost and as a result under investment by the business.
Agreed - I’ve spent months working on a deep tech problem before I could even show a functional core, let alone putting it into production. Some problems take time, experience and foresight to solve and the solution only dovetails at the end.
Sorry, I should have perhaps put a disclaimer in my original comment. I work for a company called StorReduce and built our replication feature* (an intelligent, continuous "sync" effectively). We currently have a patent pending for our method, so I'm not sure if I can offer any real insight unfortunately.
I haven't looked at your project, but based on what you've said I agree the way you're doing it is conceptually as fast as it can be (massively parallel and leveraging metadata) whilst being a general purpose tool that "just works" and has no external dependencies or constraints.
That depends on how you're storing the files. I was really just trying to highlight that for deduplication across files you need to deduplicate before you encrypt.
This approach works well enough for relatively small amounts of objects. Once you start getting in to the millions (and significantly higher) then it begins to break down. Every "sync" operation has to start from scratch, comparing source and target (possibly through an index) on a file by file basis. There are definitely faster ways of doing it that scale to much larger object counts, but then they have their own drawbacks.
It's a shame the S3 Api doesn't let you order by modified date, or this would be trivial to do efficiently.
How about if a single byte changes? Now the encrypted output looks 100% unique, and you have to treat it as an entirely new file. You lose the ability to do diffs on a file by file basis or proper deduplication across all files.