It was designed with large-scale environments in mind. I’m aware of several deployments managing hundreds of petabytes of data and billions of objects, which is why lakeFS does not use Git’s Merkle tree / directory tree approach.
That’s the current RBAC implementation: agents by default can make no API calls. the only way for them to contact the outside world is through a forward proxy configured in the sandbox. that proxy only allows making requests to destinations explicitly allowed (host, path, method)
very very fast: proportional to the count of objects modified, but not their size. Every commit represents a snapshot - an immutable listing of objects that represents the repository. reverting is essentially applying the inverse of the diff introduced by the reverted commit.
This is metadata only as the objects themselves are immutable.
Imagine an agent dropping a directory with 1m images in it. just figuring out what happened and what got dropped, restoring it one by one, etc. - doable, but ergonomics are a bit lacking.
Regarding pricing - that's indeed a great question and we don't have an answer yet. It will very likely be based on consumption and should be competitive to similar solutions.
Atomic commits are based on snapshotting done by lakeFS under the hood. Each sandbox run produces a new atomic commit to a hidden "main" branch. Updating that branch is optimistically concurrent, with lakeFS checking for conflicts - multiple writers updating the same object.
It uses lakeFS under the hood, so the unit of conflict would be a single file (object, under the hood). Resolving conflicts requires "picking" a winning side, or rerunning a conflicting job. Would you see a use case for merging changes into the same file? Interested to hear about your use case!
It provides a filesystem abstraction, which agents are really good at interacting with. Because it's just a POSIX filesystem - you can put a sqlite database directly on it and get those same transactional capabilities for that too.
it should absolutely scale to that. The filesystem is backed by lakeFS, where every sandbox automatically branches out, and mounts that branch. so you get isolation from lakeFS and the scale of an underlying object store (S3, in Tilde).
If that database is stored on the transactional filesystem available to the sandboxes, yes! Instead of backing up, it utilizes an efficient snapshot mechanism (lakeFS under the hood).
Part of the appeal (subjective, I know) of versioning is stuff like human-in-the-loop approvals. Think of a pull request: a change is requested by an agent, a human approves, changes get merged atomically. Even if other changes were applied since creation.
Hey! It doesn't necessarily have to be "data heavy", but any form of state (from code to binary files) that an agent might use for automation.
Agents are really good at interacting with files and directories (text in, text out!). This adds a layer for those that allows managing that state in a transactional, versioned way.
In a perfect world, every system and external API would expose a standardized interface for versioning its own immutable state, so you'd be able to rollback and time travel across multiple such systems.
Not sure what else we can do in this world other than tightly control outbound requests and provide enough visibility into those requests for a human|agent to try and undo changes.
Happy to hear your thoughts - what would you like to see us take this?
It was designed with large-scale environments in mind. I’m aware of several deployments managing hundreds of petabytes of data and billions of objects, which is why lakeFS does not use Git’s Merkle tree / directory tree approach.
Disclaimer: I’m one of the project’s co-creators.