I've built clipper.dev - the fastest container repository in the world. (Big claim, I know.) It deduplicates at a file/chunk level, rather than at the layer level.
I get around 10-20% faster pulls than docker (when using containerd for both) for fresh pulls.
For some workflows, I see 80-90% faster patch pulls (think v0.17.5 -> 0.17.6).
Here's a best case scenario for me (and worst case for dockerhub)
docker> time sudo nerdctl pull ollama/ollama:0.17.5
docker.io/ollama/ollama:0.17.5:
elapsed: 244.1s total: 2.9 Gi (12.3 MiB/s)
real 4m4.172s
user 0m0.115s
sys 0m0.227s
docker> time sudo nerdctl pull ollama/ollama:0.17.6
docker.io/ollama/ollama:0.17.6:
elapsed: 234.9s total: 2.8 Gi (12.3 MiB/s)
real 3m54.965s
user 0m0.051s
sys 0m0.275s
---
clipper> time sudo clipper pull clipper/ollama:0.17.5
using containerd runtime at /run/containerd/containerd.sock
fetching manifest clipper.dev/clipper/ollama:0.17.5 ...
image created: clipper.dev/clipper/ollama:0.17.5
fetched 46 chunks (2.2 GB, 11.7 MB/s), took 3m16.134s
done — clipper.dev/clipper/ollama:0.17.5
real 3m17.532s
user 0m0.006s
sys 0m0.012s
clipper> time sudo clipper pull clipper/ollama:0.17.6
using containerd runtime at /run/containerd/containerd.sock
fetching manifest clipper.dev/clipper/ollama:0.17.6 ...
image created: clipper.dev/clipper/ollama:0.17.6
fetched 6 chunks (599.2 MB, 7.7 MB/s), took 1m17.48s
done — clipper.dev/clipper/ollama:0.17.6
real 1m19.061s
user 0m0.005s
sys 0m0.016s
The first two layers of 0.17.6 are shared with 0.17.5, but everything else has to be transported again with docker, but not with clipper. The exact speedup depends on how fast my wifi is feeling at the time, but I'm pulling 1.5 GB less than dockerhub can do. This is _huge_ for robotics workflows or anyone else pulling docker images on poor connections. (This is not even as fast as it could be - there's other optimizations like chunking within files or using BytePort that could make this go even faster).
I built this because at multiple companies I've been at, pulling new images for updates has been...very painful. Robotics Dockerfiles tend to have a lot of dependencies stuffed into a small amount of layers. Adding one new dependency means shipping the bytes for every other dependency, even with perfect pinning. Additionally, the Dockerfiles are oftentimes...not well written, compounding the problem. So I decided to fix the tool to fit the user, rather than the inverse.
There's some other side benefits to using clipper:
- Moving a file or changing permissions is "free": another layer already has the bytes
- The on-disk structure lends itself much better to P2P device updates than docker's layout
I do intend this to be a commercial product, but pricing is still up in the air. I'm currently targeting robotics companies as they feel the most pain here. There's some other possible fields that I'm interested in that might find use in this: IoT, edge AI, space datacenters/etc, developmer tooling.
Please give it a try. If you don't want to upload an image, these repos should be publically pullable once you register an account:
I've built clipper.dev - the fastest container repository in the world. (Big claim, I know.) It deduplicates at a file/chunk level, rather than at the layer level.
I get around 10-20% faster pulls than docker (when using containerd for both) for fresh pulls. For some workflows, I see 80-90% faster patch pulls (think v0.17.5 -> 0.17.6).
Here's a best case scenario for me (and worst case for dockerhub)
---
The first two layers of 0.17.6 are shared with 0.17.5, but everything else has to be transported again with docker, but not with clipper. The exact speedup depends on how fast my wifi is feeling at the time, but I'm pulling 1.5 GB less than dockerhub can do. This is _huge_ for robotics workflows or anyone else pulling docker images on poor connections. (This is not even as fast as it could be - there's other optimizations like chunking within files or using BytePort that could make this go even faster).
I built this because at multiple companies I've been at, pulling new images for updates has been...very painful. Robotics Dockerfiles tend to have a lot of dependencies stuffed into a small amount of layers. Adding one new dependency means shipping the bytes for every other dependency, even with perfect pinning. Additionally, the Dockerfiles are oftentimes...not well written, compounding the problem. So I decided to fix the tool to fit the user, rather than the inverse.
There's some other side benefits to using clipper: - Moving a file or changing permissions is "free": another layer already has the bytes - The on-disk structure lends itself much better to P2P device updates than docker's layout
I do intend this to be a commercial product, but pricing is still up in the air. I'm currently targeting robotics companies as they feel the most pain here. There's some other possible fields that I'm interested in that might find use in this: IoT, edge AI, space datacenters/etc, developmer tooling.
Please give it a try. If you don't want to upload an image, these repos should be publically pullable once you register an account: