Disk space usage leak in Docker for Mac(github.com)
github.com
Disk space usage leak in Docker for Mac
https://github.com/docker/for-mac/issues/371
24 comments
(I'm a Docker employee working on this very issue. I guess my plan to take a break by reading HN failed!)
You're completely right -- it's a problem caused by lack of TRIM in the storage path. In the next beta of Docker for Windows (beta 31 due today hopefully) TRIM should be enabled. The Mac will take a little longer as we need to switch protocols and do more work on the host side -- unfortunately the default Apple filesystem doesn't support sparse files so we can't "cheat" by simply passing the TRIM down to the filesystem layer. We'll probably need some kind of explicit block-level compaction to shuffle blocks from the end of the file into holes that have been created by TRIM.
You're completely right -- it's a problem caused by lack of TRIM in the storage path. In the next beta of Docker for Windows (beta 31 due today hopefully) TRIM should be enabled. The Mac will take a little longer as we need to switch protocols and do more work on the host side -- unfortunately the default Apple filesystem doesn't support sparse files so we can't "cheat" by simply passing the TRIM down to the filesystem layer. We'll probably need some kind of explicit block-level compaction to shuffle blocks from the end of the file into holes that have been created by TRIM.
Keep in mind the Apple file system is going to change early 2017! (APFS)
It's going to be a long time before it's widely deployed.
I'm looking forward to trying APFS, particularly support for sparse files. But you're right, it'll take while before we can rely on it.
Does virtio-scsi and TRIMs work around it?
We'll probably use AHCI since hyperkit https://github.com/docker/hyperkit (the Mac hypervisor based on xhyve based on bhyve) lacks a virtio-scsi implementation at the moment. When the VM sends a TRIM ("these blocks aren't used any more, you can free them") we need to make use of this on the host. On Linux/BSD you could tell the host OS that the blocks aren't needed in the file using something like FALLOC_FL_PUNCH_HOLE and the kernel would take care of it e.g. by send TRIMs to the real storage device or shuffling filesystem metadata around. Unfortunately the Mac filesystems in common use don't support this so we'll need to manage this ourselves, e.g. by moving blocks from the end of the file into the middle and shrinking the file.
Maybe it doesn't but it makes the docker Mac port basically unusable for anyone that runs it on a laptop without much space free.
But wait there's more... the mounted volume performance makes it unusable for anyone using mounted volumes for stuff like say dev work. It's a complete joke.
https://forums.docker.com/t/file-access-in-mounted-volumes-e...
https://forums.docker.com/t/file-access-in-mounted-volumes-e...
Everything old is new again. I posted a Docker issue about the default storage driver (DeviceMapper) not correctly freeing unused space nearly three years ago: https://github.com/docker/docker/issues/3182
From what I can tell, that issue was never truly resolved. The advice is like the adage about finding romance, "Have enough storage, don't run out of storage." As long as you follow that advice, you will never have a problem.
Oh, you installed Docker on a cloud provider with limited local disk, or on your laptop? Silly you for thinking that a finite amount of storage and the default configuration of Docker was adequate.
From what I can tell, that issue was never truly resolved. The advice is like the adage about finding romance, "Have enough storage, don't run out of storage." As long as you follow that advice, you will never have a problem.
Oh, you installed Docker on a cloud provider with limited local disk, or on your laptop? Silly you for thinking that a finite amount of storage and the default configuration of Docker was adequate.
Same here. I've also bumped into various storage issues with docker. Really drove me mad a couple of times... And like parent stated, the Docker docs about these issues reminded me of the XFS faq. Which stated about data loss for unclean shutdown (powerloss, panic): "If you know it hurts don't do it". In the end docker is just not as robust as for example SmartOS / Triton with ZFS.
I've given up with Docker on mac. It's a complete joke that they even consider it out of beta. The windows version is faster but doesn't handle file change events yet (or something like that). I just said screw it and switched my dev machine over to Ubuntu.
I haven't given up on Docker for Mac yet, but I could not agree more about them taking it out of beta. It is not ready for primetime yet. The shared volume driver, osxfs, is complete garbage in terms of performance. I use this solution as what I hope to be a temporary work around: https://github.com/EugenMayer/docker-sync.
No kidding, same experience here with docker for mac. Days wasted chasing down problems with a completely un-usable tool. Literally full stop critical bugs that are clear as day yet they release it and call it 1.0? Docker is ruining their reputation with this sort of garbage.
Agreed. Forget usage leaks, Docker for Mac doesn't even work on my system, and five days later after reporting it, nobody cares. It used to work, many versions ago, but they've somehow broken it entirely between redesigning the UI 2-3 times and switching from VirtualBox (which worked) to OSX Virtualization (which so far doesn't). See https://github.com/docker/for-mac/issues/984
In addition, they don't seem to care that the whole thing is useless in China - https://github.com/docker/docker/issues/28791 - also after emailing, no response.
I have contributed to Docker since very early days but am frankly not using it now because the project has for all intents and purposes entered a toxic-to-the-community stage where hype and marketing exceed capacity to resolve issues, total fails are occurring for me on all platforms, and AFAIK nobody in their over-funded San Franciscan office bubble seems to care. It's a typically arrogant startup: not listening to users, heading for a fall.
It's like they've decided to re-invent downloads (badly), re-invent cross-platform hypervisors (badly), re-invent orchestration (badly), re-invent storage (badly) and roll it all up in branded glue. I can't help but wonder if an approach with broader applicability and longevity would segregate the OS (anything container or VM-like at any layer, from Erlang to BSD jails to diskless clusters), and environment paradigm (container-based, paravirtualization-based, bare metal) from the workload and truly enable infrastructure agnoticism by removing the dependency on a single shifting-sands component, and allowing people to A/B test identical workloads on disparate paradigm infrastructures. I tried a few years ago, it worked: http://stani.sh/walter/cims
In addition, they don't seem to care that the whole thing is useless in China - https://github.com/docker/docker/issues/28791 - also after emailing, no response.
I have contributed to Docker since very early days but am frankly not using it now because the project has for all intents and purposes entered a toxic-to-the-community stage where hype and marketing exceed capacity to resolve issues, total fails are occurring for me on all platforms, and AFAIK nobody in their over-funded San Franciscan office bubble seems to care. It's a typically arrogant startup: not listening to users, heading for a fall.
It's like they've decided to re-invent downloads (badly), re-invent cross-platform hypervisors (badly), re-invent orchestration (badly), re-invent storage (badly) and roll it all up in branded glue. I can't help but wonder if an approach with broader applicability and longevity would segregate the OS (anything container or VM-like at any layer, from Erlang to BSD jails to diskless clusters), and environment paradigm (container-based, paravirtualization-based, bare metal) from the workload and truly enable infrastructure agnoticism by removing the dependency on a single shifting-sands component, and allowing people to A/B test identical workloads on disparate paradigm infrastructures. I tried a few years ago, it worked: http://stani.sh/walter/cims
> they don't seem to care that the whole thing is useless in China
You filed a GitHub issue on the docker repo about their infrastructure deployment.
They kindly (and correctly) pointed out the better way to resolve the issue and you insulted them.
Sorry you haven't heard a response to email. Hopefully that works in China too.
You filed a GitHub issue on the docker repo about their infrastructure deployment.
They kindly (and correctly) pointed out the better way to resolve the issue and you insulted them.
Sorry you haven't heard a response to email. Hopefully that works in China too.
Realistically, how they choose to divide their issue resolution is irrelevant except in terms of how much friction it causes with users and developers.
From a consumer perspective, if you can't docker pull then docker is ~useless, so being a key software feature within the docker binary it's not unrealistic to expect the average person to report directly to docker on github.
The fact we have to report fundamental issues that should never have passed basic testing in an extremely well funded company is the really amazing thing here.
Even if they wish to maintain a separate issue resolution process via email, a better resolution would be "I have forwarded this to the email support team, please expect an email" (since github IDs all have email associated anyway, this should be easy).
Right now they have ~maximum friction, and no response at all.
You will be pleased to note that email works fine in China (obviously - otherwise how do you think global trade operates?), unless you use NSA/Google as your provider.
From a consumer perspective, if you can't docker pull then docker is ~useless, so being a key software feature within the docker binary it's not unrealistic to expect the average person to report directly to docker on github.
The fact we have to report fundamental issues that should never have passed basic testing in an extremely well funded company is the really amazing thing here.
Even if they wish to maintain a separate issue resolution process via email, a better resolution would be "I have forwarded this to the email support team, please expect an email" (since github IDs all have email associated anyway, this should be easy).
Right now they have ~maximum friction, and no response at all.
You will be pleased to note that email works fine in China (obviously - otherwise how do you think global trade operates?), unless you use NSA/Google as your provider.
What's frustrating is if you submit an issue it will eventually be closed or linked to someone else's issue that's unrelated. Usually, the solution is to uninstall and re-install the beta. You shouldn't have to uninstall or reboot every time the daemon crashes.
Docker for mac was the most blatantly not-ready-for-primetime piece of software I've installed on my mac the past 10 years.
I suppose this is better than when the disk usage wouldn't grow at all.
It's not all that terrible to manage this yourself under light-to-medium usage, but if you're constantly experimenting with new machines and run into this more than once a week I'd say it's time to use docker remotely.
It's not all that terrible to manage this yourself under light-to-medium usage, but if you're constantly experimenting with new machines and run into this more than once a week I'd say it's time to use docker remotely.
I've gave up from Docker for Mac the first time I've tried it.
Docker for Mac has issues that make it unusable? You don't say!
Switching to `virtio-scsi` and sending regular TRIMs could fix this issue. It looks like they're also allowing people to configure the maximum size of the qcow2 image, which puts a hard upper bound on how much space the VM will take.