Docker Performance Evaluation Across Operating Systems(mdpi-res.com)
mdpi-res.com
Docker Performance Evaluation Across Operating Systems
https://mdpi-res.com/d_attachment/applsci/applsci-14-06672/article_deploy/applsci-14-06672.pdf?version=1722406713
37 comments
Ops person here, I'm confused what meaningful question this paper is supposed to answer?
Windows is suboptimal for running Linux containers? Sure, Docker on Windows is meant as development tool. If you have Windows Server but need to run Linux containers in production, you should run Hyper-V, install Linux and run container inside that VM.
MacOS has some funkiness for running containers as well that makes me think that's its disk operations are not truly backed. Also, Docker Desktop is again for Development, not actually running it in production since that would be really expensive server.
This entire paper could be a blog article. Did someone need to publish for some reason and threw this out there? What knowledge is being gained here? I WANT TO KNOW.
Windows is suboptimal for running Linux containers? Sure, Docker on Windows is meant as development tool. If you have Windows Server but need to run Linux containers in production, you should run Hyper-V, install Linux and run container inside that VM.
MacOS has some funkiness for running containers as well that makes me think that's its disk operations are not truly backed. Also, Docker Desktop is again for Development, not actually running it in production since that would be really expensive server.
This entire paper could be a blog article. Did someone need to publish for some reason and threw this out there? What knowledge is being gained here? I WANT TO KNOW.
A tech report that should have been a blog post is a great way to juice your citations. I know because I have done it twice.
Development performance does matter in situations where, for example, you're running an elaborate test suite on your laptop. In that case they should also test OrbStack though.
Development performance does matter in situations where, for example, you're running an elaborate test suite on your laptop. In that case they should also test OrbStack though.
Abstract: Docker has gained significant popularity in recent years. With the introduction of Docker Desktop for Windows and macOS, there is a need to determine the impact of the operating system on the performance of the Docker platform. This paper aims to investigate the performance of Docker containers based on the operating system. One of the fundamental goals of this study is to conduct a comprehensive analysis of the Docker architecture. This technology utilizes Linux kernel virtualization mechanisms such as namespaces and cgroups. Upon analyzing the distribution of Docker Desktop for Windows and Docker Desktop for macOS, it was discovered that running the Docker environment on these requires a lightweight virtual machine that emulates the Linux system. This information suggests that the additional virtualization layer may hinder the performance of non-Linux operating systems hosting Docker containers. The paper presents a performance test of the Docker runtime on Linux, Microsoft Windows, and macOS. The test evaluated specific aspects of operating system performance on a MacBook computer with an ×86/64 processor architecture. The experiment carried out examined the performance in terms of CPU speed, I/O speed, and network throughput. This test measured the efficiency of software that utilizes various system resources.
Is it normal for abstracts to not include any more detailed results or outcomes, or did they only realize that it needs to run a Linux-layer for the containers and that’s it? Isn’t this a well-known technical aspect of Docker?
Not-so-well-known technical aspect of Docker Engine is the support of Windows Server Containers which don't run Linux as name suggests. Since it's documentation [1] points only at Docker Desktop for awhile it's rather confusing to get all the info from MS [2].
[1] https://docs.docker.com/engine/install/
[2] https://learn.microsoft.com/en-us/virtualization/windowscont...
[1] https://docs.docker.com/engine/install/
[2] https://learn.microsoft.com/en-us/virtualization/windowscont...
> Ubuntu shows its fragility, especially in random writing. macOS outperforms Ubuntu by a substantial margin, exceeding its performance by a factor exceeding 40
They talk about this "fragility" like it's a common, known thing. I must be living in the woods, so to speak and feel silly. What's this known Ubuntu fragility they mention, anyone know?
They talk about this "fragility" like it's a common, known thing. I must be living in the woods, so to speak and feel silly. What's this known Ubuntu fragility they mention, anyone know?
It’s a bit more ridiculous because Ubuntu (and most Linux distros) do not implement their own filesystem semantics from scratch. They’re using the Linux kernel which many distros build off of. So it’s more correct to say “Linux shows its fragility” rather than “Ubuntu”. It’s all just a bit awkward, but I guess I’m being pedantic.
I wish I could address your question, but I don’t think there is “fragility” in how Linux does random writing. It’s extremely durable. MacOS (and therefore Darwin) has some pretty awesome optimizations by owning both the hardware and software, but I don’t know if random writes on Mac “succeed” more often than on Linux. It’s also a bit nonsensical since the hardware really matters when we’re talking about write durability.
I wish I could address your question, but I don’t think there is “fragility” in how Linux does random writing. It’s extremely durable. MacOS (and therefore Darwin) has some pretty awesome optimizations by owning both the hardware and software, but I don’t know if random writes on Mac “succeed” more often than on Linux. It’s also a bit nonsensical since the hardware really matters when we’re talking about write durability.
This "performance evaluation" is seriously flawed. I don't see how the results presented can be interpreted in any meaningful way.
> This barrier could be overcome with the help of the T2-Ubuntu [32] distribution, designed to be installed on Apple computers with T2-chipped.
That reference leads to https://github.com/t2linux/T2-Debian-and-Ubuntu-Kernel?tab=r..., which provides instructions on how to use a non-Ubuntu kernel to work around a hardware support issue on the particular hardware they used.
So they didn't actually test Ubuntu at all. They tested a third party produced kernel hack used to get support for their hardware. They then falsely assume that this is somehow representative of running Docker in...what? Production? Or for development purposes? They seem to have neglected to think about their use case. You'd think that the hardware they used for this test would be representative of a developer use case - but then why does performance matter?
If you consider container technology of the form that Docker uses, the kernel used is paramount for performance. Host OS userspace doesn't matter since it isn't active - the container userspace is provided by the container!
So not using an official Ubuntu kernel means that they didn't actually test anything provided by Ubuntu at all in practice. They've just tested whatever Linux kernel Docker Desktop uses inside its VMs on Windows and macOS against this Linux kernel hack running natively.
Their abstract also says:
> Upon analyzing the distribution of Docker Desktop for Windows and Docker Desktop for macOS, it was discovered that running the Docker environment on these requires a lightweight virtual machine that emulates the Linux system.
This is inherently known to anyone who can remotely claim to understand container technology. If this was a revelation to them, then this would explain why they think they tested Ubuntu when they didn't, and why they were unable to spot this flaw.
Disclosure: I'm biased in favour of Ubuntu. But I think the flaws I've described above stand for themselves.
> This barrier could be overcome with the help of the T2-Ubuntu [32] distribution, designed to be installed on Apple computers with T2-chipped.
That reference leads to https://github.com/t2linux/T2-Debian-and-Ubuntu-Kernel?tab=r..., which provides instructions on how to use a non-Ubuntu kernel to work around a hardware support issue on the particular hardware they used.
So they didn't actually test Ubuntu at all. They tested a third party produced kernel hack used to get support for their hardware. They then falsely assume that this is somehow representative of running Docker in...what? Production? Or for development purposes? They seem to have neglected to think about their use case. You'd think that the hardware they used for this test would be representative of a developer use case - but then why does performance matter?
If you consider container technology of the form that Docker uses, the kernel used is paramount for performance. Host OS userspace doesn't matter since it isn't active - the container userspace is provided by the container!
So not using an official Ubuntu kernel means that they didn't actually test anything provided by Ubuntu at all in practice. They've just tested whatever Linux kernel Docker Desktop uses inside its VMs on Windows and macOS against this Linux kernel hack running natively.
Their abstract also says:
> Upon analyzing the distribution of Docker Desktop for Windows and Docker Desktop for macOS, it was discovered that running the Docker environment on these requires a lightweight virtual machine that emulates the Linux system.
This is inherently known to anyone who can remotely claim to understand container technology. If this was a revelation to them, then this would explain why they think they tested Ubuntu when they didn't, and why they were unable to spot this flaw.
Disclosure: I'm biased in favour of Ubuntu. But I think the flaws I've described above stand for themselves.
OrbStack is where it's at on macOS these days. It's so much better than Docker Desktop, it's not even funny. It's in a whole different league.
I'd take this article with a grain of salt when it doesn't even mention it, or any of the other alternatives.
I'd take this article with a grain of salt when it doesn't even mention it, or any of the other alternatives.
It has the same license issues like Docker Desktop when using it in a commercial setting (per user subscription) if this is something that matters.
What can you do with OrbStack that docker can't?
You mean what Docker for Desktop can't do? It does have a few unique features (for example automatic HTTPS for containers) but that's not the point. The point is that it's much faster, the fan runs quieter, the battery drains less, memory gets hogged less, it crashes less, the UI is smoother, the updates actually work, and it keeps improving at a steady pace. The fundamentals just work a lot better.
I second this. We switched everyone at Instacart over from Docker Desktop to OrbStack because it was so much faster, easier on the battery, and simple to use. It was particularly noticeable on our older Intel MacBooks. Simply running Docker Desktop at all left mine noticeably hot and fans would regularly spin up for simple containers.
The difference is much less stark on Apple Silicon, but last time I checked OrbStack was still dramatically faster across multiple metrics.
Our local dev process was pretty heavy though. If you're doing simple, small containers and not copying a ton of things in, then you probably won't notice a difference. It's nice that the GUI app is not bloated and slow though.
The difference is much less stark on Apple Silicon, but last time I checked OrbStack was still dramatically faster across multiple metrics.
Our local dev process was pretty heavy though. If you're doing simple, small containers and not copying a ton of things in, then you probably won't notice a difference. It's nice that the GUI app is not bloated and slow though.
The macOS IO benchmark and pgbench (also IO) makes me think macOS cheats somehow, for example it doesn't actually cause the SSD to write to flash when the app performs fsync, and some other "optimizations".
Pretty much - https://news.ycombinator.com/item?id=30370551
Has IO through the hypervisor improved on macOS? Last I tried, the `./node_modules` directory absolutely destroyed the ability for me to use Docker as a development environment on my macOS workstations for Rails development, even with the special caching flags enabled.
A lot has change since then, like I stay away from node and `./node_modules` as much as possible via importmaps, but even then the FS performance wasn't enough for my needs.
A lot has change since then, like I stay away from node and `./node_modules` as much as possible via importmaps, but even then the FS performance wasn't enough for my needs.
I do dev work on mostly node/web projects that use npm. Pretty much everything I do involves docker. I typically write a docker file that copies package.json/package-lock.json into the image, then run npm install. That way the rebuild cycle is fast if the package.json file doesn't change and the node_modules are inside the container. Then I bind-mount all the src files that I'm working on so that live reload will work.
Also Rosetta x86 emulation, enabled in Docker, is highly performant
Ouch .... from the research:
"From the data gathered on our configuration, macOS is considered the most versatile system for operating with Docker containers."
"From the data gathered on our configuration, macOS is considered the most versatile system for operating with Docker containers."
The author does not have enough experience to interpret the results. See the comments about MacOS “cheating” on IO.
It might be a versatile OS for docker containers, but it’s not exactly cheap.
Important caveat: "Apple MacBook Pro with Intel chip
was used in tests that met all requirements specified in this section". What I really want to see is performance evaluations between current generation x86 Linux, x86 Windows, and ARM macOS.
They really should have run the benchmarks not only in Docker containers but also uncontainerized (as well as one or two KVM VM configurations) as baseline to approximate the overhead imposed by the Docker runtime itself. That would have made the article actually interesting.
Would also have been interesting to compare different runtimes (podman,crun,runc) and drivers.
I hope the authors follow up with a more fleshed out version because the chosen benchmarks and setup could make for some interesting results with some more rigorous methodology and analysis.
Would also have been interesting to compare different runtimes (podman,crun,runc) and drivers.
I hope the authors follow up with a more fleshed out version because the chosen benchmarks and setup could make for some interesting results with some more rigorous methodology and analysis.
I jusst used Docker for the first time - I can't believe I didn't know about it before. I thought the only way to run Linux on macOS was using some sort of Virtualbox-esque solution.
I just don't like how the containers are ephemeral.. I want to use it like a VM.
I just don't like how the containers are ephemeral.. I want to use it like a VM.
> I jusst used Docker for the first time
> I just don't like how the containers are ephemeral.. I want to use it like a VM
I suspect you’ve just pulled some images and run them locally?
If you want your own custom image to run whenever you want for whichever purpose, take a look into building your own containers with a Dockerfile
https://docs.docker.com/get-started/docker-concepts/building...
That gets you the base “everything I need is already installed when I start it”.
After that it’s volume mappings for accessing local data within the running container
https://docs.docker.com/engine/storage/volumes/
> I just don't like how the containers are ephemeral.. I want to use it like a VM
I suspect you’ve just pulled some images and run them locally?
If you want your own custom image to run whenever you want for whichever purpose, take a look into building your own containers with a Dockerfile
https://docs.docker.com/get-started/docker-concepts/building...
That gets you the base “everything I need is already installed when I start it”.
After that it’s volume mappings for accessing local data within the running container
https://docs.docker.com/engine/storage/volumes/
What if I used an bare image, installed some stuff and can't remember what all I did to get it in a usable state for my application.
I wish it could automatically create a Dockerfile for me.
I wish it could automatically create a Dockerfile for me.
```
tail .zsh_history > setup.sh
echo COPY setup.sh >> Dockerfile
echo RUN ./setup.sh >> Dockerfile
```
See ‘docker container commit’ [0]
> Create a new image from a [running] container's changes
You might then be able to run ‘docker image history’ [1] to get the changes, which you might be able to write into your own Dockerfile [2]
but I’ve never tested this workflow because the best practice is just to write a Dockerfile first
The tool (docker) was designed in a specific way, learn how to use the tool’s workflow instead of wishing for a different workflow and you’ll have a much easier time.
[0]: https://docs.docker.com/reference/cli/docker/container/commi...
[1]: https://docs.docker.com/reference/cli/docker/image/history/
[2]: can’t remember how docker tracks internal container file system changes and whether the history command will pick them up
Edit — see sibling comment https://news.ycombinator.com/item?id=41371867 — basically… learn to build a dockerfile (they’re very simple and easy once you know how).
> Create a new image from a [running] container's changes
You might then be able to run ‘docker image history’ [1] to get the changes, which you might be able to write into your own Dockerfile [2]
but I’ve never tested this workflow because the best practice is just to write a Dockerfile first
The tool (docker) was designed in a specific way, learn how to use the tool’s workflow instead of wishing for a different workflow and you’ll have a much easier time.
[0]: https://docs.docker.com/reference/cli/docker/container/commi...
[1]: https://docs.docker.com/reference/cli/docker/image/history/
[2]: can’t remember how docker tracks internal container file system changes and whether the history command will pick them up
Edit — see sibling comment https://news.ycombinator.com/item?id=41371867 — basically… learn to build a dockerfile (they’re very simple and easy once you know how).
you can but it's not recommended because it's basically impossible to audit changes made and is unrepeatable
https://stackoverflow.com/a/46917993/89373
https://stackoverflow.com/a/46917993/89373
There are several solutions for running LINUX on macOS
https://mac.getutm.app/
https://tart.run/
https://medium.com/@salah.mahmud/quick-introduction-into-usi...
https://medium.com/@salah.mahmud/quick-introduction-into-usi...
> I thought the only way to run Linux on macOS was using some sort of Virtualbox-esque solution.
On macOS, Docker uses HyperKit (which is a hypervisor built on top of macOS's Hypervisor.framework) to run a Linux virtual machine behind the scenes, so your containers actually are running in a VM, even if you don't (usually) interact directly with that VM.
On macOS, Docker uses HyperKit (which is a hypervisor built on top of macOS's Hypervisor.framework) to run a Linux virtual machine behind the scenes, so your containers actually are running in a VM, even if you don't (usually) interact directly with that VM.
Containers aren't ephemeral unless you want by default though. Are you passing in --rm on run?
You can use volumes for that.
just mount the whole root fs as a volume? or use bwrap or crun or runc to run your OCI bundle. Sadly the latter tools have some warts for this purpose...
Could use a [pdf] in the title if someone wants to add that
It would appear from this paper's conclusion that installing software with default settings may lead to poor performance.
"it was discovered" as if the VM behind the docker engine running on those operating systems was hidden to the human race.
> The dynamite plots.
Please don't create dynamite plots. Use a box plot, or plot the dots of the different repetitions with some jitter to avoid overlap.
> The evaluation
As reported in other comments apparently OSX cheats when saying it's writing to disk. If the authors wanted to compare docker executions it would be useful for the discussion to see if the difference in performance was due to the docker overhead or to the operating system, by showing what's the write performance of a non-containerized program across operating systems.
In my opinion the article should not have been published as is. It should have been rejected or asked for major revisions. MPDI Applied Sciences is quite a bad journal already for "Computer Science Applications" https://www.scimagojr.com/journalsearch.php?q=21100829268&ti... so it's not surprising to see this published.