Ask HN: Is anyone using ArchLinux for server production environments?
11 comments
Same. But for me there has been one particular recurring kind of breakage. Python version upgrades (most recently 3.8 -> 3.9) invalidate all manually installed packages and virtualenvs. Mitigated by using Arch packages for Python packages instead...
Like Jay says https://www.youtube.com/watch?v=ytfohob38vM It really depends on what do you want to run on and including also how often you plan to upgrade dependencies. But being honest, Arch upgrades are transparent. For example I never got crashes or similar issues upgrading from Arch OS/Kernel dependencies in my daily basis desktop. The issues that I have only faced were with few dependencies coming from the AUR channel that I have installed on demand.
So it's not a "crazy" idea to consider to give a try ArchLinux for server-side stuff. Not in 2020.
> I very much like the process of steadily keeping up to date bit by bit, instead of stagnating on a stable base and doing a big bang update every few years. I also like the fact that I don't need to know which version of which release am I going to get, it's always the latest one and it's the one that upstream released.
Yeah, that's also the motivation of this thread from my side. In fact, moving between majors for example has also "risks" to tackle (E.g CentOS 6 to 7 to 8). You know, things that you need to take time to evaluate and test. But in the other hand, having rolling release upgrades makes that headache disappears.
So IMO I believe the Arch rolling release approach will be the next way to upgrade our software on a non-distant future. It's time to try out and test ArchLinux on hot places like servers.
> I very much like the process of steadily keeping up to date bit by bit, instead of stagnating on a stable base and doing a big bang update every few years. I also like the fact that I don't need to know which version of which release am I going to get, it's always the latest one and it's the one that upstream released.
Yeah, that's also the motivation of this thread from my side. In fact, moving between majors for example has also "risks" to tackle (E.g CentOS 6 to 7 to 8). You know, things that you need to take time to evaluate and test. But in the other hand, having rolling release upgrades makes that headache disappears.
So IMO I believe the Arch rolling release approach will be the next way to upgrade our software on a non-distant future. It's time to try out and test ArchLinux on hot places like servers.
Interestingly, the rolling release approach is already the way companies like Facebook or Google are releasing their services: when all tests pass it goes on master, and whatever is on master goes to prod. There are no releases, no backporting, no humongous release notes, etc... In a sense I like the idea that my server acts the same way, because the server is there to provide a service the same way Google provides a search service.
I really like Arch, but I wouldn't use it in production.
Yes, the packages are up to date and there are not many problems with them. But did you try to rollback to previous version of a package?
I had the problem on my personal laptop last year. There was no possibility to get a different version than the newest, which was broken (upstream's fault in this case). Luckily I had the previous version in my pacman cache and was able to install it. The package maintainers did a good job and the issue was resolved in something like 24-48 hours (from patching upstream to pacman). So nothing tragic for personal use.
In a productive environment? A very big no. One to two full days are simply not acceptable in any kind and/or size of business. Yes, you would (and should) test the update before rolling it out to production and be able to mitigate the problem, but you can't be sure, because you can't pin the version you want to install. Ok, you could build your own repo and package everything you need yourself. But it takes time and costs a lot of money.
Also automated deployment of Arch is a little bit of a headache.
Also automated deployment of Arch is a little bit of a headache.
All FPGA synthesis runs for https://webfpga.com are on Arch Linux based containers. There isn't a lot of traffic. We get around 5-10 synthesis runs per day from .edu accounts. It's been reliable, lightweight, and stable.
When developing the platform, we tried Ubuntu/Debian/CentOS, and at the time (about 1.5 years ago), Arch Linux was the most reliable OS for getting the 32-bit deps to work with existing commercial chains, and new toolchain compiles (IceStorm, NextPnr, etc). I think this is because Arch typically includes the development headers by default.
As long as the regression tests continue to pass, we will likely continue to roll with Arch Linux. We only tend to rebuild images once a month or so, since our backend is fairly feature complete already.
--
I think the trick is:
* Use containers, e.g. dockerhub's archlinux
* Assume that you will never be able to run pacman on a live system. Instead, rebuild a new image with a Dockerfile with your new packages -- then deploy that.
When developing the platform, we tried Ubuntu/Debian/CentOS, and at the time (about 1.5 years ago), Arch Linux was the most reliable OS for getting the 32-bit deps to work with existing commercial chains, and new toolchain compiles (IceStorm, NextPnr, etc). I think this is because Arch typically includes the development headers by default.
As long as the regression tests continue to pass, we will likely continue to roll with Arch Linux. We only tend to rebuild images once a month or so, since our backend is fairly feature complete already.
--
I think the trick is:
* Use containers, e.g. dockerhub's archlinux
* Assume that you will never be able to run pacman on a live system. Instead, rebuild a new image with a Dockerfile with your new packages -- then deploy that.
I'm a fan of Arch on the desktop, and I wouldn't recommend this.
The strength of Arch on the desktop for me is that it is a rolling update distro - that you are always up to date with the newest driver updates, mesa, etc.
This is exactly why I wouldn't use it on a server. Every time you install or update, you run the risk of getting a major software update that could break your app or server config.
For servers I want predictability - the same software versions over an extended period with just bugfixes & security fixes, and for me to decide when I want to change major versions. Arch doesn't make this predictability an option.
The strength of Arch on the desktop for me is that it is a rolling update distro - that you are always up to date with the newest driver updates, mesa, etc.
This is exactly why I wouldn't use it on a server. Every time you install or update, you run the risk of getting a major software update that could break your app or server config.
For servers I want predictability - the same software versions over an extended period with just bugfixes & security fixes, and for me to decide when I want to change major versions. Arch doesn't make this predictability an option.
Please don't use ArchLinux for your production environments. Unless you want to risk being murdered by future employees of your company of course.
now don't set any CentOS Machine anymore too: it is becoming rolling release as ArchLinux!
Hardly. The centos streams definition of rolling release bears no resemblance to arch’s definition. Centos streams will effectively be a point release ahead of RHEL and substantially behind on security releases.
I do upgrade my packages from time to time. It's almost always happened smoothly. The few times problems happened were due to 2 reasons:
- the software itself has a buggy upgrade procedure. I probably would have had the same issue with any other distribution, assuming they would contain the latest version
- it's an Arch-specific upgrade issue, but I have so few packets that this happens at most twice a year and it's always documented on the front page of the arch wiki
I have to admit that since it's only my own server, there is no pressure for services to stay up. In practice in the years I've used this setup I've never had an issue because a package was too recent and buggy; it was always a misconfiguration from my part.
I very much like the process of steadily keeping up to date bit by bit, instead of stagnating on a stable base and doing a big bang update every few years. I also like the fact that I don't need to know which version of which release am I going to get, it's always the latest one and it's the one that upstream released.