I think more people should know about the existence of ZRAM on modern Linux distributions. It's really changed the way I look at swap configs.
ZRAM is a compressed block device that is stored in RAM. It's great!
Previously, if I ever had high memory pressure situations, I really dreaded the slowdowns. Now, with swap sitting on top of /dev/zram0 it's a completely different experience.
I have ZRAM enabled on all of my personal machines, both laptops with limited memory, and desktops with 64 or 128GB of RAM. It's rarely used, but it is nice to have that extra room sometimes.
The performance of a zram device is so much faster than even the latest NVMe drives.
Make sure you have solid Linux system monitoring in general. About 50% of running Ceph successfully at scale is just basic, solid system monitoring and alerting.
Not sure what you mean about Ceph wanting to be in a single rack.
I run Ceph at work. We have some clusters spanning 20 racks in a network fabric that has over 100 racks.
In a typical Leaf-Spine network architecture, you can easily have sub 100 microsecond network latency which would translate to sub millisecond Ceph latencies.
We have one site that is Leaf-Spine-SuperSpine, and the difference in network latency is barely measurable between machines in the same network pod and between different network pods.
What did the provider do? Did they put your IMEI onto some list of other customers that complained, where all of you get better network prioritization?
I've lived most of my adult life in houses with forced air furnaces (albeit powered via natural gas, not propane), and what you are saying is inaccurate regarding indoor air pollution unless your furnace is in need of immediate replacement.
A modern furnace works via a heat exchanger, where the combustion produced pollutants never mix with the indoor air being pushed through. All pollutants are expelled outside via a property functioning chimney. This is one reason why you should have the furnace (and chimney function) inspected annually. Aging heat exchangers will show hotspots before there is a possibility of air being mixed, giving plenty of time to plan for a replacement. Of course there is a possibility of failure, which is why you should have a carbon monoxide detector.
I know I'm going to sound crazy here, but there is one more alternative. How about: Reduce, Reuse, Repair, Recycle?
I recently got a sewing machine for an unrelated project and around the same time I ordered it I had one of these cloth reusable bags rip, because I put too many heavy things in it. When I got the sewing machine, for practice I decided to see if I could fix the bag. It turned out to be surprisingly quick and easy. I didn't use any extra material besides the thread, and I believe the bag is much stronger now.
While most of what you speak of re Ceph is correct, I want to strongly disagree with your view of not filling up Ceph above 66%. It really depends on implementation details. If you have 10 nodes, yeah then maybe that's a good rule of thumb. But if you're running 100 or 1000 nodes, there's no reason to waste so much raw capacity.
With upmap and balancer it is very easy to run a Ceph cluster where every single node/disk is within 1-1.5% of the average raw utilization of the cluster. Yes, you need room for failures, but on a large cluster it doesn't require much.
80% is definitely achievable, 85% should be as well on larger clusters.
Also re scale, depending on how small we're talking of course, but I'd rather have a small Ceph cluster with 5-10 tiny nodes than a single Linux server with LVM if I care about uptime. It makes scheduled maintenances much easier, also a disk failure on a regular server means RAID group (or ZFS/btrfs?) rebuild. With Ceph, even at fairly modest scale you can have very fast recovery times.
Source, I've been running production workloads on Ceph at fortune-50 companies for more than a decade, and yes I'm biased towards Ceph.
Done correctly, Ceph is extremely reliable, resilient, and fast. Once you get over the initial learning curve, dare I say, even a joy to work with.