I think it's pretty fair to test each OS with its filesystem of choice. I'm aware that you can use ZFS on Linux, but I'm not (yet) brave enough to recommend ZFS+Linux. And yes there's btrfs but would you trust it with your data ? :)
Before the benchmark I did some test with FreeBSD+ZFS with LZ4 on/off. The LZ4 CPU overhead was pretty negligible and performance was slightly better than without compression, due to lower IO usage. That's the reason I've chosen LZ4 for the actual benchmark.
Hello, OP here. I'm certain that you can fine tune every OS for specific use case. I may indeed do that in a future blogpost. The question is what to compare ? Should I compare Linux kernel versions, PostgreSQL versions, filesystems (and features like compression, block size, ...) ? As you can see the permutations are endless and thats why I compared stock OSes with their default filesystems of choice.
I don't think that a Linux distribution is just a variable and the only thing that differs is the kernel version. Each distro made its own choices, for better or worse...
As for the clients connecting over the network - that was exactly my point. My idea was to benchmark in conditions similar to production deployment. I doubt that many production systems connect over unix socket.
And for the warmup period, as you can see in the benchmarking script there is a 30 min warmup period before I start to record the results.