Self-hosting on a Raspberry Pi cluster(savannahostrowski.com)
savannahostrowski.com
Self-hosting on a Raspberry Pi cluster
https://savannahostrowski.com/posts/raspberry-pi-cluster/
19 comments
For fun is okay, but you can buy a single computer more powerful than those 4x Pi for 500 bucks. Pi has stopped being cost effective a long while ago.
And know that virtually 100% of software will run on it, rather than having to hope there's an ARM build of it (which most major FOSS software does have, but there's still that slice that doesn't)
And even if you want to experiment with multi-node HA stuff, I personally prefer to do that with VMs than wrangling actual physical machines
And even if you want to experiment with multi-node HA stuff, I personally prefer to do that with VMs than wrangling actual physical machines
i have a raspberry pi3 (4gb, 50 euros) and a pi5 (8gb, 100 euros). i run a k0s cluster (for fun) w/ pihole, plex, some monitoring and homeassistant. that for me is cost effective
You can get a 9th gen intel pc for that today in the us. Maybe not in eu ebay but in us for sure. New n100 mini pc from china companies also clock in around that total price.
I have some Raspberry Pis that I self host stuff on. Gitlab on one, Runtipi on another, and pi-hole + Nginx + a game I made on a third. Some of it is publicly accessible on my own domains, but the only traffic I actually get is from myself.
I connect my Pis to my WiFi router and I've struggled with the adapters, so a POE switch might be interesting. I should containerize stuff, but I enjoy running it directly as well. I need to figure out backups some day.
The biggest issue I've had is that I've lost 3 SD cards in 6 years. Twice due to power failure, and once to heavy disk writes. I can't stand randomly losing stuff and having to re-setup a pi, so I switched to booting from USB SSDs. I haven't had an issue since. The cost is a bit obnoxious since I don't trust a cheap drives, but the perceived peace of mind is nice.
Has anyone else had this issues with losing SD cards while self hosting on Pis?
I connect my Pis to my WiFi router and I've struggled with the adapters, so a POE switch might be interesting. I should containerize stuff, but I enjoy running it directly as well. I need to figure out backups some day.
The biggest issue I've had is that I've lost 3 SD cards in 6 years. Twice due to power failure, and once to heavy disk writes. I can't stand randomly losing stuff and having to re-setup a pi, so I switched to booting from USB SSDs. I haven't had an issue since. The cost is a bit obnoxious since I don't trust a cheap drives, but the perceived peace of mind is nice.
Has anyone else had this issues with losing SD cards while self hosting on Pis?
Only time I've ever had an issue is when I manually disconnected power from a Pi 4 when I overreacted to a network outage. I was dumb and thought my Pi's DNS was screwing things up for the whole network, but it turns out that it was just Spectrum screwing things up for the entire region. Fortunately there was only one minor corrupted file, it didn't bork the SD card, and I was able to repair the setup manually.
I've never had any problem as long as I've stuck with the `sudo shutdown` command, which powers down in a controlled manner so writes don't get interrupted. But I've also never had an issue after a power outage, so I think somehow power outage shutdowns are more graceful than just yanking the power cable out.
If you're really concerned, you can always configure your Pi to use a read-only root filesystem. Combine that with a USB SSD to store your _actual_ data and you should be OK indefinitely. Or just boot from that USB SSD -- just remember to configure fstab correctly for your intended behaviour! You might be surprised to discover that fstab can delay a boot indefinitely if mounting expectations do not match reality.
I've never had any problem as long as I've stuck with the `sudo shutdown` command, which powers down in a controlled manner so writes don't get interrupted. But I've also never had an issue after a power outage, so I think somehow power outage shutdowns are more graceful than just yanking the power cable out.
If you're really concerned, you can always configure your Pi to use a read-only root filesystem. Combine that with a USB SSD to store your _actual_ data and you should be OK indefinitely. Or just boot from that USB SSD -- just remember to configure fstab correctly for your intended behaviour! You might be surprised to discover that fstab can delay a boot indefinitely if mounting expectations do not match reality.
I’ve had 3 SD cards fail on me in the last year. I now avoid using them as serious long term storage.
To be fair, these SD cards were exposed to fairly extreme Texas temperatures. One in a car dashcam, the other in an outdoor camera.
To be fair, these SD cards were exposed to fairly extreme Texas temperatures. One in a car dashcam, the other in an outdoor camera.
As to your direct question: I used some Pis for TV dashboards at work and had some random bit flips on the SD card and corrupted files for the dashboards. It may be "rare," but seems inevitable on a long enough timeline. For toy projects where you can re-image the SD card it's alright, but even for my trivial personal projects it made me uneasy.
100% to backups. I know we all put off doing it, but you'll rest a lot easier, even with personal data you don't think you care about. It's not only about a hardware failure, but even a fluke sysadmin error where you accidentally nuke something. I'd recommend getting a account for Backblaze B2, and setting up restic on each Pi to at least daily backup the data directories and stuff you care about. For your Gitlab it's a bit less risky since presumably you also have a clone of each repo on some other machine.
I love that people are building small datacenters out of Pis. I haven't done the math as far as TCO, but instead of multiple Pis for self-hosting, I have a lonely secondhand Dell Precision with an old 8th gen Intel CPU (6C/12T), 64GB of RAM, and several TB of NVMe plus some spinning rust for the long term stuff. It's just a crazy amount of horsepower. Most trusted workloads run as containers, and my other experiments can run as VMs, and I have capacity in all the right places (I need disk and RAM more than CPU). Not as exciting as building a cluster, but I have the excess capacity to spin up multiple VMs on that one machine, if I want to play with that. It can get very Inception-like, what if I'm running VMs in KubeVirt on top of Kubernetes that is running on a cluster of VMs that are ultimately on a single machine, but while delegating whatever extra /64 IPv6 prefixes Comcast gave me to each of the bottom-layer VMs so that each pod still gets a globally routable IPv6 address. Cool times for the homelab stuff, and helped me understand things like Kubernetes and IPv6 to a much greater depth.
100% to backups. I know we all put off doing it, but you'll rest a lot easier, even with personal data you don't think you care about. It's not only about a hardware failure, but even a fluke sysadmin error where you accidentally nuke something. I'd recommend getting a account for Backblaze B2, and setting up restic on each Pi to at least daily backup the data directories and stuff you care about. For your Gitlab it's a bit less risky since presumably you also have a clone of each repo on some other machine.
I love that people are building small datacenters out of Pis. I haven't done the math as far as TCO, but instead of multiple Pis for self-hosting, I have a lonely secondhand Dell Precision with an old 8th gen Intel CPU (6C/12T), 64GB of RAM, and several TB of NVMe plus some spinning rust for the long term stuff. It's just a crazy amount of horsepower. Most trusted workloads run as containers, and my other experiments can run as VMs, and I have capacity in all the right places (I need disk and RAM more than CPU). Not as exciting as building a cluster, but I have the excess capacity to spin up multiple VMs on that one machine, if I want to play with that. It can get very Inception-like, what if I'm running VMs in KubeVirt on top of Kubernetes that is running on a cluster of VMs that are ultimately on a single machine, but while delegating whatever extra /64 IPv6 prefixes Comcast gave me to each of the bottom-layer VMs so that each pod still gets a globally routable IPv6 address. Cool times for the homelab stuff, and helped me understand things like Kubernetes and IPv6 to a much greater depth.
I get it, self hosting is fun (until user count is > 1), but I wonder how long a $6 VPS would be able to handle the same load before you'd "break even" with the cost of 4 Pi's, as well as the ~12W required to run them, as well as the 5-7W required to power the switch.
Assuming it's 20W in total, that's 14.5 kWh/month, which even as US prices of $0.15/kWh adds up to $2.15 per month. In Europe you're closer to €5/month (€0.35/kWh)
Assuming it's 20W in total, that's 14.5 kWh/month, which even as US prices of $0.15/kWh adds up to $2.15 per month. In Europe you're closer to €5/month (€0.35/kWh)
$6 vps probably has no storage. E.g 5tb hetzner box is more like $14 a month. Pis are absurd though for homelabbing today, prices have not fallen relative to other cheap compute. For the price of a $90 pi you can get a used mini pc with an i5 8500 with memory and storage on ebay today. Double that and you can get a new asus minipc with an 8 core 9700t. All with a case and psu. The asus minis are even designed to stack nicely as a cluster arrangement.
From my experience, being comfortable with networks and systems outside the cloud is a valuable skill that not a ton of people possess. It may be surprising to some of you, but there is an entire generation of syadmin/devops that have never touched a switch or a bare metal server.
yup, i belong to the already-cloud generation, having a capable router+ a couple of raspberries have taught me a lot
When you put your data on a third party host, you can basically kiss it goodbye -- "trusting" that host means you have to trust every single employee in it.
For the peace of mind, I'd even pay more -- but it's significantly cheaper in the long run to self host since those 'cheap shared-resource' virtual servers are always oversold.
But to each his own.
For the peace of mind, I'd even pay more -- but it's significantly cheaper in the long run to self host since those 'cheap shared-resource' virtual servers are always oversold.
But to each his own.
> When you put your data on a third party host, you can basically kiss it goodbye -- "trusting" that host means you have to trust every single employee in it.
Or you encrypt your data as well as backup your data somewhere else.
Besides, on the scale we're talking here, a workload that can be handled by 3 RPI's (or a VPS), do you really have such sentivie data ?
Or you encrypt your data as well as backup your data somewhere else.
Besides, on the scale we're talking here, a workload that can be handled by 3 RPI's (or a VPS), do you really have such sentivie data ?
Cool stuff! I built a similar cluster of Orange Pi 5s with NVMe drives. For IP masking I'm paying for a $3.50 VPS to have nginx forward raw packets.
Fwiw Cloudflare Proxy and, to a lesser extent Tunnels, are de facto MITM attacks. I don't exactly trust large unprofitable corporations to do the right thing when they decrypt all of my and my users' traffic.
Not that CF is a malicious entity, but, the potential for them to be used as one is a looming threat. Then again, so is any VPS you host on :^)
Fwiw Cloudflare Proxy and, to a lesser extent Tunnels, are de facto MITM attacks. I don't exactly trust large unprofitable corporations to do the right thing when they decrypt all of my and my users' traffic.
Not that CF is a malicious entity, but, the potential for them to be used as one is a looming threat. Then again, so is any VPS you host on :^)
Super fun that you have more worker nodes than websites lol
Neat to see Docker Swarm in use. Don’t hear much about it since K8s took over.
Neat to see Docker Swarm in use. Don’t hear much about it since K8s took over.
Industry-wise Docker Swarm is somewhere between "on life support" and "shambling corpse" and has been for a while.
I'd say for at least as long as since the Mirantis acquisition in 2019.
I mean in all honesty running docker swarm instead of k3s or k8s seems ... i dunno, lol, i've slowly moved away from using docker even on my single hosts i run podman now instead, i like using systemd for containers :S
the hosting a blog on kubernetes joke managed to get a giggle out of me
https://twitter.com/dexhorthy/status/856639005462417409
https://twitter.com/dexhorthy/status/856639005462417409