My Home Server Journey – From Raspberry Pi to Ryzen(blog.viktorpetersson.com)
blog.viktorpetersson.com
My Home Server Journey – From Raspberry Pi to Ryzen
https://blog.viktorpetersson.com/2024/05/04/home-server-journey.html
16 comments
Unfortunately that board, like pretty much all consumer-like boards, suffers from a lack of PCI-Express lanes.
It has:
- 1x Gen4 x16
- 1x Gen4 x4
- 1x M.2 Gen3 x1
That M.2 slot can do only 0.985 GB/s, so literally all NVMe SSDs on the market will be limited by that. Maybe useful for a boot drive, but that's about it.
Assuming you want somewhat-decent storage your only realistic option is getting an x16-to-4x-M.2 adapter card, and stick at least two NVMe drives in there.
The board has pretty lackluster network interfaces, so you want to get an external NIC. That probably means grabbing a cheap second-hand 40G NIC from Ebay. But those are designed for PCI-E Gen3, so sticking it in your only remaining (x4) slot means it is limited to 31.5Gbps - in a best-case scenario...
With consumer CPUs it's all about compromises, and nobody seems willing to invest in making a low-volume homelab-focused board.
It has:
- 1x Gen4 x16
- 1x Gen4 x4
- 1x M.2 Gen3 x1
That M.2 slot can do only 0.985 GB/s, so literally all NVMe SSDs on the market will be limited by that. Maybe useful for a boot drive, but that's about it.
Assuming you want somewhat-decent storage your only realistic option is getting an x16-to-4x-M.2 adapter card, and stick at least two NVMe drives in there.
The board has pretty lackluster network interfaces, so you want to get an external NIC. That probably means grabbing a cheap second-hand 40G NIC from Ebay. But those are designed for PCI-E Gen3, so sticking it in your only remaining (x4) slot means it is limited to 31.5Gbps - in a best-case scenario...
With consumer CPUs it's all about compromises, and nobody seems willing to invest in making a low-volume homelab-focused board.
> Unfortunately that board, like pretty much all consumer-like boards, suffers from a lack of PCI-Express lanes.
Depending on the use case, this is true.
> That M.2 slot can do only 0.985 GB/s, so literally all NVMe SSDs on the market will be limited by that. Maybe useful for a boot drive, but that's about it.
Yeah I stated speed limited, but you are correct.
> The board has pretty lackluster network interfaces, so you want to get an external NIC
This is also true (in my post I wrote 2.5G interface, actually this was a typo, I meant 2x1G interface).
> That probably means grabbing a cheap second-hand 40G NIC from Ebay. But those are designed for PCI-E Gen3, so sticking it in your only remaining (x4) slot means it is limited to 31.5Gbps - in a best-case scenario...
Ok, this might be not the normal persons use case... I think 10G is already pretty unusual in the homelab of most people, whereas I would say that 2.5G should be standard.
> it's all about compromises
That's the point. You can't get "fast, cheap and good" - I think the MC12 is a pretty good compromise. Sure, you can get a Supermicro X13SAE or ASRock IMB-X1314 (Intel, W680) - probably higher quality and far more reliable, but limited bifurcation and only the board would cost ~450 bucks. Another problem with the Supermicro may be that DDR5 ECC RAM is hardly available and expensive.
Depending on the use case, this is true.
> That M.2 slot can do only 0.985 GB/s, so literally all NVMe SSDs on the market will be limited by that. Maybe useful for a boot drive, but that's about it.
Yeah I stated speed limited, but you are correct.
> The board has pretty lackluster network interfaces, so you want to get an external NIC
This is also true (in my post I wrote 2.5G interface, actually this was a typo, I meant 2x1G interface).
> That probably means grabbing a cheap second-hand 40G NIC from Ebay. But those are designed for PCI-E Gen3, so sticking it in your only remaining (x4) slot means it is limited to 31.5Gbps - in a best-case scenario...
Ok, this might be not the normal persons use case... I think 10G is already pretty unusual in the homelab of most people, whereas I would say that 2.5G should be standard.
> it's all about compromises
That's the point. You can't get "fast, cheap and good" - I think the MC12 is a pretty good compromise. Sure, you can get a Supermicro X13SAE or ASRock IMB-X1314 (Intel, W680) - probably higher quality and far more reliable, but limited bifurcation and only the board would cost ~450 bucks. Another problem with the Supermicro may be that DDR5 ECC RAM is hardly available and expensive.
I’m sorry i can’t upvote you more than once.
I have had a similar path in the past, I've started with some Raspi 1, than suffer their little horsepower, tried a minipc then suffer it's limited expandability than I switch to an assembled desktop rackmounted as a short-depth 5U. Intel in my case, but that's is.
In the end modern iron eat not too much energy, do not produce too much heat, is not super cheap, but compared to modern raspi and co prices and their spec... It's worth the investment AND we damn need homeservers in this dark uncivilized modern mainframe world.
The next real issues are:
- having IPv6 with a global per host, to ease connections;
- ready made services, for instance we have a handful self-hosted VoIP solution NO ONE OF THEM easy to deploy, with many dependencies and poor results, and no I'm not talking about Big Blue Button but simple WebRTC apps, services for IoT, meaning hw using open protocols (MQTT, the old ModBUS, Kafka, Matter, CoIoT etc) with something possibly more manageable than Home Assistant or OpenHAB;
- get people to the idea that's the time to have a homeserver per home, for their own services and internal needs.
In the end modern iron eat not too much energy, do not produce too much heat, is not super cheap, but compared to modern raspi and co prices and their spec... It's worth the investment AND we damn need homeservers in this dark uncivilized modern mainframe world.
The next real issues are:
- having IPv6 with a global per host, to ease connections;
- ready made services, for instance we have a handful self-hosted VoIP solution NO ONE OF THEM easy to deploy, with many dependencies and poor results, and no I'm not talking about Big Blue Button but simple WebRTC apps, services for IoT, meaning hw using open protocols (MQTT, the old ModBUS, Kafka, Matter, CoIoT etc) with something possibly more manageable than Home Assistant or OpenHAB;
- get people to the idea that's the time to have a homeserver per home, for their own services and internal needs.
I'd definitely recommend anyone working in tech, but especially software engineers, to run their own fleets. Mine started similar to this one a decade ago as a home lab. I learned a lot from that lab and now it's more of a single datacenter cloud running XCP-ng from what used to be a collection of SSH Docker Compose hosts.
Another approach is to use HP Microserver. It provides a "real" server with options to use Xeon, ECC RAM, iLO (remote KVM) in a relatively compact and silent package and not very expensive price.
If you want to use self-built server, you might be interested in pikvm project which allows to build a remote KVM using Raspberry Pi. Remote KVM is very useful for home servers: you can put it in the shelf and use it remotely, including things like tinkering with BIOS, reinstalling OS and resetting.
Also do not forget about fire hazards. The thing that works 24/7 can burn your house. Keep that in mind.
If you want to use self-built server, you might be interested in pikvm project which allows to build a remote KVM using Raspberry Pi. Remote KVM is very useful for home servers: you can put it in the shelf and use it remotely, including things like tinkering with BIOS, reinstalling OS and resetting.
Also do not forget about fire hazards. The thing that works 24/7 can burn your house. Keep that in mind.
Similarly, I got one of these refurbished Lenovo tiny desktops. Real PCI Express lanes, NVMe storage, etc etc.
Lenovo ThinkCentre M900 Tiny Desktop (Intel Core i5-6500T Upto -3.1 GHz, 16 GB Ram
https://a.co/d/aoZyBft
So much more capable than a cheaper single board computer.
Lenovo ThinkCentre M900 Tiny Desktop (Intel Core i5-6500T Upto -3.1 GHz, 16 GB Ram
https://a.co/d/aoZyBft
So much more capable than a cheaper single board computer.
How's the performance?
It looks like a fantastic alternative to partitioning a disk for having multiple OSs. Just buy a few of these things and run them simultaneously.
It looks like a fantastic alternative to partitioning a disk for having multiple OSs. Just buy a few of these things and run them simultaneously.
I haven't had any complaints, it's been plenty fast enough for multiple VMs, services, containers, and databases.
Yeah I did evaluate a used SuperMicro server, but couldn't find any available in the UK with the specs I wanted.
Another variable is the power draw. Commercial servers tend to have a lot larger power draw (in particular say Xeon based ones).
Good shout on pikvm. I did have a look at that some time ago, but largely forgot about it.
Another variable is the power draw. Commercial servers tend to have a lot larger power draw (in particular say Xeon based ones).
Good shout on pikvm. I did have a look at that some time ago, but largely forgot about it.
I was honestly shocked at just how much compute you get for very little money on enterprise surplus servers. $150 on eBay for a dell R620, two 16-core Xeons and 64GB of RAM.
Sure it's dated, but I still daily drive a quad core thinkpad from 2013, I'm not bothered. It's got more power than any machine I've owned up to this point. I don't have any use for more power, this guy is usually around 15% utilization. And someone just threw it away. Incredible.
Sure it's dated, but I still daily drive a quad core thinkpad from 2013, I'm not bothered. It's got more power than any machine I've owned up to this point. I don't have any use for more power, this guy is usually around 15% utilization. And someone just threw it away. Incredible.
The problem is size, power consumption, and noise. A Raspberry Pi is going to consume about 5W, and depending on your choice of case might not even have a fan at all. I literally bolted mine to the wall of a small utility closet - out of sight, out of mind.
A Dell R620 consumes about 100W at best, but more realistically 200W. Where I live that's $650 in electricity per year. It generates enough heat that you don't really want to stick it in a ventilation-less closet, and it's big enough that I'd have a hard time fitting it anyways. Worst of all, those tiny 1U fans are going to be screaming in the summer, so I really don't want a small quarter-height rack in my living spaces either.
A Dell R620 consumes about 100W at best, but more realistically 200W. Where I live that's $650 in electricity per year. It generates enough heat that you don't really want to stick it in a ventilation-less closet, and it's big enough that I'd have a hard time fitting it anyways. Worst of all, those tiny 1U fans are going to be screaming in the summer, so I really don't want a small quarter-height rack in my living spaces either.
My house is pretty old, so it has a cold cellar under the porch. It was originally intended to keep canned goods cool, I've never seen it get to 30°C, even in summer. It's also out of the way enough that I can close the door and the howling fans don't bother us in the rest of the house.
Yeah, the power use is a concern, but it's well within my hobby budget. I wish I'd gone for a bigger unit with proper 3.5" drive bays, but this is what I got. Normal load is something around 100-200W.
One day I'll upgrade to a more efficient machine, but I'm happy with this for now.
Yeah, the power use is a concern, but it's well within my hobby budget. I wish I'd gone for a bigger unit with proper 3.5" drive bays, but this is what I got. Normal load is something around 100-200W.
One day I'll upgrade to a more efficient machine, but I'm happy with this for now.
I'm hoping the Aoostar 6 bay NAS isn't vaporware, but many previous ones have been, in no particular order:
- wiretrustee NAS
- pibox 5 bay
- storaxa 5 bay (scam/company disappeared into the night)
- axzez cm4 NAS
- U-NAS (case only)[deleted]
However, there is a board, that is pretty good in all categories, the
Costs around 50-70 bucks, supports IPMI (remote management), Bifurcation, ECC-RAM, has 2.5G Ethernet and one NVMe port (although it is speed limited). With one PCIe card for about 20 bucks you can add 2 full speed NVMe. Medioker Idle Power consumption and strange stability issues are the "only" drawbacks - with a modern CPU (e.g. Ryzen Pro 5650) it's pretty hard to get under 20W Idle. Although many people report it works rock solid, there are also many reports of having issues with randomly turning off, BIOS upgrade fails and other hardware issues.
There are indeed Ryzen B550 boards, that unofficially support ECC and have lower power consumption than the MC12, but usually these boards lack internal remote management options and other server benefits.
Low power boards like Asrock N100DC-ITX are good, but lack support of SATA Ports and ECC RAM.
Industrial Boards like the ASRock IMB-X1231 (or other ASRock RACK / IMB series boards) are either way too expensive or impossible to buy anywhere.
Intel W680 Chipset boards are too expensive and ARM alternatives (e.g. Friendly Elec CM3588 NAS Kit, https://www.friendlyelec.com/index.php?route=product%2Fprodu...) don't have support for ECC RAM, although OpenMediaVault is sometimes supported.
The "real deal" in my opinion are Fujitsu Boards with C246 chipset (e.g. Fujitsu D3644-B) or Gigabyte (C246N-WU2) - despite being older tech, these are really low on power consumption with pretty good performance and pretty stable. With these boards you can build a full steam spacemachine consuming <10W idle. Fujitsu also has remote management via Intel AMT (discontinued) or iRMT (a proprietary protocol).
Be aware that Fujtisu Celsius W550 Workstations and newer contain D3417, D3517 and similar Boards, but also a proprietary power header, so you can't use normal ATX Power Supplies without Adapter (e.g. Bojiadafast from Aliexpress).
So, my advice would be to get the Gigabyte MC12-LE0 or if available, Fujitsu D3x17/D3644-B series.