Raspberry Pi Server Mark III(uplab.pro)
uplab.pro
Raspberry Pi Server Mark III
https://uplab.pro/2020/12/raspberry-pi-server-mark-iii/
172 comments
Some usb drives are much slower than a decent sd card. Also, the ultra has cooling issues.
However, there are usb3 nvme enclosures that are incredibly cheap, and blow both out of the water:
https://www.jeffgeerling.com/blog/2020/fastest-usb-storage-o...
For a build like the one in the article, I imagine something that puts two nvme sticks beside each other, sandwiched between every other rpi, with passive aluminum heat shields in the remainder of the spaces between the rpi’s. I bet it could be even denser (more pis and more storage) and quieter.
I wonder if space can be saved by making a heat shield heatsink that has a pi on both sides, and fins in the middle.
Super cool article.
However, there are usb3 nvme enclosures that are incredibly cheap, and blow both out of the water:
https://www.jeffgeerling.com/blog/2020/fastest-usb-storage-o...
For a build like the one in the article, I imagine something that puts two nvme sticks beside each other, sandwiched between every other rpi, with passive aluminum heat shields in the remainder of the spaces between the rpi’s. I bet it could be even denser (more pis and more storage) and quieter.
I wonder if space can be saved by making a heat shield heatsink that has a pi on both sides, and fins in the middle.
Super cool article.
Yeah, I recently spied the 4 node Jetson Mate carrier board on CNX that facilitates building your own Multi GPU cluster ;)
https://www.seeedstudio.com/Jetson-Mate-Cooling-Kit-p-4784.h...
They recommend Kubernetes for private cloud workloads. But I am wondering if there isn't room in the ecosystem for a new virtualization / containerization technology to orchestrate tasks on multi gpus? Have heard NVidia is developing its own OS, or rather two, one for Cloud AI and one for real time robotics, but its still research phase...
https://www.seeedstudio.com/Jetson-Mate-Cooling-Kit-p-4784.h...
They recommend Kubernetes for private cloud workloads. But I am wondering if there isn't room in the ecosystem for a new virtualization / containerization technology to orchestrate tasks on multi gpus? Have heard NVidia is developing its own OS, or rather two, one for Cloud AI and one for real time robotics, but its still research phase...
That's a slick cluster in a box. I'm not sure which modules I'd put on it, though. Unless you harvest them from the 2 GB dev kits, the total price pushes up against a single Jetson Xavier AGX, which is a better value (and has an M.2 slot for an SSD).
yeah, I'm excited to get my hands on some Jetsons too. the 4gb memory on the pi is really a big pro though. when one of these SBCs finally comes with 8GB and GPU acceleration for < 100$ we will see some real progress I think.
> when one of these SBCs finally comes with 8GB and GPU acceleration for < 100$ (...)
It might have GPU acceleration but the RPi already comes with 8GB of RAM for about 80$.
https://www.raspberrypi.org/blog/8gb-raspberry-pi-4-on-sale-...
It might have GPU acceleration but the RPi already comes with 8GB of RAM for about 80$.
https://www.raspberrypi.org/blog/8gb-raspberry-pi-4-on-sale-...
Pi comes with 8gb ram now
Example case which does away with the need for fans is:
https://www.amazon.com/Geekworm-Raspberry-Computer-Aluminum-...
https://www.amazon.com/Geekworm-Raspberry-Computer-Aluminum-...
I've been using the FLIRC case, which appears to be die-cast and looks a little bit more family friendly sitting next to the TV on your entertainment center.
https://flirc.tv/more/raspberry-pi-4-case https://flirc.tv/more/raspberry-pi-case (for RPi 2 and 3)
https://flirc.tv/more/raspberry-pi-4-case https://flirc.tv/more/raspberry-pi-case (for RPi 2 and 3)
I like this one, a little more expensive but makes the Pi into a NUC style SFFPC:
https://www.waveshare.com/pi4-case-argon-one.htm
Edit to add the fan is optional, so it can run fanless with no throttling.
https://www.waveshare.com/pi4-case-argon-one.htm
Edit to add the fan is optional, so it can run fanless with no throttling.
looks cool, and the breakout board is nice, but they don't show how it attaches to the pi. it's aluminum yeah, but does it actually do anything for heat dissipation?
Scroll down that page, there’s a photo and description... “The Extension Pillars Makes The Aluminum Case Become A Big Heatsink”
That’s good value with the 2 hdmi adapters
I use one of those for my headless dedicated Pi4B website server plus UPS controller.
Admittedly, that's a simple load, but my Pi never gets over 42 degrees C, in a tropical area, averaging around 30 degrees C.
Admittedly, that's a simple load, but my Pi never gets over 42 degrees C, in a tropical area, averaging around 30 degrees C.
>Fans are completely unnecessary on a Pi4 with an aluminum heat shield style "case". They do run hotter than with a fan, but they will not be heat throttled with recent firmware.
1 x 4k@60hz with no case and alu heat shield and my RPI4 does throttle. Latest firmware. Seems you don't punish it enough.
1 x 4k@60hz with no case and alu heat shield and my RPI4 does throttle. Latest firmware. Seems you don't punish it enough.
That's the GPU which I would expect to be fairly idle on a cluster of any sort unless you're doing GPU computing (and then why are you using a Pi?)
I used the following case/setup and can confirm. No fans are required and booting is lightning fast. https://www.prusaprinters.org/prints/40852-pi4-case-stand
USB isn't very good for high vibration environments (e.g. inside robots) though :-/ Worst abomination of a connector design I've seen in a couple decades. I wish they had locking industrial connectors on the USB ports on the Pi ... although a SATA/m.2 would be nicer.
I guess that's why you'd move to the compute module?
Does a little drop of glue fix the issue, or is there something else?
Not really, because that makes it really hard to unplug/replug for servicing.
I wish USB was designed to at least snugly support housing to housing of connectors and not have the entire connector supported by a flimsy piece of sheet metal soldered to a PCB.
I wish USB was designed to at least snugly support housing to housing of connectors and not have the entire connector supported by a flimsy piece of sheet metal soldered to a PCB.
Which USB thumb drives and which SD cards?
Do you overclock?
If you ran at 99-100% CPU on average, would you still not need fans?
Do you overclock?
If you ran at 99-100% CPU on average, would you still not need fans?
any usb 3.0 drive works, but I am using SanDisk Ultra Fits. price per byte is less and performance is similar or slightly better for reads and way better for sustained writes.
No Overclock.
99-100% CPU on all 4 cores for hours and no throttling. they do run at 75-78C instead of 65C, but I could not get them to throttle.
Even with the cheapo 3 piece heat sinks there was almost no throttling, but they did throttle occasionally after sustained stress.
No Overclock.
99-100% CPU on all 4 cores for hours and no throttling. they do run at 75-78C instead of 65C, but I could not get them to throttle.
Even with the cheapo 3 piece heat sinks there was almost no throttling, but they did throttle occasionally after sustained stress.
[deleted]
missed the SD cards. I am benchmarking against the best SD cards I can get because that is always the bottleneck on the Pi to my experience. The bandwidth and clock rate on USB3 vs sdcard bridge is just faster, but the media is also cheaper in USB3 format in general.
Hi! thanks for posting my project here ) I will be glad if you subscribe to my instagram https://www.instagram.com/uptime.lab/ I don't sell anything, but feedback is very important for me
Just did, always like subscribing to pages that add a bit of education to my feed
What could you do with a similar setup? I mean: what's the purpose and/or the advantages compared to a "classical" solution ?
Purpose: Learn linux clustering/diskless nodes.
Advantages: Cheaper than conventional servers, limited physical form factor suitable for constrained environments such as apartments, luggage, etc. Low power consumption, suitable for mobile/off-grid/renewable.
Recommended software: https://clusterlabs.org https://www.kernel.org/doc/html/latest/admin-guide/nfs/nfsro... https://wiki.gentoo.org/wiki/Diskless_nodes
Commercial advantage: Run highly cost and power efficient data centers, call centers, etc. Eliminate redundant state. Remove manual configuration rollout. Infrastructure as code. Full CI/CD with kernel-level control, per node, for any OS. Basically becoming a lost art in this day of "outsource to cloud". With SDN knowledge, confidently build your own cloud provider.
Advantages: Cheaper than conventional servers, limited physical form factor suitable for constrained environments such as apartments, luggage, etc. Low power consumption, suitable for mobile/off-grid/renewable.
Recommended software: https://clusterlabs.org https://www.kernel.org/doc/html/latest/admin-guide/nfs/nfsro... https://wiki.gentoo.org/wiki/Diskless_nodes
Commercial advantage: Run highly cost and power efficient data centers, call centers, etc. Eliminate redundant state. Remove manual configuration rollout. Infrastructure as code. Full CI/CD with kernel-level control, per node, for any OS. Basically becoming a lost art in this day of "outsource to cloud". With SDN knowledge, confidently build your own cloud provider.
> Learn linux clustering/diskless nodes.
Honestly, could be as valid with a setup that just involves running a bunch of VMs.
> Low power consumption
Yeah, well, a typical laptop should draw around 50W, which is what, like 0.05$ a day?
> Commercial advantage: Run highly cost and power efficient data centers
Hu, no, in terms of power efficiency per unit of work, you will not beat specialized hardware.
> Infrastructure as code. Full CI/CD with kernel-level control, per node, for any OS. With SDN knowledge, confidently build your own cloud provider.
Sure, also, blockchain, 5G, serverless, microwaves, rust, wasm
Lets come back to earth for a second. There is objectively no purpose for this, except it's fun and nerdy. Do we really have to find excuses?
Honestly, could be as valid with a setup that just involves running a bunch of VMs.
> Low power consumption
Yeah, well, a typical laptop should draw around 50W, which is what, like 0.05$ a day?
> Commercial advantage: Run highly cost and power efficient data centers
Hu, no, in terms of power efficiency per unit of work, you will not beat specialized hardware.
> Infrastructure as code. Full CI/CD with kernel-level control, per node, for any OS. With SDN knowledge, confidently build your own cloud provider.
Sure, also, blockchain, 5G, serverless, microwaves, rust, wasm
Lets come back to earth for a second. There is objectively no purpose for this, except it's fun and nerdy. Do we really have to find excuses?
> > Learn linux clustering/diskless nodes.
> Honestly, could be as valid with a setup that just involves running a bunch of VMs.
Slightly less valid. You don't have the same IO latency between nodes using VMs unless you start messing around with artificial attempts at inducing latency on each node individually.
It's much easier to accurately simulate real world networking by not bothering to simulate and just using real world networking.
There's a huge difference between latency on a wire vs latency within a hypervisor, and you won't know what you've papered over through lower latency via VM until you move to real hardware.
> Honestly, could be as valid with a setup that just involves running a bunch of VMs.
Slightly less valid. You don't have the same IO latency between nodes using VMs unless you start messing around with artificial attempts at inducing latency on each node individually.
It's much easier to accurately simulate real world networking by not bothering to simulate and just using real world networking.
There's a huge difference between latency on a wire vs latency within a hypervisor, and you won't know what you've papered over through lower latency via VM until you move to real hardware.
To be clear, the commercial utility refers to the skills, not the hardware.
> Yeah, well, a typical laptop should draw around 50W, which is what, like 0.05$ a day?
At least where I live that would be 1.2kWh and about 13 cents per day, around $4 per month, or around $50 per year if running 24x7.
Pacific Northwest has I believe some of the cheaper power in the U.S. at 10-11 cents per kilowatt hour.
At least where I live that would be 1.2kWh and about 13 cents per day, around $4 per month, or around $50 per year if running 24x7.
Pacific Northwest has I believe some of the cheaper power in the U.S. at 10-11 cents per kilowatt hour.
[deleted]
Thanks for this, the other comments had left me, a layman, thinking this was actually useful instead of just for fun.
If you want to learn clustering with physical hardware, and you don't have the power and size constraints, you're better off buying some off-lease business desktops. These i3 and i5 systems with 4 to 8GB of RAM can be had for less than $100 each.
When you're done with the cluster, you can likely sell them for nearly what you paid for them.
When you're done with the cluster, you can likely sell them for nearly what you paid for them.
i think you will lose in the end this way if you actually factor in power costs.
Yup, 6x old i5 thinkpads will require more space, more power and will likely be more expensive than 6x Pi.
I'd take 6 old ThinkPad over 6 new Pi any day. :)
I host a lot out of my house on a T430S with a cracked screen.
I host a lot out of my house on a T430S with a cracked screen.
I know there are thinkpad hoarders with like 10 or 20+ old thinkpads. Seems to be a nice hobby, but you will rarely find one who builds a thinkpad cluster.
In the case of Raspberry Pi it is pretty normal nowadays to connect a bunch of them into a cluster and have some fun. In fact, it has become so popular, that you u can find stackable cluster cases and other Pi cluster hardware on amazon.
In the case of Raspberry Pi it is pretty normal nowadays to connect a bunch of them into a cluster and have some fun. In fact, it has become so popular, that you u can find stackable cluster cases and other Pi cluster hardware on amazon.
Not when you factor in performance per watt.
I’d love to see the M1 on this list.
[deleted]
You could rent a couple of
1USD/month VPS, or if really on a budget rent a bunch of containers with ipv6 for just a few cents
You absolutely could. But you wouldn't learn how to build a system that recovers from yanking a power or network cable, wouldn't own it, and wouldn't be able to run it on your terms. You'd need a credit card with an American-run payments network to operate, with a declared identity and a bank account, for which a prerequisite is a few months of stable living and income in a particular jurisdiction, itself something large portions of the world do not have access to. Further you'd need to be happy with the fact none of your data was in your country, under your physical control, and never would be. But feel free to ignore the rest of the world and the fact that all this rental crap runs on open source software, just throw in the towel and be a rent-payer: a strategy that has its perks and efficiencies, for sure. But don't discourage those who seek to learn more.
This sounds like some kind of fantasy; how many people without stable living conditions or without a bank account that can be processed by the US are concerned with...cluster computing?
"What about all the homeless Iranian Ferrari owners!" doesn't seem like much of a rallying cry.
"What about all the homeless Iranian Ferrari owners!" doesn't seem like much of a rallying cry.
Surely if there is even 1 unbanked person in existence that is interested in cluster computing, then this is a worthwhile endeavour.
No, absolutely not.
Or maybe to Don Quixote, but he was famously delusional.
Or maybe to Don Quixote, but he was famously delusional.
Is there any number of unbanked people showing interest in cluster computing that would change your opinion?
I "show an interest" in space, but I'm not entitled/deluded enough to expect someone to deliver a shuttle to me, because that would be a waste of time/materials/energy/money.
It's the same here.
It's the same here.
If you had spoken to people in the 1960s and told them how many transistors we would have on a chip in 2020, they probably would have called you delusional.
Also you are comparing a space shuttle to a few raspberry pis, hardly apples with apples.
Also you are comparing a space shuttle to a few raspberry pis, hardly apples with apples.
https://ipv6onlyhosting.com/ is the first result, and accepts payment by bank transfer.
Many other small VM hosts accept payment by Bitcoin.
Many other small VM hosts accept payment by Bitcoin.
Someone needs to invent a way to drop things off your desk onto a power switch via ssh.
> You'd need a credit card with an American-run payments network to operate, with a declared identity and a bank account, for which a prerequisite is a few months of stable living and income in a particular jurisdiction,
None of this is true.
None of this is true.
Try being from a random South Asian or African village. Tell me how you go buying cloud services with your nonexistant banking infrastructure and personal credit line.
The physical location of a person isn't generally the key barrier to being unbanked; instead it's poverty.
"The most common one was having too little money to use an account. Two-thirds cited this as a reason for not having a financial institution account, and roughly a fifth cited it as the sole reason. Cost and distance were each cited by about a quarter of those responding to the question, and a similar share said they do not have an account because a family member already has one. Lack of documentation and distrust in the financial system were both cited by roughly a fifth of adults without a financial institution account, and religious concerns by 6%."
https://www.cashmatters.org/blog/globally-17-billion-people-...
(And to state the obvious: poverty is also a barrier to owning physical infrastructure)
"The most common one was having too little money to use an account. Two-thirds cited this as a reason for not having a financial institution account, and roughly a fifth cited it as the sole reason. Cost and distance were each cited by about a quarter of those responding to the question, and a similar share said they do not have an account because a family member already has one. Lack of documentation and distrust in the financial system were both cited by roughly a fifth of adults without a financial institution account, and religious concerns by 6%."
https://www.cashmatters.org/blog/globally-17-billion-people-...
(And to state the obvious: poverty is also a barrier to owning physical infrastructure)
No, try buying cloud services while living in a remote village where clothing is non-existent.
One issue I have with going the VPS route is that what I'm charged is often not what I'm expecting. I will use estimation calculators and figure my bill will be $25/month (which I get for a couple months) and then suddenly get a $120 bill. This has happened more than once and has had me loose all confidence in these services for hobby projects. I'd rather just buy a Pi cluster at this point.
Your agreement was $25/month, so if they start to charge you $120 you pay them $25 and then cancel the service. This is very annoying but do happen dealing with unscrupulous businesses. Although if they have 500% margin eg. you pay $5 for something you can get for $1 elsewhere they will care more about keeping you as a customer. Compared to for example a cloud bill, renting a VPS is both cheaper and more predictive. If you are however paying $25/month you should start looking at dedicated servers, where all resources are preserved for you.
From my experience the cheapest reliable ones in terms of storage, cpu, ram performance come at about 3$/month (+-50 cent). I can recommend Hetzner.
Any links to an ok $1 VPS that would work? I tried a few a while back and always have an issue with RAM.
What's the best way to go about clusters since I think you need 1gb RAM per node and lowest I've seen is $5.
Thanks
What's the best way to go about clusters since I think you need 1gb RAM per node and lowest I've seen is $5.
Thanks
Where does one find VPSs this cheap?
A few years ago I could vouch for LowEndBox: https://lowendbox.com/. I've not frequented the site in a very long time though, so can longer be sure.
RamNode has some starting at $15/year on their OpenVZ platform. Have been happy enough with that for several years now, although it's quite limited as to what it can do in terms of performance. But as another commentor mentioned, check out LowEndBox. It's where I discovered this.
Is the power consumption really that good? I mean comparing to just getting a single 5950x or something would this actually win in terms of performance per watt?
I bought a Turing Pi 1 . It's a 7 node mini-cluster, not very expensive (I think about £300 fully populated), that fits into an MiniITX-style case with a single small power supply and single ethernet exposed to the world. You can easily pick the thing up with one hand. It's certainly not powerful, each node is a very modest ARM processor with "only" a gigabyte of RAM and 8 GB of local storage.
The alternatives are much more powerful, but far more expensive and not very attractive in other ways. Blade server? Huge, noisy, power-hungry and I wouldn't get much change from many thousands of pounds. Rack of NUCs? Powerful of course but far more expensive, large, separate power bricks, ethernet switch and lots of wiring required. VMs running on a desktop? Still more expensive, larger, and not the same tangible experience as having a real cluster. VMs on a laptop? Laptops at the same price point would have too little RAM to run lots of VMs well.
So these Raspberry Pi clusters fit a niche which is not well served by other solutions.
The alternatives are much more powerful, but far more expensive and not very attractive in other ways. Blade server? Huge, noisy, power-hungry and I wouldn't get much change from many thousands of pounds. Rack of NUCs? Powerful of course but far more expensive, large, separate power bricks, ethernet switch and lots of wiring required. VMs running on a desktop? Still more expensive, larger, and not the same tangible experience as having a real cluster. VMs on a laptop? Laptops at the same price point would have too little RAM to run lots of VMs well.
So these Raspberry Pi clusters fit a niche which is not well served by other solutions.
Its cool and fun and teaches you a skill at a low price point. There is essentially no functional reason to do this over a single high power server unless for some reason you need a bunch of ARM servers for something.
All fourteen Pis, even v4s (which are significantly hungrier), would be lower power (as in electrical draw) than even a feeble (as in compute power) traditional machine.
If you can distribute your workload (or it's not intensive anyway), i.e. having no single machine with a faster processor, then I think it's a good choice, cheaper to run.
Edit: I'm happy to be corrected if down-voters would like to reply... A Pi 4 draws 6-7W at full load. A former desktop PC turned server likely idles over 100W.
They're different beasts, so I'm not saying 'a Pi is better than a PC', I'm saying 'if your load could also be handled by a (bunch of) Pi(s), then all else being equal that might be better than the PC'. Of course all else is never equal, ymmv, you may already have the old PC, you may need very many Pis, so even if pennies cheaper to run it might take years and years to break even on the Pi acquisition cost.
If you can distribute your workload (or it's not intensive anyway), i.e. having no single machine with a faster processor, then I think it's a good choice, cheaper to run.
Edit: I'm happy to be corrected if down-voters would like to reply... A Pi 4 draws 6-7W at full load. A former desktop PC turned server likely idles over 100W.
They're different beasts, so I'm not saying 'a Pi is better than a PC', I'm saying 'if your load could also be handled by a (bunch of) Pi(s), then all else being equal that might be better than the PC'. Of course all else is never equal, ymmv, you may already have the old PC, you may need very many Pis, so even if pennies cheaper to run it might take years and years to break even on the Pi acquisition cost.
Don't forget that Pis also draw quite some power while idle. The Pi 4 draws around 3.4 Watts [1], or around 2.7 Watts since they are running headless in a server setup you'd probably disable HDMI [2]. Times fourteen that's 37.8 Watts just sitting there. An idling headless Desktop PC will be below that for sure.
[1] https://magpi.raspberrypi.org/articles/raspberry-pi-specs-be...
[2] https://www.pidramble.com/wiki/benchmarks/power-consumption
[1] https://magpi.raspberrypi.org/articles/raspberry-pi-specs-be...
[2] https://www.pidramble.com/wiki/benchmarks/power-consumption
It's not that difficult to have a reasonably modern desktop PC idling more in the 10-20W range. For 100W you need to do something seriously wrong, or add extras that would blow up the Pi usage too (e.g. a home server full of hard disks will of course use more power, but so would Pis with the same number of drives)
[deleted]
Did not downvote but desktops idle way below 100w.
Exactly. Take a look at the "ENERGY STAR ® Power and Performance Data Sheet" for Dell "office" desktops. Many are around 10 watts for idle.
Idle as in powered on, not sleeping, just 'no' workload? I stand corrected if so, I didn't realise that was the case.
I also like the idea of having an easy homogeneous setup, each Pi having one disk, vs. my alternative (old PC parts) being something like '4 SATA ports on this one, and 6 on that one, but one's broken, less one on each for the boot disk' - I quite like the idea of having a bunch of Pis, and one image that I apply to all of them, which can include details like /dev/sda being its only data disk. That's not really a functional advantage, just appealing for keeping it simple to come back to and remember/understand how I set it up when I need to tweak something (and really easy to add another one increasing available cpu/ram/disk).
I also like the idea of having an easy homogeneous setup, each Pi having one disk, vs. my alternative (old PC parts) being something like '4 SATA ports on this one, and 6 on that one, but one's broken, less one on each for the boot disk' - I quite like the idea of having a bunch of Pis, and one image that I apply to all of them, which can include details like /dev/sda being its only data disk. That's not really a functional advantage, just appealing for keeping it simple to come back to and remember/understand how I set it up when I need to tweak something (and really easy to add another one increasing available cpu/ram/disk).
This high end system draws ~35W idle: Intel Core i9-10900K @ 5,0 GHz, AMD Radeon RX 6900 XT (16 GB), Asus ROG Maximus XII Hero Wi-Fi, 2× 16 GB G Skill Trident Z Royal 3 600 MHz 16-16-16-36, Samsung 970 Evo M.2 1 TB, Samsung 860 Evo 1 TB, Seasonic Prime Ultra Titanium, 1 000 W.
Picture in Swedish, from graphics card review comparing average power draw for the entire system for various GPUs, idle and while benchmarking in Metro Exodus, numbers rounded to nearest multiple of 5: https://cdn.sweclockers.com/artikel/diagram/22452?key=2c6f21...
If the direct picture link doesn't work, it's under "Effektmätning" here: https://www.sweclockers.com/test/30908-amd-radeon-rx-6900-xt...
Picture in Swedish, from graphics card review comparing average power draw for the entire system for various GPUs, idle and while benchmarking in Metro Exodus, numbers rounded to nearest multiple of 5: https://cdn.sweclockers.com/artikel/diagram/22452?key=2c6f21...
If the direct picture link doesn't work, it's under "Effektmätning" here: https://www.sweclockers.com/test/30908-amd-radeon-rx-6900-xt...
A good functional reason is when you need to have high redundancy for low-intensity jobs while keeping power consumption ridiculously low. As long as you don't use SD cards, Pis can run for years without stability issues.
lol? what if you don't need a heap of compute power but require a lot of redundancy? A single server is not good for that...Also not every site has internet so yes having small redundant easily replaceable nodes on-site does have plenty of use cases.
Y'all negative folks against this seem to have very restricted use cases in mind. Not all redundant rack servers live in a 36U case in a air-conditioned internet connected server farm. There are plenty of use cases outside of this.
Y'all negative folks against this seem to have very restricted use cases in mind. Not all redundant rack servers live in a 36U case in a air-conditioned internet connected server farm. There are plenty of use cases outside of this.
It’s fun to hack together. Not everything needs a clear business case with strong ROI.
This. I think the Hacker News mindset has substantially shifted in the last few years from the "Hacker" mentality of doing things because they are fun and knowledgeable, and more towards Enterprise News..
Not everything needs a business case, some things (most things) should be done just because you want to see if you can.
If someone really needs to think in items of ROI and that everything needs a business case, then this ideology of doing things for fun and testing limits is the only way to really innovate.
Not everything needs a business case, some things (most things) should be done just because you want to see if you can.
If someone really needs to think in items of ROI and that everything needs a business case, then this ideology of doing things for fun and testing limits is the only way to really innovate.
Clustering SBC's is a great learning experience, and makes for a great budget friendlier home lab setup.
I have a friend that uses similar setups to tech kubernetess but previously he also used it as example for all sorts of distributed computing (when Hadoop was still a cool thing to do)
Could be used for testing hardware+firmware. Example connect 14 hardware revisions of a $SOME_HARDWARE to each RPi and test firmware changes.
You can neatly mount your Pis in your rack where you have the rest of your gear supported by network switches, power delivery, cabling, cooling, good internet connectivity connectivity (if housed in a machine room or colo) etc.
If you need a cheap dedicated (for privacy) server to run a mail server and perhaps a private cloud with Nextcloud, this is ideal.
Nextcloud would be much cheaper to run on a x86 board.
Which x86 board do you have in mind?
The RPi4 with a USB SSD should be fast enough and use little power.
The RPi4 with a USB SSD should be fast enough and use little power.
Can you expand on why?
My guess would be Nextcloud put a lot of load on both CPU and disk (for thumbnail generation, file encryption, etc.) making Raspberry Pi which lack performance in both front not very suitable for the task.
I tried running Nextcloud on Raspberry Pi 4 (4GB) a while ago, and hitting performance issue on SD card very fast (initial DB import take very long). I tried to solve this by moving all Nextcloud data into iSCSI disk but RAM & CPU became a problem once I started adding more files. After a week or so, I tore everything down and went back to a VPS instance.
It might works better with Raspberry Pi cluster on Kubernetes, though. (This very Raspberry Pi is now happily running as a DVB streaming server.)
I tried running Nextcloud on Raspberry Pi 4 (4GB) a while ago, and hitting performance issue on SD card very fast (initial DB import take very long). I tried to solve this by moving all Nextcloud data into iSCSI disk but RAM & CPU became a problem once I started adding more files. After a week or so, I tore everything down and went back to a VPS instance.
It might works better with Raspberry Pi cluster on Kubernetes, though. (This very Raspberry Pi is now happily running as a DVB streaming server.)
Try using a SSD connected via USB 3.0 on the RPi 4.
Thanks. I keep meaning to try it out but booting from an SSD instead. I was hoping anything intensive (e.g. thumbnail generation) would be a one-time thing so not noticeable in daily usage. Perhaps I will re-think, I have a ThinkCentre M75q I could use but that seems overkill.
I find you can get a much better experience with a cheap celeron NUC than a RPi because everything from single thread performance to IO speeds favor x86 right now.
[deleted]
That is a damn good looking 3d print wow.
My only beef with RPIs as servers is disk I/O is terrible. If that's been solved, I'd love to know! If the SOC has PCI-E lanes that were accessable externally I feel like the server side would really take off
My only beef with RPIs as servers is disk I/O is terrible. If that's been solved, I'd love to know! If the SOC has PCI-E lanes that were accessable externally I feel like the server side would really take off
The Compute Modules have (optionally I think) eMMC which is not especially fast, but fast enough and appears much more reliable than SD cards.
The RPi 4 also has support for USB 3 disks using UAS. I've attached a couple of SSDs to mine. https://rwmj.wordpress.com/2020/09/24/raspberry-pi-4-running...
The RPi 4 also has support for USB 3 disks using UAS. I've attached a couple of SSDs to mine. https://rwmj.wordpress.com/2020/09/24/raspberry-pi-4-running...
I think recent firmwares would allow Sd-card-less pxe boot. I was wondering the other day if you could then run the whole system from RAM disk only and/or maybe mount volumes via iSCSI from SAN (keeping in mind limitations of RPi 1Gb nic).
Edit: someone has done this already. Of course! :) ( https://tech.xlab.si/blog/pxe-boot-raspberry-pi-iscsi/ )
Edit: someone has done this already. Of course! :) ( https://tech.xlab.si/blog/pxe-boot-raspberry-pi-iscsi/ )
Could also use the sd card to boot only, then pass everything to a USB3 SSD.
or just boot usb :)
Yes, with the compute module. https://pipci.jeffgeerling.com/
Surely one could make do with passive heatsinks on the Pis and use the back coolers?
Posted before I saw this comment. Yes passive cooling is fine with recent firmware and a good heatsink case. I have stress tested with the following case for hours with all cores pegged at 100% and there is no CPU throttling with passive cooling. They do run hotter, but they do not throttle. Old firware and bad cases will though.
https://www.amazon.com/Geekworm-Raspberry-Computer-Aluminum-...
You wouldn't need that much in their case (sorry) as they have through flow.
The official PoE 'hat' (add-on module) comes with that fan, I believe that's what's shown.
The fan official PoE hat is way too small so annoyingly noisy, and because it's so small replacements are expensive
There's room for a much larger fan so it seems a case of bad design by Raspberry Pi
There's room for a much larger fan so it seems a case of bad design by Raspberry Pi
This hat seems to be built for a fanless case. Pretty sure a heatsink protruding through the hat orifice would be enough in their case design.
The Raspberry Pi Compute Module 4 with support for PCI Express will be ideal for something like this.
If only there was a standard way of communicating the cost-effectiveness of these things ...
PoE Ethernet would mean just one cable to each device.
Hmm ... seems like you can also network boot them over DHCP/TFTP:
https://www.raspberrypi.org/documentation/hardware/raspberry...
Hmm ... seems like you can also network boot them over DHCP/TFTP:
https://www.raspberrypi.org/documentation/hardware/raspberry...
keep in mind that PoE requires an additional PoE hat. That adds another 20$ to the bill for each Pi.
https://www.sparkfun.com/products/14882
https://www.sparkfun.com/products/14882
What are some use cases for this ?
Pi's don't last as long as traditional hardware, but if you are running a self-healing cluster of machines, why wouldn't you use this for production environments?
Why there is so much love for such project where someone connected a couple of ready made things (maybe except the case) and everyone gets excited?
The hotswap 3D printed mechanism is pretty neat (reminds me of the swappable harddrive days). As for Rpi clusters, its just a neat idea to think about. It's not entirely trivial hardware or software wise to load balance 14 Rpis for "one purpose".
Why are pop music, fast food and sitcoms so extremely popular? They maximize a small amount of appeal across a broad group.
Every step deeper, or obscure loses you more and more mass appeal.
Therefore, projects with high max appeal make it to the top of lists like HN. It's no conspiracy.
Every step deeper, or obscure loses you more and more mass appeal.
Therefore, projects with high max appeal make it to the top of lists like HN. It's no conspiracy.
Pop music is extremely difficult to make although the end product looks "simple".
I think for me there's just a hobby/cool factor. I have no intention to build one nor can I think of a good use case but it's still neat to look at someone else's handiwork.
You are describing the elements of most of the best ideas in history.
I tried to setup something like this, but let's be frank. The performance of this thing makes this largely pointless. I was able to learn much more by setting up virtual machines and configuring them into a cluster. Only advantage of RPi cluster is that you can physically pull the Ethernet plug and test the failures. But that's more like a "fun" factor rather that utility.
Very cool. Do I understand correctly that there's no traditional backplane on this, power and data are both provide via ethernet?
The Pi does not do poe natively. POE is available as a hat, but I have not seen one for the Pi4 yet. The hats are nearly as expensive as the Pi itself.
I'm using the existing PoE hat on several Pi4's in a production environment and it works great. Never had shutdowns or issues with power. Note that I'm not drawing extra power from any of the USB ports, so perhaps the existing PoE hat isn't sufficient when you have USB devices attached to the pi4.
Good to know. I couldn't see any reason that they wouldn't work for a Pi4, but I couldn't find a single manufacturer that said Pi4 was actually supported.
Hat on the official site [1] explicitly says the 4B is supported.
[1] https://www.raspberrypi.org/products/poe-hat/
[1] https://www.raspberrypi.org/products/poe-hat/
The PoE Hats are $15-30 on Amazon. That makes the most expensive one 30% of the price of a Pi4 8gb.
This is so exciting. Is this type of "movement" (trend) likely to see data centers coming back on premises for enteprise markets? (ie. back from cloud)
Cloud vs. datacenter is entirely about administration, rather than hardware.
I would say it's more about elasticity and the ability to scale horizontally if you take the minimum amount of precautions when designing your app.
This is what an engineer would say and that's important but, behind the scenes it's all about operating costs (e.g. administration). I worked with Cisco on a contract, they had about 22 folks managing the day to day operation for a specific on premise integration. Imagine if Cisco didn't have to pay 22 folks to manage the day to day? Let's say we cut that need in half but the licensing price remained the same or you could repurpose good engineers to deliver value elsewhere?
All of the elastic and horizontal stuff is great, but that's covered under SLA.
All of the elastic and horizontal stuff is great, but that's covered under SLA.
I don't think this will have (or is meant to have) any effect on professional applications. This is just for the educational and cool factor.
Right now, yeah. But if this tech trend (miniature, low power, cheap, commodity) continues and we can run our apps on them ... it might be a game changer. No?
Hu no.
"Low power" , "cheap" and "commodity" already describes retail computers since almost 30 years.
Why do so many people want to see a higher purpose in any fun experiment? Can't it just be that, a fun and cool hobby project?
Why do so many people want to see a higher purpose in any fun experiment? Can't it just be that, a fun and cool hobby project?
Just for a different perspective, I grew up in a very low income household, and the Pi was the only computer I was able to convince my mum to get me as an early teenager. "Cheap" is relative, and I disagree that the Pi wasn't a big step forward in terms of price on the very low-end.
Depending what you are doing. For use cases like Dropbox cloud was never an option, other use cases can realize 30% year over year saving just by moving to the cloud. I think where RPIs make the most sense is soho, office building use cases, onsite backups, NAS, hosting infra services (DNS server for example) where you can use many nodes without much power and with a good amount of redundancy.
How "fast" and "strong" can those servers be?
Compared to something we all know, e.g. MBP 16" 2019, how does this rack of Raspberry Pi's stack up?
Compared to something we all know, e.g. MBP 16" 2019, how does this rack of Raspberry Pi's stack up?
I'm just trying to understand the power of those machines compared to the only piece of equipment I daily use and can compare to.
I know it can be like comparing apples to oranges but I was just curious how fast this can be.
I know it can be like comparing apples to oranges but I was just curious how fast this can be.
The best Raspberry Pi available right now is a bit faster than a modern entry-level PC. It has 4 ARM cores running at 1.5 GHz and 8 gigs of RAM.
That is not true at all. For compute, it's about 5x slower than the cheapest Intel processor you can buy.
It's certainly much better than the original Raspberry Pi (which felt like a 486), but it is not using recent technology and it just doesn't compare to what is available today. It is for projects where you would normally use a microcontroller, but want HDMI and Linux out of the box. People using the Pi for art projects, signage, dashboards, controlling their sprinkler system, etc. have the right idea. People using them to serve interactive traffic are going to have a bad time. They are really, really slow.
It's certainly much better than the original Raspberry Pi (which felt like a 486), but it is not using recent technology and it just doesn't compare to what is available today. It is for projects where you would normally use a microcontroller, but want HDMI and Linux out of the box. People using the Pi for art projects, signage, dashboards, controlling their sprinkler system, etc. have the right idea. People using them to serve interactive traffic are going to have a bad time. They are really, really slow.
It’s not the same money, but I like pcengines apu for this. Lots of intel Ethernet, mpcie msata, etc. Intel amd64 makes dev simple
https://archive.vn/sVYW3
The HN hug is strong this morning
The HN hug is strong this morning
How it started: Lets create a small cheap computer that poor kids can use for education.
How its going: Raspberry Pi Server Mark III
How its going: Raspberry Pi Server Mark III
making something like this with eg, amlogic socs, would be significantly faster and require no fans and less power. it would be cool to see someone do something like this with the not terrible rpi broadcom chips
Imagine a Beowulf cluster of-- shit, am I too late?
What is a use case for this if any?
There isn't one. For actually delivering an application, a traditional server is better in every way, unless your application is so small that you are wasting your hardware and get killed on power costs.
It's more or less the same as the people in /r/homelab who have full server racks in their basements but never do anything with most of their eBay servers other than install CentOS on them. At least those can serve movies via Plex, though.
It's more or less the same as the people in /r/homelab who have full server racks in their basements but never do anything with most of their eBay servers other than install CentOS on them. At least those can serve movies via Plex, though.
I mean there are probably a few super edge cases for something like this. A digital art display, or theater production for instance. Something you can custom 3d print yourself is just about perfect for that.
lol brrrrt. Actually there's heaps of cases. Traditional servers are large, power hungry and generally 10-100x the compute power most jobs need if you buy a piece of hardware for a site. Traditional server also requires expensive monkey like me to service.
RPI rack on other hand. plug and play. Cheap as chips to keep failover as you can keep backup rigs in a cupboard. It doesnt require expensive monkey to unplug bits. Most sites I serve have minimal if any internet at all. So cloud compute not an option everything has to be done on-site. RPI is perfect. It has more than enough grunt, costs bugger all, performance is rock solid.
RPI rack on other hand. plug and play. Cheap as chips to keep failover as you can keep backup rigs in a cupboard. It doesnt require expensive monkey to unplug bits. Most sites I serve have minimal if any internet at all. So cloud compute not an option everything has to be done on-site. RPI is perfect. It has more than enough grunt, costs bugger all, performance is rock solid.
Disks, man. Where are your disks??? Compute by itself is next to useless for anything serious. You need data. What do you need a compute node for if you don't have any data? You could just do it client-side in the browser.
Can someone explain what the point is? Can't you just gets a multicore xeon off eBay really cheap and get more performance for less money?
cloud style microservice architectures. saves a lot of power and no single node is carrying all the weight. distributed computing is scalable. 1 big xeon is not.
Looks like fun ... but I am getting the new Mac Mini.
https://uplab.pro/2020/10/raspberry-pi-mini-case/ got you covered ;)
That is nice print quality, especially that top sheen from printing on glass.
Thanks for the deserved downvotes.
But in terms for actual cost and performance; how does this compare in terms of peak performance, power consumption and actual total cost to mass produced quality hardware like the Apple Mac Mini?
I don't the actual numbers but I have the feeling it won't fare that well.
But in terms for actual cost and performance; how does this compare in terms of peak performance, power consumption and actual total cost to mass produced quality hardware like the Apple Mac Mini?
I don't the actual numbers but I have the feeling it won't fare that well.
Obviously in terms of single-core performance your Mac (or any desktop system for that matter) will always do better. As for actual power consumption, it always depends on the load and additional devices connected to the Pis. The only easily comparable aspect is the price, but it's meaningless on its own.
For starters it has 14x 8GB RAM, 7TB SSD, 14x GBit and 14x4 A72 cores. Not too shabby.
For practical purposes 8 high performance cores will be much more performant that 56 cores with lower performance; in particular if the cores are spread over 14 physical machines.
That depends on the task.
I challenge you to find a single application where a setup of 14 raspberry pi's will beat the new mac mini.
It was apparently built for a very specific use case (doing CI for some application) so I guess it has to be ARM Linux. You can run Linux in a VM guest on Mac mini, but some limitations remain:
- The project was obviously started before the Mac mini was available
- Linux guests are still rather experimental and might not behave the same way as a real machine.
- 32-bit Arm applications do not run on the M1, it is a 64-bit-only CPU
- Nested virtualization is said to not work yet under MacOS, so you cannot run KVM guests inside of the Linux guest
- If you need to spin up lots OS instances simultaneously to test various setups and each of them need 8GB, you can have at most 2 instances on a Mac.
For most use cases, none of those apply, so you would be better off with a Mac, a Graviton instance or an off-the-shelf PC.
- The project was obviously started before the Mac mini was available
- Linux guests are still rather experimental and might not behave the same way as a real machine.
- 32-bit Arm applications do not run on the M1, it is a 64-bit-only CPU
- Nested virtualization is said to not work yet under MacOS, so you cannot run KVM guests inside of the Linux guest
- If you need to spin up lots OS instances simultaneously to test various setups and each of them need 8GB, you can have at most 2 instances on a Mac.
For most use cases, none of those apply, so you would be better off with a Mac, a Graviton instance or an off-the-shelf PC.
Maybe. But you have a network bottle neck between the 14 separate computers that you have on a single computer. If you have a computational task you split in 56 pieces and run it across the setup, you can utilies the cores fully. However even in that case I think the Mac Mini will outperform this.
You can use it as a USB-hub though as I suppose you get 2 USB-ports pr. raspberry pi.
You can use it as a USB-hub though as I suppose you get 2 USB-ports pr. raspberry pi.
Here's one off the top of my head:
Serving web pages. A single Raspberry Pi 4 can manage 943 Mbit on its Gigabit port.
You might be able to buy (expensive) 40 Gigabit network adapters for the two TB4 ports of the Mac Mini. You'd still be slower.
Serving web pages. A single Raspberry Pi 4 can manage 943 Mbit on its Gigabit port.
You might be able to buy (expensive) 40 Gigabit network adapters for the two TB4 ports of the Mac Mini. You'd still be slower.
Maybe some ram based application? Opening 100gb csv in pandas? Redis with more than 8gb data?
I doubt it - each cpu core only has access to the 4 or 8 GB that is on its machine - to communicate to other parts of the setup it will have to go over network or a shared drive. I think that will be much slower than a highly integrated device like the mac mini just using its hard drive.
Controlling several hundred WS2812B LED strips, perhaps?
Or a kiosk with 14 HDMI displays.
Or a kiosk with 14 HDMI displays.
Sure. But wouldn't you then put the computer inside the display?
Something IO-bottlenecked maybe?
USB thumb drive boot disks are better performance and cheaper than SD cards.
Fans are completely unnecessary on a Pi4 with an aluminum heat shield style "case". They do run hotter than with a fan, but they will not be heat throttled with recent firmware.