Setting Up a Cluster of Tiny PCs for Parallel Computing(kenkoonwong.com)
kenkoonwong.com
Setting Up a Cluster of Tiny PCs for Parallel Computing
https://www.kenkoonwong.com/blog/parallel-computing/
36 comments
I appreciate the author's work in doing this and writing it all up so nicely. However every time I see someone doing this, I cannot help but wonder why they are not just using SLURM + Nextflow. SLURM can easily cluster the separate computers as worker nodes, and Nextflow can orchestrate the submission of batch jobs to SLURM in a managed pipeline of tasks. The individual tasks to submit to SLURM would be the users's own R scripts (or any script they have). Combine this with Docker containers to execute on the nodes to manage dependencies needed for task execution. And possibly Ansible for the management of the nodes themselves to install the SLURM daemons and packages etc.. Taken together this creates a FAR more portable and system-agnostic and language-agnostic data analysis workflow that can seamlessly scale over as many nodes and data sets as you can shove into it. This is a LOT better than trying to write all this code in R itself that will do the communication and data passing between nodes directly. Its not clear to me that the author actually needs anything like that, and whats worse, I have seen other authors write exactly that in R and end up re-inventing the wheel of implementing parallel compute tasks (in R). Its really not that complicated. 1) write R script that takes a chunk of your data as input, processes it, writes output to some file, 2) use a workflow manager to pass in chunks up the data to discrete parallel task instances of your script / program and submit the tasks as jobs to 3) a hardware-agnostic job scheduler running on your local hardware and/or cloud resources. This is basically the backbone of HPC but it seems like a lot of people "forget" about the 'job scheduler' and 'workflow manager' parts and jump straight to glueing data-analysis code to hardware. Also important to note that most all robust workflow managers such as Nextflow also already include the parts such as "report task completion", "collect task success / failure logs", "report task CPU / memory resource usages", etc.. So that you, the end user, only need to write the parts that implement your data analysis.
If you want to do the same thing, build a cluster of complete machines, in a ligh-weight manner, just for the learning experience, you can use incus to create containers in the same manner.
As they are all complete machines and you bridge them, you get DHCP allocation identical to running a little PC on the local LAN. If you have ssh set up in the image, as described, you can run the identical scripts.
As a plus, if you run them on ZFS with de-dup, even the disk cost for new machines is miniscule.
As a plus, if you run them on ZFS with de-dup, even the disk cost for new machines is miniscule.
Just started working with Incus and it is great. Have you tried IncusOS yet?
No I have not. Changing my base image is a hassle but next time I set up another NUC or such, I will for sure.
https://linuxcontainers.org/incus/ apparently
So someone re-implemented the Beowulf cluster. I guess the other Slashdot memes are ready to come back.
Score: 1 (Insightful)
I love that you created this account just for this joke. Now we just need Natalie Portman to join.
Don't forget the ultimate /. achievement!
Score: 5 (Troll)
Score: 5 (Troll)
Hot Grits as a Service
> So someone re-implemented the Beowulf cluster.
Darn already. For I was thinking to myself: "Imagine a Beowulf cluster of these!".
Darn already. For I was thinking to myself: "Imagine a Beowulf cluster of these!".
Can it play Crysis/Doom?
Someone needs to submit it to this awesome archive/gallery of Doom ports:
https://canitrundoom.org/
Something like proxmox [1] could make wiping everything and restarting a lot easier. There really isn't a huge penalty between bare metal and a VM now so you get the ability to create/deploy/monitor all from a reasonable interface. If the standard clustering stuff isn't enough their datacenter version looks like it is designed more for this. Never used it though. (no ties to proxmox here, just a guy that likes it!)
[1] https://www.proxmox.com/en/products/proxmox-virtual-environm...
[1] https://www.proxmox.com/en/products/proxmox-virtual-environm...
Congratulations on learning about distributed electronic computers. (This is worth tinkering with. This is how people actually get good at HPC.)
Pay attention to your SRAM (L3 unified cache), DRAM and swap space tilings.
[Snark] In practice: With memory access latency depending on both the square root of the memory size and the physical lengths of the wires in your cluster this sounds like a case for Adam Drake:
https://adamdrake.com/command-line-tools-can-be-235x-faster-...
Pay attention to your SRAM (L3 unified cache), DRAM and swap space tilings.
[Snark] In practice: With memory access latency depending on both the square root of the memory size and the physical lengths of the wires in your cluster this sounds like a case for Adam Drake:
https://adamdrake.com/command-line-tools-can-be-235x-faster-...
Very interesting!
For completeness, the job in the article is not strictly parallel, instead, they are separable tasks, which fall under high-throughput computing. HTCondor is a popular choice and is widely used in physics. If the cluster has a shared filesystem such as NFS, it does not even need ssh to each node. Moreover, nodes can share the same R installation on NFS, remove the complexity of installing same set of R packages on all nodes. HTCondor is available on Ubuntu.
For completeness, the job in the article is not strictly parallel, instead, they are separable tasks, which fall under high-throughput computing. HTCondor is a popular choice and is widely used in physics. If the cluster has a shared filesystem such as NFS, it does not even need ssh to each node. Moreover, nodes can share the same R installation on NFS, remove the complexity of installing same set of R packages on all nodes. HTCondor is available on Ubuntu.
I'm planning to set up something similar (but for compiling code). The difference is that my systems came without storage, so I intend to netboot them, which adds a whole other level of complication. I'm planning to use NixOS, with something like nixos-netboot-serve[0].
https://github.com/DeterminateSystems/nix-netboot-serve
https://github.com/DeterminateSystems/nix-netboot-serve
Something I've been playing with: what's the cheapest semi-reasonable computer that I could make a toy cluster from? Currently eyeing buying a bunch of Raspberry Pi Zeros but I suspect there are cheaper options (maybe some tiny openwrt thing). Too bad ESP32s don't have an MMU:D
Really depends on the details of your goals, but why not just VMs?
The "problem" with Pi-like devices is that they're usually not very "normal". The process of provisioning is different, IIRC they only "recently" supported booting off something other than the SD Card, and in the case of the Zeros, you'll either be using Wifi or an external USB Ethernet dongle (over USB OTG no less). Sometimes they need specially compiled version of linux, so you're stuck far from mainline (this was a big component of the RPis success) This may be distracting from your goals of learning about clustering.
I suspect the $10 Pi Zero is about as cheap as you'll get though, depending on your personal costs of case + ethernet dongle + USB power supply, etc.
The "problem" with Pi-like devices is that they're usually not very "normal". The process of provisioning is different, IIRC they only "recently" supported booting off something other than the SD Card, and in the case of the Zeros, you'll either be using Wifi or an external USB Ethernet dongle (over USB OTG no less). Sometimes they need specially compiled version of linux, so you're stuck far from mainline (this was a big component of the RPis success) This may be distracting from your goals of learning about clustering.
I suspect the $10 Pi Zero is about as cheap as you'll get though, depending on your personal costs of case + ethernet dongle + USB power supply, etc.
I recently built a Plan 9 cluster of 8 raspberry pi zero (2w). From my supplier (digikey) the 2W was the same price and has 4X the cores!
I think it looks quite cool: https://x.com/andreer/status/2007509694374691230
Instead of having to use lots of dongles and usb ethernet, I just wired them all up using brass rods, a small 5V power supply in the base, and boot them over WiFi (just the kernel and wifi config on the sd card).
Raspberry pi's are cheap, easily available, and there is an absolutely massive trove of information about them on the internet. And the scale means that the linux implementation is very stable and "just works" to a degree that is extremely hard for other SBCs to match.
Sure, VMs are the logical choice, but not everything has to be logical. Real hardware does feel more real :-)
I think it looks quite cool: https://x.com/andreer/status/2007509694374691230
Instead of having to use lots of dongles and usb ethernet, I just wired them all up using brass rods, a small 5V power supply in the base, and boot them over WiFi (just the kernel and wifi config on the sd card).
Raspberry pi's are cheap, easily available, and there is an absolutely massive trove of information about them on the internet. And the scale means that the linux implementation is very stable and "just works" to a degree that is extremely hard for other SBCs to match.
Sure, VMs are the logical choice, but not everything has to be logical. Real hardware does feel more real :-)
You can get intel NUCs way cheaper if you look around, or the Lenovo mini PCs. Small clusters will never beat a decent CPU but you can probably make a cluster of old mini PCs for less than the price of one.
So obviously it depends on goals; I absolutely agree that up to a surprisingly high point, the best way to actually get the most compute is to go buy the most powerful single machine that will do what you want off eBay or whatever. My goal is the opposite; I want to specifically get my hands dirty figuring out how to actually provision and manage and operate a large number of hosts, even if they're so low resource as to be effectively worthless individually and are still weak clustered. To that end specifically, digikey claims to be willing to sell me a rather large number of pi zero 2w at $15/each - https://www.digikey.com/en/products/detail/raspberry-pi/SC11... - and even cheap used boxes on eBay seem to start at double that. Obviously you need a little bit more to actually run them, but I believe the only thing you actually need is a USB cable and then a single host computer can provide power and boot them over usb.
Just be aware that you’ll be stuck with wifi, or spending far more on ethernet adapters for them. If you want cheap with ethernet there’s other devices supported by Arabian in the same price range which have ethernet. It’s fun for sure to have a huge number of very cheap machines, I have an old arm cluster made of Odroid boards for that purpose.
I actually think I can run network over the USB connections to the head node anyways, though WiFi is also fine.
Actually, you raise a good point: I should spend some time browsing the Armbian supported hardware list...
Actually, you raise a good point: I should spend some time browsing the Armbian supported hardware list...
Wanted to reply to you directly also to increase the chance you see this because I think I had exactly the same intrusive thought as you, and actually built such a cluster recently. Would love to hear what you think: https://x.com/andreer/status/2007509694374691230/photo/1
I think that's amazing, surprisingly physically nice looking, and now you've gone and reminded me that Plan 9 is an option, which is kinda another tangent I didn't need:), but IIRC Plan 9 is really low resource so it might be a good fit (aside it lending itself to distributed computing). Have you written up the build anywhere?
No, not beyond what I already put in the twitter thread. I wanted to wait until I had some cool distributed software running on it too, but then I ran into trouble with the plan 9 wifi drivers for the rpi being unstable and so I'm still working on fixing that. It does serve as a great test bed for that purpose too, as with 8 nodes I can get much more reliable data about how often the driver gets stuck
I feel you, but something like an R730 or 7810 with a pair of E5-2690 v4 and 128GB RAM can be had for under $400. Not the most power efficient, but you'd have to run it quite a while to make up the difference in energy cost. Plus there's way less work in getting it all set up.
I have a bunch of old Rpi 3B's in a 19" 1RU rack space i use for toy clustering. PoE splitters from a PoE switch to power them which also provides the network uplink. All very neat and self contained and when it comes times to upgrade/replace if they all fail (i have a couple of spares) I can just pull out the 1RU for disposal.
Intel NUC's are probably much better value for money these days but the 3B's were pretty cheap at the time I bought them.
Intel NUC's are probably much better value for money these days but the 3B's were pretty cheap at the time I bought them.
I work in e-waste recycling, and specialize in micro desktops. The most common model I have, Optiplex Micro 3050 (7th gen i5, 8gb RAM, 128gb SSD), goes for about $60 each.
I've got a couple of Lenovo M715q's under my desk for some custom testing and used they're great value for money too
Think https://www.servethehome.com/introducing-project-tinyminimic... was where I originally came across them
Think https://www.servethehome.com/introducing-project-tinyminimic... was where I originally came across them
I mean if you want to learn distributed systems stuff, two Erlang BEAM processes on virtually any general purpose computer are more than enough....
If the author had googled better they might have discovered https://www.learnpdc.org/
Kudos for Rscript.