ITX-sized 7-node ARM-64 cluster(forum.pine64.org)
forum.pine64.org
ITX-sized 7-node ARM-64 cluster
https://forum.pine64.org/showthread.php?tid=5461
39 comments
The situation's gotten much better since Armbian started building images[1], but it's still running a stranded kernel for now. This is expected[2] to change within the next few months. I've heard on IRC of people successfully running mainline Linux on the A64 SOC, but I don't believe it's recommended.
As far as China single-board computers go, I'm pretty pleased with Pine's build quality and attention to detail with regards to electrical design that's reflected in their schematics. Community-provided Linux support has come a long way in the last year+, it's worth checking out.
[1] http://linux-sunxi.org/Linux_mainlining_effort [2] https://www.armbian.com/pine64
As far as China single-board computers go, I'm pretty pleased with Pine's build quality and attention to detail with regards to electrical design that's reflected in their schematics. Community-provided Linux support has come a long way in the last year+, it's worth checking out.
[1] http://linux-sunxi.org/Linux_mainlining_effort [2] https://www.armbian.com/pine64
Moreover, ayufan's images (still BSP) are very solid and contain all the features you'd want and expect:https://github.com/ayufan-pine64/linux-build/releases
As a side-note, armbian's mainline is perfectly usable for headless applications.
As a side-note, armbian's mainline is perfectly usable for headless applications.
I'd go with H5 based boards. A64 is targetted at tablets, while H5 is for OTT devices. That means, you'll get much better USB experience with H5. It also supposedly has ARM cores with less quirks than A64. I'm hapilly running a secondary desktop on H5 with no serious issues on mainline kernel + some patches. It runs non-stop with no stability issues. But I have a very good power supply and custom made cables, which matters a lot.
https://linux-sunxi.org/Linux_mainlining_effort#Status_Matri...
See A64 there. Not quite there yet, but closer than a year ago.
Certainly not proud of the purchase... software support is still this bad despite two years already.
See A64 there. Not quite there yet, but closer than a year ago.
Certainly not proud of the purchase... software support is still this bad despite two years already.
I certainly hope it's gotten better, I just ordered a rock64 O_0
The Rock64 uses a different SOC and there is a lot of progress on the development front (because its a new board).
Yay for new and shiny I guess. Thanks!
Not to diminish this feat, but I wonder how the Phoronix benchmarks compare between an ARM cluster and a single x86 chip. Consider the OpenSSL benchmark.
A Sopine A64 compute module costs $29 and performs 15.05 OpenSSL signs / second [0].
An AMD Ryzen 1700 costs ~$300 and performs 986.73 OpenSSL signs / second [1].
So, if you spent $300 on 10 Pine64's, you would achieve approximately 150 signs / second, but for the same cost you could just buy the AMD chip and achieve approximately 6.5x better performance for the dollar.
[0] https://www.phoronix.com/scan.php?page=article&item=pine-64-...
[1] https://www.phoronix.com/scan.php?page=article&item=amd-ryze...
A Sopine A64 compute module costs $29 and performs 15.05 OpenSSL signs / second [0].
An AMD Ryzen 1700 costs ~$300 and performs 986.73 OpenSSL signs / second [1].
So, if you spent $300 on 10 Pine64's, you would achieve approximately 150 signs / second, but for the same cost you could just buy the AMD chip and achieve approximately 6.5x better performance for the dollar.
[0] https://www.phoronix.com/scan.php?page=article&item=pine-64-...
[1] https://www.phoronix.com/scan.php?page=article&item=amd-ryze...
I don't think openssl in phronix benchmarks use armv8 instructions, which makes those benchmarks unfair. There are benchmarks on cloudfare with openssl-devel, with hardware armv8, where arm performed decently. We have to wait and see how this pine64 does.
https://blog.cloudflare.com/arm-takes-wing/
https://blog.cloudflare.com/arm-takes-wing/
This is entirely to lacking an optimized version for the new aarch64 instruction set. Similar problem noted in Go benchmarks awhile back, see 100x speed up here: https://blog.minio.io/accelerating-sha256-by-100x-in-golang-...
armv7 and armv8 is just 64 bit, so I guess a corner case like openssl signs could benefit massively (lots of bigint) but normal workloads only marginally, while taking a massive hit on memory (don't want the 1 GiB model).
The ARM they tested at Cloudflare is also of course nothing like the mobile chip that powers the pine.
The ARM they tested at Cloudflare is also of course nothing like the mobile chip that powers the pine.
AArch64 is actually a completely new and much improved instruction set. More registers, less warts that inhibit out-of-order execution, etc.
I suspect this is a more inexpensive/interesting way to get a small cluster together. (You could also look at user/surplus PCs, but then you probably will use a lot of space and power, and not necessary get a cluster of exactly the same boards and CPUs and what not)
This was my thought in looking at this post. I could get a several node cluster going with low power and small space and test various ideas out.
> I wonder how the Phoronix benchmarks compare between an ARM cluster and a single x86 chip.
I don't think the focus here would be performance, but the cluster operations. Also, having slow hardware helps with spotting some performance issues (there are things we get away with Xeons that we simply can't with ARMs) even though fast hardware will help uncover some concurrency bugs that wouldn't happen on ARM.
As for virtualizing a cluster inside a bigger Xeon box, I worked with server provisioning software and the kind of nasty bugs you find with real hardware is pretty surprising. Some nodes refuse to shut down, some refuse to start, some take longer than expected (please see http://bazaar.launchpad.net/~maas-committers/maas/trunk/revi... for the smallest commit ever - a single bit flip - that actually solves a bug) and so on.
I'll probably put mine inside an empty HP workstation pizza box I have with a proper power supply, along with a couple Pi's, a PiHAT cluster, a CHIO and an early Pine A64. Having a heterogeneous cluster is a necessity for testing.
I don't think the focus here would be performance, but the cluster operations. Also, having slow hardware helps with spotting some performance issues (there are things we get away with Xeons that we simply can't with ARMs) even though fast hardware will help uncover some concurrency bugs that wouldn't happen on ARM.
As for virtualizing a cluster inside a bigger Xeon box, I worked with server provisioning software and the kind of nasty bugs you find with real hardware is pretty surprising. Some nodes refuse to shut down, some refuse to start, some take longer than expected (please see http://bazaar.launchpad.net/~maas-committers/maas/trunk/revi... for the smallest commit ever - a single bit flip - that actually solves a bug) and so on.
I'll probably put mine inside an empty HP workstation pizza box I have with a proper power supply, along with a couple Pi's, a PiHAT cluster, a CHIO and an early Pine A64. Having a heterogeneous cluster is a necessity for testing.
Interesting to see a thing I worked on 30 years ago come back around: https://en.m.wikipedia.org/wiki/Transputer#TRAMs
Right place at the wrong time? I worked in a start up that was webex/go2meeting/bluejeans eons before these things existed. Our potential customers wouldn't put dow their black berries though and we tanked. It's hard to be a visionary.
Well Microsoft has msn messenger and shut it down, not there are WhatsApp, telegram and many others but I'd pay good money to have msn messenger back
I remember the coverage in Byte and was convinced it was gonna go big...
Yup. I always say it's disappointing we are all running the bastard offspring of either Unix or VMS on overgrown IBM 5150 PC's featuring enhanced 8080's at their core ;-)
That now it's possible to have a computer whose sole processor is a Xeon Phi fills my heard with a tiny little bit of hope.
That now it's possible to have a computer whose sole processor is a Xeon Phi fills my heard with a tiny little bit of hope.
I've recently been fantasizing about something like this, but in laptop form. Have a dedicated console/display node, a few compute nodes, a 'router' node, and a gige (or 10gbe) fabric connecting them.
I was looking into getting a cheap laptop recently, turns out you can get a decent one for €200 nowadays (probably less in the US), which outperforms and outprices anything you can build with cheap ARM chips. A decent laptop-sized screen is already €100, and the whole PiTop setup (HD display, RasPi 3, 6-8 hour battery, case, touchpad, keyboard and charger) costs $320. The €200 laptop outperforms it and even comes with a Windows 10 license. And no crapware as far as I could tell.
With a 32 GB of RAM and a beefy x86, you can have a fairly decent cluster inside your laptop. Even a fully configured Macbook will probably be able to host a dozen gigabyte-sized nodes, if you refrain from running Slack and iTunes.
Sure, and I have that in each of the half-dozen ThinkPads laying about my place.
That doesn't negate a desire for a unique architecture portable.
edit: Particularly of interest to me is being able to isolate nodes. Your router and console are on the same physical board perhaps, but only interconnect via an onboard Ethernet switch (itself managed as a 'slave' under either the router or console node, depending on where one landed philosophically). Compute nodes could be powered off entirely when not in use.
That doesn't negate a desire for a unique architecture portable.
edit: Particularly of interest to me is being able to isolate nodes. Your router and console are on the same physical board perhaps, but only interconnect via an onboard Ethernet switch (itself managed as a 'slave' under either the router or console node, depending on where one landed philosophically). Compute nodes could be powered off entirely when not in use.
I think a laptop with a server-grade ARM processor would be a lot of fun.
Seeing them stacked up like DIMMs make me think of the S100 backpane popularized by the Altair.
https://en.wikipedia.org/wiki/S-100_bus
Makes a guy somewhat nostalgic of when computers were a collection of boards rather than a single chip.
https://en.wikipedia.org/wiki/S-100_bus
Makes a guy somewhat nostalgic of when computers were a collection of boards rather than a single chip.
I wonder how these could do in a homelab environment with each node running a different VM for pihole/SMB/transmission/etc.
why add the overhead of a VM?
> To not waste the 24W on the 12V I will probably use the DC/DC converter in the pictures and go from 12V to 5V for efficiency reasons...
The more common converters "burn" the voltage difference times ampere. So 14W will be emitted as heat by the converter, a matching heatsink would be somewhat important.
Instead of 24W wasted it is now 14W wasted. I am not sure if this is worth the effort.
The more common converters "burn" the voltage difference times ampere. So 14W will be emitted as heat by the converter, a matching heatsink would be somewhat important.
Instead of 24W wasted it is now 14W wasted. I am not sure if this is worth the effort.
Linear regulators went the way of the Dodo long ago for applications like this. HF DC-DC converters are the way to go, at the currents involved here you'd be paying more for the heatsink in a linear setup than for the whole DC-DC converter.
These things you are talking about are usually called _voltage regulators_. DC/DC converter usually refers to the switching converter, they are very efficient.
A step-down DC-to-DC converter can have efficiencies in the 90% range or higher. However, a linear regulator will indeed waste that additional power.
I bought one without looking too closely into it after seeing it mentioned somewhere online by someone who said it was cheap and it was ARM64. Only after it'd arrived did I learn that it had very poor Linux support. I've got mine laying around without ever having bothered to even boot it due to this.
For example, here's a key quote from what Hackaday had to say about the Linux support of the Pine64:
> The Ubuntu experience was tremendously slow on the Pine64 and I suffered several reboots. As of this writing, I have tested all of the software distributions on the Pine64 wiki. Only the Ubuntu distribution works poorly, and right now I consider the Pine64 to be a waste of $15.
https://hackaday.com/2016/04/21/pine64-the-un-review/
Has Linux support improved recently? Anyone here doing anything useful with a Pine64?