Btw, I just cut EVE release 5.12.3-rpi-xen-arm64 (there's also 5.12.3-rpi-kvm-arm64 of course -- but this is Xen thread ;-))) that fixes a few issues that were reported ever since we got slashdotted this morning. This is your best bet in trying Xen/EVE/RPi4 combo. E.g. docker run lfedge/eve:5.12.3-rpi-xen-arm64 live > live.raw is literally all you need to get going.
One of the guys behind this hack here. The other is Stefano (and most credit goes to him actually). Would love for everyone to give it a try and help us improve it. I'm working on a few ready-made VMs that you can easily deploy on Raspberry Pi 4 with Project EVE (including Windows 10 VM and Plan9 VM). If you're interested -- drop me a note (I may even be able to do a few bounties/etc. for things we need to improve ASAP). Otherwise -- feel free to ask questions, etc. in this thread. It looks small -- but the whole thing took us a month or so (not full time of course) so there's quite a bit of subtlety there.
hi crawshaw, I've started playing with Tailscale as potential alternative for how we're using LISP in Project EVE (Edge Computing Mesh Network usecase) and was wondering whether there's any way today to make Linux clients negotiate (even if through a manual process without) going through your control plane?
I'll take Buildroot, and Bitbake first (Yocto itself is slight different we'll get to that later). Both Buildroot, Bitbake share the ultimate goal with all these news systems like linuxkit and bottle rocket. They all aim at producing single-image, pruprose built linux distros based on the very same usual suspects of upstream components. So the question really two-fold: out-of-the box availability of said upstream components (this is where we should stop talking about Bitbake and talk about Yocto -- Buildroot kind of commingles the two) AND the usability of the build harness itself.
So on #1, if you look very casually at Buildroot and Yocto -- they will clearly come on top over this next generation of systems. It appears they have WAY more upstream components already available for you to chose from. Compared to them the list here looks almost laughable https://github.com/bottlerocket-os/bottlerocket/tree/develop... and https://github.com/linuxkit/linuxkit/tree/master/pkg The problem though is combinatoric explosion of how you can compose all these upstream components. The canonical example here is the choice of your init system. You pick one -- and your choice in everything else gets severely restricted. So to some extent that apparent embarrassment of riches that Buildroot and Yocto offer is misleading.
These next generation systems, on the other hand, don't pretend that you can build a host OS in any shape or form you want (hence very few base packages) but rather that you build "just enough of Linux to run containerd" -- the rest of what you would typically put into your baseOS goes into various containers. This is a very different approach to constructing the bootable system, but subtly so -- which I don't think a lot of people on either side of this debate appreciate.
I honestly think that what makes Yocto and Buildroot difficult is that they want to be all things to all people and they want it at the level of baseOS -- complexity-wise, this is a wrong approach these days.
That scores one point for these next generation systems in my book.
The question #2 is not even a comparison. In Buildroot most of the integration/package logic is implemented in Makefiles and usability-wise (if you're trying to actively change the system or add a new package) it falls apart pretty quickly (it is still great if you're just using what's already there btw). In Bitbake -- the codebases is REALLY complex Python which suffers from the same issue. Contrast it with Linuxkit/Project EVE where all that logic is golang and bottle rocket which uses Rust and ask yourself whether you would rather debug a complex issue with a dozen of Makefiles all full of non-trivial recipes or look over go/rust codebase (yes, I know all these things are turing complete and thus equivalent -- but life is to short to debug Makefile).
If you don't quite believe me, there's been a number of studies about using Buildroot and Yocto for building containers. Pretty much all of them came back with the same conclusion -- the usability aspect of extending them makes it a non-starter. Here's the one from the last Kubecon that VMWare guys did: https://blogs.vmware.com/opensource/2020/02/27/distribution-...
Started to dig into the implementation. First impressions so far: loving all the Rust harness - really nicely done and way better than buildroot/yocto/etc for creating tight, single-purpose linux images. Speaking of tight, here comes the bad news: really NOT loving all the over-engineered upstream components like D-Bus and systemd that seem to be there by default. In that sense #linuxkit with its Alpine base and strong attention to how bit the image is still comes way on top.
One more thing on the good side: the TUF implementation in Rust seems really interesting. I'll be digging some more and may actually steal it for linuxkit (and by extension Project EVE)
Fun fact: a lot of the patches you will find in more system level packages like grub seems to trace their lineage to CoreOS (and potentially Project EVE) but I haven't seen acknowledgments anywhere. This is of course all fine from licensing perspective -- but I still would be curious to know whether it is indeed where it was taken from.
This looks pretty sweet. It seems to be a continuation of the same trend of "just enough Linux to run containerd" that CoreOS started, linuxkit continued, then Project EVE expanded to cover virtualization.
It is also interesting to note that every step on that journey seems to have picked the coolest runtime to implement it in (C/early Go, established Go, and now Rust)
Since I'm one of the original EVE's authors, let me start by saying this: EVE is different from Balena in two fundamental ways:
1. EVE believes in Edge Virtualization (hardware assisted virtualization) as the only way to build secure-by-default edge computing systems
2. EVE supports a variety of workloads (VMs, Unikernels and Docker/OCI containers) and we’re in a process of defining a spec for this called Edge Containers
You can read more about EVE's architecture (and please let us know if anything is not clear so we can improve our documentation) here: https://github.com/lf-edge/eve/blob/master/docs/README.md)
EVE does have a pretty decent support for various gateways: https://wiki.lfedge.org/display/EVE/Hardware+Platforms+Suppo... but as you can see there’s a heavy tilt towards Intel-based systems (simply because they are the ones with way more existing workloads available for them).
Now, let's talk about business side of things: Balena and ZEDEDA are very different when it comes to the value add on top of the open source platform. Balena aims at being more of a PaaS solution (sort of like Heroku of that space) where ZEDEDA is clearly an IaaS/CaaS (or better yet ECasS) solution. That said, both companies have their Cloud Based portions. Balena has https://www.balena.io/cloud/ and ZEDEDA has https://zedcontrol.zededa.net/
At ZEDEDA we believe that we’re providing the most scalable controller to our customers, however, there’s an open source controller implementation as well: https://github.com/lf-edge/adam
Adam is somewhat new and we’re still working on making it a bit more polished so all of your pull requests will be very much welcome!
Finally, there are other architectural differences that are mentioned in the doc I linked about. Things like Balena is build on top of Yocto, while EVE leverages amazing linuxkit project (the same one that powers Docker Desktop product). I'm only mentioning this because ironically enough, while Balena was a pioneer of Docker containers in IoT space, the way the system itself is put together is still very much embedded-like.