frzb · HackerTrans

8 years agodiscuss

A standard Ubuntu installation plus the mentioned proprietary GPU-Driver package are using ~4GB of the available file system capacity. Thats way to much to fit into a ramdisk while having enough RAM left for OS. Most of the nodes in the GPU mining farms I know have only 4GB of RAM at all. So there is some optimisation needed to shrink the filesystem usage by getting rid of the things that are not necessary on the filesystem to run the OS. Source files, man pages, caches and so on. At the end of this optimisations you have a system which is much more lightweight but does still have the same well known usability. Having a more lightweight storage footprint than a "classic linux distro" is the key to be usable for network booting and diskless operation at scale on hundreds of nodes. Coinboot also has a highly customized Early Userspace (https://github.com/frzb/coinboot-debirf/blob/427b5fb8f3b9313...) so after extracting the root fs onto the ramdisk, additional file archives a.k.a. Coinboot Plugins can be downloaded and extracted on the root file system. This is how the plugin system of Coinboot works, which enables the extension of the core system without repackaging the Initramfs archive.

frzb

8 years agodiscuss

Docker is used to provide a working setup (DHCP-, PXE-, Webserver, the configuration and needed files) in the most easy way.

frzb

8 years agodiscuss

Hello, I am the creator of Coinboot and I will try to answer some of your questions.

Coinboot was initially created with only one use case in mind: To run the nodes of GPU-crypto-mining farms diskless. Because you don't need disks for mining. But these farms have only cheap commodity 1Gbit/s network hardware so the filesizes to transfer need to be as small as possible.

Being stateless is also a benefit for operations - all software + configurations are loaded in the RAM during boot, so you always have the most recent version in place and don't have to use a configuration management. There is also a plugin system which makes it easy to extend the functionality with further software. Just put the updated plugin archives in the plugin directory on the Coinboot server and reboot your nodes.

Coinboot is still in Beta phase but has been running for months on hundreds of GPU nodes.

My plan is to make the first stable release by the end of the year.

Centralised logging is already on the roadmap, also load-balancing the file download via local Bit-Torrent during the boot phase, and a Web-UI plus many other things. Ubuntu was chosen as base because the proprietary GPU driver for OpenCL has only support for Ubuntu and RHEL. RHEL was no option. Coinboot does use TFTP only for the NBP, all other file transfers are happening over HTTPS. If your hardware is using UEFI HTTP, then all file transfers are happening over HTTP. Boot time in the production environment is less then 15s.

Running diskless is done since decades and seems to be quite common in HPC - ages before everything got *-less. ;-) There was no implementation which fitted the requirements of the use case or was not outdated. Coinboot is also not using NFS - just because there is only cheap commodity 1Gbit/s network hardware in place. The whole root file system is put onto a TMPFS. After boot ~500MB of the RAM are used for the root filesystem. CoreOS is very impressive but there image size ~380MB was way to big. Coinboot is using currently ~130MB for the kernel + initramfs archives and the GPU driver Coinboot plugin.

comments