[OP] A quick read highlighting some key open source projects available to machine learning platform engineers.
HPC-like systems can now be implemented in cloud-native environments (using Kubernetes), but the stack of tools for doing so is quickly evolving and requires continued investment.
> why do people continue to do their work in one?
Most nb users don't use vscode. I'd say main reason is you can't open a notebook without starting a kernel which takes >1 second. It also is harder to setup.
In terms of a flat file formats, there is actually a mark down extension which I would say is better than vscode's format because it is more standardised:
Thought I'd drop my xmas project in here: A visual disk usage analyser for docker images.
I have regularly found Disk Inventory X to be illuminating when fighting congestion on my macbook, but when it came to de-congesting Python docker images there was no similar tool.
Unlike Disk Inventory X and other similar tools, Whaler works from the command-line both on local dirs and images. It is built using modern front end tech: D3, Canvas, React, making it portable to all platforms including headless environments.
I have tested the tool with directories containing up to 500K nodes which should be fine for most Python Docker images. Beyond that it may chug/crash.
HPC-like systems can now be implemented in cloud-native environments (using Kubernetes), but the stack of tools for doing so is quickly evolving and requires continued investment.