Pipelines as code is the way to go, no matter whether it is CI/CD, data or ML. It’s unfortunate that a lot of pipeline tools use YAML. Glad to see projects like Dagger that use Python instead.
However it is not clear for me what is the benefit of using it instead of calling commands like docker, pytest and kubectl from Python with Plumbum or similar library. Add Fire and it is trivial to create a complete DevOps CLI for your app that can run locally or called from GitHub actions.
Books, always having several books available on my Kindle so I can grab it and continue reading instead of consuming social media on my phone. I read 3-4 books at once and don't put pressure on myself to continue reading or finish any book. If I get bored with one just switch to another one.
There is no shortage of book recommendations. I usually get them from people interviewed on podcasts.
We tried to use Postgres with TimescaleDB plugin for high frequency data several TB in size. It was unusable. Switched to Clickhouse, which was roughly 50-100 faster on the same hardware and 10 times less disk space. They use very different storage engines with different functionality so check the docs to see what fits your use case.
I met guys like you, skillful and hardworking software engineers who deliver results but are unable to build positive relations with management. Without it there is not enough trust for promotion, investment. The fastest way to learn this is to observe and adapt behavior of more successful people in your environment. Books on the topic often ignore the differences in national and organizational culture.
Perhaps EU should own software companies developing open-source software that each country can adapt to its needs. It is common with state-owned energy and water utilities. Why not software?
There are plenty of ML applications in industries like farming, energy, transportation, construction etc. ML is one of the tools along with others like mathematical programming and simulations that allow for more optimal use of resources such as materials, energy, land and human resources. These applications are domain-specific and less known to the general public but have substantial sociatal impact.