May I recommend looking into flyte.org, it is open source kubernetes native "orchestration" style tool, but essentially and infrastructure component that is geared to making your ML Engineers and Data scientists more productive.
I think iteration velocity, dynamic infrastructure management and trackability are really important and fundamentally different needs of such products.
PS. I am a maintainer at flyte.org (thoughts are my own)
I do agree it makes sense fornscale, but if your data fits in memory Flyte native constructs shine. For example it will ensure your data is stored / serialized correctly. Allows you to use polars, vaex, duckdb etc. tbh I am a huge proponent of vertical scaling till you can get the mileage.
It also supports - gpu allocation, spot instances and collaboration across multiple users. I do not think it is a wrong choice if you feel your complexity will grow.
@tomrod, thank you for the callout. By the way we are integrating mlflow into Flyte in a way that you do not need to start the web server to view the logs. They are available
Locally and statically in Flyte Ui. Ofcourse you cal also use mlflow server
Some of these were the core problems that we wanted to address as part of https://flyte.org. We started with a team first and multi-tenant approach at the core. For example, each team can have separate IAM roles, secrets are restricted to teams, tasks and workflows are shareable across teams, without making libraries. and it is possible to trigger workflows across teams.
Each teams workflows are tasks and grouped using a construct called projects. It is even possible to separate execution clusters per team, per workflow onto separate k8s clusters. Also the platform is built to be managed and easily deployed.
I remember having this feeling a few years ago. What I realized is that airflow has taught us a few bad habits and also brought ahead an interesting paradigm of the vertical workflow engine.
I agree airflow is old, legacy and ideally folks should not use it, reality is there is a lot of pipelines already built with it - sadly. I think as a community we have to start moving away from it for more complicated problems.
Disclaimer: I created Flyte.org and heavily believe in decentralized development of DAGs and centralized management of infrastructure
Thus users could continue using RED, and if they want to scale to multiple machines or want resilience, you could allow them to switch out the backend to Flyte.
Disclosure: I am maintainer of Flyte. This is just a suggestion. Great work!
Thank you. So at Lyft, there exists an AirflowFlyteOperator. It was designed to interop with Airflow. I know the community is working on open sourcing it. Keep a lookout for the same
PS. I am a maintainer at flyte.org (thoughts are my own)