Pure python scripts, maybe using the #%%-convention (https://code.visualstudio.com/docs/python/jupyter-support-py...) so you get the best of both notebooks and scripts, in a right-sized instance/container/machine.
And if you need to run jobs in parallel, then orchestrate using make, like so: https://www.sumsar.net/blog/makefile-recipe-python-data-pipe...