Containers are meant to be stateless infrastructure. By downloading something at startup, you're breaking that contract implicitly. Secondly, depending on where you're deploying, downloads from S3 (and then loading to memory) may take a non-negligible amount of time that can impact the availability of your pods (again, depending on their configuration).
Synchronicity everywhere may cause request loss if your ML pipeline is not very reliable, which in most cases it isn't. Relying on a message queuing system will also increase system observability because it's easier to expose metrics, log requests, take advantage of time travelling for debugging, etc.
As a DevOps Engineer working for a ML-based company and have had worked for others in the past, these are my quick suggestions for production readiness.
DOs:
If you are doing any kind of soft-realtime (i.e. not batch processing) inference, by exposing a model on a request-response lifecycle, use Tensorflow Serving for concurrency reasons.
Version your models and track their training. Use something like MLFlow for that. Divise a versioning system that makes sense for your organization.
If you are using Kubernetes in Production, mount NFS in your containers to serve models. Do not download anything (from S3, for instance) on container start up time unless your models are small (<1Gb).
If you have to write some sort of heavy preprocessing or postprocessing steps, eventually port them to a more efficient language than Python. Say Go, Rust, etc.
DO NOTs:
Do NOT make your ML engineers/researchers write anything above the model stack. Don't make them write queue management logic, webservers, etc. That's not their skillset, they will write poorer and less performant code. Bring in a Backend Engineer EARLY.
Do NOT mix and match if you are working on an asynchronous model, i.e. don't have a callback-based API and then have a mix of queues and synchronous HTTP calls. Use queues EVERYWHERE.
DO NOT start new projects in Python 2.7. From past experiences, some ML engineers/researchers are quite attached to the older versions of Python. These are ending support in 2020 and it makes no sense to start a project using them now.
Synchronicity everywhere may cause request loss if your ML pipeline is not very reliable, which in most cases it isn't. Relying on a message queuing system will also increase system observability because it's easier to expose metrics, log requests, take advantage of time travelling for debugging, etc.