I've been thinking about this quite a bit lately. I've run DevOps for a few organizations and learned quite a bit through that.
Ultimately you can engineer your systems, even if they are quite complex, to be manageable by a single person. It's not one thing though. It's years of experience and gut feel. It's also totally distinct from technology.
Some things that come to mind:
- use queues for background tasks that may need to be retried. If things go down and you have liberal retry policies, things should recover.
- use boring databases. Just stay away from mongo and use something like rds which is proven and reliable.
- be careful in your code about what an error is. Log only things at the error level you need to look at.