Hey man, thanks for the question. Regarding the problem of whether an app will still be healthy in three months: it's a matter of ongoing verification. I'm targeting early next week for launching a public testnet, where you can try deploying random shit to test it and/or try to break things. It would be welcomed and appreciated, actually :)
Here are some things I can say that we have thought about and built from the start:
- Per-tenant observability via Prometheus through a metrics API, with counters at hopefully all meaningful decision points -- ingress decisions, auth delegations, internal events, rates, etc.
- The audit/telemetry pipeline with methods that help ensure consistent performance at scale and graceful degradation under extreme and increasing load.
- Requests are assigned ID's that propagate end-to-end across hops in the mesh (e.g. internal peer service calls), so you can trace requests across them.
- ACID DB transactions ACROSS all internal service calls, spanning the entire service-to-service call graph.
- Incident response and recovery are first class considerations, but I don't want to go on and on. There are documented runbooks, rollbacks, transactional migrations for applying updates to code+data, etc etc.
Here are some things I can say that we have thought about and built from the start:
- Per-tenant observability via Prometheus through a metrics API, with counters at hopefully all meaningful decision points -- ingress decisions, auth delegations, internal events, rates, etc.
- The audit/telemetry pipeline with methods that help ensure consistent performance at scale and graceful degradation under extreme and increasing load.
- Requests are assigned ID's that propagate end-to-end across hops in the mesh (e.g. internal peer service calls), so you can trace requests across them.
- ACID DB transactions ACROSS all internal service calls, spanning the entire service-to-service call graph.
- Incident response and recovery are first class considerations, but I don't want to go on and on. There are documented runbooks, rollbacks, transactional migrations for applying updates to code+data, etc etc.
---
Hope that helps.