Heavy Rotation of Relational Hashicorp Vault Database Secrets in Spring(secrets-as-a-service.com)
secrets-as-a-service.com
Heavy Rotation of Relational Hashicorp Vault Database Secrets in Spring
https://secrets-as-a-service.com/posts/hashicorp-vault/rotate-dynamic-relational-database-connection-in-spring-at-runtime/
5 comments
Thanks for your feedback.
I agree that the post does not offer a generally applicable solution for the rotation of configuration. Unfortunately, it looks like in this moment of time there is no such solution available (at least not that I'm aware of). On the other hand, the reason for writing this post was not to solve this issue but to help developers of Spring applications to use Hashicorp Vault for generating dynamic database credentials.
A solution for the more generally applicable, but also much more complex problem of configuration rotation without downtime would likely also solve the issue of the blog post. But as long as this solution is not available a more specific approach could at least address the challenges for some of us. The quite narrow scope of the blog post is bound to relational databases and HikariCP and I hope that I could point out that this still addresses a lot of use-cases (at least as long as SQL databases are not dead ;) ).
Additionally, I would like to increase the awareness that Spring itself is currently not automatically addressing the issue of rotating expiring secrets provided by Hashicorp Vault. Because Spring provides so many out-of-the-box production ready solutions for a lot of use-cases a lot of developers don’t know and even think about the edge cases which could still create pain.
I’m really interested in more details about the proxy-based approach you were writing about. If I’m able to build a more general solution with this approach I would be really happy to share this approach in another blog post. Or do you know if it there already something planed in the Spring backlog to make this easier?
I agree that the post does not offer a generally applicable solution for the rotation of configuration. Unfortunately, it looks like in this moment of time there is no such solution available (at least not that I'm aware of). On the other hand, the reason for writing this post was not to solve this issue but to help developers of Spring applications to use Hashicorp Vault for generating dynamic database credentials.
A solution for the more generally applicable, but also much more complex problem of configuration rotation without downtime would likely also solve the issue of the blog post. But as long as this solution is not available a more specific approach could at least address the challenges for some of us. The quite narrow scope of the blog post is bound to relational databases and HikariCP and I hope that I could point out that this still addresses a lot of use-cases (at least as long as SQL databases are not dead ;) ).
Additionally, I would like to increase the awareness that Spring itself is currently not automatically addressing the issue of rotating expiring secrets provided by Hashicorp Vault. Because Spring provides so many out-of-the-box production ready solutions for a lot of use-cases a lot of developers don’t know and even think about the edge cases which could still create pain.
I’m really interested in more details about the proxy-based approach you were writing about. If I’m able to build a more general solution with this approach I would be really happy to share this approach in another blog post. Or do you know if it there already something planed in the Spring backlog to make this easier?
The approach is currently a design draft to approach credential rotation from a generic perspective. It spans over a couple of components and we need to identify first on which level we want to address the requirement of credentials rotation. So it will take a bit until we have a more clear picture.
So there will be a generic solution provided by Spring?
I'm really looking forward to this. Is there I way I can be kept up to date about the progress or how I could potentially participate?
I'm really looking forward to this. Is there I way I can be kept up to date about the progress or how I could potentially participate?
Author here. Happy to answer any questions or hear any feedback about this post.
If we look at what happens after credential expiry, we experience a quite diverse field. Some databases just keep working on existing connections as these remain authenticated. In other cases, databases validate the security context on each operation, which requires disposal of connections.
While injecting updated configuration is possible in some cases, it's not an approach that works for all technologies. Rather, we should come up with an approach that helps to understand and solving the problem for all components (or at least the majority).
The root of all this is that configuration binding and the components being configured just do not expect rotating credentials. We need to work around this fact. While this approach works for Hikari, it might not work for other pool implementations and that is the problem I have with the posted approach.
Instead, I would recommend a Proxy-based approach that can work with the majority of components. A pool is a DataSource after all and with the usage pattern of opening and closing connections, an implementation can track resources in use. So instead of exposing a HikariCP pool directly as data source, it would make sense to expose a proxy that routes getConnection() calls to the currently active and valid pool implementation.
Once credentials expire (or are about to expire), the implementation would fetch new credentials, pass these to a DataSource factory function and obtain a new, healthy pool. Also, once all connections of the previous pool are released, the old pool can be closed.
For technologies that do not expose a releasable connection (MongoDB, Cassandra), we require a different release mechanism, probably timeout-based. Still, the general proxy idea (proxying the Mongo or Session object) would remain the same.