Do you have any insight into comparing
SurrealDB (https://surrealdb.com/docs/introduction/start) to DuckDB (https://shell.duckdb.org/)? I haven't done that comparison yet, I might now based on your recommendation. I saw DuckDB at Data Day Texas 2023 and their demo was intriguing. That's even for our use case which was not very SQL-y.
This is a fantastic take (seriously, no sarcasm here). I'll add that serverless platforms have not actually shown the ability to scale _efficiently_. It feels like every software architect has been talking about the 2023 March post^ from Amazon Prime Video on migrating from serverless to serverful. That is a direct response to the inability of serverless to scale because under the hood it is serverful but limited. Besides AWS Lambda and Google Cloud Run others have tried OpenFaaS and Cloudflare Workers. I think only Cloudflare Workers have "scaled" but that's because it's limited scale for each service at the edge with lots of edge devices.
I do not know if this is solvable: a permanent service with light autoscaling handles even inconsistent load so well and reduces operational complexity so much that I don't know if serverless will be anything more than toys. It is no coincidence that AWS Lambda's first language was a frontend language* (Node.js). Those use cases have low to moderate scale (because after that you split frontend and backend).
- Apache Spark starts its relatively slow decline in favor of vendor and in-house solutions.
- Cost drives cloud systems away from managed solutions (think AWS Batch) towards semi-managed solutions (think self-run k8s deployments but on AWS EKS).
- There are no improvements in autoscaling k8s clusters.
- Integration testing is replaced by testing in the development environment.
Kind of funny about that last bit of advice, as that is the opposite of the advice for other creative endeavours. In https://savethecat.com/ the book talks about telling your story ideas to everyone so that your creative juices get excited.
I do think you're right about tech. Try it yourself, build it out, have some fun, be a dork. Then tell people.