I'm working on a video / post on how to solve the 1 billion row challenge (https://github.com/gunnarmorling/1brc) and get a competitively fast result while keeping the code readable and maintainable.
So far I'm within spitting distance of the winning entries without using any unsafe code or bit twiddling tricks or custom JVMs or anything like that, and having all the concerns nicely separated and modularized.
Just off the top of my head I can think of a few cases where we easily have 100-1000x that on a single table in shared node on kube, with an ebs volume (point being, bog standard, if a tad large, commodity hardware). If you properly index things and your workload isn't too crazy in a way that needs access to all of this at once (e.g. point lookups with some recency bias), classic RDBMSs have this use case covered - and it's achievable by anyone!
Holy shit, I think the front page signinless-demo is using the css visited hack to figure out popular sites that I've visited, and showing them to me in the demo. Devious, but brilliant.
So far I'm within spitting distance of the winning entries without using any unsafe code or bit twiddling tricks or custom JVMs or anything like that, and having all the concerns nicely separated and modularized.
Excited to share soon!