Agree. And thing I noticed is that tools like #apache spark have become the de-facto standard for any data engineer work even when data size does not require it. Result is that many jobs are much harder to mantain and often slower (due to all the shuffling) than running on a single node.
What is a real-time data cache and why do you need it?
Have you ever found yourself re-inventing the wheel as a data engineer to ultimately build some APIs?
Why not use Hasura, flink, elasticsearch and Airbyte.
All of the above are great tools and they are power packed with features. But as a data engineer, you still have to integrate many tools and put together an end-to-end solution. Check out this article where we describe our approach to deploying APIs.