Hey! Just wanna wave a flag for Coralogix (I work there, disclaimer). We've built a ton of cost optimization because we know that the industry is just ridiculous right now. Your assessment is absolutely correct, and there are more than a few multi-billion dollar companies whose bottom line is predicated on their customers wasting money and being inefficient. We're not one of them!
I was gonna make a post but this took the words out of my mouth. I have a whole talk about this exact topic, but the summary is that the paradigm of hot storage and then 2 weeks later, compressed archive, is the most wasteful way we could possibly organize this data. I discuss this at length in the talk below:
Hey! If you’re looking for open source friendly with really straight forward cost, check out Coralogix.com.
Great features for logs, metrics & traces, total compatibility with open telemetry, cost optimization tools built in (DataDog leavers typically save around 50%), and much more!
Check out our site, and you can find me on LinkedIn (or indeed reply here !) if you want to ask further questions.
In defense of Grafana Loki, it's awesome... if you have a lot of information about your telemetry data up front - i.e volumes, use cases, how you wish to query it etc.
A broader solution like Coralogix (https://coralogix.com/) is more appropriate if you're venturing into the unknown and you need more of a data discovery capability. The problem with most of THESE "all in one" platforms is they tried flexibility and feature breadth for cost. Coralogix is a wee bit different, and it gives users an awesome set of cost optimization tools, as well as a simple pricing model, to offer data discovery without a sudden increase in costs (or worse, surprise overages!).
Check out Coralogix! It comes in at a fraction of the price of DataDog (typically a 40-70% cost reduction for migrating customers), no service tiering (so you instantly get support, managed onboarding etc) with a complete, end to end, open source experience (even archived logs are stored in Parquet).
For your use case, Coralogix is awesome. It comes with a managed Grafana instance, if you wish to use that interface, or a custom dashboarding solution, metric driven alarms, release tagging, and much more.
https://coralogix.com/