> Do wide events really have to take up this much space?
100PB is the total volume of the raw, uncompressed data for the full retention period (180 days). compression is what makes it cost-efficient. on this dataset, we see ~15x compression, so we only store around 6.5PB at rest.
we use k8s + otel filelog receiver. in this case you don't have to connect to the clickhouse instance to collect what it's writing to stdout/stderr, just tail /var/log/pods/*/*/*.log.
> Anecdotally I know robust tracing data can help me find problems in 5-15 minutes that would have taken hours or days with manual probing and scouring logs.
exactly. high-cardinality, wide structured events are the way.