Yep! Other than random reads (~p99=200ms on larger ranges), it's essential to get good download performance of a single file. A single (range) request can "only" drive ~500 MB/s, so you need multiple offsets.

https://github.com/sirupsen/napkin-math

Sirupsen

·5 माह पहले·discuss

Due to the batching, this will only consume a few million class B per month. They are $5/million

Sirupsen

·5 माह पहले·discuss

(cofounder of tpuf here)

We don't have a relational database, otherwise that would work great for a queue! You can imagine us continuing to iterate here to Step 5, Step 6, ... Step N over time. The tradeoff of each step is complexity, and complexity has to be deserved. This is working exceptionally well currently.

Sirupsen

·8 माह पहले·discuss

Our query planner has that built in! We've spent a lot of time making high recall with any selectivity in the fitler work.