We are using dedicated clusters to push data over the links.
A member of our team proposed a session for Google NEXT '19 that goes into the architecture we are using in a lot more depth. We're waiting to see if that talk is accepted or not, but in any case we're planning to share more soon.
Indeed, these workloads are bursty. They also tend to involve running lots of different processing frameworks over the same data, which makes the value proposition for separating compute and storage stronger.
One of the interesting benchmarks was GridMix performance at scale. And overall performance for throughput-oriented workloads like Hadoop was generally strong. Most cloud benchmarking I have seen sticks to micro-benchmarks and single-node benchmarks, so they miss this.
A member of our team proposed a session for Google NEXT '19 that goes into the architecture we are using in a lot more depth. We're waiting to see if that talk is accepted or not, but in any case we're planning to share more soon.