Interesting. It might be a by-product of having worked in larger companies, but my experience has pretty much always been: Product manager > "We need this to work for 10k r/s at launch".
Actual launch throughput (and all time peak): 100 r/s
Now I always mostly just assume (capacity asked / 10) is more than enough!
I personally wouldn't call Parse a "failure" and without going into details, this has absolutely nothing to do with why Parse shutdown.
That said the real challenge, that other such platforms might not have, was handling all the very different DB workloads for all the apps we were hosting.
I worked on some of the pieces of that auto indexer. In most cases, this is a feature that was both necessary for us and extremely useful for our customers who didn't know how to manage their own DB. What was arguably missing was a way to expose the indexing operations to the developers, although this would have brought it's share of other challenges for obvious reasons.
If we created 17 indexes on the same collection (in some cases, it was way! more), that's because there were query families issued that needed those 17 indexes. I can't say this is your case, but in almost all instances I've seen, this was a result of poorly designed DB schemas and query patterns. Of course, for developers who know what they're doing, it's hard to design properly when you're dealing with a blackbox.
Yes, amplification and DB size are an issue when over indexing but our auto indexer was under constant tweaking and wasn't creating indexes "just because".
I have used Locust heavily in the past few month for loadtesting various apps and APIs. I've tried a couple different alternatives before settling with it, including ab and Vegeta.
Having the power of Python for scripting my loadtest was probably what sealed the deal for me. It allowed me to create a very powerful/reusable loadtesting framework that I can easily adapt very quickly to any app. Python scripting also allowed me to add features not build in to Locust's core itself like additional logging or metrics reporting and reading API endpoints from file/Redis/...
Also, being able to completely test all endpoints of an app at once is really useful. Most of the time, what I really want to know is can the whole backend handle the full traffic I'm expecting. Having clustering built-in helped a lot here. I was able to scale the loadtest to several thousands of RPS by just adding a couple more slaves...
The only feature I would like to see added is built-in Vegeta style graphs of latency/RPS over time. By default, Locust only gives you real time stat of the last second during the load test and a final average results in csv files. Combining metrics reporting with a graphing engine like Graphite can fix that though.
Now I always mostly just assume (capacity asked / 10) is more than enough!