I intended that as a reference to an XKCD strip[0]. Folks often talk about public goods like scientific research, as if the "real stuff" couldn't possibly happen without private investment. ¯\_(ツ)_/¯
Barstool is referenced more than once in the piece, but not directly by name. Deadspin did not fail, it was murdered.
And I know you aren't saying it directly, but this really shouldn't be about The Left and The Right. The problem is about punching up versus punching down.
A crawler has two high level options: parse the page, or render the page.
Most of our parser-based crawling is done by Heritrix (crawler.archive.org) and most of our render-based crawling is done by a proxy-based recorder similar to what you theorize (https://github.com/internetarchive/brozzler).