>Over the past year, we’ve meticulously scraped all Worldcat records. At first, we hit a lucky break. Worldcat was just rolling out their complete website redesign (in Aug 2022). This included a substantial overhaul of their backend systems, introducing many security flaws. We immediately seized the opportunity, and were able scrape hundreds of millions (!) of records in mere days.
>After that, security flaws were slowly fixed one by one, until the final one we found was patched about a month ago. By that time we had pretty much all records, and were only going for slightly higher quality records.
OCLC carelessly fiddlefarted around with their moat and lost it. Poof!
Seems to be similar to the direction taken by John De Goes (using algebras to describe your problem domain, building programs from those algebras via free monads/applicatives/etc., interpreting those programs using natural transformations). I think there is definitely something to this kind of thinking.
Both authors argue that modern programming languages do not have the best ergonomics for this style of programming, albeit it is at least possible to do most of it in languages such as Haskell and Scala.
>After that, security flaws were slowly fixed one by one, until the final one we found was patched about a month ago. By that time we had pretty much all records, and were only going for slightly higher quality records.
OCLC carelessly fiddlefarted around with their moat and lost it. Poof!