Absolutely, but it turns out to be a really common misconfiguration found in default configs and tutorials. Definitely something to be aware of when configuring FastCGI.
My thoughts exactly. Turning a living creature into a plaything and masquerading it as educational is just sick. It degrades the creature, and it degrades the science.
If you want to learn about GP, the Genetic Programming Field Guide - http://www.gp-field-guide.org.uk/ - is an awesome book, it taught me a lot. In fact, I liked it so much, I bought a hard copy. Highly recommended!
For this particular project I needed access to more than just the page itself - I needed to interact with scripts and have access to the page resources on a level which matched that of a browser. What you're seeing is only the tip of a much larger iceberg, for which efficiency wasn't the main objective.
You're correct that you can write more efficient spiders if you don't want to do anything fancy (indeed, I've written simple scrapers and spiders in various languages), but this project leant itself better to a full browser environment.
Thank goodness for that. Countless hours have been lost debating how best to implement this pointless law, and the amount of business lost due to unsightly and confusing consent banners must have been huge.
I quite like these. Regarding the negative comments, these stories were not intended to conform strictly to the traditional kōan form, and are clearly not expected be taken entirely seriously (the About page makes this clear!)
As a personal bonus for me, the author makes reference to the book "Gödel, Escher, Bach", of which I was somehow not previously aware. It sounds like it's right up my street.