nmlracx

·3 anni fa·discuss

It seems to me that Twitter's robots.txt only allows Googlebot:

https://twitter.com/robots.txt

Therefore, this disallows other bots in "maschinenlesbarer Form" and the scraping is illegal.