HackerTrans
TopNewTrendsCommentsPastAskShowJobs

Robots.txt Won't Save You(rb.ax)

2 points·by rbax·2 ปีที่แล้ว·1 comments
rb.ax
Robots.txt Won't Save You

https://rb.ax/blog/robots.txt-wont-save-you/

2 comments

JohnFen·2 ปีที่แล้ว
> This matters because in an ideal world all client requests would have the correct stated User-Agent so that website administrators could manage appropriate access.

The reason that it's necessary to spoof the user-agent is that so many websites mishandle it to begin with. Websites in general brought this state of things on themselves.
[deleted]·2 ปีที่แล้ว