HackerTrans
TopNewTrendsCommentsPastAskShowJobs

gavinhking

no profile record

comments

gavinhking
·vor 2 Jahren·discuss
I made this, let me know if you have questions or feedback.
gavinhking
·vor 3 Jahren·discuss
Yup, here is an example with all listed agents, although you probably want to modify it to not fully block all of them: https://darkvisitors.com/robots-txt-builder
gavinhking
·vor 3 Jahren·discuss
Yeah they respect robots.txt: https://commoncrawl.org/faq
gavinhking
·vor 3 Jahren·discuss
Thank you, will try to confirm this.
gavinhking
·vor 3 Jahren·discuss
You're not missing anything, they're just for any humans reading the file who want to know more about the agent. This is also just an example file.
gavinhking
·vor 3 Jahren·discuss
Here it is: https://darkvisitors.com/robots-txt-builder
gavinhking
·vor 3 Jahren·discuss
Right here: https://darkvisitors.com/robots-txt-builder

This example blocks everything, which is probably not what you want, but it's meant to be a starting point.
gavinhking
·vor 3 Jahren·discuss
It's updated daily. Thanks for posting by the way.
gavinhking
·vor 3 Jahren·discuss
Ha thanks, it was a shower thought
gavinhking
·vor 3 Jahren·discuss
Fixed, thanks!
gavinhking
·vor 3 Jahren·discuss
Hi all, I made this site. I wasn't expecting this post, but happy to answer questions and take feedback.

Hopefully it's pretty self-explanatory but I made this website as a simple resource for people who want to stay up to date with the ever-changing cast of AI user agents.

Feel free to sign up with the Google Form to get notified when this list is updated. And if you know of any agents I'm missing, please submit them. Thanks!