Thanks, appreciate the star. If you hit any snag deploying it, let me know. Curious what you're mainly after, the AI-referral alerts or the crawler side?
I help a lot of client with their ecommerce websites (mostly WooCommerce), attacks became so common recently, could be AI, but I found the best way to deal with this is to trace the patterns in access log and block the same patterns of checkout submission, this have worked really well for me. There are a lot of card testing attacks that Stripe doesn't care to handle as well as a lot of other fraud techniques, but there is always a pattern, especially automated ones. There is country, IP range, certain behavior (eg; no js, or direct api calls..etc). I really think it's easy to deal with this if you're willing to look deeper than a dashboard.
I did use AI to organize my ideas but I didn't think it was that bad, I'll modify and make it easier to read.
Anyway, in my test I saw zero requests from any Google UA after multiple Gemini and AI mode prompts that should have triggered grounding, so the working interpretation is that Gemini served from its own index/cache rather than doing a live provider-side fetch. The original phrasing was fuzzier than it should have been.
Added $http_accept and re-ran. None of them use text/markdown. Results:
ChatGPT-User/1.0 text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9
Claude-User/1.0 /
Perplexity-User/1.0 (empty, no Accept header)
PerplexityBot/1.0 (empty, no Accept header)
ChatGPT sends a Chrome-style Accept string. Claude sends a wildcard. Perplexity sends nothing at all. Gemini didn't fetch in my test.
Also worth noting: Claude-User hit /robots.txt before the page.
Yeah, that the main reason I never use services like Google Cloud if I don't have to, it's impossible to have a hard cap, and anyone pretending to be an expert, is just off.
Google says that they can't provide a hard cap because that would mean shutting down all your services..bla bla, but at least give users the option.
I thought I had something wrong within my setup, I could never use Codex 5.3 while everyone else was praising it. It uses some weird terms and complex jargon and doesn't really make it clear what it was doing or planning to do unlike Opus which makes things clear, this allows me to give accurate feedback and change plans and make proper decision.
Not bad, but it sacrifices accuracy and there are risks of causing more hallucinations from having incomplete data or agent writing bad extraction logic. So the whole MCP assumes Claude is smart enough to write good extraction scripts AND formulate good search queries. I'm sure thing could expand in the future to something better, but information preservation is a real issue in my experience.
There are 4 models, all receiving the exact same prompts a few times a day, required to respond with a specific action.
In the first experiment I used gemini-3-pro-preview, it spent ~$18 on the same task where Opus 4.5 spent ~$4, GPT-5.1 spent ~$4.50, and Grok spent ~$7. Pro was burning through money so fast I switched to gemini-3-flash-preview, and it's still outspending every other model on identical prompts. The new experiment is showing the same pattern.
Most of the cost appears to be reasoning tokens.
The takeaway here is: Gemini spends significantly more on reasoning tokens to produce lower quality answers, while Opus thinks less and delivers better results. The per-token price being lower doesn't matter much when the model needs 4x the tokens to get there.