ChatGPT creates phisher's paradise by serving the wrong URLs for major companies(theregister.com)
theregister.com
ChatGPT creates phisher's paradise by serving the wrong URLs for major companies
https://www.theregister.com/2025/07/03/ai_phishing_websites/
27 comments
I feel a bit icky, but whenever I see people work with such a disregard for quality, I'm actually rooting for their products to break and get hacked.
In 2015 it was copy and pasting code from stackoverflow, in 2020 it was npm install left-pad, in 2025 it's vibecoding.
I refuse to join them, and I patiently await the day of rapture.
In 2015 it was copy and pasting code from stackoverflow, in 2020 it was npm install left-pad, in 2025 it's vibecoding.
I refuse to join them, and I patiently await the day of rapture.
It may already be here but in large moneyed organizations no one wants to take responsibility and speaking against management’s orders to lean in on AI may be a political suicide.
So a lot of people are quietly watching lots of ships slowly taking water.
So a lot of people are quietly watching lots of ships slowly taking water.
leftpad was 2016
I confess to vibe coding. Specially with api work. And it has gotten so badly that I have to actually either send api pdfs and links to api documentation.
A cynic. I like it.
This is, in turn, making the world of comment and forum spam much worse. Site operators could tag all user-submitted links as "nofollow," making their sites useless for SEO spammers. But spammers have learned that most LLM content scraper bots don't care about "nofollow," so they're back to spamming everywhere.
It reminds me of non-radioactive steel, the kind you can only get from ships sunk before the atomic bomb. Someday, we’ll be scavenging for clean data the same way: pre-AI, uncontaminated by the AI explosion of junk.
I’m not sure if even for traditional search engines “nofollow” means that the scraper doesn’t follow the link, or that it just does not include it in the PageRank or whatever graph but still uses it for to discover new pages. (Of course, LLMs are far too impenetrable for such a middle ground to exist.)
The study they link to: https://www.netcraft.com/blog/large-language-models-are-fall...
It seems they are selling some product to protect against this. So, I could believe the headline here, but less confident that this is an unbiased look at the problem.
It seems they are selling some product to protect against this. So, I could believe the headline here, but less confident that this is an unbiased look at the problem.
Yep. Just try recreating their results yourself. Not happening unless you explicitly select bad old models.
Note this wording: "Netcraft ran an experiment using a GPT-4.1 family of models."
Why GPT 4.1? Nobody uses that. Also 'family'? That just seems like a way to hide that they used GPT 4.1-mini rather than the full model.
Note this wording: "Netcraft ran an experiment using a GPT-4.1 family of models."
Why GPT 4.1? Nobody uses that. Also 'family'? That just seems like a way to hide that they used GPT 4.1-mini rather than the full model.
I've been using Phind lately and I think they do a really good job avoiding this problem. I don't think I've ever run into a fake URL using it. As a search engine, I think I still prefer Kagi but Phind is a great if you want a free option.
I wonder if Cloudflare's new plan for blocking AI from scraping the "real" sites...
You wonder if the plan is (or does) what?
Not just URL's. Google's AI snippet at the top of a search result page recently fed me the wrong phone number for Starlink support. It connected me to a scam being run at scale out of an overseas call center.
After questions like "Is your router plugged in", "What lights are on", etc., they eventually asked for my account email address and password (I was already getting suspicious by that point and of course did not provide). I could hear other agents in the background having similar conversations with other callers.
Based on some probing questions I asked, I think the agents themselves didn't even realize they weren't really working for Starlink.
I was in a hurry and should have double checked the number before calling, rather than blindly trusting the AI shoveled slop.
I tried but wasn't able to recreate the search query that produced the result, and a Google search for the number itself came up blank. A few weeks later, the number just rings with no answer.
After questions like "Is your router plugged in", "What lights are on", etc., they eventually asked for my account email address and password (I was already getting suspicious by that point and of course did not provide). I could hear other agents in the background having similar conversations with other callers.
Based on some probing questions I asked, I think the agents themselves didn't even realize they weren't really working for Starlink.
I was in a hurry and should have double checked the number before calling, rather than blindly trusting the AI shoveled slop.
I tried but wasn't able to recreate the search query that produced the result, and a Google search for the number itself came up blank. A few weeks later, the number just rings with no answer.
It's not wrong, you're just a luddite who is refusing to embrace the wonder of AI ... and if the guy on the other side of the phone denies being Starlink support you should tell him the same thing :-)
LLMs dont seem to hallucinate my niche products and company (sometimes the name of the company doing the product yes, but not the product name, not the url of the company), and according to CloudFlare Radar I'm only between 200000 and 500000 top domains https://radar.cloudflare.com/scan
I see a few visits from ChatGPT to pages that don't exist on my website. Not only wrong URLs, but topics I don't cover. ChatGPT is not merely stealing my work and audience; it's also adding my name to made-up information.
> “Crims have cottoned on to a new way to lead you astray”
Was - was the article written by AI?
Was - was the article written by AI?
No, it just sounds British.
Australian. "Crims" isnt said much in the UK, for us that's "thugs" or "yobs".
What about “cottoned”? Is that an actual word used for “caught”?
"cottoned on" means to figure something out
It's wild how many of the links are hallucinations.
Maybe the error rate is consistent with everything else, we just eisegesis our way into thinking it's not
Maybe the error rate is consistent with everything else, we just eisegesis our way into thinking it's not
I’ve had Claude hallucinate options for a lot of things, most recently DOM method arguments, and Wrangler configuration. It’s like, very reasonable things you might expect to exist. But they don’t.
I must be holding it wrong. How are people working with these tools to produce quality software?
I must be holding it wrong. How are people working with these tools to produce quality software?
In case you're not being sarcastic: they're not.
Most of the time AI will auto complete the line or couple lines of code I’m writing with perfect correctness. You still need to be able to notice when it hallucinates something wrong and understand why it’s wrong and how to fix it. But the amount of time saved by simply not having to type so much is so great.
I have renewed faith in the universe. It totally makes sense that vibe coding would be poisoned into useless oblivion so early in this game.