95% of AI Pilots Fail(selector.ai)
selector.ai
95% of AI Pilots Fail
https://www.selector.ai/blog/95-of-ai-pilots-fail-heres-how-to-be-the-5/
56 comments
Here's the list I have - but I think there were others?
The AI vibe shift is upon us - https://news.ycombinator.com/item?id=45003052 - Aug 2025 (100 comments)
95% of Companies See 'Zero Return' on $30B Generative AI Spend - https://news.ycombinator.com/item?id=44974104 - Aug 2025 (415 comments)
95 per cent of organisations are getting zero return from AI according to MIT - https://news.ycombinator.com/item?id=44956648 - Aug 2025 (14 comments)
95% of generative AI pilots at companies are failing – MIT report - https://news.ycombinator.com/item?id=44941118 - Aug 2025 (167 comments)
The AI vibe shift is upon us - https://news.ycombinator.com/item?id=45003052 - Aug 2025 (100 comments)
95% of Companies See 'Zero Return' on $30B Generative AI Spend - https://news.ycombinator.com/item?id=44974104 - Aug 2025 (415 comments)
95 per cent of organisations are getting zero return from AI according to MIT - https://news.ycombinator.com/item?id=44956648 - Aug 2025 (14 comments)
95% of generative AI pilots at companies are failing – MIT report - https://news.ycombinator.com/item?id=44941118 - Aug 2025 (167 comments)
IMO the step most of these projects fail on is UTILITY. I have yet to hear an AI project start with, "We need to do X and I think AI might help." Every single project meeting I've sat in started with an non-technical exec asking, "what can we use AI for?" They've got a solution and go looking for a problem so they can say they're using AI.
At my last company thankfully I was able to limit the losses to just under $450k which sounds like a lot, and it is, but it could have been about $5m/yr. I didn't sabotage anything, either. It was mostly through simply asking a lot of questions early, repeatedly suggesting we write plans down and make sure everyone on the project is up to day, and suggesting we test drive components before buying the whole car.
Spent a bunch of money, rolled out the pilot project to 70 folks, and began 30 day checkins. After 30 days we had a retention rate over 80%. By the 60 day mark, we were down to 40%. By day 90 we were at 22% and landed at 11% after 4 months and never went higher. By month 8 we cancelled the project and the 8 people still using the tool were ok letting it go, feeling it didn't help that much.
All because someone from the board said, "we need to use AI, anyone not using AI in a year will be out of business." The CEO asked what we should use it for, and the board member said, "I have no idea, that's your job. But we need AI somewhere. Look here, it can buy a plane ticket for me now!" and then wasted the next 40 minutes of the meeting talking about how amazing AI agents were. We're not a tech company, he was definitely NOT a tech person.
At my last company thankfully I was able to limit the losses to just under $450k which sounds like a lot, and it is, but it could have been about $5m/yr. I didn't sabotage anything, either. It was mostly through simply asking a lot of questions early, repeatedly suggesting we write plans down and make sure everyone on the project is up to day, and suggesting we test drive components before buying the whole car.
Spent a bunch of money, rolled out the pilot project to 70 folks, and began 30 day checkins. After 30 days we had a retention rate over 80%. By the 60 day mark, we were down to 40%. By day 90 we were at 22% and landed at 11% after 4 months and never went higher. By month 8 we cancelled the project and the 8 people still using the tool were ok letting it go, feeling it didn't help that much.
All because someone from the board said, "we need to use AI, anyone not using AI in a year will be out of business." The CEO asked what we should use it for, and the board member said, "I have no idea, that's your job. But we need AI somewhere. Look here, it can buy a plane ticket for me now!" and then wasted the next 40 minutes of the meeting talking about how amazing AI agents were. We're not a tech company, he was definitely NOT a tech person.
>Every single project meeting I've sat in started with an non-technical exec asking, "what can we use AI for?"
To be fair, if the goal is to learn and discover how a new tool works and when to use it, then a legitimate strategy is to solve a bunch of problems with this new tool
To be fair, if the goal is to learn and discover how a new tool works and when to use it, then a legitimate strategy is to solve a bunch of problems with this new tool
Realistically that isn't the goal of almost any company these days
The goal is just "make all the money at all costs"
The goal is just "make all the money at all costs"
I totally agree but AI is such a broad category it's like asking "what can we do with electricity?" Well, that depends, what problems are you looking to solve?
You always start with the problem that needs to be solved, not the solution. I'm concerned we'll have another bubble pop like the dotcom crash. A lot of companies are spending money like mad to have any kind of AI thing they can market about without regard to "Does it work right? Does is solve a problem?"
It took my former company a year and half a million dollars to learn that what we actually needed was a completely different tool that promised to do a lot less, but did what we needed it to do quite well. Mgmt got stars in their eyes with all the things the first tool might be able to do someday, and didn't pay attention to what it could do today.
We switched to the other tool at $70/month/user and saw a retention rate over 90% at 6 months. Exact same test group, exact same type of tool, but this tool was more focused and tangible. We went from spending $450k per year at with the first company to $59k/yr with the second company.
AI isn't magic, You can't skip all the project planning steps you'd normally do just because this is AI. Don't fast track it, don't ignore testing data or feedback, etc. It's hard to push back on excitement but it's important that we do. When our emotions carry us away that's when we make worse decisions.
You always start with the problem that needs to be solved, not the solution. I'm concerned we'll have another bubble pop like the dotcom crash. A lot of companies are spending money like mad to have any kind of AI thing they can market about without regard to "Does it work right? Does is solve a problem?"
It took my former company a year and half a million dollars to learn that what we actually needed was a completely different tool that promised to do a lot less, but did what we needed it to do quite well. Mgmt got stars in their eyes with all the things the first tool might be able to do someday, and didn't pay attention to what it could do today.
We switched to the other tool at $70/month/user and saw a retention rate over 90% at 6 months. Exact same test group, exact same type of tool, but this tool was more focused and tangible. We went from spending $450k per year at with the first company to $59k/yr with the second company.
AI isn't magic, You can't skip all the project planning steps you'd normally do just because this is AI. Don't fast track it, don't ignore testing data or feedback, etc. It's hard to push back on excitement but it's important that we do. When our emotions carry us away that's when we make worse decisions.
Umm, was your management really willing to spend ~500USD/month/dev (if you hadnt intervened)?? Thats insane. Most companies spend way less (I hope) for AI (even regular IT companies, like where I work). Though I do believe big tech is the craziest, burning all that money for data centers and making it look like sustainable investments on their balance sheets (which they are not).
Eh, you can go through $500 in one day of coding if you're using anthropic Claude API and don't pay attention. (Not sub)
But aside from that, were did you get that number? I didn't see any direct reference nor anything that would let you break it down to $/dev
But aside from that, were did you get that number? I didn't see any direct reference nor anything that would let you break it down to $/dev
It’s interesting. On the one hand you’re clearly right.
On the other hand though, with hindsight, imagine the board member saying the same thing about the PC, the internet, mobile computing, cloud computing…
I think both of you are right and it’s only a matter of timing. Time will tell.
On the other hand though, with hindsight, imagine the board member saying the same thing about the PC, the internet, mobile computing, cloud computing…
I think both of you are right and it’s only a matter of timing. Time will tell.
probably more than 95% of early internet projects failed, it took like a decade before people really started to have intuitions about what it was good for.
It’s easy to imagine that we’re going to make the AI analog of paypal or amazon, but the vast majority of people flailed at doing anything useful with the web until patterns had been established
It’s easy to imagine that we’re going to make the AI analog of paypal or amazon, but the vast majority of people flailed at doing anything useful with the web until patterns had been established
> imagine the board member saying the same thing about the PC, the internet, mobile computing, cloud computing…
For the deployment of the PC, I think the vast majority of companies start from an environment where there were already terminal systems or shared workstations available, and the adoption of PCs happened as they became powerful enough to accomplish the tasks that were already being done on the larger computers. It wasn't really a "thou must use PCs, though I know not how they are useful" mandate.
A better example of such a mandate can be found in education, and most of my personal experience with such thou-must-use-technology mandates has been that they've been similarly ineffective as the AI mandate.
For the deployment of the PC, I think the vast majority of companies start from an environment where there were already terminal systems or shared workstations available, and the adoption of PCs happened as they became powerful enough to accomplish the tasks that were already being done on the larger computers. It wasn't really a "thou must use PCs, though I know not how they are useful" mandate.
A better example of such a mandate can be found in education, and most of my personal experience with such thou-must-use-technology mandates has been that they've been similarly ineffective as the AI mandate.
'What can we use x for' is a dumb approach for all of those xs today too.
Oh, I agree, I'm not saying "AI will always be garbage" but I WOULD say, "yep, mobile computing just isn't ready yet" in 1996. I also said every single day for the past 25 years that "cloud computing is a tool, not a solution." With how many gigs I've had moving workloads back on prem to reduce costs because someone else did a bad lift-and-shift, I think I'm right in tempering excitement. It's just hard to do. Everyone wants new and shiny.
On the other hand, imagine them saying the same thing about blockchain (lots did; there was a lot of "we have been reliably informed this 'block chain' thing is magic; pls use it for something"), or expert systems, or palmpilots, or Minitel, or Segways, or...
Step one. Don't absentmindedly twirl your mustache while mumbling how you can finally get rid of all these troublesome employees.
But I had a two hour conversation with ChatGPT and we couldn’t come up any scenario where I didn’t end up winning.
This article's structure and wording feel like it was barely tweaked after coming out of ChatGPT.
Anyway, it misses one of the current biggest problems with all things AI in the workplace: It has attracted every resume-builder and ladder-climber who don't know what they're doing. They just want an AI initiative for their resume.
As long as it's not too public (like the Taco Bell AI drive through disaster) it doesn't really matter if it's successful or not. They can spin it as a success and the next company they apply to won't be able to check.
The same thing happened a few years ago when every PM and rising manager was looking for a way to put blockchain into products. Before that it was "big data", and so on.
Anyway, it misses one of the current biggest problems with all things AI in the workplace: It has attracted every resume-builder and ladder-climber who don't know what they're doing. They just want an AI initiative for their resume.
As long as it's not too public (like the Taco Bell AI drive through disaster) it doesn't really matter if it's successful or not. They can spin it as a success and the next company they apply to won't be able to check.
The same thing happened a few years ago when every PM and rising manager was looking for a way to put blockchain into products. Before that it was "big data", and so on.
It's wild to see would-be influencers continue to build upon a pile of sand, each trying to out pace the lingo on one hand and clinging to anecdotes and conjecture on the other. I wish like other technologies there were first principals that everyone could agree on but you still see it here with people offering what worked for them as some kind of objective fact. Someone should study how these influencer tactics are short circuited by this stuff?
It’s a piece of marketing content for an “AI Ops” company that will, presumably, sell you a solution to stop your AI pilot failing.
> This article's structure and wording feel like it was barely tweaked after coming out of ChatGPT.
eugh, yeah that's definitely the case. I stopped reading once I got to the first repeated paragraph about data quality.
eugh, yeah that's definitely the case. I stopped reading once I got to the first repeated paragraph about data quality.
this is an advertisement for a company promising to make your pilot one of the 5% that succeed
I mean sure, but it is based on an actual MIT study. Which they are clearly trying to spin in their favor, but real data nonetheless.
> based on an actual MIT study
Speaking of which, anyone have a link to the study itself? Since the linked marketing post doesn't even provide a link to it, just a bunch of links to their own platform and social media accounts.
Probably better for the submission to link directly to the study, without the extra call-to-action fluff at the end?
Speaking of which, anyone have a link to the study itself? Since the linked marketing post doesn't even provide a link to it, just a bunch of links to their own platform and social media accounts.
Probably better for the submission to link directly to the study, without the extra call-to-action fluff at the end?
It was mentioned and discussed in a fair amount of other articles and blog posts on HN as well. So it was fairly easy to find: https://mlq.ai/media/quarterly_decks/v0.1_State_of_AI_in_Bus...
i found this analysis interesting - https://www.youtube.com/watch?v=5QzqyrnL010
specifically, a study done by the NANDA group at MIT
https://nanda.media.mit.edu/
they seem to be pushing their own take on how to implement agentic AI
frankly this study and the headlines surrounding it all seem like a bunch of marketing spin
https://nanda.media.mit.edu/
they seem to be pushing their own take on how to implement agentic AI
frankly this study and the headlines surrounding it all seem like a bunch of marketing spin
I still remember the 2015 alexa 100M$ fund and the buzz around everything voice first.
https://www.geekwire.com/2015/amazon-makes-echos-alexa-avail...
of course it wasn't as big as this new AI push, but since the 80s AI over promised and under devlivered, it's not the fault of AI, but the greed of the business people.
https://www.geekwire.com/2015/amazon-makes-echos-alexa-avail...
of course it wasn't as big as this new AI push, but since the 80s AI over promised and under devlivered, it's not the fault of AI, but the greed of the business people.
remember Cortana?
I found this company today https://www.usemotion.com/ai-employees/ai-sdr and they are selling AI employees. I understand the need to sell. But if you are selling tools as employees, then there is something that will break eventually. I agree with most of the sentiment, that don't try to invest thinking people will be replaced. Make your employees efficinet with tools.
There's a huge opportunity to pair this with gig workers to embody AI intelligence with human presence. Kind of like the LinkedIn version of Surroo[1].
1. https://surroo.me/
1. https://surroo.me/
stop putting them in control of aircraft then
I made this joke (https://news.ycombinator.com/item?id=44941459) last time (https://news.ycombinator.com/item?id=44941118).
How about if it's just to land the aircraft?
https://www.garmin.com/en-US/autonomi/autoland/
https://www.garmin.com/en-US/autonomi/autoland/
Isn't this accomplished via an almanac of published ILS Category III airports and their runways?
I thought this system just picked the closest runway and broadcasts "get out of my way, I'm making an emergency landing!"
I thought this system just picked the closest runway and broadcasts "get out of my way, I'm making an emergency landing!"
The main reason pilots fail is that people don't see the value.
If they did, they might have addressed many of the issues described in the report as the cause of the failures.
If they did, they might have addressed many of the issues described in the report as the cause of the failures.
Many of the same issues would equally trip up none AI project. One reason these project might fail is because, like so many other things in business, AI is thrown at a project to cover up organisations short coming, terrible procedures, poor data management and many other issues.
I'm not the biggest fan of current AI technology, but claiming that it's a failure because it doesn't magically solves your problems, which was always more organisations/managerial in nature rather than technical, is a little disingenuous.
I'm not the biggest fan of current AI technology, but claiming that it's a failure because it doesn't magically solves your problems, which was always more organisations/managerial in nature rather than technical, is a little disingenuous.
What I wouldn't give to live in a world where people had long term goals.
[deleted]
For that you need the emperor to fuse himself with sandworms so that he can live thousands of years.
To me this is an indictment of the leadership style of the day of charging in, laying everyone off, then putting you on an AI death march project. That just destroys institutional knowledge, your credibility as a leader, while inevitably the project fails (and few on the team believed in it anyway).
Sure this type of leadership can work in limited short term circumstances where you have a very clear and meaningful objective. But if you can’t quickly get the team itself to believe in the project - and make it THEIR project not YOUR project - you’re going to fail.
Sure this type of leadership can work in limited short term circumstances where you have a very clear and meaningful objective. But if you can’t quickly get the team itself to believe in the project - and make it THEIR project not YOUR project - you’re going to fail.
That may be true but what happens when 5% grow exponentially.
if 5% grows exponentially, it'll become 0.025%, then 0.00125%, ...
I only read the headline, so my reaction was "they can't even drive cars right yet!!! Now AI is flying airplanes???"
The main problem is that Ai "intelligence" has vastly outpaced the associated tooling.
Am I the only one who feels like adding LLMs to many products is the equivalent of bringing Path of Exile crafting to the real world? All hail RNG?
95% of Companies See 'Zero Return' on $30B Generative AI Spend - https://news.ycombinator.com/item?id=44974104 - Aug 2025 (415 comments)
95% of generative AI pilots at companies are failing – MIT report - https://news.ycombinator.com/item?id=44941118 - Aug 2025 (167 comments)