IMO the step most of these projects fail on is UTILITY. I have yet to hear an AI project start with, "We need to do X and I think AI might help." Every single project meeting I've sat in started with an non-technical exec asking, "what can we use AI for?" They've got a solution and go looking for a problem so they can say they're using AI.
At my last company thankfully I was able to limit the losses to just under $450k which sounds like a lot, and it is, but it could have been about $5m/yr. I didn't sabotage anything, either. It was mostly through simply asking a lot of questions early, repeatedly suggesting we write plans down and make sure everyone on the project is up to day, and suggesting we test drive components before buying the whole car.
Spent a bunch of money, rolled out the pilot project to 70 folks, and began 30 day checkins. After 30 days we had a retention rate over 80%. By the 60 day mark, we were down to 40%. By day 90 we were at 22% and landed at 11% after 4 months and never went higher. By month 8 we cancelled the project and the 8 people still using the tool were ok letting it go, feeling it didn't help that much.
All because someone from the board said, "we need to use AI, anyone not using AI in a year will be out of business." The CEO asked what we should use it for, and the board member said, "I have no idea, that's your job. But we need AI somewhere. Look here, it can buy a plane ticket for me now!" and then wasted the next 40 minutes of the meeting talking about how amazing AI agents were. We're not a tech company, he was definitely NOT a tech person.
>Every single project meeting I've sat in started with an non-technical exec asking, "what can we use AI for?"
To be fair, if the goal is to learn and discover how a new tool works and when to use it, then a legitimate strategy is to solve a bunch of problems with this new tool
I totally agree but AI is such a broad category it's like asking "what can we do with electricity?" Well, that depends, what problems are you looking to solve?
You always start with the problem that needs to be solved, not the solution. I'm concerned we'll have another bubble pop like the dotcom crash. A lot of companies are spending money like mad to have any kind of AI thing they can market about without regard to "Does it work right? Does is solve a problem?"
It took my former company a year and half a million dollars to learn that what we actually needed was a completely different tool that promised to do a lot less, but did what we needed it to do quite well. Mgmt got stars in their eyes with all the things the first tool might be able to do someday, and didn't pay attention to what it could do today.
We switched to the other tool at $70/month/user and saw a retention rate over 90% at 6 months. Exact same test group, exact same type of tool, but this tool was more focused and tangible. We went from spending $450k per year at with the first company to $59k/yr with the second company.
AI isn't magic, You can't skip all the project planning steps you'd normally do just because this is AI. Don't fast track it, don't ignore testing data or feedback, etc. It's hard to push back on excitement but it's important that we do. When our emotions carry us away that's when we make worse decisions.
Umm, was your management really willing to spend ~500USD/month/dev (if you hadnt intervened)?? Thats insane. Most companies spend way less (I hope) for AI (even regular IT companies, like where I work). Though I do believe big tech is the craziest, burning all that money for data centers and making it look like sustainable investments on their balance sheets (which they are not).
It’s interesting. On the one hand you’re clearly right.
On the other hand though, with hindsight, imagine the board member saying the same thing about the PC, the internet, mobile computing, cloud computing…
I think both of you are right and it’s only a matter of timing. Time will tell.
probably more than 95% of early internet projects failed, it took like a decade before people really started to have intuitions about what it was good for.
It’s easy to imagine that we’re going to make the AI analog of paypal or amazon, but the vast majority of people flailed at doing anything useful with the web until patterns had been established
> imagine the board member saying the same thing about the PC, the internet, mobile computing, cloud computing…
For the deployment of the PC, I think the vast majority of companies start from an environment where there were already terminal systems or shared workstations available, and the adoption of PCs happened as they became powerful enough to accomplish the tasks that were already being done on the larger computers. It wasn't really a "thou must use PCs, though I know not how they are useful" mandate.
A better example of such a mandate can be found in education, and most of my personal experience with such thou-must-use-technology mandates has been that they've been similarly ineffective as the AI mandate.
Oh, I agree, I'm not saying "AI will always be garbage" but I WOULD say, "yep, mobile computing just isn't ready yet" in 1996. I also said every single day for the past 25 years that "cloud computing is a tool, not a solution." With how many gigs I've had moving workloads back on prem to reduce costs because someone else did a bad lift-and-shift, I think I'm right in tempering excitement. It's just hard to do. Everyone wants new and shiny.
On the other hand, imagine them saying the same thing about blockchain (lots did; there was a lot of "we have been reliably informed this 'block chain' thing is magic; pls use it for something"), or expert systems, or palmpilots, or Minitel, or Segways, or...
At my last company thankfully I was able to limit the losses to just under $450k which sounds like a lot, and it is, but it could have been about $5m/yr. I didn't sabotage anything, either. It was mostly through simply asking a lot of questions early, repeatedly suggesting we write plans down and make sure everyone on the project is up to day, and suggesting we test drive components before buying the whole car.
Spent a bunch of money, rolled out the pilot project to 70 folks, and began 30 day checkins. After 30 days we had a retention rate over 80%. By the 60 day mark, we were down to 40%. By day 90 we were at 22% and landed at 11% after 4 months and never went higher. By month 8 we cancelled the project and the 8 people still using the tool were ok letting it go, feeling it didn't help that much.
All because someone from the board said, "we need to use AI, anyone not using AI in a year will be out of business." The CEO asked what we should use it for, and the board member said, "I have no idea, that's your job. But we need AI somewhere. Look here, it can buy a plane ticket for me now!" and then wasted the next 40 minutes of the meeting talking about how amazing AI agents were. We're not a tech company, he was definitely NOT a tech person.