"Offering managers didn’t have technical backgrounds and sometimes came up with ideas for new products that were simply impossible."
Sounds like they drank their own kool-aid, e.g., "Products That Enhance and Amplify Human Expertise," rather than understand the actual limitations and possibilities of ML. And it seems to me that they're still doing it with this nonsense about a human-level "AI" debating stack.
The oversell seems a real shame in light of how much good can be done with EMR and machine learning / NLP.
That's surely part of the problem, but the catalyst is the marketing strategy that is used to brainwash the employees. In essence; sell the experience, not the product.
This works well for IBM generally (the products are shit) but especially well for Watson because it's extremely easy to sell AI without getting bogged down in details. You want to identify brain tumors? We'll just teach Watson to do it.
Whilst IBM research might be able to pull it off, it'll never get to market because there is nobody capable of making good products at IBM anymore.
> Whilst IBM research might be able to pull it off, it'll never get to market because there is nobody capable of making good products at IBM anymore.
As an ex-IBMer this is so true and so frustrating at the same time. Engineers are thrashed about on a nearly sprintly basis by PM's with short attention spans and no understanding of how disruptive their continuously changing requirements are.
It doesn't help that IBM consistently puts the cart before the horse is even born and pivots multiple teams all at the same time such that nothing you build upon is stable or consistent. Working there was maddening.
Sounds like the way non-tech companies operate. Pretty damning that IBM can't understand why leadership of a tech company should have a tech background.
> This works well for IBM generally (the products are shit) but especially well for Watson because it's extremely easy to sell AI without getting bogged down in details.
The cynic in me says that every use of the term AI in any capacity is to sell experience and not functionality. When was the last time you used a product billed as 'AI' and thought 'wow, this is a huge game changer'? Siri is cool, but it's ultimately not super useful. Google translate is incredible, but it can only do what it can do because of the absolutely mind-boggling amount of training data that google can access. Most disciplines have the problem of not enough data, despite what 'big-data' folks say. In contrast, humans can extrapolate and make reliable predictions about the future based on really small sample sizes. We can pick up a new skill or recognize a new pattern with a high degree of accuracy really effing fast compared to a computer. This gives humans an enormous advantage. If IBM and anyone else in this space were really focused on delivering excellent real-world results, step 0 is building out world-class data integration and search tools (which we still actually suck at, weirdly.)
I use & depend upon plenty of products that are built upon AI - GMail spam filtering & categorized inbox, Google image search, YouTube & Netflix recommendations, cheque OCR at my ATM, predictive keyboards on my phone, Amazon's "people also buy with this product" feature, Google translate, computer opponents in games that I play, and all of the signals that feed into Google Search.
The irony is that not one of these bills itself as AI. It's just "a product that works", and the company that produces it is happy to keep the details secret and let users enjoy the product. So you may be right that the term "AI" itself is pure salesmanship. When it starts to work it ceases to be AI.
Also - humans only look like we're fast at picking up new domains because we apply a helluva lot of transfer learning, and most "new" domains aren't actually that different from our previous experiences. Drop a human in an environment where their sensory input is truly novel - say, a sensory deprivation tank where all visual & auditory stimulation is random noise - and they will literally go insane. I've got a 5-month-old and a project where I'm attempting to use AI to parse webpages, and I will bet you that I can teach my computer to read the web before I can teach my kid to do so.
>The irony is that not one of these bills itself as AI. It's just "a product that works"
I think you are on to something, put differently:
If you need to use the term "AI" to enhance the marketability of the product it is probably because the product sucks.
And employees. Google's embrace of the term "AI" isn't because they need help developing or selling AI-powered products, it's to encourage all the kids to go into computer science and all the existing developers to learn TensorFlow. They can then pick off the best of them as potential employees without having to train them up themselves.
None of the things you mentioned are even close to AI. They’re applied statistics, and they mostly use techniques we’ve known about for decades but have only now found a use case because computing and storage is cheap enough to make them viable.
The recommendation, translation, & image classification algorithms are all done with deep-learning; that's considered AI now.
There was a time, not all that long ago, when SVMs, Bayesian networks, and perceptrons were considered AI. That's behind the spam filters, predictive keyboards, and most of the search signals.
There was a time, a bit longer ago, when beam search and A* were considered AI. That's behind the game opponents.
As the linked Wikipedia article says, "AI is whatever we don't know how to do yet." There will be a time (rapidly approaching) where deep learning and robotics are common knowledge among skilled software engineers, and we won't consider them AI either. We'll find something else to call AI then, maybe consciousness or creativity or something.
This is my point: the term AI has always been BS. It was BS when beam search was AI, it was BS when expert systems were AI, and it is equally as BS when applied to neural networks. It comes to the same thing: the 'AI' tools we use are increasingly good function approximators. That's it. It's still reaching the moon by building successively taller ladders.
As much as I look into what’s being done with deep learning, I see they’re all stuck there on the level of associations. Curve fitting. That sounds like sacrilege, to say that all the impressive achievements of deep learning amount to just fitting a curve to data. From the point of view of the mathematical hierarchy, no matter how skillfully you manipulate the data and what you read into the data when you manipulate it, it’s still a curve-fitting exercise, albeit complex and nontrivial.
And
I left the arena to pursue a more challenging task: reasoning with cause and effect. Many of my AI colleagues are still occupied with uncertainty. There are circles of research that continue to work on diagnosis without worrying about the causal aspects of the problem.
the 'AI' tools we use are increasingly good function approximators
Nothing in the definition of AI says that AI has to work the same way the human brain does... and as far as that goes, we're probably not 100% sure that, in the end, the brain is anything more than a really good function approximator and some applied statistics.
I would say the canonical definition of AI, to the extent that there is one, is roughly something like "making computers do things that previously only humans could do". If people think "AI is bullshit" I'd say it's because they're applying their own definition to the term, where there definition imposes much more stringent requirements.
This is an interesting comment - where would you draw the line between AI and applied statistics? A lot of AI which happens to be ML (not saying there is non-ML AI, just that a significant chunk of AI being practiced today is ML) also happens to be applied statistics. Or have statistical interpretations.
Also, because something has been around for decades does not make it not AI. For ex the cheque OCR mentioned probably runs off (or can feasibly run off) of a neural network. I think the parent's comment holds well - not sure about the last line though ...
The line is clear: everything today branded "AI" is just applied statistics. AI is a buzzword. I don't know what the definition of intelligence is, but I have a feeling it doesn't rest anywhere near concepts like function approximation, and that's all even the most sophisticated "AIs" at Google or Facebook or Apple boil down to.
What was not clear from your earlier comment, and is now, is that when you say AI you don't mean AI as is practiced by most of academia and the industry but the vision of Artificial General Intelligence (AGI). If so, yes, that's a good point to make. However, it is debatable whether the path of statistical learning wont lead to AGI, or is not how our brains function, or the truth partly does comprise of statistical learning and part of something else. The Norvig-Chomsky debate is an example of the arguments on both sides.
I didn't make an earlier comment. You're replying to my one and only comment.
> when you say AI you don't mean AI as is practiced by most of academia and the industry but the vision of Artificial General Intelligence (AGI).
What I actually mean is people practicing what they call "AI" in academia and the industry have co-opted the name to make what they do sound more interesting. First it was called "statistics". Then it was called "pattern matching". Then it was called "machine learning". Now it's called "AI". But it hasn't changed meaningfully through any iteration of these labels.
If you can definitely a problem rigorously, you've essentially defined a function. So "function approximation" is basically "general problem solving approximation".
I don't really think that characterization is fair, for example GANs, there is no data set of correct input output pairs for the function that is learned.
Something that actually learns on its own and is not completely stumped when it encounters something new but actually learns. When it recognizes failure it should go and start learning by itself, i.e. try to get more data and analyze that and do its own trial and error - so that it actually grows in capabilities (on its own).
I would argue that all of your examples have failed to be anything even remotely resembling AI, just data crunching to fit most use cases. I don't use GMail but I do regularly use Google image search, Translate, YouTube, Netflix and predictive typing via SwitfKey. And IMHO they all suck horribly (SwiftKey still sucks pretty bad after 8 years of learning from me). Google Translate is getting better and I have recently started using first-pass Google Translate before correcting the mistakes... instead of everything by hand. YT/Netflix Recommendations are always bullshit. I wish there was a way to say "never show me anything like this ever again" because I often feel like 90% of the recommendations make absolutely no sense. Sometimes I think that someone else must be logged into my account clicking on things just to mess with my recommendations. I usually spend a minimum of 30 minutes searching, often giving up out of frustration (and I always have an IMDB tab open to check details because all of the IMDB rating plugins for Firefox stop/ped working). Maybe I'm an edge case living outside the U.S.? Are their algorithms only tuned for English-speaking countries?
The most creative, intelligent and least frustrating "AI" I've ever encountered was in some games, such as Dota2 or many years ago F.E.A.R. They were frustrating but only due to unpredictability, even after hundreds of hours of playtime. YouTube and NetFlix AI after hundreds/thousands of hours invested are also very unpredictable and frustrating, but that's the opposite experience I am looking for in those situations.
Completely have to agree. YouTube has so much content, far more than Spotify, Vimeo and everybody else in the space, which is why I use it. But the recommendations are an offense. YT is only good at 'recommending' stuff I already watched or listened to. What's the point?
Translate can be useful at times...like once a year when I want to comprehend a Japanese website, usually I close the tab after 2 minutes.
I used GMail for many years and still do to some degree but I'm moving to a different mail provider. GMail's spam filter is great!
Not sure, since 2 years it became acceptable to make no difference between ML and AI. ML appears smart because of bizillions of training samples and I feel very impressed when I hear of that. But yeah, at the end of the day it doesn't have exactly the biggest impact on me... ;)
> humans can extrapolate and make reliable predictions about the future based on really small sample sizes
You severely underestimate the bandwidth of your eyes and ears and other senses, and the volume of your brain's memory (despite it's uber-loosy compression). That's terabytes a day probably, if not big data than I dunno what is. Yeah, 99% of it is thrown away at passing through the first few hundreds of layers of your neural networks, but they still know what to throw away...
To get a digital computer on "equal" terms with the zillions of hacky optimizations your semi-analog brain uses you need a shitton of raw power and data volume ("if you don't know what to throw away of the input data, you need to just sift through all/more of it") to compensate for the fact that you don't have N million years of evolution to devise similar hacky optimizations.
Also, humans work as a "network of agents", that's also recurrent (aka "culture"). Current sub-human-level AI agents are far from any sort of reliable interop.
My guess is that we'll get human level performance levels at AGI tasks when we learn to build swarms of AI-agents that cooperate well and "model each other", and few people are working on this... Heck, when it happens it will probably be an "accident" of some IoT optimizations thing, like, "oops, the worldwide network of XYZ industrial monitoring agents just reached sentience and human level intelligence bc it was the only way it could solve the energy-efficiency requirement goals it was tasked to optimize for" :)
In fact, I would say apart from maybe self-driving cars, almost all of the biggest gains from machine learning are in unsexy, hidden backend problems, like automatically rectifying disparate data, optimizing resource utilization, flagging difficult-to-articulate events or triggers in a stream of data too large for human evaluation, machine translation, and other “unsexy” things.
Product interfaces usually offer simple features to users and the value proposition is easy to see. Effective use of machine learning is well hidden upstream in a bunch of unsexy preprocessing or heavy lifting to get to the interface. Not something you’d ever need to emphasize in marketing, except maybe at tech meetups or in recruiting materials, but not to the end consumer.
It just makes pop references to AI-powered products more egregious.
> We can pick up a new skill or recognize a new pattern with a high degree of accuracy really effing fast compared to a computer.
Evolution by natural selection is the OG genetic algorithm, and it's been "running" on billions of organisms in parallel for hundreds of millions of years. The intuition that we take for granted such as the abstract concept of a shape is all hard-coded in our brains from trial and error.
Over the past 10 years I’ve been surprised when anyone smart would join IBM. I understand why people hung on, but why join that dying ship? Now I can’t think of any still there.
Who says IBM is dying? They aren’t anymore (since a long time) at the bleeding edge of research, but they still have a solid consulting and integration business. They also offer decent salaries and good opportunities for sales-oriented technical people.
It wouldn’t be my first choice of employer, however I can see how some people would enjoy working there.
Dying may be extreme. Perhaps it’s safer to say that they are in the mode of returning capital to investors rather than growing.
I’m at a large customer of theirs and they are bleeding the customer for every dollar as they get phased out. Very low caliber of services professionals too.
Agree. Consultancies rarely sell products or solutions. Instead they sell project management by making it sound like you are a more safe bet than a smaller product studio who actually do make products work. Its a real shame but i mostly blame the zero mistake KPI culture primarily fueled by how managers on the client side are promoted.
There's a pretty good podcast interview with Eugene Dubossarsky that has relevant discussion about issues with management and data science in general. https://www.datafuturology.com/podcast/1
Here are a few of my notes (my words not the interviewee's):
- in order to use data science, you have to have creative people thinking about data on the front end
- they don't have to be data scientists, but they need to be creative and want data to support decisions and iteration via feedback loops
- that creativity and desire will lead to "doing good data science"
- management on the receiving end of data science output must be intelligent in terms of synthesizing many inputs and have a strong desire to puzzle through the implications. If management is asking the data science to actually make the decisions - the situation is broken
- data science must be done with provisions for decision support and feedback loops; this is the output that is helping drive the business.
- Lack of desire for decision support and feedback loops leads to "fancy pets" and management using data science as a means to brag about what they are doing; but the data science might not being doing anything to drive the business meaningfully.
- data science that attempts to actually make decisions vs providing decision support is likely in the category of "commodity data science". Corollary : non-commodity data science is the kind that supports decisions in executing higher-level business strategy. Strategy at that level has rather unique attributes and is embedded in unique circumstances for a particular business. This requires a good data scientist to help tackle.
BTW - in listening to that podcast I found a lot of parallels with database design.
whenever I'm asked to design a database for an early-stage system (I work in early stage tech ventures), I ask the following:
- what are the questions that this database should answer for you? How are those questions supporting your business goals 3,6,12 months out? (I'm trying to get to the business requirements here)
- who will be asking those questions (I'm trying to put together some user personas in my head)
- how frequently will they be asking these questions? corollary: how often will historical data be needed? (I'm thinking hot vs cold and complexity of retrieval, minimally required performance)
- how much data to we anticipate is needed to answer the questions (this is really tricky in new ventures - often the answer is more data than what will actually occur in practice in the first year)?
- finally, what systems & tools are people using to ask the questions and be notified of events? (I'm thinking about interfacing, apis)
its all an attempt to stay very focused on the questions and business drivers and the people who use the answers.
This is a great summary. I think his guideposts are helpful for most decision support initiatives, whether you're using data to try and support decisions, or reaching out to humans.
We run prediction markets inside companies and find that if we don't establish a good lifecycle of asking forecasting questions, having people respond with probabilities, then decision makers REACTING to those probabilities in some way (whether they agree with them or not, just acknowledge their existence) the likelihood of the project failing is far higher.
It's double whammy for the Watson stuff, because not only did IBM lose track with AI, they also had that whole cloud thing whoosh by them. So not only do you have marketing telling outrageous lies about the abilities of their AI systems, they're also exaggerating the cloud angle where Watson is a cornerstone.
Yep. But if your marketing and product management are all focused on selling AI instead of IR, then they're not really working toward finding a way to deliver the IR value they have to the people who need it.
I'd actually like to give Watson a spin for an IR problem I'm looking at, but, thanks to their hype machine being set to overdrive, they've got the thing priced in the "The Bold Leaders of the Future Creating a Bright New Tomorrow Full of People in Glasses Staring Wistfully Toward the Right Edge of the Photograph, While Blue Curvy Streaks Wave Through the Background and Random Zeroes and Ones Float Around Their Heads" tier. Sadly, I've only got a "businesses solving business problems" sized budget.
Jeopardy is an information retrieval problem in a game show format, with the minor twist that the query is phrased as a declarative sentence and the response is phrased as an interrogative one.
AI is not simply things that a computer can't do yet. But I think most of people who aren't currently trying to sell a piece of software would expect AI to include some things that you don't need to do to play Jeopardy. I'd want to see general-purpose pattern recognition, for example.
Other reports seemed to indicate that Watson (and maybe all AI) requires a lot of very careful, slow, and somewhat arduous data entry and testing to get good results.
I can imagine someone who doesn't know at IBM selling a product:
"Hey we will solve all these problems like magic!"
Then IBM comes back:
"Hey do you have all this data in a specific format and a ton of time to enter and test it and maybe we'll get back to you???"
That's a big loss of trust there with the customer if you come back with that.
It seems like these are products where a lot of caveats needs to be made clear to customers and a real careful technical partnership formed with them to succeed long term. You have to bring the customer along for the ride and exploration and keep them excited for a long time it sounds to make it work.
Are the articles at ieee.org usually technically competent?
To see one of their articles with the common press confusion of mixing different definitions and interpretations of AI (correct or incorrect) doesn’t help build confidence.
For example, what was used to play Jeopardy vs. approaches being taken to improve cancer treatment, are just so different, it seems almost disingenuous to throw it all haphazardly into one conceptual bucket.
The article does IBM a disservice in some ways. They come off looking bad overall but some of the failed projects mentioned like MD Anderson, failed for reasons beyond any control they had, other than recognizing some obvious red flags earlier and detaching their name and participation from it.
On the other hand I believe the article lets them offf the hook to easily when they bring out the old trope they’ve been using for years, which is encapsulated here:
“IBM Watson has great AI” [one engineer said] “It’s like having great shoes, but not knowing how to walk—they have to figure out how to use it.”
It doesn’t make sense to say, xyz is great we just have to figure out how to use it, as a stand alone argument. It’s nonsensical unless you mention something about the seeming implied untapped potential, specific innovations, novel approach, or whatever makes it great.
I’m not familiar with all their IP so maybe there are some great things, you just don’t get to claim that and get off the hook so many times in the press without providing at least some detail or reference point.
The article only mentions the engineers being laid off, not the managers who botched it, nor the executives who hired with incompetence. The reward structure of such a corporate environment would seem counter-intuitive to success.
IMO, this is a bigger issue in the industry. People are hyping up ML/AI to the point where the actual application is either impossible or extremely difficult. Just look at how many people are so fearful of AI/ML taking away jobs and replacing humans. Anyone in the field knows that AI/ML can take away jobs but it is more of the low-end jobs and everything happens gradually rather than immediately. AI/ML is being more as a tool to enhance human productivity rather than as a direct replacement for entire occupations unless the job is very basic to begin with.
following is a text snippet from the Article . I do not know which part of the following is AI , it is pure "data analysis" writing few SQL queries . This is the Biggest problem of IBM, they bill these kind of things as AI .
> A clinic could use the system to search its patient records and find, for example, all the men over age 45 who were overdue for a colonoscopy, and then use an autocall to remind them to schedule the dreaded appointment.
Being directly involved in the execution of the example you gave, I believe ML would be unnecessary and error-prone.
Even if successful, a system which could "interpret" a health record (such as a freetext note) using anything other than properly codified data would set the health industry back a decade. Moving doctors away from freetexting their notes is the only way to advance the industry.
It will be a generational shift. You won't get current doctors (over 40) to change their ways, ever.
My previous doctor (who was probably mid-50s) didn't use email or any kind of secure electronic messaging system. Everything had to be faxed to him.
My new doctor who is younger uses all kinds of digital tools including a voice recorder with a pre-trained text-to-speech engine that understands medical terminology and codifies the transcription based on keywords.
So it's not entirely getting away from freetext but at least it's extracting some structured data from it automatically.
That is my biggest gripe with this as well. We certainly don't want an another AI winter, especially not in health, where I'm hoping (perhaps too optimistically) that it will allow better and cheaper care.
Sounds like they drank their own kool-aid, e.g., "Products That Enhance and Amplify Human Expertise," rather than understand the actual limitations and possibilities of ML. And it seems to me that they're still doing it with this nonsense about a human-level "AI" debating stack.
The oversell seems a real shame in light of how much good can be done with EMR and machine learning / NLP.