Time will tell, but as a machine learning engineer, when you see results this good it's more probable that a mistake was made. They could be reporting the training error on an overfit model or data leakage could be occuring due to an improper train-test spilt of the data.
Also, it is definitely appropriate to use the term AI in this case. AI is not a technical term so it's really in the eye of the beholder, but I think it's safe to say that ML is a subset of AI. Perhaps people are conflating AI with AGI?
It's a 14-person training set with an 8-person test set, so my guess is that it can pretty accurately predict seizures in the small group of people it is trained on. Whether the model could be kernelized for a useful general deployment is unclear. It still requires many electrodes attached to the scalp, so there is a still a ways to go before it can be integrated into a watch, for example.
They acknowledge this in the article - the system will have to be transfer learned for every patient. Which IMO is ideal, but you will need training/validation data, which in this case sounds like it'd be extremely expensive. Furthermore, the system could get automatically better over time _for that patient_, if properly designed and fed clean training samples.
Yeah watches can't even detect whether you are sleeping or not. The products in the market are mainly accelerometer based and aren't really reliable if you e.g. are awake but don't move.
They're too good and also bad at the same time. Most laypeople don't realize that "99.6%" accuracy is 1/250th chance of making an error. If you do inferences every second, that's 14.4 errors per hour. Now granted, some of those errors are false negatives, but I'm not sure what's worse in this case. Depending on which action is expected after receiving an alarm, this could render the device completely useless.
I think you meant AI is a subset of ML. Agree AGI is a red herring here. Mostly this is a marketing problem, but the pattern rec., ML, "AI", etc. branding has been shifting around for decades.
This set is far too small to say anything strong about the results, but the problem is interesting anyway.
I think I disagree in two regards. One that unless we stretch the meaning of AI to extremes, there is plenty of ML that is not AI. I guess that presupposes we sort out the stats vs. ML issues but that's what you get with all these fuzzy terminologies floating around. So it isn't really useful to think of it as a subset, in my opinon.
Secondly, while I know of (mostly historically) a small amount of serious non-ML AI and AGI work being done, it has almost nothing to do with the common parlance to day, which is nearly entirely ML. Is this what you mean when you talk about non-ML AI or is there something I'm missing?
For what it's worth, I'm happy with thinking of them as overlapping, although I do think the AI terminology is almost useless at the moment, and ML is slightly better defined.
I was not referring to the inconsistent way in which these terms are used to sell products and obtain funding for startups, but to their academic definitions.
While most recent successes in the field of AI have been brought about from advances in the subfield of Machine Learning, even today's most advanced AI systems have components that are not Machine Learning (e.g. Alpha Go still requires tree search techniques from "classic AI" to work).
In the definitions below, AI refers to any "intelligent agent", whereas ML refers to the subset of techniques that achieve this through learning/experience/data.
ML:
"Tom M. Mitchell provided a widely quoted, more formal definition of the algorithms studied in the machine learning field: "A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E."[0]
AI:
"Computer science defines AI research as the study of "intelligent agents": any device that perceives its environment and takes actions that maximize its chance of successfully achieving its goals."
Ok, we are bogged down in semantics, but I see where you are coming from. I don't buy the proper subset argument for reasons above (i.e. I don't buy the broading of AI to fit that idea of "intelligent agent", as it includes too many things that don't really fit, imo).
Unless something has changed radically since I stopped paying as much attention there is not actual agreement on these terminologies, at least broadly, in academic circles.
I certainly agree many current systems with a core ML component include other techniques from lots of areas including what you call "classic AI" as well as optimization, etc., but the ML is still the fundamental part of nearly everything recent I've seen. As pretty much every successful system of this type is a hybrid in the sense you mean, I don't find differentiating them from some putative "pure ML" approach very interesting.
There was some good work in very different approaches in the 70s through 80s, but that seems to have tapered off in the 90s really. I'm not very current though and would love to hear of newer interesting things in that vein.
> Also, I've noticed the definition of the word "AI" has grown to encompass pretty much any type of software that does something with data.
Afaik AGI research is still a separate thing, so it's not really that misleading to use "AI" for applications like this were machine learning is involved, resulting in a very specialized "artificial intelligence" that can spot otherwise hidden patterns.
I think it's ok to call this AI, but I agree that it's a little misleading. I think a better way to describe much of AI these days is pattern recognition. Incredible pattern recognition, to be clear, but pattern recognition.
Agreed about the sample size. It's not clear from the article whether the high accuracy was based on testing against the training set, too. I assume not, as that would be exceptionally sloppy. But it's not clear.
What, do you think that's too low? Sounds pretty normal for this type of study.
Of course, that does mean the results don't necessarily generalize to the whole population, which is why you do eventually need much larger sample sizes when you're working toward FDA approval.
> Also, I've noticed the definition of the word "AI" has grown to encompass pretty much any type of software that does something with data.
That's definitely the case in conferences like O'Reilly Strata. They seem to be 100% focused on AI/ML, which seems really limiting. I'm starting to skip attendance until things get a bit more balanced.
Good catch, I don’t think they mention that in the main article (at least in reader view).
In this case I guess the “AI” label is relatively well applied (compared to the scope of ALL things I’ve seen it haphazardly slapped on), but yeah, still - people need to stop calling any kind of data analysis “AI”.
That's 3 false positives in a month, depending on how many seizures a patient suffers from, this might actually not that bad. Afaik a false positive doesn't have any direct consequences except "be careful and keep medication ready"?
How does this become a treatment, assuming the results actually hold up? Is it possible to prevent seizures with a timely dose of a powerful epilepsy medication? Are there portable EEG rigs that can produce sufficiently powerful readings?
I assume it’d be used in an implant like the RNS, which is used to treat (mostly) focal seizures. It’s a small device implanted in your skull to record localized EEG and provide neurostimulation to quell burgeoning seizures. Currently, it’s periodically adjusted and individualized by neurologists based on the the EEG it records. Maybe this could speed up or automate that process.
At least it lets you get into a safe place and position. Perhaps you can even get someone to watch over and be ready to call 911 if it comes to that.
Edit: Perhaps this hints at some medication that is not currently viable but could be administered if the seisure was predicted: "Notably, seizures are controllable with medication in up to 70 percent of these patients."
Its interesting that service animals like dogs or monkeys can sometimes be trained to anticipate seizures. And in many of these cases its not clear what the signal is- a change in activity, odor etc.
"predict the occurrence of seizures up to one hour before onset"
"We are currently working on the design of an efficient hardware [device] that deploys this algorithm, considering many issues like system size, power consumption, and latency to be suitable for practical application in a comfortable way to the patient"
Also, it is definitely appropriate to use the term AI in this case. AI is not a technical term so it's really in the eye of the beholder, but I think it's safe to say that ML is a subset of AI. Perhaps people are conflating AI with AGI?