It was resolved just over 100 years ago by Wittgenstein. Either you fully define "understanding", in which case you've answered your question, or you don't clearly define it, it in which case you can't have a meaningful discussion about it because you don't even have an agreement on exactly what the word means.
That sounds clever, but it is actually devoid of both insight and information. There is nothing to be learned from that wit, witty as it is. Contrast with the question it is cleverly dismissing, which could potentially help us move technology forward.
Betrand paradox (in probability) (https://en.wikipedia.org/wiki/Bertrand_paradox_(probability)) is a great counter example of how you can have a meaningful discussion with a poorly defined question. A bit of a meta example, as the meaningful discussion of how what appears to be a well defined question isn't, and how specifically defining a question can give different answer. It also shows that there are multiple correct answers for different definitions of the question. While they disagree, they are all correct within their own definition.
Back to the topic of neural networks, just talking about why the question is hard to clearly define can be a meaningful discussion.
Didn't Wittgenstein shift a bit later in career. All 'word' play is built on other words, and language turns into a self-referential house of cards? Didn't his grand plan to define everything fall apart and he gave up on it?
I thought problem was you couldn't get anywhere because of the question, but you're saying the problem is the question didn't get an answer that satisfies your taste?
The problem, as stated by the parent, is that it is "still pretty unclear at a fundamental level what it means to understand, learn, or conceptualize things."
Which Wittgenstein didn't resolve, he describes how to kick the can down the road. Which is fine, every science needs to make assumptions to move on, but in no way is that a "resolution" to the problem of "what it means to understand, learn, or conceptualize things."
A strict definition is almost never required outside maths. We got so far being unable to define "woman" it turns out.
The most naive meaning of understanding, such as "demonstrating ability to apply a concept to a wide range of situations" is good enough for many cases, Goedel be damned.
That's a pretty useless answer. Just because you cannot fully define something doesn't mean you cannot define parts of it or have different useful definitions of it.
