> it would be much better to examine the internal state and activity of a "thinking system" directly
The particular problem here is complexity management. The 'problem' with thinking is it is a system, very possibly one of the largest and most complex systems we know about. Thinking as we know it in DNA based life is something that's at least 500 million years old and developed a few bits at a time. There is no unwinding the different components of the thinking from each other. Motor skills, reactions, learning, etc are all compressed and mixed together in the same code.
So, going from top down isn't working. But going the other way isn't working either. The computational complexity of a system that gets anywhere close to a biological thinking system requires quadrillions of calculations.
The particular problem here is complexity management. The 'problem' with thinking is it is a system, very possibly one of the largest and most complex systems we know about. Thinking as we know it in DNA based life is something that's at least 500 million years old and developed a few bits at a time. There is no unwinding the different components of the thinking from each other. Motor skills, reactions, learning, etc are all compressed and mixed together in the same code.
So, going from top down isn't working. But going the other way isn't working either. The computational complexity of a system that gets anywhere close to a biological thinking system requires quadrillions of calculations.