There are many humans that can't manipulate the world much if at all. The blind, the handicapped, the disabled. are those people unintelligent ?

It's not like LLMs can't be successfully used to control robots.