It’s a toy example of solving an abstract task that using classical techniques would have been impossible, but by combining the abstract semantic capabilities of LLMs with classical techniques becomes fully achievable. The problem with classical agents and other techniques is they fail hard when faced with a landscape that was unanticipated or requires a comprehensive semantic awareness beyond the models domain. LLMs provide a space where abstract semantic “reasoning” can happen. It isn’t a deterministic optimizer in any way, no matter the attempts to show it can do basic math tasks etc with larger models and context lengths. But why does it have to? There are tasks, class as classification, description, semantic analysis, and others it does very well at. By combining techniques you can create agents that are much more robust and autonomous in many different situations that wasn’t practical before.
This example could very well have been a factory robot that can use LLM to assist in solving unanticipated challenges, expanding its generalized capabilities beyond a very narrow and specific task set. For instance, the same machinery can have the ability to assemble many different things with minimal additional programming by providing it with specifications, drawn plans, quality control criteria and examples, etc. The LLM can use those to encode instruction to its specialized sub systems that operate with classical techniques to achieve some goal based optimization. The “glue” so to speak to achieve the abstract task is supplied by the LLM. As it performs if it’s doing things wrong reprogramming is as simple as promoting it with the error and examples of better. Through reinforcement techniques the more it performs the tasks and encounters edges and errors the better it’ll get. Compared to current factory floor robotics this would be a remarkable advancement as floors could be reconfigured more rapidly, and the set of specialized machines could be reduced to a more general and commodity set.
This example could very well have been a factory robot that can use LLM to assist in solving unanticipated challenges, expanding its generalized capabilities beyond a very narrow and specific task set. For instance, the same machinery can have the ability to assemble many different things with minimal additional programming by providing it with specifications, drawn plans, quality control criteria and examples, etc. The LLM can use those to encode instruction to its specialized sub systems that operate with classical techniques to achieve some goal based optimization. The “glue” so to speak to achieve the abstract task is supplied by the LLM. As it performs if it’s doing things wrong reprogramming is as simple as promoting it with the error and examples of better. Through reinforcement techniques the more it performs the tasks and encounters edges and errors the better it’ll get. Compared to current factory floor robotics this would be a remarkable advancement as floors could be reconfigured more rapidly, and the set of specialized machines could be reduced to a more general and commodity set.