LLM+A: Grounding Large Language Models in Physical World with Affordance Prompting

Abstract

While large language models (LLMs) are successful in completing various language processing tasks, they easily fail to interact with the physical world properly such as generating control sequences. We find that the main reason is that LLMs are not grounded in the physical world. Existing LLM-based approaches circumvent this problem by relying on additional pre-defined skills or pre-trained sub-policies, making it hard to adapt to new tasks. In contrast, we aim to address this problem and explore the possibility to prompt pre-trained LLMs to accomplish a series of robotic manipulation tasks in a training-free paradigm. Accordingly, we propose a framework called LLM+A(ffordance), where the LLM serves as both the sub-task planner (that generates high-level plans) and the motion controller (that generates low-level control sequences). To ground these plans and control sequences on the physical world, we develop the \textit{affordance prompting} technique that stimulates the LLM to 1) predict the consequences of generated plans and 2) generate affordance values for relevant objects. Empirically, we evaluate the effectiveness of LLM+A in various robotic manipulation tasks with natural language instructions and demonstrate that our approach substantially improves the performance by enhancing the feasibility of generated plans and control.