RoboGPT : An intelligent agent of making embodied long-term decisions for daily instruction tasks

TL;DR — RoboGPT agent solves daily insturction task with long-term decisions through LLM planning and low-level policy

Abstract

Robotic agents must master common sense and long-term sequential decisions to solve daily tasks through natural language instruction. The developments in Large Language Models (LLMs) in natural language processing have inspired efforts to use LLMs in complex robot planning. Despite LLMs' great generalization and comprehension of instructional tasks, LLM-generated task plans sometimes lack feasibility and correctness. To address the problem, we propose a RoboGPT agent for making embodied long-term decisions for daily tasks, with two modules: 1) LLM-based planning with Re-Plan to break the task into multiple sub-goals; 2) RoboSkill individually designed for sub-goals to learn better navigation and manipulation skills. The LLM-based planning is enhanced with a new robotic dataset and re-plan, called RoboGPT. The new robotic dataset of 67k daily instruction tasks is gathered for fine-tuning the LLaMA model and obtaining RoboGPT. RoboGPT palnner with strong generalization can plan hundreds of daily tasks, and re-plan based on the environment, thereby addressing the nomenclature diversity challenge. Additionally, a low-computational Re-Plan module is designed to allow plans to flexibly adapt to the environment. The proposed RoboGPT agent outperforms SOTA methods on the ALFRED daily tasks. Moreover, RoboGPT palnner exceeds SOTA LLM-based planners like ChatGPT in task-planning rationality for hundreds of unseen daily tasks, and even other domain tasks, while keeping the large model's original broad application and generality.