Benchmarking Large Language Models as AI Research Agents

TL;DR — we propose MLAgentBench, the first benchmark for evaluating AI research agents capable of open-ended decision-making on ML research tasks, as well as a prototype LLM-based research agent.

Abstract

Human researchers can perform scientific experimentation loops – planning, experimenting, observing the results, and generating inferences. Can we build AI research agents to perform the same? To take a step towards building and evaluating research agents capable of such open-ended decision-making, we focus on the problem of having agents perform machine learning (ML) tasks given a research problem description and dataset. In this paper, we propose MLAgent- Bench, a suite of ML tasks for benchmarking AI research agents. Agents can perform actions like file system operations, executing code, and inspecting outputs. With these actions, agents could run experiments, analyze the results, and modify the code of entire machine learning pipelines, such as data processing, architecture, training processes, etc. The benchmark then automatically evaluates the agent’s performance objectively over various metrics related to performance and efficiency. We also design an LLM-based research agent to automatically perform experimentation loops in such an environment. Empirically, we find that a GPT-4- based research agent can feasibly build compelling ML models over many tasks in MLAgentBench, displaying highly interpretable plans and actions. However, the success rates vary considerably; they span from almost 90% on well-established older datasets to as low as 10% on recent Kaggle Challenges – unavailable during the LLM model’s pretraining – and even 0% on newer research challenges like BabyLM. Finally, we identify several key challenges for LLM-based research agents such as long-term planning and hallucination. Our code is released at https://anonymous.4open.science/r/MLAgentBench/.