Thư viện kỹ năng và đặc vụ Minecraft Voyager

A skill-library agent is an AI agent that saves successful behaviors as reusable units, usually small programs, and retrieves them later instead of solving every problem from scratch. Voyager is the best-known example. It was introduced in the paper 'Voyager: An Open-Ended Embodied Agent with Large Language Models' by Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi 'Jim' Fan and Anima Anandkumar, posted to arXiv on 25 May 2023, with authors from NVIDIA, Caltech, UT Austin, Stanford and other universities. The authors describe it as the first large-language-model-powered lifelong learning agent in Minecraft, meaning it keeps acquiring skills over a long session without a human setting goals.

Minecraft suits this research because it is open-ended. A player starts with nothing, gathers wood, crafts tools, mines stone and iron, and eventually reaches diamond tools, a sequence called the tech tree. There is no single score to maximize. Voyager does not see pixels or press keys. It controls the game through Mineflayer, a JavaScript library for Minecraft bots, on top of the MineDojo research framework. Its actions are programs: GPT-4 writes JavaScript functions that call Mineflayer commands such as mining a block or crafting an item.

Voyager has three components that run in a loop. The first is an automatic curriculum. GPT-4 receives the agent's inventory, equipment, nearby blocks, biome, time of day and lists of completed and failed tasks, and proposes the next task under the overall goal of discovering as many diverse things as possible. If the agent is in a desert, it might learn to harvest sand and cactus before iron. The second is the iterative prompting mechanism. GPT-4 writes a program for the task, the program runs, and three kinds of feedback come back: in-game messages, for example a note that a stone shovel cannot be crafted until two more sticks are available, JavaScript error traces, and a verdict from a second GPT-4 instance acting as a critic that checks whether the task succeeded. The code is revised with that feedback, for up to four rounds, before the curriculum moves on.

The third component is the skill library. When the critic confirms success, the program is stored. Its key is an embedding, a numeric vector capturing meaning, of a short description of the program written by GPT-3.5 and embedded with OpenAI's text-embedding-ada-002. When a new task arrives, Voyager embeds the task plus a GPT-3.5 suggestion for solving it, and retrieves the five most similar stored skills to place in GPT-4's prompt. New programs can call old ones, so a function like crafting an iron pickaxe can reuse smelting and stick-crafting skills. Because skills are saved code rather than changed model weights, learning a new skill cannot overwrite an old one, which the authors present as a way around catastrophic forgetting.

The paper's results are large relative to its baselines, which were prompting agents adapted to the same setup: ReAct, which interleaves reasoning and actions; Reflexion, which adds self-reflection; and AutoGPT, a popular autonomous task loop. Voyager obtained 3.3 times more unique items, traveled 2.3 times longer distances, and unlocked the wooden, stone and iron tool tiers 15.3, 8.5 and 6.4 times faster in prompting iterations. ReAct and Reflexion unlocked no tier within 160 iterations. In a new world with an empty inventory, Voyager solved all four unseen tasks (diamond pickaxe, golden sword, lava bucket, compass) in three of three trials each, while the baselines solved none within 50 iterations. Ablations showed that replacing the automatic curriculum with a random one cut discovered items by 93 percent, removing self-verification cut them by 73 percent, and using GPT-3.5 instead of GPT-4 for code produced 5.7 times fewer unique items.

The comparison with other agent designs clarifies what is new. Reinforcement learning agents for Minecraft learn low-level control from rewards or videos and adjust neural network weights; Voyager uses a frozen model through an API and learns only by accumulating text and code. Pure prompting agents such as ReAct reason and act step by step but keep no durable library, so each episode starts fresh. Later Minecraft agents took other routes: JARVIS-1, posted in November 2023, uses a multimodal memory of past experience with pixel-level perception and reports completing over 200 tasks. The common thread is that memory outside the model weights lets an agent improve with experience.

The limitations are spelled out in the paper. GPT-4 was 15 times more expensive than GPT-3.5 at the time, yet weaker models could not write good enough code. The agent sometimes gets stuck, and the critic occasionally misjudges success, for example failing to treat spider string as proof of beating a spider. The curriculum can hallucinate impossible goals such as a 'copper sword', and the code writer sometimes uses invalid game logic, such as burning cobblestone as fuel, or calls functions that do not exist. Voyager also depends on Mineflayer's high-level text interface, so it sidesteps the hard perception and motor-control problems that physical robots face.

Running Voyager yourself requires Python 3.9 or later, Node.js 16.13 or later, a Minecraft Java installation with specific Fabric mods (tested on version 1.19), and an OpenAI API key with GPT-4 access, according to the GitHub repository. Saved skill libraries can be reloaded to run new tasks without relearning.