概述
Agents may cooperate, compete, or have mixed incentives, and each agent’s learning changes the environment faced by others.
深入探讨
Lowe et al. describe MADDPG, which uses centralized critics during training for multi-agent continuous control and decentralized actors at execution. Rashid et al.’s QMIX uses a monotonic mixing network over individual utilities; Sunehag et al.’s VDN sums individual action-values. These are methods for particular cooperative or mixed settings, not interchangeable solutions. Markov games extend the MDP framing to multiple agents with state transitions and agent-specific actions/rewards.
现实世界的实施
In a cooperative benchmark, agents can share a team reward while observing different parts of the environment.
Centralized-training methods may use global state during training and local observations during execution.
Value decomposition methods such as VDN and QMIX represent team values using individual agent values under different structural assumptions.
Self-play can train against copies or versions of a policy, as evaluated in particular competitive-game studies.
多智能体强化学习的未来
Choose methods by the environment’s observability, reward structure, communication limits and evaluation task. Report benchmark and training conditions because a result on one multi-agent scenario does not guarantee transfer to another. The cited papers do not establish broad current deployments in warehouses, traffic or healthcare.
付诸实践
为什么这很重要
- 更清晰的判决. 它可以帮助您将清晰的技术声明与营销语言分开。
- 成本与预算. 在花费金钱或时间之前，您可以提出更好的实施问题。
- 团队与工作流程. 具有共同理解的团队可以做出更好的产品、政策和学习决策。
留意
- 不同的团队可能会以不同的方式使用同一术语，因此请尽早定义范围。
- 基准测试可能看起来很强大，但实际性能却参差不齐。
- 忽视数据质量和评估计划通常会产生脆弱的结果。
后续步骤
- 从您需要的结果的简单语言定义开始。
- 在测试之前选择一种成功指标和一种失败条件。
- 使用代表性数据运行小型试点，而不是完善的演示集。
- 记录多智能体强化学习在哪些方面有帮助以及在哪些方面更简单的方法更好。
不断探索
常见问题
什么是多智能体强化学习？
Multi-agent reinforcement learning studies settings where multiple decision-making agents learn in a shared environment. Agents may cooperate, compete, or have mixed incentives, and each agent’s learning changes the environment faced by others.
从 MARL 中一个智能体的角度来看，是什么使得环境“不稳定”？
由于每个智能体在训练期间都会更新其策略，因此每个智能体实际上都面临着一个移动目标——环境的动态随着其他智能体的学习而变化。
“集中培训分散执行”（CTDE）是什么意思？
MADDPG 和 QMIX 等 CTDE 方法利用全局信息进行稳定学习，而每个代理仅使用自己的观察结果执行。
哪种数学框架将单智能体 MDP 推广到多智能体？
马尔可夫博弈（也称为随机博弈）通过跨多个决策者的联合行动和每个智能体奖励来扩展 MDP。
在纯粹合作的 MARL 环境中，奖励通常是如何构成的？
合作设置为代理提供了共同的目标，因此挑战变成了协调行动并在团队内分配信用。
Which training approach did OpenAI Five use to improve at Dota 2?
The OpenAI Five paper describes large-scale self-play, including games against versions of the policy. AlphaStar also used human-game data in its training, so this item is specific to OpenAI Five.