概述
Agents may cooperate, compete, or have mixed incentives, and each agent’s learning changes the environment faced by others.
深入探討
Lowe et al. describe MADDPG, which uses centralized critics during training for multi-agent continuous control and decentralized actors at execution. Rashid et al.’s QMIX uses a monotonic mixing network over individual utilities; Sunehag et al.’s VDN sums individual action-values. These are methods for particular cooperative or mixed settings, not interchangeable solutions. Markov games extend the MDP framing to multiple agents with state transitions and agent-specific actions/rewards.
現實世界的實施
In a cooperative benchmark, agents can share a team reward while observing different parts of the environment.
Centralized-training methods may use global state during training and local observations during execution.
Value decomposition methods such as VDN and QMIX represent team values using individual agent values under different structural assumptions.
Self-play can train against copies or versions of a policy, as evaluated in particular competitive-game studies.
多智能體強化學習的未來
Choose methods by the environment’s observability, reward structure, communication limits and evaluation task. Report benchmark and training conditions because a result on one multi-agent scenario does not guarantee transfer to another. The cited papers do not establish broad current deployments in warehouses, traffic or healthcare.
付諸實踐
為什麼這很重要
- 更明確的決策. 它可以幫助您將清晰的技術聲明與行銷語言分開。
- 成本與預算. 在花費金錢或時間之前,您可以提出更好的實施問題。
- 團隊與工作流程. 具有共同理解的團隊可以做出更好的產品、政策和學習決策。
留意
- 不同的團隊可能會以不同的方式使用相同術語,因此請儘早定義範圍。
- 基準測試可能看起來很強大,但實際效能卻參差不齊。
- 忽視數據品質和評估計劃通常會產生脆弱的結果。
後續步驟
- 從您需要的結果的簡單語言定義開始。
- 在測試之前選擇一種成功指標和一種失敗條件。
- 使用代表性資料運行小型試點,而不是完善的演示集。
- 記錄多智能體強化學習在哪些方面有幫助以及在哪些方面較簡單的方法較好。
不斷探索
常見問題
What is Multi-Agent Reinforcement Learning?
Multi-agent reinforcement learning studies settings where multiple decision-making agents learn in a shared environment. Agents may cooperate, compete, or have mixed incentives, and each agent’s learning changes the environment faced by others.
從 MARL 中一個智能體的角度來看,是什麼使得環境「不穩定」?
由於每個智能體在訓練期間都會更新其策略,因此每個智能體實際上都面臨著一個移動目標——環境的動態隨著其他智能體的學習而變化。
「集中培訓分散執行」(CTDE)是什麼意思?
MADDPG 和 QMIX 等 CTDE 方法利用全域資訊進行穩定學習,而每個代理程式僅使用自己的觀察結果執行。
哪一種數學框架將單智能體 MDP 推廣到多智能體?
馬可夫博弈(也稱為隨機博弈)透過跨多個決策者的聯合行動和每個智能體獎勵來擴展 MDP。
在純粹合作的 MARL 環境中,獎勵通常是如何構成的?
合作設定為代理商提供了共同的目標,因此挑戰變成了協調行動並在團隊內分配信用。
Which training approach did OpenAI Five use to improve at Dota 2?
The OpenAI Five paper describes large-scale self-play, including games against versions of the policy. AlphaStar also used human-game data in its training, so this item is specific to OpenAI Five.