Mixture-of-Agents Aggregation
Mixture-of-Agents (MoA) is a technique where several language models draft answers and then an aggregator model fuses their best ideas into one improved response.
Overview
It lets a team of open models rival or beat a single top-tier model.
Deep Dive
Introduced in a 2024 paper from Together AI, Mixture-of-Agents organizes multiple LLMs into layers. In the first layer, several 'proposer' models each independently answer the prompt. Their outputs are then concatenated and handed to the next layer, where models again respond, now conditioned on all the previous drafts. After one or more such rounds, a final 'aggregator' model synthesizes everything into a single answer. The core insight, which the authors call the 'collaborativeness of LLMs', is that models produce better responses when shown peers' answers, even imperfect ones. On the AlpacaEval 2.0 benchmark, a MoA built entirely from open-source models reportedly surpassed GPT-4 Omni's score, demonstrating that careful aggregation of diverse, cheaper models can beat a single frontier system.
Technical Insight
MoA differs from simple majority voting: rather than picking one answer, the aggregator reads all candidate responses as context and generates a fresh synthesis, blending strengths and filtering errors. Diversity among proposers helps, so mixing different model families is valuable. The structure is layered, like a deep network where each layer's 'neurons' are whole LLM calls. The trade-off is latency and cost: each layer multiplies the number of inference calls, so MoA spends more compute to lift quality.
Strategic Impact
Speed and scale
Language workflows can move faster without sacrificing consistency.
Access and reach
It expands access across languages and communication styles.
Clearer decisions
Teams can spend more time on judgment while automation handles repetition.
The Future of Mixture-of-Agents Aggregation
Expect MoA-style aggregation to spread as inference gets cheaper and orchestration frameworks mature. Research directions include learning which proposers to trust per query (routing), reducing the latency penalty by running proposers in parallel and pruning weak ones early, and combining MoA with tool-using agents so the aggregator fuses not just text but actions and retrieved evidence. As open models proliferate, ensembling them intelligently becomes an increasingly practical path to frontier-level quality without a single giant model.
Real-World Implementation
Combining three different open chat models as proposers, then using a strong aggregator to produce one polished customer-support reply.
Boosting instruction-following scores on AlpacaEval-style benchmarks using only open-source models.
Fusing diverse code suggestions from several models into a single, more robust function implementation.
Running an open-weights pipeline that approaches frontier quality for a privacy-sensitive deployment where data cannot leave a company's servers.
Risks & Guardrails
Hallucinated facts can quietly enter reports, support flows, or research outputs.
Prompt sensitivity can create inconsistent results across similar requests.
Sensitive text data may be exposed if access controls are weak.
Implementation Roadmap
Define output format, tone, and quality standards before rollout.
Ground responses with trusted sources whenever accuracy matters.
Keep a human review checkpoint for high-stakes outputs.
Track failure patterns and retrain prompts or workflows regularly.
Keep Exploring
Free newsletter
Keep up with AI in 3 minutes a day
One short email each weekday with the three AI stories that actually matter. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Mixture-of-Agents Aggregation quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Next guide
Mixture of Depths
Frequently asked questions
What is Mixture-of-Agents Aggregation?
Mixture-of-Agents (MoA) is a technique where several language models draft answers and then an aggregator model fuses their best ideas into one improved response. It lets a team of open models rival or beat a single top-tier model.
In Mixture-of-Agents, what is the role of the aggregator model?
The aggregator reads all candidate responses and generates one fused, higher-quality answer.
What key property of LLMs does the MoA paper rely on?
The authors observe that models improve their answers when conditioned on other models' drafts, even imperfect ones.
How does MoA differ from simple majority voting?
Unlike voting, MoA does not just select one existing answer; it generates a fresh response that blends the candidates.
Why is diversity among proposer models valuable in MoA?
Varied proposers cover more ground, giving the aggregator a richer set of ideas to fuse and errors to filter.
What is the main practical cost of the MoA approach?
Each layer adds more LLM calls, so MoA trades extra compute and time for improved answer quality.