What happened
A paper revised on arXiv on August 28 proposes Mixture of Activations, or MoA, for Transformer-based language models. The design uses lightweight, input-dependent gates to mix several activation functions for individual tokens while sharing the same linear projections. The authors also introduce learnable activations that combine functions without token-dependent gates.
The authoritative source is an arXiv record for “More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations.” It says the paper was first submitted on May 26, 2026, and revised on August 28, placing the substantive update inside the current news window. The authors are Mingze Wang, Jinbo Wang, Yikuan Xia, Kai Shen, and Shu Zhong. The source identifies the work as a machine-learning and artificial-intelligence paper, with language models as its direct subject.
The paper starts from a limitation the authors identify in common Transformer feedforward network, or FFN, layers. Earlier designs used functions such as ReLU and GELU, while later gated designs include SwiGLU, but the source says most FFNs still apply one fixed nonlinear transformation to every token. The proposed Mixture of Activations instead uses a dictionary of activation functions and lightweight gates whose values depend on the input. The same linear projections are shared, while the mixture of nonlinear functions can vary from token to token.
The source also describes learnable activations, abbreviated LA, as an input-independent counterpart. LA forms linear combinations of activation functions in both ReLU-type and SwiGLU-type FFNs. The authors state that their finite-width theory establishes strict expressive separations: LA strictly contains fixed-activation FFNs, while MoA strictly contains LA because it adds input-dependent nonlinear hybridization. These are theoretical expressivity claims about the functions the layers can represent, not evidence that a deployed language model is generally more capable or reliable.
For empirical testing, the authors say they conducted extensive pretraining experiments on dense and mixture-of-experts language models ranging from 0.12 billion to 2 billion parameters. They report testing different token budgets, optimizers, and learning-rate schedules. According to the abstract, MoA consistently achieved lower terminal loss and showed more favorable scaling behavior than well-tuned baselines, with minimal parameter and computational overhead. The source record does not provide the numerical loss differences, the names of the datasets or baselines, hardware details, or the precise meaning of “minimal” overhead.
Why it matters
Feedforward layers contain a large share of the parameters and nonlinear behavior in many language models. If token-adaptive activation mixing improves training without materially increasing compute or parameters, it could offer a relatively small architectural change with implications for model scaling. The evidence remains limited to the authors’ reported pretraining experiments and does not establish better downstream performance or production efficiency.
The proposal targets a consequential part of current language-model architecture. The source says FFN layers account for a large fraction of Transformer model parameters and nonlinear expressivity. That makes them an important place to seek improvements: a change to the nonlinear transformation could affect how efficiently a model uses its existing width and projections, rather than requiring an entirely different model family.
MoA’s design is potentially practical because it retains shared linear projections and adds lightweight input-dependent gating. In principle, this could allow different tokens to receive different nonlinear treatment while preserving much of the surrounding FFN structure. However, the source does not establish that existing models can be upgraded without retraining, nor does it report implementation details sufficient to determine whether the added gates improve real-world throughput, memory use, or energy consumption.
The reported lower terminal loss is relevant because training loss is a central measure of how well a model fits its pretraining data. The claimed scaling behavior could also matter if the method continues to improve as model size or training compute increases. But terminal loss is not the same as usefulness to people. The source gives no results for factuality, reasoning, coding, multilingual performance, safety, calibration, robustness, or downstream task accuracy, so the practical significance of the reported training gains remains uncertain.
The evidence is also preliminary. It comes from a revised preprint, and the source record presents the authors’ theoretical and empirical conclusions rather than independent validation. The tested range ends at 2 billion parameters, which is materially smaller than many widely deployed language models. The record does not state whether the experiments used multiple random seeds, how the baselines were tuned, whether the method changes inference latency, or whether its gains persist outside the reported pretraining settings.
What to watch next
The important next evidence is the size and consistency of the reported gains, especially beyond the tested 0.12B-to-2B range. Readers should look for exact compute, memory, latency, and parameter overheads; results on downstream tasks; independent replication; and evidence that the method remains useful across architectures, datasets, optimizers, and longer training runs.
The next useful check is quantitative detail. The full paper or later work should show exact loss curves, confidence or run-to-run variation, parameter counts, operation counts, memory requirements, and wall-clock training costs for MoA, LA, and fixed-activation baselines. Because the abstract says the comparisons were against well-tuned baselines, the tuning procedure and compute budget will be important for judging whether the advantage comes from the architecture rather than unequal optimization.
Scale is another unresolved issue. The source reports models from 0.12B to 2B parameters, but it does not say whether the same pattern holds in substantially larger dense or mixture-of-experts systems. Future experiments should test larger models, longer training runs, additional token budgets, and different data mixtures. They should also clarify whether the claimed favorable scaling behavior is measured by loss at a fixed compute budget, by loss at a fixed parameter count, or by another comparison.
Operational costs deserve close attention. MoA introduces a token-dependent gate and a dictionary of activation functions, even if the added parameter and compute burden is described as minimal. Independent measurements should determine whether this produces meaningful changes in accelerator utilization, memory traffic, batching, inference latency, training stability, or energy use. A small theoretical overhead can have a larger systems cost if it disrupts efficient hardware execution.
Finally, readers should look for independent replication and broader evaluation. Results on downstream language, coding, reasoning, and multilingual tasks would show whether lower pretraining loss transfers to capabilities that users notice. Evaluations of reliability and safety would test whether token-adaptive nonlinear behavior introduces new failure patterns. Until those results are available, the paper is best understood as a promising architectural research claim, not a demonstrated production improvement or an immediately available product.