What happened
Three researchers posted an arXiv preprint reporting a layer-by-layer sensitivity study of a 35-billion-parameter Mixture-of-Experts model, disabling low-magnitude experts and measuring output quality on a cross-lingual code translation benchmark. They report that early and middle layers degrade sharply under masking while the last five layers tolerate having half their experts switched off.
A preprint posted to arXiv (arXiv:2608.13565, listed under Artificial Intelligence and submitted 25 June 2026) by Pradeep Kumar Sharma, Shantanu Godbole and Hritvik Shrivastava reports a systematic layer-wise sensitivity analysis of a Mixture-of-Experts language model. The paper's stated target is Qwen3.6-35B-A3B, which the abstract describes as having 40 Mixture-of-Experts layers, 256 experts per layer and top-8 routing — 10,240 experts in total, of which eight are activated per token per layer. The evaluation task is XLCoST, a cross-lingual code translation benchmark. According to the abstract, the experiments ran at 100-, 300- and 500-prompt scales across three H100 GPU servers.
The technique is described as magnitude-based expert masking: experts are ranked by weight magnitude and the lowest-ranked ones within a chosen set of layers are switched off, so that routing can no longer select them. The abstract lists "physical weight surgery" as future work, which indicates the masked experts were disabled rather than deleted from memory in these experiments. The authors compare a flat policy — masking the same fraction in every layer — against depth-targeted policies that concentrate masking in later layers.
The headline numbers reported in the abstract are comparative. Flat masking of 30 percent of experts across all layers retained 150 of 300 outputs rated "Good+Similar" at the 300-prompt scale. Late-focused policies retained 249 to 255 of 300 while masking between 640 and 1,145 experts. On a later 500-prompt held-out validation slice, the narrowest policy tested — layers 35 through 39 at 50 percent masking — retained 419 of 500 Good+Similar outputs while masking 640 of the 10,240 total experts, roughly 6 percent of the model's experts. The authors characterise early layers (0-9) and middle layers (10-29) as highly fragile and very-late layers (35-39) as the most tolerant.
The paper also reports a separate lever: cutting routing width from eight active experts per token to six. On a 100-prompt probe the abstract describes a "large observed wall-clock reduction" with no loss of Good+Similar outputs, but states plainly that this change "does not yet compose cleanly with aggressive expert masking" — that is, the two savings do not simply add up.
Several things are not established by the material available. This assessment is based on the arXiv abstract and listing page, not the full PDF. The abstract does not report the unmasked baseline score, so the share of the 419-of-500 result attributable to masking cannot be determined from it; it does not define how outputs were graded as "Good" or "Similar", or by whom or what; it does not quantify the wall-clock reduction; it reports no memory savings; and it names no comparison against existing Mixture-of-Experts pruning methods. The listing shows no indication of peer review or journal acceptance.
Read the primary source: arxiv.org ↗
Why it matters
Mixture-of-Experts models are the dominant way to scale large language models cheaply, but there is little settled guidance on which experts can be cut. The finding suggests the intuitive default — pruning every layer at the same rate — may be the worst option at a given budget, though the paper masks experts rather than removing them, so the hardware savings remain unproven.
Mixture-of-Experts architectures are now the standard way to add parameters to a large language model without a proportional increase in compute per token: only a small subset of experts fires for each token. That design has spread quickly through both open-weight and commercial models, but it changes the compression problem. Classic pruning research targets dense networks where every weight participates in every forward pass. In a sparse model, the question is not only how much to cut but where — and the field has comparatively little published guidance on which experts in which layers actually carry load.
The practical claim here is about that allocation. Uniform pruning is the obvious default, and on this model and benchmark it performed worst: the flat 30 percent policy lost roughly half of usable outputs, while depth-targeted policies masking a comparable or larger number of experts kept over 80 percent. If that pattern generalises, it is a cheap change for anyone compressing a Mixture-of-Experts model — a different choice of which layers to touch, not a new training run or architecture. Depth-aware allocation is also easy to test against an existing pipeline.
The limits on that impact are real and the paper is candid about one of them. Masking an expert stops it being used; it does not free the memory its weights occupy. For Mixture-of-Experts serving, memory is often the binding constraint precisely because all experts must be resident even though few fire per token. Until the weights are physically removed and re-measured, the reported result is evidence about redundancy, not a demonstrated efficiency win. The authors frame weight surgery as the next step rather than a completed one.
For readers outside research, the stake is where these models can run. Compression is one of the main routes by which capable models reach smaller hardware — a single server, an on-premises deployment, a laptop — rather than a rented cluster. But quality cost matters: even in the best reported configuration, about one in six outputs fell outside the Good+Similar categories, and without the unmasked baseline it is not possible to say how much of that gap the masking caused. Code translation is also a task with a clear correctness notion; results there do not automatically transfer to open-ended writing or reasoning.
The depth finding is directionally consistent with a broader line of interpretability work suggesting later transformer layers carry more redundancy than early ones, which do heavier lifting on basic representation. Consistency is not confirmation. This is one model, one benchmark family and one masking criterion, from a preprint whose full methodology has not been independently checked.
What to watch next
The decisive follow-up is whether masking converts into real memory and latency savings once weights are physically removed, and whether the depth pattern holds on other Mixture-of-Experts families and on tasks beyond code translation. Also worth watching: disclosure of the unmasked baseline and grading method, code release, and any peer review.
The clearest test is whether masking survives contact with hardware. The authors name physical weight surgery as the next step; a follow-up that removes the 640 identified experts and reports measured memory footprint, throughput and latency would turn a redundancy claim into an efficiency claim. If removal produces little saving — because of how expert weights are laid out or sharded — the practical case weakens considerably even though the sensitivity finding stands.
Generalisation is the second question. The study covers a single Mixture-of-Experts model with 256 experts per layer and top-8 routing. Whether the same depth gradient appears in models with different expert counts, routing widths, shared-expert designs or training recipes is untested here. So is task coverage: code translation is one benchmark family, and masking that leaves code output intact may still damage multilingual performance, long-context behaviour or multi-step reasoning.
Methodological disclosure matters for interpreting the numbers. Watch for the unmasked baseline, a precise definition of the Good and Similar grades and who or what assigned them, the size of the wall-clock reduction from narrowing routing, and whether magnitude-based selection beats the activation-based expert scoring the authors list as future work. Release of code and the masking configurations would let others replicate the comparison directly.
The composition problem the abstract flags is worth tracking on its own. Routing-width reduction and expert masking are different savings mechanisms, and the paper reports they do not yet stack cleanly. A method that combines them without compounding quality loss would be more consequential than either alone; a finding that they fundamentally interfere would be an equally useful negative result.
Finally, an untouched gap: the study measures output quality on a code benchmark and says nothing about safety-relevant behaviour. Compression changes what a model does, and whether depth-targeted masking affects refusals, calibration or susceptibility to jailbreaks is not addressed. Anyone deploying a masked model would need that evaluated separately, and follow-up work reporting it would be a meaningful addition.
