Paper Finds Late Layers of a Mixture-of-Experts Model Tolerate Heavy Expert Masking
A preprint reports that disabling low-magnitude experts in the last five layers of a 35-billion-parameter Mixture-of-Experts model preserved far more usable code-translation outputs than spreading the same cuts across all layers. It covers one model and one benchmark, and the abstract reports no unmasked baseline.