What happened
Researchers analyzed five language models from four model families using multilingual grade-school math problems in English, German, French, Spanish, Russian and Chinese. They used sparse autoencoders and feature-swapping experiments to study whether models rely on shared internal representations across languages.
The preprint by Igor Bogdanov and Changcheng Huang investigates whether language models solve equivalent mathematical problems through common internal features or through language-specific computations that merely produce similar answers. The researchers used the Multilingual Grade School Math dataset and retained problems for which the models produced valid reasoning traces in all six tested languages: English, German, French, Spanish, Russian and Chinese. They replayed those traces through each model and recorded internal representations at multiple layers. The source describes the work as an examination of five models from four model families.
The researchers first used Centered Kernel Alignment, or CKA, to identify layers where representations were aligned across languages. At each selected layer, they trained two sparse autoencoders. One was a reconstruction-only baseline. The other, introduced in the paper, was called a Geometry-Invariant Sparse Autoencoder, or GI-SAE. GI-SAE added an Information Noise-Contrastive Estimation loss designed to make the encoder produce similar activations for traces representing the same problem, even when those traces used different languages or token positions. This gave the researchers a way to identify features that appeared geometrically shared across languages.
The paper then tested whether those apparently shared features actually played interchangeable roles. During a model’s forward pass, the researchers swapped feature values between languages and measured the resulting output change using Kullback-Leibler divergence per feature. According to the source, GI-SAE produced higher CKA and Jaccard similarity at nearly every layer. However, the increase in geometric similarity did not consistently produce greater functional interchangeability. The reported pattern was model-specific: GI-SAE strengthened cross-language structure in Qwen, produced no functional benefit in Gemma, and had mixed, layer-dependent effects in Llama and Phi. The work was accepted as a poster at the ICML 2026 Workshop on Mechanistic Interpretability.
Taken together, the experimental design compared representation geometry with the results of direct feature interventions. The multilingual traces provided matched problem contexts across English, German, French, Spanish, Russian and Chinese, while the sparse autoencoders provided the feature spaces used for comparison. CKA and Jaccard similarity described geometric alignment, and Kullback-Leibler divergence per feature described output change after swapping. The comparison therefore asked whether the features that looked shared also behaved as interchangeable components. Its answer varied by model family and layer: the geometric effect was broadly stronger with GI-SAE, while functional effects were not consistently stronger. The result is reported for the five models from four model families in the workshop paper.
Read the primary source: arxiv.org ↗
Why it matters
The findings suggest that multilingual reasoning may be partly shared inside a model, but the location and practical usefulness of that sharing depend on the model architecture. This complicates efforts to interpret, audit or improve multilingual AI systems using surface-level similarity measures.
The central implication is that similar-looking internal representations should not automatically be treated as evidence that a model uses the same computation across languages. In this study, a method specifically designed to amplify shared geometry generally increased measured similarity, but that similarity did not reliably predict what happened when features were functionally intervened on. For people evaluating multilingual AI, the distinction matters: a model can organize information similarly across languages without allowing the corresponding internal features to be substituted safely or usefully.
The findings also point to architecture as an important variable in multilingual reasoning. The source reports that cross-language feature sharing appeared at different depths in different models, and that the practical effect of GI-SAE varied across Qwen, Gemma, Llama and Phi. That makes broad claims about how all multilingual language models reason less reliable. Interpretability tools may need to be calibrated to individual model families and layers rather than applied as if internal representations had a common structure.
There is a practical research value in separating geometric alignment from functional interchangeability. Feature-level interventions are often used to investigate what a model is doing internally, and the paper suggests that similarity metrics alone may give an incomplete picture of whether a representation has a causal or operational role. The source does not establish that the method improves model accuracy, translation quality, safety or deployment performance. It reports an interpretability result in a controlled mathematical-reasoning setting, so its direct effect on users remains unknown.
What to watch next
The main open questions are whether the result holds beyond grade-school mathematics and the six tested languages, whether the reported patterns replicate across model versions, and whether shared features can improve reliability or safety in deployed multilingual systems.
Further scrutiny should focus on the exact models, checkpoints, layer locations, dataset sizes and effect sizes used in the experiments. Those details are not included in the source text provided here. The source also does not say whether the authors released code, trained autoencoders or evaluation data, so independent replication will be important for determining how robust the reported model-specific pattern is.
A key test will be whether the result extends beyond the Multilingual Grade School Math dataset. Follow-up work could examine harder mathematics, factual question answering, translation, coding or other reasoning tasks, as well as languages with different scripts and linguistic structures. The current source does not establish that the same cross-language patterns occur outside the six-language mathematical setting or beyond the particular models studied.
Researchers should also test whether feature swapping changes measurable behavior such as answer accuracy, reasoning validity or error patterns, rather than only output distributions summarized by KL divergence. If shared features can be linked to reliable behavioral effects, they might become useful for multilingual debugging or targeted interventions. If not, the study’s main contribution will remain a caution that internal geometric resemblance is not, by itself, proof of shared computation.
The paper’s workshop acceptance provides a venue for discussion, but it does not resolve the study’s broader limitations. Important unknowns include how sensitive the results are to the selected reasoning traces, the choice of sparse-autoencoder settings and the choice of intervention layers. A full peer-reviewed evaluation and tests across more models would help establish whether the reported architecture-dependent behavior is a general property of multilingual language models.
