Sarraynta iyo Polysemanticity ee Fasiraadda AI

A neural network represents what it knows as activations, lists of numbers produced inside each layer. Interpretability researchers call the meaningful properties those numbers encode features: for example, the idea that a sentence is in French, that an image contains a curve, or that a piece of code has a security bug. The hope behind early interpretability work was that each neuron, one entry in a layer's activation list, would correspond to one feature. Sometimes that holds: vision models contain clean curve detectors. Often it does not. A polysemantic neuron is one that responds to several unrelated inputs. The 2020 Distill article Zoom In, by Chris Olah and colleagues then at OpenAI, gave a famous example: neuron 4e:55 in the image model InceptionV1 fires for cat faces, the fronts of cars and cat legs.

Superposition is the leading explanation for why this happens. The idea is that a model needs to track far more features than it has neurons, so it does not give each feature its own neuron. Instead, each feature gets a direction, a particular pattern across many neurons, and those directions are allowed to overlap a little. Two facts from mathematics make this workable. In a space with n dimensions only n directions can be exactly at right angles, but exponentially many can be almost at right angles, a result related to the Johnson-Lindenstrauss lemma. And compressed sensing shows that a vector squeezed into fewer dimensions can still be recovered if it is sparse, meaning most of its entries are zero. Real features are sparse: any given sentence mentions very few of the things a model knows about.

The cost of overlapping directions is interference. When feature A is active, a little of its signal leaks onto feature B's direction. If features rarely occur together, the leak is usually small and a nonlinearity such as ReLU, which zeros out negative values, can filter it away. Polysemanticity is what this looks like from the point of view of a single neuron: because each neuron participates in many feature directions, it lights up for all of those unrelated features. So superposition is a claim about how features are stored, and polysemanticity is the symptom seen when examining neurons one at a time.

The key evidence came from Toy Models of Superposition, published by Anthropic and Harvard researchers on 14 September 2022, with Nelson Elhage as a lead author and Olah as corresponding author. The team trained very small ReLU networks to compress synthetic inputs into fewer dimensions and reconstruct them. Each input feature had a sparsity, the probability of being zero, and an importance, a weight on its reconstruction error. With dense features, the model behaved like principal component analysis and simply kept the most important ones. As features became sparser, it switched abruptly to storing extra features in superposition, a transition the paper describes as a phase change. The stored features arranged themselves into regular geometric shapes such as pairs, triangles, pentagons and tetrahedra. The paper also showed that at least simple computation, an absolute value function, can be carried out while in superposition, and found preliminary links between superposition and adversarial examples.

If features are directions rather than neurons, the practical problem becomes finding those directions. The main method is dictionary learning with a sparse autoencoder (SAE): a separate small network trained to rewrite a model's activations as a sparse combination of many more learned directions than there are neurons. Anthropic's Towards Monosemanticity, published 4 October 2023, applied this to a one-layer transformer with a 512-neuron MLP layer. Trained on 8 billion activation samples, its dictionaries ranged from 512 to 131,072 features and found interpretable ones for DNA, base64, Hebrew and Arabic script. A monosemantic feature is one that responds to a single understandable concept. The paper also showed feature splitting: one feature in a small dictionary breaks into several finer features in a larger one.

The method then scaled to production models. Anthropic's Scaling Monosemanticity, published 21 May 2024, trained SAEs with about 1 million, 4 million and 34 million features on the middle layer of Claude 3 Sonnet. It found features for the Golden Gate Bridge, brain sciences and tourist attractions, as well as safety-relevant features for code backdoors, bias, deception, sycophancy and dangerous content, and showed that clamping a feature to a high value changed the model's behavior. In June 2024 OpenAI researchers reported a 16 million latent SAE trained on GPT-4 activations over 40 billion tokens, and in August 2024 Google DeepMind released Gemma Scope, open SAEs covering all layers of Gemma 2 2B and 9B.

Compared with alternatives, the superposition view explains why simply reading neurons, the approach of early feature visualization, hits a ceiling. The obvious fix, designing models that never use superposition, did not work out: the 2023 paper reports that forcing activation sparsity during training, even down to one active neuron at a time, removed superposition but still left polysemantic neurons, because a model can lower its loss by letting one neuron stand for several mutually exclusive features. Probing, training a small classifier to detect whether a known concept is present, still works but only finds concepts researchers already thought to look for. SAEs are unsupervised, so they can surface unexpected features.

The limits are real. Superposition is a hypothesis supported by toy models and indirect evidence, not a proven description of large models. SAE features are only an approximation: in Scaling Monosemanticity the reconstructions explained at least 65% of the variance of the activations, so a sizable remainder went unexplained, and about 65% of the features in the 34 million SAE were dead, meaning they never fired on a sample of 10 million tokens. The paper cautions that a feature about lying is not evidence that a model lies. Feature labels come from people or models reading examples, which can be wrong. And training SAEs is expensive: compute grows with the product of the number of features and the number of training tokens, so covering every layer of a frontier model is a large engineering project.