Technische GIDS

Superpositie en polysemanticiteit in de interpreteerbaarheid van AI

Superpositie in AI-interpreteerbaarheid is de manier waarop neurale netwerken veel functies in gedeelde richtingen verpakken, waardoor individuele neuronen er polysemantisch en moeilijker uit te leggen uitzien.

Luister naar deze gidsAI-verteld · ~8 min

Op deze pagina9 minuten lezen
  1. Belangrijkste inzichten
  2. Diepe duik
  3. Packing three features into two dimensions
  4. Implementatie in de echte wereld
  5. De toekomst van superpositie en polysemanticiteit in de interpreteerbaarheid van AI
  6. Het in de praktijk brengen
  7. Bronnen en verdere literatuur
  8. Blijf verkennen
  9. Veelgestelde vragen

Belangrijkste inzichten

  1. A polysemantic neuron fires for several unrelated inputs, such as InceptionV1 neuron 4e:55 responding to cat faces, car fronts and cat legs.
  2. Superposition is the hypothesis that networks store more features than neurons by giving each feature a slightly overlapping direction in activation space.
  3. Anthropic's 2022 toy models showed superposition appears abruptly as features become sparser, with features forming shapes such as pentagons and tetrahedra.
  4. Sparse autoencoders rewrite activations as sparse combinations of many learned directions, recovering far more interpretable features than there are neurons.
  5. Anthropic trained sparse autoencoders with up to 34 million features on Claude 3 Sonnet in 2024 and steered behavior by clamping individual features.
  6. SAE features are approximations: reconstructions leave variance unexplained, many features can be dead, and labels for features can be wrong.

Diepe duik

A neural network represents what it knows as activations, lists of numbers produced inside each layer. Interpretability researchers call the meaningful properties those numbers encode features: for example, the idea that a sentence is in French, that an image contains a curve, or that a piece of code has a security bug. The hope behind early interpretability work was that each neuron, one entry in a layer's activation list, would correspond to one feature. Sometimes that holds: vision models contain clean curve detectors. Often it does not. A polysemantic neuron is one that responds to several unrelated inputs. The 2020 Distill article Zoom In, by Chris Olah and colleagues then at OpenAI, gave a famous example: neuron 4e:55 in the image model InceptionV1 fires for cat faces, the fronts of cars and cat legs.

Superposition is the leading explanation for why this happens. The idea is that a model needs to track far more features than it has neurons, so it does not give each feature its own neuron. Instead, each feature gets a direction, a particular pattern across many neurons, and those directions are allowed to overlap a little. Two facts from mathematics make this workable. In a space with n dimensions only n directions can be exactly at right angles, but exponentially many can be almost at right angles, a result related to the Johnson-Lindenstrauss lemma. And compressed sensing shows that a vector squeezed into fewer dimensions can still be recovered if it is sparse, meaning most of its entries are zero. Real features are sparse: any given sentence mentions very few of the things a model knows about.

The cost of overlapping directions is interference. When feature A is active, a little of its signal leaks onto feature B's direction. If features rarely occur together, the leak is usually small and a nonlinearity such as ReLU, which zeros out negative values, can filter it away. Polysemanticity is what this looks like from the point of view of a single neuron: because each neuron participates in many feature directions, it lights up for all of those unrelated features. So superposition is a claim about how features are stored, and polysemanticity is the symptom seen when examining neurons one at a time.

The key evidence came from Toy Models of Superposition, published by Anthropic and Harvard researchers on 14 September 2022, with Nelson Elhage as a lead author and Olah as corresponding author. The team trained very small ReLU networks to compress synthetic inputs into fewer dimensions and reconstruct them. Each input feature had a sparsity, the probability of being zero, and an importance, a weight on its reconstruction error. With dense features, the model behaved like principal component analysis and simply kept the most important ones. As features became sparser, it switched abruptly to storing extra features in superposition, a transition the paper describes as a phase change. The stored features arranged themselves into regular geometric shapes such as pairs, triangles, pentagons and tetrahedra. The paper also showed that at least simple computation, an absolute value function, can be carried out while in superposition, and found preliminary links between superposition and adversarial examples.

If features are directions rather than neurons, the practical problem becomes finding those directions. The main method is dictionary learning with a sparse autoencoder (SAE): a separate small network trained to rewrite a model's activations as a sparse combination of many more learned directions than there are neurons. Anthropic's Towards Monosemanticity, published 4 October 2023, applied this to a one-layer transformer with a 512-neuron MLP layer. Trained on 8 billion activation samples, its dictionaries ranged from 512 to 131,072 features and found interpretable ones for DNA, base64, Hebrew and Arabic script. A monosemantic feature is one that responds to a single understandable concept. The paper also showed feature splitting: one feature in a small dictionary breaks into several finer features in a larger one.

The method then scaled to production models. Anthropic's Scaling Monosemanticity, published 21 May 2024, trained SAEs with about 1 million, 4 million and 34 million features on the middle layer of Claude 3 Sonnet. It found features for the Golden Gate Bridge, brain sciences and tourist attractions, as well as safety-relevant features for code backdoors, bias, deception, sycophancy and dangerous content, and showed that clamping a feature to a high value changed the model's behavior. In June 2024 OpenAI researchers reported a 16 million latent SAE trained on GPT-4 activations over 40 billion tokens, and in August 2024 Google DeepMind released Gemma Scope, open SAEs covering all layers of Gemma 2 2B and 9B.

Compared with alternatives, the superposition view explains why simply reading neurons, the approach of early feature visualization, hits a ceiling. The obvious fix, designing models that never use superposition, did not work out: the 2023 paper reports that forcing activation sparsity during training, even down to one active neuron at a time, removed superposition but still left polysemantic neurons, because a model can lower its loss by letting one neuron stand for several mutually exclusive features. Probing, training a small classifier to detect whether a known concept is present, still works but only finds concepts researchers already thought to look for. SAEs are unsupervised, so they can surface unexpected features.

The limits are real. Superposition is a hypothesis supported by toy models and indirect evidence, not a proven description of large models. SAE features are only an approximation: in Scaling Monosemanticity the reconstructions explained at least 65% of the variance of the activations, so a sizable remainder went unexplained, and about 65% of the features in the 34 million SAE were dead, meaning they never fired on a sample of 10 million tokens. The paper cautions that a feature about lying is not evidence that a model lies. Feature labels come from people or models reading examples, which can be wrong. And training SAEs is expensive: compute grows with the product of the number of features and the number of training tokens, so covering every layer of a frontier model is a large engineering project.

03Uitgewerkt voorbeeld

Packing three features into two dimensions

  1. Take three features A, B and C and a layer with only two neurons, so there is room for just two exactly perpendicular directions.

  2. Assign each feature a unit-length direction 120 degrees apart, like the corners of a triangle; any two directions now have a dot product of minus 0.5.

  3. Activate only feature A with value 1: the two neurons hold A's direction, and reading back gives A = 1, B = minus 0.5 and C = minus 0.5.

  4. Apply a ReLU, which turns negative values into zero: the readout becomes A = 1, B = 0, C = 0, so the interference is filtered away.

  5. Now activate A and B together: reading back C gives minus 1 and the ReLU removes it, but A and B each read 0.5 instead of 1, so recovery is only partial.

  6. Look at one neuron alone: it carries part of all three features, so in isolation it seems to respond to three unrelated things.

Wat het laat zien

A hand-built superposition of three features in two neurons. The check: single features decode exactly after the ReLU, overlapping features decode with error, and each neuron looks polysemantic. This mirrors why superposition only pays off when features are sparse.

Implementatie in de echte wereld

InceptionV1 neuron 4e:55, described in the 2020 Distill article Zoom In, responds to cat faces, fronts of cars and cat legs, a textbook polysemantic neuron in an image classifier.

Anthropic's 2022 Toy Models of Superposition trained tiny ReLU networks on synthetic sparse data and watched five features pack into two dimensions as a regular pentagon.

Towards Monosemanticity (October 2023) used a sparse autoencoder to pull tens of thousands of interpretable features, such as DNA sequences, base64 text and Arabic script, out of a 512-neuron layer.

Scaling Monosemanticity (May 2024) trained sparse autoencoders with up to 34 million features on Claude 3 Sonnet and found a Golden Gate Bridge feature that could steer the model when clamped high.

Google DeepMind's Gemma Scope (August 2024) released open sparse autoencoders for every layer of Gemma 2 2B and 9B so outside researchers can study features without training their own.

De toekomst van superpositie en polysemanticiteit in de interpreteerbaarheid van AI

Open problems are mostly about whether SAE features are the right units. Anthropic's 2024 paper flags cross-layer superposition, where a feature is spread across several layers, as a reason it worked on the residual stream, and notes that a large share of model variance remains unexplained by its dictionaries. Feature splitting, reported in 2023, means the number of features depends on dictionary size, so there may be no single true list to recover. Researchers are also working on cheaper and better-behaved training, as shown by OpenAI's 2024 k-sparse autoencoders and DeepMind's JumpReLU SAEs in Gemma Scope, and on moving from isolated features to circuits that show how features combine to produce an answer. Whether these tools can reliably detect deception or other risky behavior in frontier models remains untested at scale. The 2022 toy models paper also left open whether superposition explains phenomena such as adversarial examples, which it linked only with preliminary evidence.

Het in de praktijk brengen

Waarom het ertoe doet

  • Kosten en budget. Architectuurbeslissingen bepalen jarenlang de prestaties en bedrijfskosten.
  • Duidelijkere beslissingen. Technisch onderwijs helpt teams bij het kiezen van de juiste stapel, niet alleen de nieuwste.
  • Kwaliteitscontrole. Betere technische keuzes verminderen het aantal betrouwbaarheidsincidenten in de productie.

Pas op voor

  • Het optimaliseren van één benchmark kan bredere systeemzwakheden verbergen.
  • Infrastructuur- en onderhoudskosten worden vaak onderschat.
  • De lacunes op het gebied van beveiliging en waarneembaarheid kunnen groter worden naarmate systemen complexer worden.

Volgende stappen

  1. Definieer latentie-, kwaliteits- en kostendoelen vóór implementatie.
  2. Benchmark onder realistische belasting- en gegevensomstandigheden.
  3. Instrumentbewaking op fouten, drift en gebruikersimpact.
  4. Bereid rollback- en incidentresponspaden voor voordat u gaat schalen.

Bronnen en verdere literatuur

  1. Transformer Circuits Thread (Anthropic)Speelgoedmodellen van superpositie
  2. Transformer Circuits Thread (Anthropic)Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
  3. Transformer Circuits Thread (Anthropic)Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
  4. DistillZoom In: An Introduction to Circuits
  5. arXiv (OpenAI)Scaling and evaluating sparse autoencoders
  6. arXiv (Google DeepMind)Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2

Blijf verkennen

Gratis nieuwsbrief

Ontvang de dagelijkse AI-briefing

Drie verifieerde AI-verhalen elke weekdagmorgen, geschreven in eenvoudig Engels. Gratis voor altijd, geen advertenties.

Eén e-mail elke weekdag. Afmelden met één klik. We verkopen of delen je adres nooit.

Test jezelf

Take the Superposition and Polysemanticity in AI Interpretability quiz

Directe feedback op elke antwoord, en een deelbare certificaat met een verifieerbare ID zodra je een cursus hebt afgerond.

Quiz starten

Ondersteun gratis AI-onderwijs. AI Understanding is een 501(c)(3) non-profit — geen advertenties, geen betaalmuur, nooit. Doe een donatie

Veelgestelde vragen

Wat is superpositie en polysemanticiteit in AI-interpreteerbaarheid?

Superpositie in AI-interpreteerbaarheid is de manier waarop neurale netwerken veel functies in gedeelde richtingen verpakken, waardoor individuele neuronen er polysemantisch en moeilijker uit te leggen uitzien.

Waarnaar verwijst 'superpositie' in neurale netwerken?

Superpositie is de strategie voor het coderen van meer afzonderlijke kenmerken dan dimensies door gebruik te maken van bijna orthogonale, overlappende richtingen in de activeringsruimte.

Welke voorwaarde zorgt ervoor dat superpositie goed werkt ondanks interferentie van kenmerken?

Omdat de meeste functies zelden tegelijkertijd actief zijn, komen botsingen niet vaak voor, waardoor de interferentiekosten laag blijven.

Wat is een 'polysemantisch' neuron?

Polysemantische neuronen vuren op meerdere niet-gerelateerde kenmerken, wat het waarneembare gevolg is van superpositie.

Welk Anthropic artikel introduceerde een gecontroleerd onderzoek naar dit fenomeen in 2022?

'Toy Models of Superposition' gebruikte kleine, controleerbare netwerken om te demonstreren wanneer en hoe functies samenkomen.

Als een laag meer objecten moet bevatten dan afmetingen heeft, wat is dan geometrisch onvermijdelijk?

Er kunnen niet meer onderling orthogonale vectoren passen dan dimensies, zodat objecten uiteindelijk evenveel bijna orthogonale richtingen opleveren met enige overlap.