本頁閱讀時間 9 分鐘
重點摘要
- A polysemantic neuron fires for several unrelated inputs, such as InceptionV1 neuron 4e:55 responding to cat faces, car fronts and cat legs.
- Superposition is the hypothesis that networks store more features than neurons by giving each feature a slightly overlapping direction in activation space.
- Anthropic's 2022 toy models showed superposition appears abruptly as features become sparser, with features forming shapes such as pentagons and tetrahedra.
- Sparse autoencoders rewrite activations as sparse combinations of many learned directions, recovering far more interpretable features than there are neurons.
- Anthropic trained sparse autoencoders with up to 34 million features on Claude 3 Sonnet in 2024 and steered behavior by clamping individual features.
- SAE features are approximations: reconstructions leave variance unexplained, many features can be dead, and labels for features can be wrong.
深入探討
A neural network represents what it knows as activations, lists of numbers produced inside each layer. Interpretability researchers call the meaningful properties those numbers encode features: for example, the idea that a sentence is in French, that an image contains a curve, or that a piece of code has a security bug. The hope behind early interpretability work was that each neuron, one entry in a layer's activation list, would correspond to one feature. Sometimes that holds: vision models contain clean curve detectors. Often it does not. A polysemantic neuron is one that responds to several unrelated inputs. The 2020 Distill article Zoom In, by Chris Olah and colleagues then at OpenAI, gave a famous example: neuron 4e:55 in the image model InceptionV1 fires for cat faces, the fronts of cars and cat legs.
Superposition is the leading explanation for why this happens. The idea is that a model needs to track far more features than it has neurons, so it does not give each feature its own neuron. Instead, each feature gets a direction, a particular pattern across many neurons, and those directions are allowed to overlap a little. Two facts from mathematics make this workable. In a space with n dimensions only n directions can be exactly at right angles, but exponentially many can be almost at right angles, a result related to the Johnson-Lindenstrauss lemma. And compressed sensing shows that a vector squeezed into fewer dimensions can still be recovered if it is sparse, meaning most of its entries are zero. Real features are sparse: any given sentence mentions very few of the things a model knows about.
The cost of overlapping directions is interference. When feature A is active, a little of its signal leaks onto feature B's direction. If features rarely occur together, the leak is usually small and a nonlinearity such as ReLU, which zeros out negative values, can filter it away. Polysemanticity is what this looks like from the point of view of a single neuron: because each neuron participates in many feature directions, it lights up for all of those unrelated features. So superposition is a claim about how features are stored, and polysemanticity is the symptom seen when examining neurons one at a time.
The key evidence came from Toy Models of Superposition, published by Anthropic and Harvard researchers on 14 September 2022, with Nelson Elhage as a lead author and Olah as corresponding author. The team trained very small ReLU networks to compress synthetic inputs into fewer dimensions and reconstruct them. Each input feature had a sparsity, the probability of being zero, and an importance, a weight on its reconstruction error. With dense features, the model behaved like principal component analysis and simply kept the most important ones. As features became sparser, it switched abruptly to storing extra features in superposition, a transition the paper describes as a phase change. The stored features arranged themselves into regular geometric shapes such as pairs, triangles, pentagons and tetrahedra. The paper also showed that at least simple computation, an absolute value function, can be carried out while in superposition, and found preliminary links between superposition and adversarial examples.
If features are directions rather than neurons, the practical problem becomes finding those directions. The main method is dictionary learning with a sparse autoencoder (SAE): a separate small network trained to rewrite a model's activations as a sparse combination of many more learned directions than there are neurons. Anthropic's Towards Monosemanticity, published 4 October 2023, applied this to a one-layer transformer with a 512-neuron MLP layer. Trained on 8 billion activation samples, its dictionaries ranged from 512 to 131,072 features and found interpretable ones for DNA, base64, Hebrew and Arabic script. A monosemantic feature is one that responds to a single understandable concept. The paper also showed feature splitting: one feature in a small dictionary breaks into several finer features in a larger one.
The method then scaled to production models. Anthropic's Scaling Monosemanticity, published 21 May 2024, trained SAEs with about 1 million, 4 million and 34 million features on the middle layer of Claude 3 Sonnet. It found features for the Golden Gate Bridge, brain sciences and tourist attractions, as well as safety-relevant features for code backdoors, bias, deception, sycophancy and dangerous content, and showed that clamping a feature to a high value changed the model's behavior. In June 2024 OpenAI researchers reported a 16 million latent SAE trained on GPT-4 activations over 40 billion tokens, and in August 2024 Google DeepMind released Gemma Scope, open SAEs covering all layers of Gemma 2 2B and 9B.
Compared with alternatives, the superposition view explains why simply reading neurons, the approach of early feature visualization, hits a ceiling. The obvious fix, designing models that never use superposition, did not work out: the 2023 paper reports that forcing activation sparsity during training, even down to one active neuron at a time, removed superposition but still left polysemantic neurons, because a model can lower its loss by letting one neuron stand for several mutually exclusive features. Probing, training a small classifier to detect whether a known concept is present, still works but only finds concepts researchers already thought to look for. SAEs are unsupervised, so they can surface unexpected features.
The limits are real. Superposition is a hypothesis supported by toy models and indirect evidence, not a proven description of large models. SAE features are only an approximation: in Scaling Monosemanticity the reconstructions explained at least 65% of the variance of the activations, so a sizable remainder went unexplained, and about 65% of the features in the 34 million SAE were dead, meaning they never fired on a sample of 10 million tokens. The paper cautions that a feature about lying is not evidence that a model lies. Feature labels come from people or models reading examples, which can be wrong. And training SAEs is expensive: compute grows with the product of the number of features and the number of training tokens, so covering every layer of a frontier model is a large engineering project.
03工作範例
Packing three features into two dimensions
Take three features A, B and C and a layer with only two neurons, so there is room for just two exactly perpendicular directions.
Assign each feature a unit-length direction 120 degrees apart, like the corners of a triangle; any two directions now have a dot product of minus 0.5.
Activate only feature A with value 1: the two neurons hold A's direction, and reading back gives A = 1, B = minus 0.5 and C = minus 0.5.
Apply a ReLU, which turns negative values into zero: the readout becomes A = 1, B = 0, C = 0, so the interference is filtered away.
Now activate A and B together: reading back C gives minus 1 and the ReLU removes it, but A and B each read 0.5 instead of 1, so recovery is only partial.
Look at one neuron alone: it carries part of all three features, so in isolation it seems to respond to three unrelated things.
它顯示了什麼
A hand-built superposition of three features in two neurons. The check: single features decode exactly after the ReLU, overlapping features decode with error, and each neuron looks polysemantic. This mirrors why superposition only pays off when features are sparse.
現實世界的實施
InceptionV1 neuron 4e:55, described in the 2020 Distill article Zoom In, responds to cat faces, fronts of cars and cat legs, a textbook polysemantic neuron in an image classifier.
Anthropic's 2022 Toy Models of Superposition trained tiny ReLU networks on synthetic sparse data and watched five features pack into two dimensions as a regular pentagon.
Towards Monosemanticity (October 2023) used a sparse autoencoder to pull tens of thousands of interpretable features, such as DNA sequences, base64 text and Arabic script, out of a 512-neuron layer.
Scaling Monosemanticity (May 2024) trained sparse autoencoders with up to 34 million features on Claude 3 Sonnet and found a Golden Gate Bridge feature that could steer the model when clamped high.
Google DeepMind's Gemma Scope (August 2024) released open sparse autoencoders for every layer of Gemma 2 2B and 9B so outside researchers can study features without training their own.
人工智慧可解釋性中疊加和多義性的未來
Open problems are mostly about whether SAE features are the right units. Anthropic's 2024 paper flags cross-layer superposition, where a feature is spread across several layers, as a reason it worked on the residual stream, and notes that a large share of model variance remains unexplained by its dictionaries. Feature splitting, reported in 2023, means the number of features depends on dictionary size, so there may be no single true list to recover. Researchers are also working on cheaper and better-behaved training, as shown by OpenAI's 2024 k-sparse autoencoders and DeepMind's JumpReLU SAEs in Gemma Scope, and on moving from isolated features to circuits that show how features combine to produce an answer. Whether these tools can reliably detect deception or other risky behavior in frontier models remains untested at scale. The 2022 toy models paper also left open whether superposition explains phenomena such as adversarial examples, which it linked only with preliminary evidence.
付諸實踐
為什麼這很重要
- 成本與預算. 多年來,架構決策決定著效能和營運成本。
- 更明確的決策. 技術教育幫助團隊選擇正確的堆疊,而不僅僅是最新的堆疊。
- 品質管控. 更好的工程選擇可以減少生產中的可靠性事故。
留意
- 優化一項基準測試可以隱藏更廣泛的系統弱點。
- 基礎設施和維護成本常常被低估。
- 隨著系統變得更加複雜,安全性和可觀察性差距可能會擴大。
後續步驟
- 在實施之前定義延遲、品質和成本目標。
- 在實際負載和資料條件下進行基準測試。
- 儀器監控錯誤、漂移和使用者影響。
- 在擴展之前準備回滾和事件回應路徑。
資料來源與延伸閱讀
- Transformer Circuits Thread (Anthropic)疊加玩具模型
- Transformer Circuits Thread (Anthropic)Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
- Transformer Circuits Thread (Anthropic)Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
- DistillZoom In: An Introduction to Circuits
- arXiv (OpenAI)Scaling and evaluating sparse autoencoders
- arXiv (Google DeepMind)Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2
不斷探索
常見問題
什麼是人工智慧可解釋性中的疊加和多義性?
人工智慧可解釋性中的疊加是神經網路將許多特徵打包到共享方向的方式,這使得單一神經元看起來具有多語義性並且更難以解釋。
神經網路中的「疊加」指的是什麼?
疊加是透過在活化空間中使用近正交、重疊的方向來編碼比維度更獨特的特徵的策略。
儘管存在特徵幹擾,什麼條件才能使疊加正常運作?
由於大多數功能很少同時處於活動狀態,因此衝突很少發生,因此幹擾成本保持較低。
什麼是「多語意」神經元?
多語意神經元會激發多個不相關的特徵,這是疊加的可觀察結果。
哪一篇 Anthropic 論文介紹了 2022 年這現象的對照研究?
「疊加玩具模型」使用小型可控網路來展示特徵何時以及如何組合在一起。
如果一個層必須儲存比其尺寸更多的特徵,那麼在幾何上什麼是不可避免的?
您無法擬合比維度更多的相互正交向量,因此特徵最終會出現盡可能多的幾乎正交的方向,並且有一些重疊。