概述
表示工程是一個更廣泛的研究項目,在 2023 年的一篇論文中被命名,旨在尋找這些方向,讀出它們並使用它們來監視或控制模型,而無需重新訓練。
重點摘要
- 激活引導在推理時添加或刪除模型內部激活的方向,以改變行為而無需重新訓練。
- 引導向量通常是根據對比提示之間的激活差異構建的,並使用 PCA 進行平均或匯總。
- 推理時間介入透過轉移一些注意力頭,將 Alpaca 的 TruthfulQA 真實性從 32.5% 提高到 65.1%。
- 發現 13 種開放聊天模式中的拒絕取決於一個方向,可以將其刪除以繞過安全訓練。
- 更強的指導通常會降低幫助或流暢性,因此每個向量都需要仔細調整和評估。
深入探討
Inside a transformer language model, every token is represented at every layer by a long list of numbers called an activation vector. The running sum of these vectors that each layer reads from and writes to is called the residual stream. Researchers have found that many human-meaningful properties, such as positive versus negative sentiment, truthfulness or whether the model is about to refuse, correspond to directions in that vector space. Activation steering exploits this: by adding or removing a direction during the forward pass, the computation that produces each token, it shifts the model's behavior while leaving its weights untouched. Representation engineering (RepE) is the name Andy Zou and 20 coauthors gave in October 2023 to the broader approach of studying and controlling models through these population-level representations rather than through individual neurons or circuits.
The basic recipe has three steps. First, write contrasting inputs that differ mainly in the property of interest, for example a prompt about love and one about hate, or honest and dishonest answers to the same question. Second, run them through the model and record activations at a chosen layer. Third, turn the differences into a single direction, either by taking the mean difference or by finding the main axis of variation with principal component analysis (PCA). To steer, add that vector, multiplied by a strength coefficient, to the activations at that layer while the model generates text. To read, take the dot product of a new activation with the vector to score how much of the property is present.
The idea took shape in a burst of papers in 2023. Alexander Matt Turner and colleagues introduced Activation Addition (ActAdd) in August 2023, computing a steering vector from a single prompt pair such as Love versus Hate and reporting state-of-the-art results for shifting sentiment from negative to positive and for detoxification, without any optimization. Kenneth Li and colleagues' Inference-Time Intervention (ITI), a NeurIPS 2023 spotlight, found truthfulness-related directions in a limited set of attention heads using a few hundred examples and raised Alpaca's TruthfulQA truthfulness from 32.5% to 65.1%. The RepE paper followed in October 2023 with a method it calls Linear Artificial Tomography, which applies PCA to differences between paired stimuli, and applied it to honesty, harmlessness and power-seeking. In December 2023, Contrastive Activation Addition (CAA) by Nina Panickssery and colleagues averaged activation differences over many multiple-choice examples to steer Llama 2 Chat and reported that it worked alongside fine-tuning and system prompts with minimal loss of capability.
Two results in 2024 widened public attention. Anthropic's interpretability team used features extracted from Claude 3 Sonnet, described in its Scaling Monosemanticity work, and on May 23, 2024 released Golden Gate Claude, a demo in which one bridge-related feature was turned up so strongly that the model described itself as the bridge. In June 2024, Andy Arditi and colleagues showed that refusal in 13 open chat models is mediated by one direction: removing it made models comply with harmful requests, and adding it made them refuse harmless ones. They used this as a white-box jailbreak and argued that current safety fine-tuning is brittle.
Steering sits alongside more familiar controls. Prompting and system prompts are cheap and need no model access but can be ignored or overridden. Fine-tuning and reinforcement learning from human feedback change weights permanently and need more data and compute; ITI's authors contrast their few hundred examples with the extensive annotation RLHF requires. Steering needs access to the model's internals, so it applies to open-weight models or to developers of closed ones, but it is cheap, reversible and adjustable at inference time. The RepE paper compared several control methods for honesty on Llama-2-Chat: a per-input contrast vector raised 13B accuracy on TruthfulQA multiple choice from 35.9% to 54.0%, outperforming ActAdd and a fixed reading vector, while a fine-tuning variant called LoRRA reached 47.5%.
The method has real failure modes. Strength is a balancing act: ITI's authors report that stronger intervention reduces helpfulness, and Anthropic described Golden Gate Claude as prone to unexpected and even jarring behavior. A direction found from a few prompt pairs may capture something other than the intended concept, such as wording or length, and a vector that works at one layer may do nothing or harm fluency at another. Results on multiple-choice benchmarks do not always carry over to open-ended generation. The same tools also cut both ways: the refusal-direction work shows that anyone with access to an open model's weights can remove a safety behavior with a simple edit.
In practice, steering requires reading and writing a model's activations, which open-source tooling supports for open-weight models through forward hooks in frameworks such as PyTorch. The compute cost is small: recording activations for tens to hundreds of contrast examples, computing a mean or PCA, and adding one vector per token at inference. The larger cost is evaluation, since every vector, layer and coefficient needs testing on held-out tasks to confirm the intended effect and to check for side effects on capability, fluency and safety.
04工作範例
建構和測試情緒引導向量
選擇一個可以在本地運行的小型開放權重模型,並選擇中間層,例如 12 層模型的第 6 層。
寫 20 對僅情感不同的短句,例如“這部電影很棒”和“這部電影很糟糕”。
使用前向鉤子記錄所選層每個句子的最後一個標記處的剩餘流激活。
將正負差平均以獲得轉向向量 v。
產生中性提示的延續,例如“我去了公園”,沒有轉向,然後在強度為 2、4 和 8 時添加 v。
使用現成的情緒分類器對每個延續進行評分,並記下文字重複或不符合語法的地方。
重複減去 v 以確認效果逆轉。
它顯示了什麼
學習者最終會得到一個可重複使用的方向和一個針對轉向強度的情緒分數小表。工作向量應該隨著強度的增加而提高正面情緒,並在減少時降低正面情緒,而流暢度會在某個高強度處下降,這標誌著可用範圍。
現實世界的實施
Anthropic 的金門 Claude 是一個於 2024 年 5 月 23 日發布、持續約 24 小時的研究演示,它放大了與金門大橋相關的 Claude 3 Sonnet 的一個內部特徵,因此該模型將大多數答案引向金門大橋。
NeurIPS 2023 年焦點論文 Inference-Time Intervention 改變了少數注意力頭的激活,並將基於 LLaMA 的 Alpaca 模型的 TruthfulQA 真實性得分從 32.5% 提高到 65.1%。
對比激活加法透過平均行為的正面和負面示例(例如事實答案與幻覺答案)之間的激活差異來引導 Llama 2 聊天,然後將該向量添加到選定的強度。
Arditi 及其同事在 2024 年發現,在 13 個開源聊天模型中,多達 720 億個參數的拒絕是由單一方向調節的,研究人員可以刪除該方向以阻止拒絕,也可以添加該方向以導致拒絕。
表示工程論文報告稱,添加誠實的對比向量將 Llama-2-13B-Chat 的 TruthfulQA 多項選擇準確率從 35.9% 提高到 54.0%。
啟動指導和表示工程的未來
Research is moving from single hand-built vectors toward features learned at scale. Anthropic's Golden Gate Claude demo in May 2024 used a feature found by its Scaling Monosemanticity work on Claude 3 Sonnet, and Anthropic said the same techniques might one day adjust safety-related features tied to dangerous code, criminal activity or deception. Whether feature-based steering is more precise and reliable than difference-of-means vectors is an open empirical question. A second direction is monitoring. The RepE paper (2023) framed reading vectors as a way to detect states such as dishonesty, and the ITI authors (2023) suggested models may internally track whether statements are true even when their outputs are false. Turning such probes into dependable monitors would need evidence that they generalize across prompts, languages and models. The security picture is mixed. Arditi and colleagues (2024) showed that a single direction controls refusal in 13 open models, which makes safety behaviors in open weights easy to remove. Open problems include steering without degrading capability, choosing layers and strengths automatically, combining several vectors without interference, and building safety that does not depend on one fragile direction.
付諸實踐
為什麼這很重要
- 成本與預算. 多年來,架構決策決定著效能和營運成本。
- 更明確的決策. 技術教育幫助團隊選擇正確的堆疊,而不僅僅是最新的堆疊。
- 品質管控. 更好的工程選擇可以減少生產中的可靠性事故。
留意
- 優化一項基準測試可以隱藏更廣泛的系統弱點。
- 基礎設施和維護成本常常被低估。
- 隨著系統變得更加複雜,安全性和可觀察性差距可能會擴大。
後續步驟
- 在實施之前定義延遲、品質和成本目標。
- 在實際負載和資料條件下進行基準測試。
- 儀器監控錯誤、漂移和使用者影響。
- 在擴展之前準備回滾和事件回應路徑。
資料來源與延伸閱讀
- arXiv表示工程:一種自上而下的人工智慧透明度方法
- arXiv透過啟動工程控制語言模型
- arXiv透過對比活化添加轉向 Llama 2
- arXiv (NeurIPS 2023)推理時間介入:從語言模型中得出真實答案
- arXiv語言模型中的拒絕是由單一方向介導的
- Anthropic金門Claude
不斷探索
常見問題
什麼是激活指導和表示工程?
激活引導透過在運行時編輯其內部激活來改變語言模型的行為方式,通常是添加與誠實、情感或拒絕等概念相關的方向向量。表示工程是一個更廣泛的研究項目,在 2023 年的一篇論文中被命名,旨在尋找這些方向,讀出它們並使用它們來監視或控制模型,而無需重新訓練。
活化轉向在模型運作的哪個點進行幹預?
在推理過程中,轉向會在模型的活化中添加或減去向量,因此不需要重新訓練,並且效果是立竿見影的。
通常如何計算轉向向量?
典型的方法是平均數差:一種行為的平均活化會減去另一種行為,以隔離該概念的方向。
哪個假設是啟動轉向運作的基礎?
轉向依賴被編碼為近似線性方向的概念,因此添加向量會移動相關特徵。
轉向向量控制的比例係數是多少?
將向量乘以較大的係數會使行為變得更加困難;負係數則以相反的方式推動。
過於激進操縱模式的已知風險是什麼?
大規模幹預可能會導致模型的活化偏離正常流形,即使目標行為發生變化,也會損害流暢性和推理能力。