Керування активацією та розробка представлення

Inside a transformer language model, every token is represented at every layer by a long list of numbers called an activation vector. The running sum of these vectors that each layer reads from and writes to is called the residual stream. Researchers have found that many human-meaningful properties, such as positive versus negative sentiment, truthfulness or whether the model is about to refuse, correspond to directions in that vector space. Activation steering exploits this: by adding or removing a direction during the forward pass, the computation that produces each token, it shifts the model's behavior while leaving its weights untouched. Representation engineering (RepE) is the name Andy Zou and 20 coauthors gave in October 2023 to the broader approach of studying and controlling models through these population-level representations rather than through individual neurons or circuits.

The basic recipe has three steps. First, write contrasting inputs that differ mainly in the property of interest, for example a prompt about love and one about hate, or honest and dishonest answers to the same question. Second, run them through the model and record activations at a chosen layer. Third, turn the differences into a single direction, either by taking the mean difference or by finding the main axis of variation with principal component analysis (PCA). To steer, add that vector, multiplied by a strength coefficient, to the activations at that layer while the model generates text. To read, take the dot product of a new activation with the vector to score how much of the property is present.

The idea took shape in a burst of papers in 2023. Alexander Matt Turner and colleagues introduced Activation Addition (ActAdd) in August 2023, computing a steering vector from a single prompt pair such as Love versus Hate and reporting state-of-the-art results for shifting sentiment from negative to positive and for detoxification, without any optimization. Kenneth Li and colleagues' Inference-Time Intervention (ITI), a NeurIPS 2023 spotlight, found truthfulness-related directions in a limited set of attention heads using a few hundred examples and raised Alpaca's TruthfulQA truthfulness from 32.5% to 65.1%. The RepE paper followed in October 2023 with a method it calls Linear Artificial Tomography, which applies PCA to differences between paired stimuli, and applied it to honesty, harmlessness and power-seeking. In December 2023, Contrastive Activation Addition (CAA) by Nina Panickssery and colleagues averaged activation differences over many multiple-choice examples to steer Llama 2 Chat and reported that it worked alongside fine-tuning and system prompts with minimal loss of capability.

Two results in 2024 widened public attention. Anthropic's interpretability team used features extracted from Claude 3 Sonnet, described in its Scaling Monosemanticity work, and on May 23, 2024 released Golden Gate Claude, a demo in which one bridge-related feature was turned up so strongly that the model described itself as the bridge. In June 2024, Andy Arditi and colleagues showed that refusal in 13 open chat models is mediated by one direction: removing it made models comply with harmful requests, and adding it made them refuse harmless ones. They used this as a white-box jailbreak and argued that current safety fine-tuning is brittle.

Steering sits alongside more familiar controls. Prompting and system prompts are cheap and need no model access but can be ignored or overridden. Fine-tuning and reinforcement learning from human feedback change weights permanently and need more data and compute; ITI's authors contrast their few hundred examples with the extensive annotation RLHF requires. Steering needs access to the model's internals, so it applies to open-weight models or to developers of closed ones, but it is cheap, reversible and adjustable at inference time. The RepE paper compared several control methods for honesty on Llama-2-Chat: a per-input contrast vector raised 13B accuracy on TruthfulQA multiple choice from 35.9% to 54.0%, outperforming ActAdd and a fixed reading vector, while a fine-tuning variant called LoRRA reached 47.5%.

The method has real failure modes. Strength is a balancing act: ITI's authors report that stronger intervention reduces helpfulness, and Anthropic described Golden Gate Claude as prone to unexpected and even jarring behavior. A direction found from a few prompt pairs may capture something other than the intended concept, such as wording or length, and a vector that works at one layer may do nothing or harm fluency at another. Results on multiple-choice benchmarks do not always carry over to open-ended generation. The same tools also cut both ways: the refusal-direction work shows that anyone with access to an open model's weights can remove a safety behavior with a simple edit.

In practice, steering requires reading and writing a model's activations, which open-source tooling supports for open-weight models through forward hooks in frameworks such as PyTorch. The compute cost is small: recording activations for tens to hundreds of contrast examples, computing a mean or PCA, and adding one vector per token at inference. The larger cost is evaluation, since every vector, layer and coefficient needs testing on held-out tasks to confirm the intended effect and to check for side effects on capability, fluency and safety.