Il prossimoProssima guida
Rappresentazioni molecolari per l'apprendimento automatico
Tecnico
GUIDA TECNICA
L'activation steering modifica il comportamento di un modello linguistico modificando le sue attivazioni interne mentre è in esecuzione, solitamente aggiungendo un vettore di direzione collegato a un concetto come onestà, sentimento o rifiuto.
Ascolta questa guidaNarrato dall'IA · ~8 min
L'ingegneria della rappresentazione è il più ampio programma di ricerca, denominato in un articolo del 2023, volto a trovare tali direzioni, leggerle e usarle per monitorare o controllare i modelli senza riaddestramento.
Inside a transformer language model, every token is represented at every layer by a long list of numbers called an activation vector. The running sum of these vectors that each layer reads from and writes to is called the residual stream. Researchers have found that many human-meaningful properties, such as positive versus negative sentiment, truthfulness or whether the model is about to refuse, correspond to directions in that vector space. Activation steering exploits this: by adding or removing a direction during the forward pass, the computation that produces each token, it shifts the model's behavior while leaving its weights untouched. Representation engineering (RepE) is the name Andy Zou and 20 coauthors gave in October 2023 to the broader approach of studying and controlling models through these population-level representations rather than through individual neurons or circuits.
The basic recipe has three steps. First, write contrasting inputs that differ mainly in the property of interest, for example a prompt about love and one about hate, or honest and dishonest answers to the same question. Second, run them through the model and record activations at a chosen layer. Third, turn the differences into a single direction, either by taking the mean difference or by finding the main axis of variation with principal component analysis (PCA). To steer, add that vector, multiplied by a strength coefficient, to the activations at that layer while the model generates text. To read, take the dot product of a new activation with the vector to score how much of the property is present.
The idea took shape in a burst of papers in 2023. Alexander Matt Turner and colleagues introduced Activation Addition (ActAdd) in August 2023, computing a steering vector from a single prompt pair such as Love versus Hate and reporting state-of-the-art results for shifting sentiment from negative to positive and for detoxification, without any optimization. Kenneth Li and colleagues' Inference-Time Intervention (ITI), a NeurIPS 2023 spotlight, found truthfulness-related directions in a limited set of attention heads using a few hundred examples and raised Alpaca's TruthfulQA truthfulness from 32.5% to 65.1%. The RepE paper followed in October 2023 with a method it calls Linear Artificial Tomography, which applies PCA to differences between paired stimuli, and applied it to honesty, harmlessness and power-seeking. In December 2023, Contrastive Activation Addition (CAA) by Nina Panickssery and colleagues averaged activation differences over many multiple-choice examples to steer Llama 2 Chat and reported that it worked alongside fine-tuning and system prompts with minimal loss of capability.
Two results in 2024 widened public attention. Anthropic's interpretability team used features extracted from Claude 3 Sonnet, described in its Scaling Monosemanticity work, and on May 23, 2024 released Golden Gate Claude, a demo in which one bridge-related feature was turned up so strongly that the model described itself as the bridge. In June 2024, Andy Arditi and colleagues showed that refusal in 13 open chat models is mediated by one direction: removing it made models comply with harmful requests, and adding it made them refuse harmless ones. They used this as a white-box jailbreak and argued that current safety fine-tuning is brittle.
Steering sits alongside more familiar controls. Prompting and system prompts are cheap and need no model access but can be ignored or overridden. Fine-tuning and reinforcement learning from human feedback change weights permanently and need more data and compute; ITI's authors contrast their few hundred examples with the extensive annotation RLHF requires. Steering needs access to the model's internals, so it applies to open-weight models or to developers of closed ones, but it is cheap, reversible and adjustable at inference time. The RepE paper compared several control methods for honesty on Llama-2-Chat: a per-input contrast vector raised 13B accuracy on TruthfulQA multiple choice from 35.9% to 54.0%, outperforming ActAdd and a fixed reading vector, while a fine-tuning variant called LoRRA reached 47.5%.
The method has real failure modes. Strength is a balancing act: ITI's authors report that stronger intervention reduces helpfulness, and Anthropic described Golden Gate Claude as prone to unexpected and even jarring behavior. A direction found from a few prompt pairs may capture something other than the intended concept, such as wording or length, and a vector that works at one layer may do nothing or harm fluency at another. Results on multiple-choice benchmarks do not always carry over to open-ended generation. The same tools also cut both ways: the refusal-direction work shows that anyone with access to an open model's weights can remove a safety behavior with a simple edit.
In practice, steering requires reading and writing a model's activations, which open-source tooling supports for open-weight models through forward hooks in frameworks such as PyTorch. The compute cost is small: recording activations for tens to hundreds of contrast examples, computing a mean or PCA, and adding one vector per token at inference. The larger cost is evaluation, since every vector, layer and coefficient needs testing on held-out tasks to confirm the intended effect and to check for side effects on capability, fluency and safety.
04Esempio lavorato
Scegli un piccolo modello a peso aperto che puoi eseguire localmente e scegli uno strato intermedio, ad esempio lo strato 6 di un modello a 12 strati.
Scrivi 20 coppie di frasi brevi che differiscono solo per sentimento, come "Il film era meraviglioso" e "Il film era terribile".
Usa un forward hook per registrare l'attivazione del flusso residuo all'ultimo token di ogni frase al livello scelto.
Calcola la media delle differenze positivo‑meno‑negativo per ottenere un vettore di steering v.
Genera continuazioni di un prompt neutro come 'Sono andato al parco e' senza steering, poi con v aggiunto a forze 2, 4 e 8.
Valuta ogni continuazione con un classificatore di sentiment preconfezionato e annota dove il testo diventa ripetitivo o non grammaticale.
Ripeti con v sottratto per confermare che l'effetto si inverta.
Cosa mostra
L'apprendente termina con una direzione riutilizzabile e una piccola tabella di punteggi di sentiment contro la forza di steering. Un vettore funzionante dovrebbe aumentare il sentiment positivo con l'aumentare della forza e ridurlo quando sottratto, con la fluidità che si deteriora a una certa forza alta, che indica l'intervallo utilizzabile.
Anthropic's Golden Gate Claude, a research demo released on May 23, 2024 for about 24 hours, amplified one internal feature of Claude 3 Sonnet tied to the Golden Gate Bridge, so the model steered most answers toward the bridge.
Inference-Time Intervention, a NeurIPS 2023 spotlight paper, shifted activations in a small number of attention heads and raised the LLaMA-based Alpaca model's TruthfulQA truthfulness score from 32.5% to 65.1%.
Contrastive Activation Addition steers Llama 2 Chat by averaging activation differences between positive and negative examples of a behavior, such as factual versus hallucinated answers, then adding that vector with a chosen strength.
Arditi e colleghi hanno scoperto nel 2024 che il rifiuto in 13 modelli di chat open-source fino a 72 miliardi di parametri è mediato da una singola direzione, che i ricercatori possono rimuovere per fermare i rifiuti o aggiungere per provocarli.
Il documento Representation Engineering riporta che l'aggiunta di un vettore di contrasto per l’onestà ha aumentato l’accuratezza a scelta multipla di TruthfulQA di Llama-2-13B-Chat dal 35,9% al 54,0%.
Research is moving from single hand-built vectors toward features learned at scale. Anthropic's Golden Gate Claude demo in May 2024 used a feature found by its Scaling Monosemanticity work on Claude 3 Sonnet, and Anthropic said the same techniques might one day adjust safety-related features tied to dangerous code, criminal activity or deception. Whether feature-based steering is more precise and reliable than difference-of-means vectors is an open empirical question. A second direction is monitoring. The RepE paper (2023) framed reading vectors as a way to detect states such as dishonesty, and the ITI authors (2023) suggested models may internally track whether statements are true even when their outputs are false. Turning such probes into dependable monitors would need evidence that they generalize across prompts, languages and models. The security picture is mixed. Arditi and colleagues (2024) showed that a single direction controls refusal in 13 open models, which makes safety behaviors in open weights easy to remove. Open problems include steering without degrading capability, choosing layers and strengths automatically, combining several vectors without interference, and building safety that does not depend on one fragile direction.
Newsletter gratuita
Tre storie verificate sull'IA ogni mattina nei giorni feriali, scritte in un inglese semplice. Gratuito per sempre, senza pubblicità.
Un'email ogni giorno feriale. Disiscriviti con un clic. Non vendiamo né condividiamo mai il tuo indirizzo.
Mettiti alla prova
Feedback immediato su ogni risposta e un certificato condivisibile con un ID verificabile una volta superato un corso.
Sostieni l'istruzione gratuita sull'IA. AI Understanding è un'organizzazione non profit 501(c)(3) — nessun annuncio, nessun paywall, mai. Fai una donazione
Lo steering delle attivazioni modifica il comportamento di un modello linguistico modificando le sue attivazioni interne durante l’esecuzione, solitamente aggiungendo un vettore di direzione legato a un concetto come onestà, sentimento o rifiuto. L’ingegneria delle rappresentazioni è il più ampio programma di ricerca, denominato in un documento del 2023, volto a trovare tali direzioni, leggerle e usarle per monitorare o controllare i modelli senza riaddestramento.
Lo sterzo aggiunge o sottrae vettori nelle attivazioni del modello durante l'inferenza, quindi non è necessaria alcuna riqualificazione e l'effetto è immediato.
Una ricetta tipica è la differenza delle medie: attivazioni medie per un comportamento meno l'altro per isolare la direzione di quel concetto.
Lo sterzo si basa su concetti codificati come direzioni approssimativamente lineari, quindi l'aggiunta di un vettore sposta la caratteristica rilevante.
Moltiplicando il vettore per un coefficiente maggiore si spinge più forte il comportamento; un coefficiente negativo spinge nella direzione opposta.
Interventi di ampia portata possono allontanare le attivazioni dalla varietà normale del modello, danneggiando la fluidità e il ragionamento anche se il comportamento target cambia.
Continua a imparare
Altre guide selezionate per questo argomento
Il prossimoProssima guida
Rappresentazioni molecolari per l'apprendimento automatico
Tecnico