Function Vectors and Task Representations
Function vectors are compact directions inside a language model's hidden states that encode an entire task, like 'translate to French' or 'return the antonym.' They reveal that models compress a demonstrated task into a portable internal signal you can extract and reinject.
Deep Dive
When you give a model a few in-context examples, it somehow infers the task and applies it to a new input. Function vector research shows that this inferred task is partly captured by a single vector living in the model's activation space. Researchers identify a small set of attention heads that, across many tasks, carry task-identity information. Averaging their outputs over example prompts yields a function vector. Remarkably, adding that vector to the hidden states during a fresh, zero-shot prompt can make the model perform the task without seeing any examples. This is strong evidence that models build reusable, abstract task representations rather than just pattern-matching surface text, and it connects to broader work on steering and interpretability.
Technical Insight
The method builds on causal mediation analysis. Researchers run the model on many demonstrations of a task, identify attention heads whose outputs causally carry the task identity, and average those head outputs to form the function vector. Injected at a particular layer, the vector shifts later computation toward executing the task. Crucially, function vectors show some transport: a vector extracted from one prompt context can trigger the task in unrelated contexts.
Strategic Impact
Speed and scale
Language workflows can move faster without sacrificing consistency.
Access and reach
It expands access across languages and communication styles.
Clearer decisions
Teams can spend more time on judgment while automation handles repetition.
The Future of Function Vectors and Task Representations
Function vectors point toward controllable, transparent steering: instead of crafting prompts, you might keep a library of task vectors and switch behaviors by addition. They could enable lightweight task adaptation without fine-tuning, safety auditing by inspecting which task a model has 'decided' to run, and composition of multiple tasks by combining vectors. Expect tighter integration with interpretability tooling and activation-steering methods as researchers map how abstract these representations really are.
Real-World Implementation
Triggering a task like 'list the capital' on a zero-shot prompt by injecting a vector extracted from earlier few-shot examples.
Auditing model behavior by checking which task vector is active to detect when a model silently switches objectives.
Building a reusable library of task directions so applications switch functions by addition instead of re-prompting.
Studying composition by adding two function vectors to see whether the model can chain operations like 'translate then uppercase'.
Risks & Guardrails
Hallucinated facts can quietly enter reports, support flows, or research outputs.
Prompt sensitivity can create inconsistent results across similar requests.
Sensitive text data may be exposed if access controls are weak.
Implementation Roadmap
Define output format, tone, and quality standards before rollout.
Ground responses with trusted sources whenever accuracy matters.
Keep a human review checkpoint for high-stakes outputs.
Track failure patterns and retrain prompts or workflows regularly.
Keep Exploring
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Function Vectors and Task Representations quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Next guide
GloVe Global Vectors
Frequently asked questions
What is Function Vectors and Task Representations?
Function vectors are compact directions inside a language model's hidden states that encode an entire task, like 'translate to French' or 'return the antonym.' They reveal that models compress a demonstrated task into a portable internal signal you can extract and reinject.
What does a function vector primarily encode inside a language model?
A function vector is a compact direction in activation space that captures the identity of a whole task the model has inferred from demonstrations.
How is a function vector typically constructed?
Researchers locate attention heads that causally carry task identity and average their outputs across demonstrations to form the vector.
What surprising thing happens when you inject a function vector into a zero-shot prompt?
Adding the vector to the hidden states can make the model execute the encoded task even with no in-context examples present.
Which analysis technique underpins finding the heads that carry a function vector?
Causal mediation analysis identifies which attention-head outputs causally determine the task identity, isolating the relevant heads.
What broader claim about language models do function vectors support?
The existence of portable task vectors suggests models form abstract, reusable representations of operations rather than pure surface matching.