Back to News
InnovationAI Understanding briefing

Exa improves embedding throughput by 47% on AWS Neuron chips to index billions of documents

Exa reports a 47% boost in embedding performance and cost efficiency on AWS Inferentia2 and Trainium2 chips after a series of kernel and compiler optimizations, enabling faster indexing of tens of billions of web documents.

4 min readRead the primary source
Source-provided image accompanying Exa improves embedding throughput by 47% on AWS Neuron chips to index billions of documents
Primary-source documentSource recorded
Publisher
aws.amazon.com
Source type
Primary document — an official announcement, paper, filing, or first-party page we read directly.
ContextUnderstand this in 60 seconds

Key terms

Embedding
A numeric vector representation that captures semantic meaning of text, images, or other data.
Memory (Agent Memory)
Stored context an AI agent uses across steps or sessions to improve continuity.
Agentic Workflow
A multi-step process where an AI system plans, executes, checks results, and iterates toward a goal.

What happened

Exa detailed how it achieved a 47% improvement in throughput and cost per million documents on AWS Neuron hardware (Inferentia2 and Trainium2). The gains came from weight‑layout optimization, autocasting, custom NKI kernels for attention and MLP layers, and shape‑specific kernel rewrites that cut FLOPS and memory stalls. The optimizations were driven by an agentic development workflow that used Flyte‑orchestrated agents to compile, verify correctness (≥0.999 cosine similarity), profile, and benchmark the model on real hardware. The final system runs on a single NeuronCore per pod, scaling across larger instances and multiple AWS regions to balance performance per dollar.

Exa’s blog post explains that the company needed to re‑embed 30 billion high‑priority web documents, temporarily doubling its workload. This prompted a reassessment of hardware, leading to the selection of AWS Inferentia2 and Trainium2 chips for inference.

Starting from a naïve FP16 data‑parallel implementation, the team applied a sequence of optimizations using the Neuron SDK and the Neuron Kernel Interface (NKI) library. First, they aligned weight layouts to eliminate unnecessary transposes and enabled autocast to down‑cast matrix multiplications while preserving dynamic range, meeting a 0.999 cosine‑similarity correctness threshold.

Next, they leveraged pre‑built NKI kernels, adding a drop‑in attention kernel that yielded a noticeable speedup. Profiling revealed memory‑spill bottlenecks from high‑bandwidth memory (HBM) stalls, which they addressed by writing a custom weight‑stationary fused SwiGLU MLP kernel that removed those stalls.

Further gains came from custom attention kernels tailored to the exact tensor shapes of Exa’s model, reducing padding‑related FLOPS and folding multiple attention steps into single matrix multiplications. These shape‑specific kernels cut inner‑loop operations and HBM round‑trips, delivering the final performance boost.

Overall, the cumulative optimizations resulted in a 47% improvement in throughput and cost per million documents embedded compared with the initial baseline. The work was orchestrated by an using Flyte, where lightweight agents performed grid searches of compiler flags, reasoning agents identified bottlenecks, and code‑generation agents produced custom kernels. Each agent’s output was validated against a cosine‑similarity correctness gate before being benchmarked on real hardware.

Source details: aws.amazon.com

Why it matters

large text corpora into vector representations is a core cost driver for semantic search engines and AI‑powered retrieval systems. By reducing the per‑document embedding cost by nearly half, Exa can index more data faster and at lower expense, which directly improves the freshness and coverage of its web‑scale knowledge index. The work also demonstrates how specialized AI chips and low‑level kernel tuning can yield substantial efficiency gains beyond generic GPU deployments, offering a roadmap for other firms that rely on massive vector databases. Moreover, the showcases a reproducible method for continuous performance engineering on emerging hardware, potentially accelerating future AI infrastructure optimizations.

cost is a dominant factor for any service that builds a vector database of web‑scale text, directly influencing the economics of semantic search, retrieval‑augmented generation, and grounding of AI agents. A 47% reduction translates into millions of dollars saved at Exa’s scale and enables more frequent re‑indexing of fresh content.

The results illustrate the practical advantage of purpose‑built AI inference chips (Neuron) over more generic GPU solutions, especially when combined with low‑level kernel customizations. This could shift industry thinking toward tighter hardware‑software co‑design for large‑scale workloads.

Exa’s agentic development methodology provides a repeatable framework for continuous performance engineering. By automating correctness checks and profiling, the approach reduces manual effort and risk of regression, which other organizations can adopt to accelerate hardware optimization cycles.

The ability to dynamically schedule pods across multiple AWS regions based on performance‑per‑dollar metrics helps mitigate spot‑instance scarcity and regional compute shortages, ensuring more reliable service delivery for downstream applications that depend on up‑to‑date embeddings.

Interactive Mechanism

Interactive Mechanism: How It Actually Works

Explore the underlying technology behind this development interactively.

Thinking Budget (Test-Time Tokens):1,024 tokens
Complex Accuracy79%Math & Code Logic
Latency3.2sTime to first full output
Inference Cost$0.0092Per query estimated
Reasoning StyleStep VerificationInternal chain depth
Active Thinking Trace:
1Deconstruct user problem into formal constraints
2Propose candidate hypotheses & step-by-step calculation
3Self-correction: Backtrack and refute subtle edge cases
4Exhaustive consistency check & final output synthesis
Core takeaway: Test-time compute fundamentally changes AI economics. Instead of only scaling during pre-training, giving reasoning models more tokens at inference time allows them to systematically solve PhD-level STEM problems.
Interactive Concept Check+10 Points
What is AI? Quiz

A route planner searches possible journeys using explicit rules. What does this illustrate about AI?

What to watch next

Future updates from Exa on scaling the optimized pipeline to additional AWS regions and instance types, as well as any public benchmarks comparing the Neuron‑optimized embeddings to GPU‑based baselines. Watch for broader adoption of the agentic development approach within AWS customers and possible extensions to other model families or downstream tasks such as retrieval‑augmented generation.

Exa may publish further performance data as it extends the optimizations to Trainium2 and other instance types, potentially revealing additional cost‑performance trade‑offs.

AWS could incorporate some of Exa’s custom kernels into the standard NKI library, benefiting a broader user base.

Monitoring whether other AI infrastructure providers (e.g., Nvidia, AMD) respond with competing optimizations for similar workloads will indicate the competitive impact of Exa’s approach.

Adoption of the by other teams within AWS or external customers could lead to a wave of hardware‑specific performance improvements across the AI ecosystem.

Related guides & quizzes

Found this useful?