What happened
Exa detailed how it achieved a 47% improvement in throughput and cost per million documents on AWS Neuron hardware (Inferentia2 and Trainium2). The gains came from weight‑layout optimization, autocasting, custom NKI kernels for attention and MLP layers, and shape‑specific kernel rewrites that cut FLOPS and memory stalls. The optimizations were driven by an agentic development workflow that used Flyte‑orchestrated agents to compile, verify correctness (≥0.999 cosine similarity), profile, and benchmark the model on real hardware. The final system runs on a single NeuronCore per pod, scaling across larger instances and multiple AWS regions to balance performance per dollar.
Exa’s blog post explains that the company needed to re‑embed 30 billion high‑priority web documents, temporarily doubling its workload. This prompted a reassessment of hardware, leading to the selection of AWS Inferentia2 and Trainium2 chips for inference.
Starting from a naïve FP16 data‑parallel implementation, the team applied a sequence of optimizations using the Neuron SDK and the Neuron Kernel Interface (NKI) library. First, they aligned weight layouts to eliminate unnecessary transposes and enabled autocast to down‑cast matrix multiplications while preserving dynamic range, meeting a 0.999 cosine‑similarity correctness threshold.
Next, they leveraged pre‑built NKI kernels, adding a drop‑in attention kernel that yielded a noticeable speedup. Profiling revealed memory‑spill bottlenecks from high‑bandwidth memory (HBM) stalls, which they addressed by writing a custom weight‑stationary fused SwiGLU MLP kernel that removed those stalls.
Further gains came from custom attention kernels tailored to the exact tensor shapes of Exa’s model, reducing padding‑related FLOPS and folding multiple attention steps into single matrix multiplications. These shape‑specific kernels cut inner‑loop operations and HBM round‑trips, delivering the final performance boost.
Overall, the cumulative optimizations resulted in a 47% improvement in throughput and cost per million documents embedded compared with the initial baseline. The work was orchestrated by an using Flyte, where lightweight agents performed grid searches of compiler flags, reasoning agents identified bottlenecks, and code‑generation agents produced custom kernels. Each agent’s output was validated against a cosine‑similarity correctness gate before being benchmarked on real hardware.
Source details: aws.amazon.com
Why it matters
large text corpora into vector representations is a core cost driver for semantic search engines and AI‑powered retrieval systems. By reducing the per‑document embedding cost by nearly half, Exa can index more data faster and at lower expense, which directly improves the freshness and coverage of its web‑scale knowledge index. The work also demonstrates how specialized AI chips and low‑level kernel tuning can yield substantial efficiency gains beyond generic GPU deployments, offering a roadmap for other firms that rely on massive vector databases. Moreover, the showcases a reproducible method for continuous performance engineering on emerging hardware, potentially accelerating future AI infrastructure optimizations.
cost is a dominant factor for any service that builds a vector database of web‑scale text, directly influencing the economics of semantic search, retrieval‑augmented generation, and grounding of AI agents. A 47% reduction translates into millions of dollars saved at Exa’s scale and enables more frequent re‑indexing of fresh content.
The results illustrate the practical advantage of purpose‑built AI inference chips (Neuron) over more generic GPU solutions, especially when combined with low‑level kernel customizations. This could shift industry thinking toward tighter hardware‑software co‑design for large‑scale workloads.
Exa’s agentic development methodology provides a repeatable framework for continuous performance engineering. By automating correctness checks and profiling, the approach reduces manual effort and risk of regression, which other organizations can adopt to accelerate hardware optimization cycles.
The ability to dynamically schedule pods across multiple AWS regions based on performance‑per‑dollar metrics helps mitigate spot‑instance scarcity and regional compute shortages, ensuring more reliable service delivery for downstream applications that depend on up‑to‑date embeddings.
Interactive Mechanism: How It Actually Works
Explore the underlying technology behind this development interactively.
A route planner searches possible journeys using explicit rules. What does this illustrate about AI?
What to watch next
Future updates from Exa on scaling the optimized pipeline to additional AWS regions and instance types, as well as any public benchmarks comparing the Neuron‑optimized embeddings to GPU‑based baselines. Watch for broader adoption of the agentic development approach within AWS customers and possible extensions to other model families or downstream tasks such as retrieval‑augmented generation.
Exa may publish further performance data as it extends the optimizations to Trainium2 and other instance types, potentially revealing additional cost‑performance trade‑offs.
AWS could incorporate some of Exa’s custom kernels into the standard NKI library, benefiting a broader user base.
Monitoring whether other AI infrastructure providers (e.g., Nvidia, AMD) respond with competing optimizations for similar workloads will indicate the competitive impact of Exa’s approach.
Adoption of the by other teams within AWS or external customers could lead to a wave of hardware‑specific performance improvements across the AI ecosystem.