Back to News
InnovationAI Understanding briefing

Inference now drives two‑thirds of AI workloads, prompting chip makers to favor commodity memory

Positron AI says inference workloads now account for about 66% of AI compute, prompting the company to ship modular, low‑power servers built on commodity memory instead of scarce high‑bandwidth memory.

4 min readRead the linked source
Source-provided image accompanying Inference now drives two‑thirds of AI workloads, prompting chip makers to favor commodity memory
Source referenceSource recorded
Publisher
siliconangle.com
Source type
Linked source — primary-source status has not been established.
ContextUnderstand this in 60 seconds

Key terms

Memory (Agent Memory)
Stored context an AI agent uses across steps or sessions to improve continuity.
Inference
The runtime phase where a trained model generates predictions or outputs.
Compute
The processing resources required to train and run models, often measured in FLOPS or GPU hours.

What happened

Positron AI Inc., a startup that raised $875 million at a $5 billion valuation, announced that workloads now represent roughly two‑thirds of AI , up from a training‑dominant mix in 2023. The company’s Atlas server, built on a programmable chip that uses commodity DRAM rather than high‑bandwidth memory (HBM), is already deployed in Oracle’s cloud, with a neocloud provider also signing on. Positron plans to tape‑out its own silicon for a next‑generation Titan server early next year. The design keeps power draw low—about 4.5 kW per server and 40 kW per rack—making it suitable for air‑cooled, modular data centers that can be deployed quickly and run on cheap megawatt‑scale power. The company argues that this approach speeds the path from land and power to monetizable token generation within 90‑120 days.

Positron AI Inc. highlighted that now accounts for about two‑thirds of AI workloads, a reversal from 2023 when training made up roughly 85% of . The claim comes from Cameron McCaskill, head of sales and business development, during an interview at the SC26 conference.

The company’s first‑generation Atlas server runs on a programmable chip that uses standard DRAM instead of high‑bandwidth memory (HBM). This design choice reduces component cost and eases supply‑chain pressure, as HBM is limited and expensive.

Oracle is deploying approximately 340 Atlas servers in its cloud this quarter, and a neocloud provider has also committed to both Atlas and the upcoming Titan server. Positron plans to tape out its own silicon for Titan early next year, moving from a programmable chip to a custom ASIC.

Power consumption is a key differentiator: each Atlas server draws about 4,500 watts, and a full rack of Titan servers stays under 40 kilowatts, allowing operation in air‑cooled rooms without specialized cooling infrastructure.

The low‑power, modular footprint aligns with the rise of modular data centers that can be deployed quickly (within 90‑120 days) and run on megawatt‑scale, inexpensive power sources, appealing to service providers seeking rapid capacity expansion.

Source details: siliconangle.com

Why it matters

The shift toward ‑first workloads reshapes where AI chips are deployed, moving demand away from large, power‑hungry training clusters toward smaller, modular data centers that can be built rapidly and operated with lower‑cost power. Positron’s use of commodity memory sidesteps the supply constraints and high cost of HBM, potentially lowering the total cost of ownership for AI inference infrastructure. By delivering servers that fit within 40 kW racks and can be air‑cooled, the company enables operators to locate in regions with cheaper, less regulated power, accelerating token generation and revenue for inference‑focused services. This trend could influence chip manufacturers to prioritize low‑power, memory‑efficient designs and may affect cloud providers’ hardware roadmaps as they seek to meet the growing demand for inference capacity.

The ‑first shift changes capital‑expenditure planning for AI infrastructure, favoring smaller, faster‑to‑deploy sites over massive training clusters that require extensive cooling and power.

By avoiding HBM, Positron reduces reliance on a constrained component, potentially lowering hardware costs and mitigating supply‑chain bottlenecks that have affected other AI chip vendors.

Low‑power, air‑cooled designs broaden the geographic options for AI deployment, allowing operators to locate in regions with cheaper electricity and fewer regulatory hurdles, which could accelerate token generation and revenue for ‑centric businesses.

If other chipmakers follow Positron’s commodity‑memory approach, the market could see a broader move toward energy‑efficient, cost‑effective hardware, influencing cloud providers’ procurement strategies and possibly reshaping the AI hardware ecosystem.

The upcoming Titan silicon tape‑out will be a test of whether custom ASICs can maintain the power and cost advantages of the commodity‑memory approach while delivering higher performance for workloads.

Interactive Mechanism

Interactive Mechanism: How It Actually Works

Explore the underlying technology behind this development interactively.

Document Size:128K tokens
Needle Placement Depth (Location in document):50% into text
Attention Context Buffer Map:
Target Fact (50%)
Equivalent Pages~320Standard book pages
Retrieval Accuracy99.9%Needle recall score
RAM / KV Cache5.1 GBMemory overhead
Prompt CachingActive~80% discount on reuse
Core takeaway: Million-token context windows allow querying whole codebases or legal archives in one prompt. However, KV cache memory scales with context length, making prompt caching crucial for real-time production.
Interactive Concept Check+10 Points
AI Models Explained Quiz

Which component of an AI application is the machine-learning model itself?

What to watch next

Watch for adoption rates of Positron’s Atlas and upcoming Titan servers across cloud and edge providers, and for any follow‑on announcements about the Titan silicon tape‑out. Monitor whether other chipmakers introduce comparable commodity‑memory designs, and track how modular data‑center deployments affect regional power markets and AI token economics.

Adoption metrics for Atlas and Titan servers across major cloud platforms and edge providers, especially any announcements from Oracle or other cloud vendors about scaling deployments.

Details of the Titan silicon tape‑out, including performance benchmarks, power consumption, and whether the custom ASIC retains the commodity‑memory advantage.

Responses from competing chipmakers—such as Nvidia, AMD, or emerging startups—regarding memory architecture choices and power efficiency for ‑focused products.

Growth of modular data‑center projects that specifically target workloads, including any partnerships with power‑utility firms or regional developers.

Regulatory or supply‑chain developments that could affect the availability or pricing of HBM versus commodity DRAM, influencing the broader market dynamics.

Related guides & quizzes

Found this useful?