Covariant RFM-1 και Robotic Foundation Models

Covariant is a robotics software company that builds the artificial intelligence that tells industrial robot arms what to grab and how. It does not mainly build robot hardware. Its product, the Covariant Brain, runs on arms from other manufacturers in warehouses, where the job is to pick an item from a bin of mixed products and place it in a box, a tote or a sorting wall. Since 2024 the company has described its core technology as robotics foundation models, borrowing the term used for large language models: one general model trained on very broad data, then applied to many tasks, instead of a separate narrow model for each task. Its best-known model is RFM-1, short for Robotics Foundation Model 1.

The company was founded in 2017 by researchers from the University of California, Berkeley. TechCrunch reports that co-founder and chief scientist Pieter Abbeel joined OpenAI in 2016 and that co-founder Peter Chen, who studied AI under Abbeel at Berkeley, joined about a month later; in a 2023 article TechCrunch noted that three of the four co-founders had a direct connection to OpenAI. Amazon's 2024 announcement names Rocky Duan as a third founder. Funding grew with deployments: an $80 million Series C in July 2021 was followed by a $75 million extension in April 2023, led by Radical Ventures and Index Ventures, which brought total funding to $222 million, according to TechCrunch. At that time Chen said the money would help turn the Covariant Brain into a more capable foundation model.

The central idea is that warehouse robots generate the kind of data that robot learning lacks. Language models can learn from text scraped from the web, but there is no web-scale collection of robots touching objects. Covariant's robots, deployed at customer sites, record camera images, video, joint angles, force readings and suction-cup strength each time they attempt a pick, along with whether the pick worked. IEEE Spectrum reported in March 2024 that RFM-1 was trained on tens of millions of trajectories, a trajectory being the record of how a robot moved during one task, and quoted Abbeel comparing this with roughly a million trajectories in Google's RT-X effort and saying Covariant collected a million every few weeks. Spectrum also reported that Covariant's picking robots were deployed in 15 countries with dozens of customers. SiliconANGLE reported that the training mix added data from the public web to this production data.

RFM-1 works as a single multimodal model in which any input type can be used to predict any other. According to IEEE Spectrum, the inputs and outputs include still images, video, robot joint commands and language, all connected in one network of 8 billion parameters. Three uses follow from that design. First, a person can give instructions in plain English, and the model maps the words to objects in the camera view and to arm motions. Second, the model can predict what a camera will see next, generating short video of a likely outcome, which lets a robot rehearse a pick before attempting it; Covariant described this to SiliconANGLE as supporting online decision-making through planning. Third, the model can produce language about its own behavior, for example reporting why an item is hard to grasp and asking for a different strategy.

The obvious comparison is with research robot foundation models from large labs. Google DeepMind's RT-2, posted to arXiv in July 2023, fine-tunes a vision-language model on robot trajectories and writes robot actions as text tokens, so the same model that answers questions about images also outputs motor commands; the paper calls this a vision-language-action model. The Open X-Embodiment project, posted in October 2023, pooled data from 22 robot types across 21 institutions covering 527 skills, to test whether one model can learn from many different robots. Covariant's approach differs mainly in its data source. Its corpus comes from commercial robots in production rather than from lab demonstrations, which gives volume and realistic clutter but concentrates the data on one family of tasks, picking and placing in logistics. Compared with traditional warehouse automation, which relies on hand-written rules for each product and gripper, a learned model aims to handle new items without reprogramming.

The limitations are significant and were flagged in contemporary coverage. IEEE Spectrum's robotics editor Evan Ackerman cautioned readers against taking Covariant's language about human-like reasoning at face value, and noted that RFM-1 learns physical behavior from experience rather than from physics, so its predictions only hold for the kinds of situations it has seen and it could not yet generalize to completely new objects or settings. TechCrunch similarly described the reasoning claim as broad and hard to verify. Generated video predictions are a model's guess, not a physics simulation, and a confident but wrong prediction can lead to a bad grasp. The training data is weighted toward suction and gripper picking in warehouses, so transfer to homes, kitchens or humanoid bodies was a stated goal, not a demonstrated result; TechCrunch noted the software was not yet on humanoid robots.

The company's trajectory changed in August 2024. Amazon announced that Abbeel, Chen and Duan, along with research scientists and engineers making up about a quarter of Covariant's staff, would join its Fulfillment Technologies and Robotics team, and that Amazon had signed a non-exclusive license to Covariant's robotic foundation models. The financial terms were not disclosed. Robotics 24/7 reported that Covariant would continue to operate under chief operating officer Ted Stinson as CEO, with Tianhao Zhang remaining, and that it cited more than 50 customers and partners and hundreds of deployed robotic systems.

In practice, the approach requires three expensive ingredients. The first is a fleet of robots already doing paid work, because the data flywheel only turns when deployments produce trajectories. The second is large-scale training compute for a multi-billion-parameter multimodal model. The third is integration engineering at each customer site, since grippers, cameras and warehouse software differ. For a buyer, the practical question is not whether a model can reason in general but its pick rate, error rate and recovery behavior on that buyer's product mix, measured on site.