公司指南

協變 RFM-1 與機器人基礎模型

Covariant 的 RFM-1 是一個機器人基礎模型,經過倉庫操作資料的訓練,因此機器手臂可以透過語言和視覺來推理和挑選不熟悉的物體。

聆聽此指南AI 語音敘述 · ~8 min

本頁閱讀時間 9 分鐘
  1. 重點摘要
  2. 深入探討
  3. Trace a language-guided warehouse pick
  4. 現實世界的實施
  5. 協變 RFM-1 和機器人基礎模型的未來
  6. 付諸實踐
  7. 資料來源與延伸閱讀
  8. 不斷探索
  9. 常見問題

重點摘要

  1. Covariant was founded in 2017 by Berkeley researchers including Pieter Abbeel and Peter Chen, three of whom had direct connections to OpenAI.
  2. RFM-1, announced on 11 March 2024, is an 8-billion-parameter model that links text, images, video, sensor data and robot actions.
  3. IEEE Spectrum reported RFM-1 was trained on tens of millions of robot trajectories collected from commercial warehouse deployments.
  4. RFM-1 can predict short videos of likely outcomes, letting a robot rehearse a pick before acting, but only within situations it has seen.
  5. In August 2024 Amazon hired three Covariant founders and about a quarter of its staff and took a non-exclusive license to its models.

深入探討

Covariant is a robotics software company that builds the artificial intelligence that tells industrial robot arms what to grab and how. It does not mainly build robot hardware. Its product, the Covariant Brain, runs on arms from other manufacturers in warehouses, where the job is to pick an item from a bin of mixed products and place it in a box, a tote or a sorting wall. Since 2024 the company has described its core technology as robotics foundation models, borrowing the term used for large language models: one general model trained on very broad data, then applied to many tasks, instead of a separate narrow model for each task. Its best-known model is RFM-1, short for Robotics Foundation Model 1.

The company was founded in 2017 by researchers from the University of California, Berkeley. TechCrunch reports that co-founder and chief scientist Pieter Abbeel joined OpenAI in 2016 and that co-founder Peter Chen, who studied AI under Abbeel at Berkeley, joined about a month later; in a 2023 article TechCrunch noted that three of the four co-founders had a direct connection to OpenAI. Amazon's 2024 announcement names Rocky Duan as a third founder. Funding grew with deployments: an $80 million Series C in July 2021 was followed by a $75 million extension in April 2023, led by Radical Ventures and Index Ventures, which brought total funding to $222 million, according to TechCrunch. At that time Chen said the money would help turn the Covariant Brain into a more capable foundation model.

The central idea is that warehouse robots generate the kind of data that robot learning lacks. Language models can learn from text scraped from the web, but there is no web-scale collection of robots touching objects. Covariant's robots, deployed at customer sites, record camera images, video, joint angles, force readings and suction-cup strength each time they attempt a pick, along with whether the pick worked. IEEE Spectrum reported in March 2024 that RFM-1 was trained on tens of millions of trajectories, a trajectory being the record of how a robot moved during one task, and quoted Abbeel comparing this with roughly a million trajectories in Google's RT-X effort and saying Covariant collected a million every few weeks. Spectrum also reported that Covariant's picking robots were deployed in 15 countries with dozens of customers. SiliconANGLE reported that the training mix added data from the public web to this production data.

RFM-1 works as a single multimodal model in which any input type can be used to predict any other. According to IEEE Spectrum, the inputs and outputs include still images, video, robot joint commands and language, all connected in one network of 8 billion parameters. Three uses follow from that design. First, a person can give instructions in plain English, and the model maps the words to objects in the camera view and to arm motions. Second, the model can predict what a camera will see next, generating short video of a likely outcome, which lets a robot rehearse a pick before attempting it; Covariant described this to SiliconANGLE as supporting online decision-making through planning. Third, the model can produce language about its own behavior, for example reporting why an item is hard to grasp and asking for a different strategy.

The obvious comparison is with research robot foundation models from large labs. Google DeepMind's RT-2, posted to arXiv in July 2023, fine-tunes a vision-language model on robot trajectories and writes robot actions as text tokens, so the same model that answers questions about images also outputs motor commands; the paper calls this a vision-language-action model. The Open X-Embodiment project, posted in October 2023, pooled data from 22 robot types across 21 institutions covering 527 skills, to test whether one model can learn from many different robots. Covariant's approach differs mainly in its data source. Its corpus comes from commercial robots in production rather than from lab demonstrations, which gives volume and realistic clutter but concentrates the data on one family of tasks, picking and placing in logistics. Compared with traditional warehouse automation, which relies on hand-written rules for each product and gripper, a learned model aims to handle new items without reprogramming.

The limitations are significant and were flagged in contemporary coverage. IEEE Spectrum's robotics editor Evan Ackerman cautioned readers against taking Covariant's language about human-like reasoning at face value, and noted that RFM-1 learns physical behavior from experience rather than from physics, so its predictions only hold for the kinds of situations it has seen and it could not yet generalize to completely new objects or settings. TechCrunch similarly described the reasoning claim as broad and hard to verify. Generated video predictions are a model's guess, not a physics simulation, and a confident but wrong prediction can lead to a bad grasp. The training data is weighted toward suction and gripper picking in warehouses, so transfer to homes, kitchens or humanoid bodies was a stated goal, not a demonstrated result; TechCrunch noted the software was not yet on humanoid robots.

The company's trajectory changed in August 2024. Amazon announced that Abbeel, Chen and Duan, along with research scientists and engineers making up about a quarter of Covariant's staff, would join its Fulfillment Technologies and Robotics team, and that Amazon had signed a non-exclusive license to Covariant's robotic foundation models. The financial terms were not disclosed. Robotics 24/7 reported that Covariant would continue to operate under chief operating officer Ted Stinson as CEO, with Tianhao Zhang remaining, and that it cited more than 50 customers and partners and hundreds of deployed robotic systems.

In practice, the approach requires three expensive ingredients. The first is a fleet of robots already doing paid work, because the data flywheel only turns when deployments produce trajectories. The second is large-scale training compute for a multi-billion-parameter multimodal model. The third is integration engineering at each customer site, since grippers, cameras and warehouse software differ. For a buyer, the practical question is not whether a model can reason in general but its pick rate, error rate and recovery behavior on that buyer's product mix, measured on site.

03工作範例

Trace a language-guided warehouse pick

  1. An operator types an instruction such as pick up the red object for a robot arm facing a bin of mixed items.

  2. The model combines the text with the current camera image and identifies which item matches the description.

  3. It proposes candidate grasps, including where to place the suction cup or gripper on the chosen item.

  4. For each candidate, the world model predicts a short video of what will happen, such as a clean lift or the item slipping.

  5. The robot executes the grasp whose predicted outcome looks most successful and records sensor data from the attempt.

  6. If the pick fails, the model reports the likely cause in plain language and the operator can suggest a different approach.

它顯示了什麼

The learner can see how one multimodal model links words, images, predicted video and motor commands in a single pick. To check understanding, compare the predicted outcome with the actual result and note that the logged attempt becomes new training data.

現實世界的實施

Covariant Brain software powers robotic arms that pick and place items in warehouses; TechCrunch reported in 2023 that it learns from a growing database of package sizes, shapes and materials gathered from real picks.

In a March 2024 demonstration reported by TechCrunch, an operator typed or spoke an instruction such as picking up the red object, and the RFM-1-driven arm selected an apple from a bin.

RFM-1 can generate short videos predicting how a scene will change, such as an object falling over on a moving conveyor belt, which IEEE Spectrum described as a learned world model.

According to SiliconANGLE, a robot running RFM-1 that struggles with an item can describe the problem in plain language and accept new motion strategies from the operator instead of new code.

In August 2024 Amazon took a non-exclusive license to Covariant's robotic foundation models to make its existing warehouse robot fleet more capable, according to Amazon's announcement.

協變 RFM-1 和機器人基礎模型的未來

The open question for robotics foundation models is whether data from deployed robots can produce a model that generalizes beyond the tasks those robots perform. Covariant said in March 2024, as reported by SiliconANGLE, that it planned to roll RFM-1 out to customers' warehouse robots in the coming months and to increase its data collection speed tenfold or more; independent measurements of that rollout have not been published. Research models point to the same goal from the other direction. RT-2 (July 2023) showed that internet-trained vision-language models can transfer some general knowledge to robot control, and Open X-Embodiment (October 2023) found positive transfer when one model learned from many robot types. Amazon said in August 2024 that it would embed Covariant's technology in its existing robot fleet, which, if reported on, would be one of the largest real-world tests of a licensed robot foundation model. Key open problems include reliable world-model predictions over long horizons, safe behavior when a prediction is wrong, standard benchmarks for comparing commercial systems, and the business question of whether model companies or robot fleet owners end up holding the most valuable data.

付諸實踐

為什麼這很重要

  • 供應商策略. 供應商路線圖會影響您的團隊接下來可以建立的功能。
  • 成本與預算. 商業條款和部署選項會影響長期成本和風險。
  • 風險與安全. 公司激勵措施塑造了產品預設、安全態勢和開放性。

留意

  • 發佈公告可能會超過實際生產工作流程的穩定性。
  • API 定價或政策轉變可能會在一夜之間打破假設。
  • 單一供應商依賴性增加了鎖定和遷移成本。

後續步驟

  1. 使用您自己的任務和資料集評估提供者。
  2. 在整合之前查看隱私、安全和法律條款。
  3. 維護跨模型或供應商的後備計劃。
  4. 監控發行說明,以便路線圖的變更不會讓團隊感到意外。

資料來源與延伸閱讀

  1. IEEE SpectrumCovariant Announces a Universal AI Platform for Robots
  2. TechCrunchCovariant is building ChatGPT for robots
  3. SiliconANGLECovariant develops video-generating AI model for powering warehouse robots
  4. TechCrunchCovariant's robotic picking AI nabs another $75M
  5. AmazonAmazon hires from AI robotics startup Covariant, licenses technology
  6. Robotics 24/7Amazon hires three of the founders of AI robotics company Covariant, licenses its technology
  7. arXiv (Google DeepMind)RT-2:視覺-語言-動作模型將網路知識轉移到機器人控制
  8. arXivOpen X-Embodiment: Robotic Learning Datasets and RT-X Models

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

參加 Covariant RFM-1 與機器人基礎模型測驗

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

什麼是協變 RFM-1 和機器人基礎模型?

Covariant 的 RFM-1 是一個機器人基礎模型,經過倉庫操作資料的訓練,因此機器手臂可以透過語言和視覺來推理和挑選不熟悉的物體。

2024 年推出的 Covariant 基礎模式的名稱是什麼?

Covariant 推出了 RFM-1,即機器人基礎模型 1,在機器人資料以及文字和影像上進行訓練。

Amazon 於 2024 年與 Covariant 的商業協議包含了什麼?

Amazon 描述了非獨占的模型授權,並表示 Covariant 的創辦人與部分員工將加入其機器人團隊;Covariant 將持續服務客戶。

RFM-1借鑒大型語言模型的核心思想是什麼?

RFM-1 將文字、圖像、感測器讀數和動作視為標記並預測下一個,像法學碩士一樣學習概括。

RFM-1 設計的目的是協助倉庫機器人完成什麼任務?

Covariant 將 RFM-1 定位於倉儲挑選,並描述其對各種物品的泛化能力;實際效能取決於機器人本身與任務。

Covariant 的著名創辦人有哪些?

Covariant 由 Pieter Abbeel、Peter Chen 和 Rocky Duan 等人工智慧研究人員共同創立,其根源於加州大學柏克萊分校和 OpenAI。