公司指南

獵鷹模型

Falcon is a family of openly released large language models from the Technology Innovation Institute (TII) in Abu Dhabi.

聆聽此指南AI 語音敘述 · ~7 min

本頁8 分鐘閱讀
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. Estimate the memory savings of multiquery attention
  5. 現實世界的實施
  6. 獵鷹模型的未來
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

Falcon-40B topped Hugging Face's open model leaderboard in 2023 and showed that carefully filtered web data alone could train a strong model; later releases moved to efficient designs such as attention-free Mamba models and hybrid attention plus state-space models.

重點摘要

  1. Falcon is a family of openly released language models from the Technology Innovation Institute in Abu Dhabi.
  2. Falcon-40B ranked first on Hugging Face's Open LLM Leaderboard at release in 2023 under the commercial-friendly Apache 2.0 license.
  3. TII's RefinedWeb work showed that heavily filtered and deduplicated web data alone could train competitive language models.
  4. Falcon-180B was trained on 3.5 trillion tokens using about 7 million GPU hours and needs roughly 640 GB of GPU memory at 16-bit precision.
  5. Falcon-H1 runs attention and state-space heads in parallel and supports contexts up to 256,000 tokens in sizes from 0.5 to 34 billion parameters.

深入探討

Falcon is the name of the large language models built by the Technology Innovation Institute (TII), a research institute in Abu Dhabi, United Arab Emirates. A large language model (LLM) is a neural network trained on large amounts of text to predict the next token, a word or word fragment, which lets it answer questions, write and summarize. TII releases Falcon models openly: anyone can download the weights, the learned numbers that make up the model, and run or fine-tune them on their own hardware. That openness, together with strong benchmark results, made Falcon one of the best-known model families from outside the United States and China.

The first widely used releases were Falcon-7B and Falcon-40B in 2023. Hugging Face reported in June 2023 that Falcon-40B was trained on 1 trillion tokens and Falcon-7B on 1.5 trillion, using 2,800 and 700 petaflop-days of compute respectively. Falcon-40B ranked first on Hugging Face's Open LLM Leaderboard at release, and both models were licensed under Apache 2.0, which allows commercial use. In September 2023 TII released Falcon-180B, trained on 3.5 trillion tokens using up to 4,096 GPUs and about 7 million GPU hours on Amazon SageMaker. At release it had the highest leaderboard score of any openly released pretrained model, 68.74, later revised to 67.85, on par with Llama 2 70B.

The central idea behind those models was data. TII's RefinedWeb paper, posted in June 2023, argued that properly filtered and deduplicated web data, and web data only, could match or beat curated mixtures of books, papers and conversations. Deduplication means removing repeated or near-identical documents so the model does not memorize them. The team extracted about 5 trillion tokens from Common Crawl, a public archive of web pages, and released a 600-billion-token portion. Hugging Face noted that over 80% of Falcon-7B and 40B's training data came from RefinedWeb. The November 2023 Falcon series paper described the 180B model's training run as the largest openly documented pretraining run at the time.

Architecture was the second theme. The early Falcon models used multiquery attention, in which all attention heads share one set of keys and values instead of each head having its own. During text generation a model stores keys and values for every past token in a key-value cache; sharing them shrinks that cache. Hugging Face's comparison for a 2,048-token context lists 240 MB for Falcon-40B against 5,400 MB for LLaMA-65B. From 2024 TII went further. Falcon Mamba 7B, described in an October 2024 paper and trained on 5.8 trillion tokens, uses the Mamba state-space architecture with no attention at all; a state-space model carries a fixed-size summary of everything read so far, so memory stays constant however long the text grows. In December 2024 the Falcon3 family added models of 1, 3, 7 and 10 billion parameters, the 7B trained on 14 trillion tokens on 1,024 H100 GPUs.

Falcon-H1, published in a July 2025 technical report, combined both approaches. Each layer runs attention heads and state-space heads in parallel, with the ratio between them adjustable. The family spans 0.5B, 1.5B, 1.5B-Deep, 3B, 7B and 34B parameters, supports contexts up to 256,000 tokens and 18 languages, and ships as more than 30 checkpoints. TII reports that the 34B model matches or beats models up to about 70 billion parameters, such as Llama 3.3 70B, and that the 0.5B model performs like typical 7B models from 2024. On January 5, 2026, TII released Falcon-H1R 7B, a reasoning model built on Falcon-H1; The Decoder reported TII's claim of a 49.5% average across four benchmarks against 46.2% for Qwen3 32B.

Compared with other open model families, Falcon's distinguishing traits are its emphasis on web-data quality, its early adoption of memory-saving attention and its turn to state-space and hybrid designs. Licenses differ by release, which matters for builders. Falcon-7B and 40B use Apache 2.0. Falcon-180B allows commercial use only under restrictive conditions that exclude hosting it as a service, according to Hugging Face. Newer models use the Falcon LLM license, which permits free and commercial use but requires following an acceptable use policy that TII can update at any time.

The limitations are those of most open models plus a few specific ones. Most performance figures for recent Falcon models are self-reported by TII, and The Decoder cautioned that benchmark scores only loosely track real-world usefulness, especially for small models. Falcon-180B's leaderboard lead was brief, and by its own revised score it was level with Llama 2 70B. Pure state-space models trade some ability to look back at exact earlier text for constant memory, which is one reason hybrid designs like Falcon-H1 reintroduce attention. As with any model trained mostly on web text, outputs can reflect web biases and errors, and openly released weights cannot be recalled once published.

Running Falcon models ranges from easy to expensive. Falcon-7B fits in about 15 GB of GPU memory, and Falcon-40B needs about 90 GB at 16-bit precision or about 27 GB at 4-bit. Falcon-180B needs about 640 GB, eight 80 GB A100 GPUs, at 16-bit precision, or 320 GB when quantized to 4-bit. The smaller Falcon3, Falcon-H1 and H1R models are designed for single GPUs, laptops and edge devices, and quantized versions are published on Hugging Face. Training a Falcon-scale model from scratch is far beyond most organizations: the 180B run used about 7 million GPU hours.

04工作範例

Estimate the memory savings of multiquery attention

  1. Pick a context of 2,048 tokens and look up Hugging Face's key-value cache figures: 240 MB for Falcon-40B and 5,400 MB for LLaMA-65B.

  2. Divide 5,400 by 240 to get a ratio of 22.5, so the Falcon cache is about 4% of the LLaMA cache at that length.

  3. Scale to a batch of 16 simultaneous users: 16 times 240 MB is about 3.8 GB, against 16 times 5,400 MB, about 86 GB.

  4. Add the model weights: about 90 GB for Falcon-40B at 16-bit precision, so the cache barely changes total memory for Falcon.

  5. Note that part of the gap reflects the different model sizes, so the ratio overstates the effect of attention design alone.

  6. Compare with Falcon Mamba 7B, whose state-space layers keep a fixed-size state with no growing cache at all.

它顯示了什麼

A rough memory budget showing why shared keys and values let Falcon serve many users or long contexts on fewer GPUs. Check it by loading a Falcon checkpoint in Transformers, generating at two context lengths and comparing peak GPU memory with the estimate.

現實世界的實施

Falcon-7B needs about 15 GB of GPU memory and was fine-tuned on a single 16 GB NVIDIA T4, according to Hugging Face, putting a capable open model within reach of small teams in 2023.

Data researchers use the 600-billion-token public extract of RefinedWeb, TII's filtered Common Crawl dataset, to study how web-only data compares with curated training corpora.

Falcon Mamba 7B, trained on 5.8 trillion tokens, lets developers test an attention-free model whose memory use does not grow with the length of the text being generated.

The Falcon-H1 family, from 0.5 to 34 billion parameters with contexts up to 256,000 tokens in 18 languages, offers hybrid models for long-document tasks on smaller hardware.

Falcon-H1R 7B, released in January 2026, is a compact reasoning model with weights and a quantized GGUF version on Hugging Face for local deployment.

獵鷹模型的未來

TII's releases since 2024 point toward smaller, more efficient models rather than ever-larger ones. After Falcon-180B in 2023, the institute moved to the attention-free Falcon Mamba 7B in 2024, the sub-10B Falcon3 family in December 2024, the hybrid Falcon-H1 family in July 2025 and the compact Falcon-H1R 7B reasoning model in January 2026. That path tracks a wider research question: whether hybrid attention and state-space designs can match pure Transformers on hard reasoning while staying cheaper for long contexts. TII's own benchmark claims are favorable, but independent evaluations at scale are still limited, and The Decoder noted the loose link between benchmark scores and real use for small models. Open issues for Falcon users include licensing, since newer models carry an acceptable use policy that TII can change, and language coverage, with Falcon-H1 trained on 18 languages. Whether Falcon remains among the leading open model families will depend on how its efficiency-focused designs perform against frequent releases from larger labs, which cannot be known yet.

付諸實踐

為什麼這很重要

  • 供應商策略. 供應商路線圖會影響您的團隊接下來可以建立的功能。
  • 成本與預算. 商業條款和部署選項會影響長期成本和風險。
  • 風險與安全. 公司激勵措施塑造了產品預設、安全態勢和開放性。

留意

  • 發佈公告可能會超過實際生產工作流程的穩定性。
  • API 定價或政策轉變可能會在一夜之間打破假設。
  • 單一供應商依賴性增加了鎖定和遷移成本。

後續步驟

  1. 使用您自己的任務和資料集評估提供者。
  2. 在整合之前查看隱私、安全和法律條款。
  3. 維護跨模型或供應商的後備計劃。
  4. 監控發行說明,以便路線圖的變更不會讓團隊感到意外。

資料來源與延伸閱讀

  1. arXiv (Technology Innovation Institute)The Falcon Series of Open Language Models
  2. arXivThe RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only
  3. Hugging FaceThe Falcon has landed in the Hugging Face ecosystem
  4. Hugging FaceSpread Your Wings: Falcon 180B is here
  5. arXivFalcon Mamba: The First Competitive Attention-free 7B Language Model
  6. Hugging FaceWelcome to the Falcon 3 Family of Open Models
  7. arXivFalcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance
  8. The DecoderAbu Dhabi's TII claims its Falcon H1R 7B reasoning model matches rivals seven times its size

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

Take the Falcon Models quiz

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

What is Falcon Models?

Falcon is a family of openly released large language models from the Technology Innovation Institute (TII) in Abu Dhabi. Falcon-40B topped Hugging Face's open model leaderboard in 2023 and showed that carefully filtered web data alone could train a strong model; later releases moved to efficient designs such as attention-free Mamba models and hybrid attention plus state-space models.

哪個組織發展了 Falcon 模型?

Falcon 由位於阿聯酋阿布達比的政府支持的研究機構 TII 開發。

用於訓練早期 Falcon 模型的 RefinedWeb 資料集有什麼值得注意的地方?

RefinedWeb 表明,規模化、嚴格過濾的網路數據可以與精心策劃的法學碩士培訓語料庫相媲美。

Falcon Transformer 模型使用什麼注意力技術來節省推理記憶體?

多查詢注意力讓許多頭共享一組鍵/值投影,從而減少內存並加快生成速度。

Falcon Mamba 在架構上與標準 Falcon 模型有何不同?

Falcon Mamba 使用選擇性狀態空間架構,以近乎恆定的記憶而不是變壓器注意力來處理長序列。

Falcon型號一般都是在什麼條件下發布的?

Falcon 模式在授權條款下公開發布,支持全球商業和研究用途。