Moodooyinka Falcon

Falcon is the name of the large language models built by the Technology Innovation Institute (TII), a research institute in Abu Dhabi, United Arab Emirates. A large language model (LLM) is a neural network trained on large amounts of text to predict the next token, a word or word fragment, which lets it answer questions, write and summarize. TII releases Falcon models openly: anyone can download the weights, the learned numbers that make up the model, and run or fine-tune them on their own hardware. That openness, together with strong benchmark results, made Falcon one of the best-known model families from outside the United States and China.

The first widely used releases were Falcon-7B and Falcon-40B in 2023. Hugging Face reported in June 2023 that Falcon-40B was trained on 1 trillion tokens and Falcon-7B on 1.5 trillion, using 2,800 and 700 petaflop-days of compute respectively. Falcon-40B ranked first on Hugging Face's Open LLM Leaderboard at release, and both models were licensed under Apache 2.0, which allows commercial use. In September 2023 TII released Falcon-180B, trained on 3.5 trillion tokens using up to 4,096 GPUs and about 7 million GPU hours on Amazon SageMaker. At release it had the highest leaderboard score of any openly released pretrained model, 68.74, later revised to 67.85, on par with Llama 2 70B.

The central idea behind those models was data. TII's RefinedWeb paper, posted in June 2023, argued that properly filtered and deduplicated web data, and web data only, could match or beat curated mixtures of books, papers and conversations. Deduplication means removing repeated or near-identical documents so the model does not memorize them. The team extracted about 5 trillion tokens from Common Crawl, a public archive of web pages, and released a 600-billion-token portion. Hugging Face noted that over 80% of Falcon-7B and 40B's training data came from RefinedWeb. The November 2023 Falcon series paper described the 180B model's training run as the largest openly documented pretraining run at the time.

Architecture was the second theme. The early Falcon models used multiquery attention, in which all attention heads share one set of keys and values instead of each head having its own. During text generation a model stores keys and values for every past token in a key-value cache; sharing them shrinks that cache. Hugging Face's comparison for a 2,048-token context lists 240 MB for Falcon-40B against 5,400 MB for LLaMA-65B. From 2024 TII went further. Falcon Mamba 7B, described in an October 2024 paper and trained on 5.8 trillion tokens, uses the Mamba state-space architecture with no attention at all; a state-space model carries a fixed-size summary of everything read so far, so memory stays constant however long the text grows. In December 2024 the Falcon3 family added models of 1, 3, 7 and 10 billion parameters, the 7B trained on 14 trillion tokens on 1,024 H100 GPUs.

Falcon-H1, published in a July 2025 technical report, combined both approaches. Each layer runs attention heads and state-space heads in parallel, with the ratio between them adjustable. The family spans 0.5B, 1.5B, 1.5B-Deep, 3B, 7B and 34B parameters, supports contexts up to 256,000 tokens and 18 languages, and ships as more than 30 checkpoints. TII reports that the 34B model matches or beats models up to about 70 billion parameters, such as Llama 3.3 70B, and that the 0.5B model performs like typical 7B models from 2024. On January 5, 2026, TII released Falcon-H1R 7B, a reasoning model built on Falcon-H1; The Decoder reported TII's claim of a 49.5% average across four benchmarks against 46.2% for Qwen3 32B.

Compared with other open model families, Falcon's distinguishing traits are its emphasis on web-data quality, its early adoption of memory-saving attention and its turn to state-space and hybrid designs. Licenses differ by release, which matters for builders. Falcon-7B and 40B use Apache 2.0. Falcon-180B allows commercial use only under restrictive conditions that exclude hosting it as a service, according to Hugging Face. Newer models use the Falcon LLM license, which permits free and commercial use but requires following an acceptable use policy that TII can update at any time.

The limitations are those of most open models plus a few specific ones. Most performance figures for recent Falcon models are self-reported by TII, and The Decoder cautioned that benchmark scores only loosely track real-world usefulness, especially for small models. Falcon-180B's leaderboard lead was brief, and by its own revised score it was level with Llama 2 70B. Pure state-space models trade some ability to look back at exact earlier text for constant memory, which is one reason hybrid designs like Falcon-H1 reintroduce attention. As with any model trained mostly on web text, outputs can reflect web biases and errors, and openly released weights cannot be recalled once published.

Running Falcon models ranges from easy to expensive. Falcon-7B fits in about 15 GB of GPU memory, and Falcon-40B needs about 90 GB at 16-bit precision or about 27 GB at 4-bit. Falcon-180B needs about 640 GB, eight 80 GB A100 GPUs, at 16-bit precision, or 320 GB when quantized to 4-bit. The smaller Falcon3, Falcon-H1 and H1R models are designed for single GPUs, laptops and edge devices, and quantized versions are published on Hugging Face. Training a Falcon-scale model from scratch is far beyond most organizations: the 180B run used about 7 million GPU hours.