Segment Anything Model (SAM)

Image segmentation means labeling exactly which pixels belong to an object, producing an outline, called a mask, rather than just a bounding box. Before 2023, segmentation models were usually trained for a fixed list of categories, such as people, cars and dogs, and needed new labeled data for each new domain. The Segment Anything project, described by Alexander Kirillov, Eric Mintun, Nikhila Ravi, Ross Girshick and colleagues at Meta AI in a paper posted April 5, 2023, set out to build a foundation model for segmentation: one model that can outline almost any object in almost any image, with no retraining.

The key design choice is the task. SAM performs promptable segmentation: it takes an image and a prompt and returns a valid mask. A prompt can be one or more foreground or background points, a box, a rough mask, or, experimentally, text. Prompts are often ambiguous. A click on a shirt could mean the shirt or the whole person. SAM handles this by returning three candidate masks, roughly the whole object, a part and a subpart, each with a predicted quality score, its estimate of intersection over union (IoU), the overlap between a predicted and true mask. The paper notes that nested masks are usually at most three levels deep.

SAM has three parts. The image encoder is a Vision Transformer (ViT), a network that processes images as a grid of patches, pretrained with the masked autoencoder (MAE) method. The largest version, ViT-H, has about 636 million parameters. It reads a 1024 by 1024 image and outputs a 64 by 64 grid of 256-dimensional features, called the image embedding. It runs once per image. The prompt encoder turns points and boxes into vectors using positional encodings plus learned type embeddings, and turns masks into features with small convolutions. The mask decoder is a lightweight two-layer Transformer that lets prompt tokens and image features attend to each other, then upsamples and predicts a mask. Because only the decoder runs per prompt, a user can click repeatedly and see new masks in about 50 milliseconds.

The data came from a 'data engine' in which model and annotators improved each other in three stages. In the assisted-manual stage, professional annotators clicked points with a SAM-powered browser tool and corrected masks with brush and eraser tools; the model was retrained six times, and 4.3 million masks were collected from 120,000 images. In the semi-automatic stage, SAM pre-filled confident masks and annotators labeled what was missing, adding 5.9 million masks from 180,000 images. In the fully automatic stage, SAM was prompted with a 32 by 32 grid of points on each of 11 million images and only confident, stable masks were kept. The result, SA-1B, holds 1.1 billion masks, 99.1% generated automatically. Released images were downsampled to 1,500 pixels on the short side, with faces and license plates blurred.

On 23 diverse segmentation datasets, the authors found SAM's zero-shot results from a single point often competitive with or better than prior fully supervised methods. Training used 256 A100 GPUs for 68 hours. The code and three checkpoints, ViT-B, ViT-L and ViT-H, are released under Apache 2.0, while SA-1B requires agreeing to a research license.

SAM 2, posted August 1, 2024 by Nikhila Ravi, Christoph Feichtenhofer and colleagues, extends promptable segmentation to video. It treats an image as a one-frame video and adds a streaming memory, so a mask drawn on one frame can be propagated and corrected through the rest. The authors reported better video accuracy with three times fewer interactions than prior approaches, and, on images, more accuracy than SAM at six times the speed. SAM 3, posted November 20, 2025 by Nicolas Carion and colleagues, adds promptable concept segmentation: given a short noun phrase such as 'yellow school bus', image examples, or both, it finds and masks every matching instance in an image or video. Its data engine produced 4 million unique concept labels, and the authors report roughly double the accuracy of existing systems on this task. Meta says SAM 3 runs at about 30 milliseconds per image with more than 100 detected objects on an H200 GPU.

Compared with classic models such as Mask R-CNN, trained on fixed label sets, SAM does not name what it segments; it only outlines. That makes it a component rather than a complete system: practitioners often pair it with a detector or language model that supplies boxes or names. Compared with dedicated interactive tools, the SAM paper itself says those methods can outperform SAM when many points are provided, since SAM aims for breadth over maximum precision.

The paper lists other limitations: SAM can miss fine structures, sometimes produces small disconnected fragments, and gives less crisp boundaries than slower methods that zoom in. Its text prompting was exploratory and not fully robust, and domain-specific tools may do better in specialized fields such as medical imaging. The heavy image encoder means the full pipeline is not real time even though prompting is. The authors also examined fairness and reported results across perceived gender presentation, age and skin tone groups. In practice, running ViT-H requires a GPU for reasonable speed, while smaller checkpoints trade accuracy for speed.