Visuele AI-GIDS

Segment Anything Model (SAM)

The Segment Anything Model (SAM) is Meta AI's promptable image segmentation model: given an image and a click, box or rough mask, it outputs a pixel-accurate outline of the object.

Luister naar deze gidsAI-verteld · ~8 min

Op deze pagina9 minuten lezen
  1. Overzicht
  2. Belangrijkste inzichten
  3. Diepe duik
  4. Segmenting one object with point prompts
  5. Implementatie in de echte wereld
  6. The Future of Segment Anything Model (SAM)
  7. Het in de praktijk brengen
  8. Bronnen en verdere literatuur
  9. Blijf verkennen
  10. Veelgestelde vragen

Overzicht

Released in April 2023 with SA-1B, a dataset of 1.1 billion masks on 11 million images, it turned segmentation into a general-purpose tool, and its successors SAM 2 and SAM 3 extend it to video and to text-named concepts.

Belangrijkste inzichten

  1. SAM, released by Meta AI in April 2023, outlines objects from point, box or mask prompts without retraining for new categories.
  2. SAM's SA-1B dataset contains 1.1 billion masks on 11 million images, and 99.1% of those masks were generated automatically.
  3. SAM runs a roughly 636 million parameter ViT-H encoder once per image, then a small decoder predicts each mask in about 50 milliseconds.
  4. For ambiguous prompts SAM returns three masks, roughly whole, part and subpart, each with a predicted IoU score.
  5. SAM 2 (2024) extends promptable segmentation to video with streaming memory, and SAM 3 (2025) segments every instance of a text-named concept.

Diepe duik

Image segmentation means labeling exactly which pixels belong to an object, producing an outline, called a mask, rather than just a bounding box. Before 2023, segmentation models were usually trained for a fixed list of categories, such as people, cars and dogs, and needed new labeled data for each new domain. The Segment Anything project, described by Alexander Kirillov, Eric Mintun, Nikhila Ravi, Ross Girshick and colleagues at Meta AI in a paper posted April 5, 2023, set out to build a foundation model for segmentation: one model that can outline almost any object in almost any image, with no retraining.

The key design choice is the task. SAM performs promptable segmentation: it takes an image and a prompt and returns a valid mask. A prompt can be one or more foreground or background points, a box, a rough mask, or, experimentally, text. Prompts are often ambiguous. A click on a shirt could mean the shirt or the whole person. SAM handles this by returning three candidate masks, roughly the whole object, a part and a subpart, each with a predicted quality score, its estimate of intersection over union (IoU), the overlap between a predicted and true mask. The paper notes that nested masks are usually at most three levels deep.

SAM has three parts. The image encoder is a Vision Transformer (ViT), a network that processes images as a grid of patches, pretrained with the masked autoencoder (MAE) method. The largest version, ViT-H, has about 636 million parameters. It reads a 1024 by 1024 image and outputs a 64 by 64 grid of 256-dimensional features, called the image embedding. It runs once per image. The prompt encoder turns points and boxes into vectors using positional encodings plus learned type embeddings, and turns masks into features with small convolutions. The mask decoder is a lightweight two-layer Transformer that lets prompt tokens and image features attend to each other, then upsamples and predicts a mask. Because only the decoder runs per prompt, a user can click repeatedly and see new masks in about 50 milliseconds.

The data came from a 'data engine' in which model and annotators improved each other in three stages. In the assisted-manual stage, professional annotators clicked points with a SAM-powered browser tool and corrected masks with brush and eraser tools; the model was retrained six times, and 4.3 million masks were collected from 120,000 images. In the semi-automatic stage, SAM pre-filled confident masks and annotators labeled what was missing, adding 5.9 million masks from 180,000 images. In the fully automatic stage, SAM was prompted with a 32 by 32 grid of points on each of 11 million images and only confident, stable masks were kept. The result, SA-1B, holds 1.1 billion masks, 99.1% generated automatically. Released images were downsampled to 1,500 pixels on the short side, with faces and license plates blurred.

On 23 diverse segmentation datasets, the authors found SAM's zero-shot results from a single point often competitive with or better than prior fully supervised methods. Training used 256 A100 GPUs for 68 hours. The code and three checkpoints, ViT-B, ViT-L and ViT-H, are released under Apache 2.0, while SA-1B requires agreeing to a research license.

SAM 2, posted August 1, 2024 by Nikhila Ravi, Christoph Feichtenhofer and colleagues, extends promptable segmentation to video. It treats an image as a one-frame video and adds a streaming memory, so a mask drawn on one frame can be propagated and corrected through the rest. The authors reported better video accuracy with three times fewer interactions than prior approaches, and, on images, more accuracy than SAM at six times the speed. SAM 3, posted November 20, 2025 by Nicolas Carion and colleagues, adds promptable concept segmentation: given a short noun phrase such as 'yellow school bus', image examples, or both, it finds and masks every matching instance in an image or video. Its data engine produced 4 million unique concept labels, and the authors report roughly double the accuracy of existing systems on this task. Meta says SAM 3 runs at about 30 milliseconds per image with more than 100 detected objects on an H200 GPU.

Compared with classic models such as Mask R-CNN, trained on fixed label sets, SAM does not name what it segments; it only outlines. That makes it a component rather than a complete system: practitioners often pair it with a detector or language model that supplies boxes or names. Compared with dedicated interactive tools, the SAM paper itself says those methods can outperform SAM when many points are provided, since SAM aims for breadth over maximum precision.

The paper lists other limitations: SAM can miss fine structures, sometimes produces small disconnected fragments, and gives less crisp boundaries than slower methods that zoom in. Its text prompting was exploratory and not fully robust, and domain-specific tools may do better in specialized fields such as medical imaging. The heavy image encoder means the full pipeline is not real time even though prompting is. The authors also examined fairness and reported results across perceived gender presentation, age and skin tone groups. In practice, running ViT-H requires a GPU for reasonable speed, while smaller checkpoints trade accuracy for speed.

04Uitgewerkt voorbeeld

Segmenting one object with point prompts

  1. Install the segment-anything package, download the ViT-B checkpoint, and load a photo containing a person wearing a shirt.

  2. Compute the image embedding once with the predictor's set_image call; this is the slow step.

  3. Pass a single foreground point on the shirt with multimask output enabled, and receive three masks with three predicted IoU scores.

  4. Display the masks: typically one covers a small region of the shirt, one the whole shirt, and one the whole person.

  5. Add a background point on the person's face and run the decoder again, reusing the cached embedding, to remove the person-level interpretation.

  6. Save the chosen mask and compare it with a hand-drawn outline by computing intersection over union.

Wat het laat zien

A binary mask of the shirt, plus a record of how added prompts resolved the ambiguity. Check quality by computing IoU against your manual outline; values close to 1 indicate a tight match, and the second decoder call should return far faster than the first embedding step.

Implementatie in de echte wereld

Interactive labeling: Meta's own annotators used SAM in a browser tool, and average time per mask fell from 34 to 14 seconds as the model improved, per the 2023 paper.

Generating training data automatically: SAM prompted with a 32 by 32 grid of points produced 99.1% of the 1.1 billion masks released in SA-1B.

Running in a web page: the SAM repository exports the lightweight mask decoder to ONNX, and the paper reports it predicts a mask in about 50 milliseconds on a CPU in a browser.

Tracking objects through video with SAM 2, which the 2024 paper reports reaches better accuracy than prior methods while using three times fewer user interactions.

Facebook Marketplace's View in Room feature uses SAM 3 and SAM 3D to show how home decor items such as lamps would look in a shopper's space, according to Meta.

Wildlife research: Meta's SA-FARI dataset, built with Conservation X Labs and Osa Conservation, provides masks for every animal in over 10,000 camera-trap videos.

The Future of Segment Anything Model (SAM)

The Segment Anything line shows a clear direction: from points on single images (SAM, April 2023), to tracked objects in video (SAM 2, August 2024), to finding every instance of a named concept (SAM 3, November 2025). Meta's March 2026 SAM 3.1 update reports tracking up to 16 objects per forward pass and raising speed from 16 to 32 frames per second on one H100 GPU for videos with a medium number of objects, which points toward real-time multi-object use. Open problems remain. The original paper said it was unclear how to design simple prompts for semantic and panoptic segmentation, and SAM 3's concept prompts are one answer whose robustness across languages, rare concepts and specialized domains is still being measured, for example through the SA-Co benchmark and partner datasets such as SA-FARI for wildlife and FathomNet for ocean imagery. Fine boundaries, thin structures and medical or scientific images where domain tools still compete are active areas. As segmentation becomes cheap enough to run on every frame, questions about privacy, such as segmenting people in public video, and about dataset licensing are likely to get more attention.

Het in de praktijk brengen

Waarom het ertoe doet

  • Snelheid en schaal. Visuele AI kan inspectie-, detectie- en taggingtaken op schaal automatiseren.
  • Bouwkeuzes. Creatieve teams kunnen concepten sneller prototypen met minder handmatige revisies.
  • Team en workflow. Bij bewerkingen kan gebruik worden gemaakt van beeld- en videosignalen die voorheen moeilijk te verwerken waren.

Pas op voor

  • Beeldrechten en toestemming kunnen juridische risico's worden als de herkomst onduidelijk is.
  • De prestaties van modellen kunnen variëren afhankelijk van de belichting, demografische gegevens en omgevingen.
  • Valse positieve resultaten kunnen onopgemerkt blijven, tenzij de vertrouwensdrempels worden gecontroleerd.

Volgende stappen

  1. Definieer acceptatiecriteria voor precisie-, terugroep- en foutkosten.
  2. Test met gegevens die overeenkomen met echte productieomstandigheden.
  3. Voeg menselijke beoordeling toe voor voorspellingen met weinig vertrouwen of hoge impact.
  4. Volg modelafwijkingen en valideer opnieuw na wijzigingen in de camera of dataset.

Bronnen en verdere literatuur

  1. arXiv (Meta AI)Segment Anything
  2. Meta AI (GitHub)facebookresearch/segment-anything
  3. arXivSAM 2: Segment Anything in Images and Videos
  4. arXivSAM 3: Segment Anything with Concepts
  5. Meta AIIntroducing Meta Segment Anything Model 3 and Segment Anything Playground

Blijf verkennen

Gratis nieuwsbrief

Ontvang de dagelijkse AI-briefing

Drie verifieerde AI-verhalen elke weekdagmorgen, geschreven in eenvoudig Engels. Gratis voor altijd, geen advertenties.

Eén e-mail elke weekdag. Afmelden met één klik. We verkopen of delen je adres nooit.

Test jezelf

Take the Segment Anything Model (SAM) quiz

Directe feedback op elke antwoord, en een deelbare certificaat met een verifieerbare ID zodra je een cursus hebt afgerond.

Quiz starten

Ondersteun gratis AI-onderwijs. AI Understanding is een 501(c)(3) non-profit — geen advertenties, geen betaalmuur, nooit. Doe een donatie

Veelgestelde vragen

What is Segment Anything Model (SAM)?

The Segment Anything Model (SAM) is Meta AI's promptable image segmentation model: given an image and a click, box or rough mask, it outputs a pixel-accurate outline of the object. Released in April 2023 with SA-1B, a dataset of 1.1 billion masks on 11 million images, it turned segmentation into a general-purpose tool, and its successors SAM 2 and SAM 3 extend it to video and to text-named concepts.

Welk kernidee maakt SAM tot een ‘fundamenteel model’ voor segmentatie?

SAM herformuleert segmentatie als prompt en is ontworpen voor zero-shot-overdracht naar objecten en afbeeldingen buiten de trainingsset.

Hoe groot is de SA-1B-dataset die wordt gebruikt om SAM te trainen ongeveer?

SA-1B bevat meer dan 1 miljard maskers op ongeveer 11 miljoen afbeeldingen, gebouwd met een model-in-the-loop annotatie-engine.

Waarom splitst SAM zijn architectuur op in een zware image-encoder en een lichtgewicht prompt-encoder/decoder?

De dure beeldcodering wordt één keer uitgevoerd; dan maken goedkope promptcodering en maskerdecodering het snel, interactief opnieuw oproepen van hetzelfde beeld mogelijk.

Hoe gaat SAM om met een dubbelzinnige prompt, zoals een enkele klik die meerdere objecten kan betekenen?

SAM voert verschillende kandidaatmaskers met scores uit, zodat dubbelzinnigheid kan worden opgelost door middel van rangschikking of aanvullende aanwijzingen.

Welk type backbone gebruikt SAM om het invoerbeeld te coderen?

De beeldencoder van SAM is een Vision Transformer, vaak vooraf getraind met gemaskeerde auto-codering, waardoor een compacte beeldinbedding ontstaat.