Polygon and Mask Annotation for Segmentation

Segmentation annotation comes in two closely related forms: polygon annotation, where a labeler clicks a series of vertices to trace an object's outline, and mask annotation, where every pixel belonging to an object is painted or selected directly. Polygons are stored as coordinate lists and are compact and editable, which makes them common for objects with fairly smooth edges like cars, buildings or road markings. Masks are stored as pixel-level bitmaps and are better suited to objects with irregular or fine-grained boundaries, such as hair, foliage, or smoke, where a polygon with straight edges between vertices would blur the true shape. Manual tracing can be time-consuming for complex objects, and quality checks may include a second reviewer checking boundary accuracy. Promptable segmentation models, including Meta's Segment Anything Model (SAM), released in 2023, changed some workflows: in supported tools, a person can click a point or draw a box and receive a proposed mask, then review and correct it. The amount of time or cost saved depends on the task, model, tool, and proposal quality. A common misconception is that segmentation always means separating an object from its background; in practice, panoptic segmentation labels every pixel in an image, including background regions like sky, road and grass, not just discrete foreground objects. Another misconception is that masks are strictly more accurate than polygons; for polygon-friendly shapes, a well-placed polygon can match mask accuracy while using far less storage and being easier for a human to verify.