Learned Feature Matching: SuperGlue and LightGlue

Classical matchers like SIFT find the nearest-neighbor descriptor for each keypoint independently, which struggles when many keypoints look locally similar, such as repeated windows on a building facade, or when a large viewpoint change distorts the local patch around a keypoint beyond what a fixed descriptor can absorb. SuperGlue, introduced by Sarlin and colleagues at CVPR in 2020, reframes matching as a joint optimization problem solved by a graph neural network. Given two sets of keypoints, each with a position and a descriptor from a separate detector such as SuperPoint, SuperGlue applies alternating layers of self-attention, where keypoints attend to other keypoints in the same image, and cross-attention, where keypoints attend to keypoints in the other image, letting the network build up contextual information about the whole scene layout rather than judging each point in isolation. The final matching decision is framed as an optimal transport problem, solved approximately with the Sinkhorn algorithm, which assigns each keypoint to at most one match in the other image while allowing points to be marked as unmatched, which naturally handles occlusion and keypoints visible in only one image. LightGlue, introduced in 2023 as a more efficient successor, keeps the same attention-based architecture but adds an adaptive mechanism that stops processing early for image pairs that are easy to match, and prunes keypoints that are confidently unmatched partway through, letting it spend less computation on easy pairs; the paper reports accuracy and speed results for its evaluated benchmarks, which should not be generalized to every dataset or device. A common misconception is that these systems detect keypoints themselves; in practice they are matchers that take keypoints already found by a separate detector, most often SuperPoint, and their entire contribution is deciding which points correspond across the two images.