CLIP and Vision-Language Models

Learning Transferable Visual Models from Natural Language Supervision introduces CLIP and reports training on about four hundred million image-text pairs. The system learns an image encoder and a text encoder whose projected vectors can be compared in a shared representation space. In a batch of N observed pairs, it computes similarities for all N times N image-text combinations. The objective favors the N observed pairings relative to the other combinations under the studied batch construction. With thirty-two images and thirty-two texts, there are 1024 score entries, including thirty-two observed paired entries. This count describes cross-modal comparisons, not the number of original training examples in the batch. The original paper evaluates modified ResNet and Vision Transformer image encoders, so a ViT is not the only CLIP architecture. The text encoder in its main method is a transformer. For zero-shot classification, candidate labels are expressed as text descriptions, encoded, and compared with an image representation. Zero-shot refers to no task-specific supervised fitting in that evaluation; the model still underwent large-scale pretraining. The result depends on labels, prompt wording, and the evaluated distribution, and a similarity score does not prove that every named object or relation is understood. CLIP can supply representations to another system. The unCLIP paper, for example, combines a prior over CLIP image embeddings with a decoder for image generation. That generative architecture is distinct from claiming that the original CLIP encoders directly create pixels or produce complete captions.