CLIP および視覚言語モデル

CLIP, short for Contrastive Language-Image Pre-training, is a pair of neural networks described by OpenAI researchers in the paper "Learning Transferable Visual Models From Natural Language Supervision", posted to arXiv on 26 February 2021. One network, the image encoder, turns a picture into a fixed-length list of numbers called an embedding. The other, the text encoder, turns a sentence into an embedding of the same length. Training pulls the embedding of an image toward the embedding of its real caption and pushes mismatched pairs apart. CLIP does not draw images or write captions. It scores how well a piece of text describes an image, and that single ability supports classification, search, content filtering and the conditioning of image generators. A vision-language model (VLM) is the broader family of systems that process images and text together, and CLIP is the encoder much of that family was built on.

The training data came from the web. The authors assembled 400 million image-text pairs from publicly available internet sources, keeping pairs whose text contained one of 500,000 search queries and capping each query at about 20,000 pairs. In every training step a batch of N pairs (32,768 in the paper) passes through both encoders. Each output is projected by a single linear layer into the shared space and normalized to unit length. The model then computes the cosine similarity of every image with every text, an N by N grid. The N true pairs sit on the diagonal and every other cell is a negative example. A cross-entropy loss asks each row to pick its correct caption and each column to pick its correct image, and the two losses are averaged. A learned temperature scales the similarities before the softmax; the paper initializes it to the equivalent of 0.07 and clips it so logits are never multiplied by more than 100.

Zero-shot classification reuses that scoring ability without any task-specific training, which is what zero-shot means. To sort an image into 1,000 categories, write one short sentence per category, such as "a photo of a dog", embed all of them with the text encoder, embed the image, and choose the sentence with the highest similarity. The paper points out that the text embeddings behave like the weights of a linear classifier, so they are computed once and cached. Wording matters. Prompt engineering combined with an ensemble of 80 context templates improved ImageNet accuracy by almost 5 percentage points over bare class names, with the ensemble alone adding 3.5 points.

The headline results set the template for later work. OpenAI trained five ResNet and three Vision Transformer (ViT) image encoders for 32 epochs each. The largest ResNet, RN50x64, took 18 days on 592 V100 GPUs; the largest ViT took 12 days on 256 V100 GPUs. The best model, ViT-L/14 given one extra epoch at 336-pixel resolution, reached 76.2% zero-shot top-1 accuracy on ImageNet, matching the original supervised ResNet-50, and 95% top-5. On seven natural distribution-shift test sets, including ImageNet Sketch and ObjectNet, zero-shot CLIP models shrank the gap between ImageNet accuracy and shifted accuracy by up to 75%. The paper also showed smooth scaling: average zero-shot error across 39 evaluations followed a log-log linear trend over a 44x range of compute.

Milestones followed quickly. Stable Diffusion v1 used a frozen CLIP ViT-L/14 as its text encoder. The OpenCLIP project reproduced the recipe on open data such as LAION-2B and DataComp-1B. Google's SigLIP paper (March 2023) swapped the softmax loss for a pairwise sigmoid loss and, combined with locked-image tuning, reported 84.5% zero-shot ImageNet accuracy after two days on four TPUv4 chips. LLaVA (April 2023) turned CLIP into the eyes of a chat assistant: features from a pretrained CLIP ViT-L/14 pass through a projection matrix into the Vicuna language model, which was then tuned on 158,000 machine-generated instruction-following samples. Its authors report an 85.1% relative score against GPT-4 on a synthetic benchmark and 92.53% on ScienceQA when combined with GPT-4. In this design CLIP no longer makes the final decision; it supplies visual features that a large language model reads alongside text.

Compared with a supervised classifier trained on a fixed label list, CLIP trades some peak accuracy on any one benchmark for flexibility, because any category that can be described in words is a valid label. The paper measured why the contrastive design was chosen: matching whole captions learned zero-shot ImageNet transfer 4x more efficiently than predicting a bag-of-words version of the caption, which was itself 3x more efficient than a transformer predicting the exact caption. ViTs were about 3x more compute-efficient than ResNets as CLIP image encoders. Fine-tuning a CLIP model on ImageNet raised ImageNet accuracy by 9.2% but erased most of its robustness advantage on shifted data. Against SigLIP, standard CLIP needs a softmax normalized over the whole batch, while the sigmoid loss treats each image-text pair as an independent yes-or-no decision and, per its authors, outperforms softmax by a large margin below batch sizes of 16,000.

The limitations are documented in the paper itself. Zero-shot CLIP is weak at fine-grained distinctions such as car models, flower species and aircraft variants, at counting objects, and at unfamiliar tasks such as estimating the distance to the nearest car, where it can be near chance. It scored 88% on MNIST handwritten digits, below logistic regression on raw pixels, because almost no digit-like images appear in its training data, and 51% on full street-number recognition (SVHN). Because CLIP learned to read text in images, OpenAI researchers described typographic attacks in the March 2021 Distill article "Multimodal Neurons in Artificial Neural Networks", where written words in a photo can steer the classification. Web-scraped captions also carry the associations and gaps of their sources, so outputs should be audited before use in sensitive settings.

In practice, running CLIP is cheap and training it is not. Classification costs one image-encoder pass per image plus one text-encoder pass per label, and label embeddings are reused across all images. Search systems store image embeddings in a vector index and compare each query by cosine similarity. Training from scratch is the expensive part: the original runs used hundreds of V100 GPUs for up to 18 days, and a batch of 32,768 forced the similarity computation to be sharded across GPUs. SigLIP found that a batch of about 32,000 is sufficient and that gains fade as batches grow toward one million, which lowers the bar for smaller labs. Most practitioners start from released weights, such as OpenAI's checkpoints or OpenCLIP models, and add a small trained linear layer when zero-shot accuracy is not enough.