視覺人工智慧指南

CLIP 和視覺語言模型

CLIP(對比語言-圖像預訓練)是一種 2021 OpenAI 模型,它訓練圖像編碼器和文字編碼器,將匹配的圖片和標題靠近放置在一個共享向量空間中。

聆聽此指南AI 語音敘述 · ~9 min

本頁閱讀時間 10 分鐘
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. 使用 CLIP 對一張照片進行零樣本分類
  5. 現實世界的實施
  6. CLIP 和視覺語言模型的未來
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

此共享空間允許軟體使用簡單語言描述對圖像進行分類、搜尋和過濾,而 CLIP 式編碼器成為許多文字到圖像和多模式聊天系統的視覺前端。

重點摘要

  1. CLIP 將圖像編碼器和文字編碼器一起訓練,因此匹配的圖像和標題最終會靠近一個共享的嵌入空間。
  2. OpenAI 在 4 億個網路圖像文字對上訓練 CLIP,批量大小為 32,768。
  3. 最好的 CLIP 模型達到了 76.2% 的零樣本 ImageNet 準確率,與有監督的 ResNet-50 相匹配,而無需使用 ImageNet 的標記訓練圖像。
  4. CLIP 對文字與圖像的匹配程度進行評分;它本身不會產生圖像或標題。
  5. Stable Diffusion v1 使用 CLIP 文字編碼器,LLaVA 使用 CLIP 影像編碼器提供語言模型。
  6. CLIP 論文記錄了計數、細粒度類別和手寫數字的弱點,在 MNIST 上得分為 88%。

深入探討

CLIP, short for Contrastive Language-Image Pre-training, is a pair of neural networks described by OpenAI researchers in the paper "Learning Transferable Visual Models From Natural Language Supervision", posted to arXiv on 26 February 2021. One network, the image encoder, turns a picture into a fixed-length list of numbers called an embedding. The other, the text encoder, turns a sentence into an embedding of the same length. Training pulls the embedding of an image toward the embedding of its real caption and pushes mismatched pairs apart. CLIP does not draw images or write captions. It scores how well a piece of text describes an image, and that single ability supports classification, search, content filtering and the conditioning of image generators. A vision-language model (VLM) is the broader family of systems that process images and text together, and CLIP is the encoder much of that family was built on.

The training data came from the web. The authors assembled 400 million image-text pairs from publicly available internet sources, keeping pairs whose text contained one of 500,000 search queries and capping each query at about 20,000 pairs. In every training step a batch of N pairs (32,768 in the paper) passes through both encoders. Each output is projected by a single linear layer into the shared space and normalized to unit length. The model then computes the cosine similarity of every image with every text, an N by N grid. The N true pairs sit on the diagonal and every other cell is a negative example. A cross-entropy loss asks each row to pick its correct caption and each column to pick its correct image, and the two losses are averaged. A learned temperature scales the similarities before the softmax; the paper initializes it to the equivalent of 0.07 and clips it so logits are never multiplied by more than 100.

Zero-shot classification reuses that scoring ability without any task-specific training, which is what zero-shot means. To sort an image into 1,000 categories, write one short sentence per category, such as "a photo of a dog", embed all of them with the text encoder, embed the image, and choose the sentence with the highest similarity. The paper points out that the text embeddings behave like the weights of a linear classifier, so they are computed once and cached. Wording matters. Prompt engineering combined with an ensemble of 80 context templates improved ImageNet accuracy by almost 5 percentage points over bare class names, with the ensemble alone adding 3.5 points.

The headline results set the template for later work. OpenAI trained five ResNet and three Vision Transformer (ViT) image encoders for 32 epochs each. The largest ResNet, RN50x64, took 18 days on 592 V100 GPUs; the largest ViT took 12 days on 256 V100 GPUs. The best model, ViT-L/14 given one extra epoch at 336-pixel resolution, reached 76.2% zero-shot top-1 accuracy on ImageNet, matching the original supervised ResNet-50, and 95% top-5. On seven natural distribution-shift test sets, including ImageNet Sketch and ObjectNet, zero-shot CLIP models shrank the gap between ImageNet accuracy and shifted accuracy by up to 75%. The paper also showed smooth scaling: average zero-shot error across 39 evaluations followed a log-log linear trend over a 44x range of compute.

Milestones followed quickly. Stable Diffusion v1 used a frozen CLIP ViT-L/14 as its text encoder. The OpenCLIP project reproduced the recipe on open data such as LAION-2B and DataComp-1B. Google's SigLIP paper (March 2023) swapped the softmax loss for a pairwise sigmoid loss and, combined with locked-image tuning, reported 84.5% zero-shot ImageNet accuracy after two days on four TPUv4 chips. LLaVA (April 2023) turned CLIP into the eyes of a chat assistant: features from a pretrained CLIP ViT-L/14 pass through a projection matrix into the Vicuna language model, which was then tuned on 158,000 machine-generated instruction-following samples. Its authors report an 85.1% relative score against GPT-4 on a synthetic benchmark and 92.53% on ScienceQA when combined with GPT-4. In this design CLIP no longer makes the final decision; it supplies visual features that a large language model reads alongside text.

Compared with a supervised classifier trained on a fixed label list, CLIP trades some peak accuracy on any one benchmark for flexibility, because any category that can be described in words is a valid label. The paper measured why the contrastive design was chosen: matching whole captions learned zero-shot ImageNet transfer 4x more efficiently than predicting a bag-of-words version of the caption, which was itself 3x more efficient than a transformer predicting the exact caption. ViTs were about 3x more compute-efficient than ResNets as CLIP image encoders. Fine-tuning a CLIP model on ImageNet raised ImageNet accuracy by 9.2% but erased most of its robustness advantage on shifted data. Against SigLIP, standard CLIP needs a softmax normalized over the whole batch, while the sigmoid loss treats each image-text pair as an independent yes-or-no decision and, per its authors, outperforms softmax by a large margin below batch sizes of 16,000.

The limitations are documented in the paper itself. Zero-shot CLIP is weak at fine-grained distinctions such as car models, flower species and aircraft variants, at counting objects, and at unfamiliar tasks such as estimating the distance to the nearest car, where it can be near chance. It scored 88% on MNIST handwritten digits, below logistic regression on raw pixels, because almost no digit-like images appear in its training data, and 51% on full street-number recognition (SVHN). Because CLIP learned to read text in images, OpenAI researchers described typographic attacks in the March 2021 Distill article "Multimodal Neurons in Artificial Neural Networks", where written words in a photo can steer the classification. Web-scraped captions also carry the associations and gaps of their sources, so outputs should be audited before use in sensitive settings.

In practice, running CLIP is cheap and training it is not. Classification costs one image-encoder pass per image plus one text-encoder pass per label, and label embeddings are reused across all images. Search systems store image embeddings in a vector index and compare each query by cosine similarity. Training from scratch is the expensive part: the original runs used hundreds of V100 GPUs for up to 18 days, and a batch of 32,768 forced the similarity computation to be sharded across GPUs. SigLIP found that a batch of about 32,000 is sufficient and that gains fade as batches grow toward one million, which lowers the bar for smaller labs. Most practitioners start from released weights, such as OpenAI's checkpoints or OpenCLIP models, and add a small trained linear layer when zero-shot accuracy is not enough.

04工作範例

使用 CLIP 對一張照片進行零樣本分類

  1. 載入已發布的 CLIP 檢查點,例如 OpenAI ViT-B/32 或 OpenCLIP 模型,及其影像預處理和分詞器。

  2. 每個候選標籤寫一個提示,例如「貓的照片」、「狗的照片」和「汽車的照片」。

  3. 透過文字編碼器運行提示並對三個生成的嵌入進行 L2 標準化;緩存它們以供重用。

  4. 使用模型的預處理調整照片大小並裁剪照片,透過影像編碼器運行它,並對嵌入進行 L2 標準化。

  5. 計算影像嵌入和每個文字嵌入之間的點積,乘以模型的學習尺度,然後套用 softmax。

  6. 讀出機率最高的標籤,然後添加更多模板措辭並平均它們的嵌入,看看預測是否穩定。

它顯示了什麼

每個標籤的機率,無需任何特定於任務的訓練即可計算。要檢查它,請通過相同的步驟運行一小部分標記的您自己的圖像,並比較有和沒有提示集成的準確性; CLIP 論文報告稱,集成對 ImageNet 有幫助。

現實世界的實施

Stable Diffusion v1-4 在凍結的、預先訓練的 CLIP ViT-L/14 文字編碼器上調節其影像產生器,因此使用者輸入的提示在任何去雜訊開始之前就變成了 CLIP 文字嵌入。

LLaVA (2023) 透過一個簡單的投影矩陣將預先訓練的 CLIP ViT-L/14 影像編碼器連接到 Vicuna 語言模型,讓聊天模型回答有關照片的問題。

零樣本圖像分類:最初的 CLIP 論文僅從類別名稱中報告了 76.2% 的 top-1 ImageNet 準確率,而無需對 ImageNet 的 128 萬張標記圖像中的任何圖像進行訓練。

對照片集進行文字到圖像搜尋:每個圖像都嵌入一次,在搜尋時嵌入「雪中的紅色自行車」等查詢,結果按餘弦相似度排名。

開放複製品,例如 OpenCLIP 在公共資料集上訓練 CLIP 風格的模型;在 LAION-2B 上訓練的 ViT-bigG-14 在專案 README 中報告了 80.1% 的零樣本 ImageNet 準確率。

CLIP 和視覺語言模型的未來

Several research directions grew directly out of CLIP's documented weaknesses. Data quality is one: the OpenCLIP project reports that the same architecture trained on different public datasets, for example LAION-2B versus DataComp-1B, reaches noticeably different zero-shot accuracy, so dataset curation has become a research topic of its own. Loss design is another: the 2023 SigLIP paper showed that a sigmoid loss decouples batch size from normalization and found little benefit beyond batches of about 32,000, shifting attention from raw scale to efficiency. The largest shift is in how the encoders are used. Since LLaVA's April 2023 release, a common recipe is to keep a contrastively trained image encoder and project its features into a large language model, so open questions now include how to preserve fine detail, text in images and spatial relations that a single global embedding discards. The weaknesses listed in the 2021 paper, such as counting, fine-grained categories and typographic attacks, remain useful test cases for any new vision-language model, and published evaluations rather than marketing claims are the reliable way to judge whether a newer system has closed them.

付諸實踐

為什麼這很重要

  • 速度與規模. 視覺人工智慧可以大規模自動化檢查、檢測和標記任務。
  • 配裝選擇. 創意團隊可以透過更少的手動修改來更快地建立概念原型。
  • 團隊與工作流程. 操作可以使用以前難以處理的影像和視訊訊號。

留意

  • 如果出處不明,肖像權和同意可能會成為法律風險。
  • 模型表現可能因光照、人口統計和環境的不同而有所不同。
  • 除非監控置信閾值,否則誤報可能會被忽略。

後續步驟

  1. 定義精確度、召回率和錯誤成本的接受標準。
  2. 使用符合實際生產條件的數據進行測試。
  3. 為低置信度或高影響力的預測添加人工審核。
  4. 追蹤模型漂移並在相機或資料集變更後重新驗證。

資料來源與延伸閱讀

  1. arXiv (OpenAI)從自然語言監督學習可轉移視覺模型
  2. arXiv (Google)语言图像预训练的 Sigmoid 损失
  3. arXiv (NeurIPS 2023)视觉指令调整
  4. LLaVA project pageLLaVA:大型語言和視覺助手
  5. GitHub (mlfoundations)OpenCLIP:CLIP 的開源實現
  6. Distill人工神經網路中的多模態神經元
  7. Hugging FaceCompVis/stable-diffusion-v1-4 模型卡

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

參加 CLIP 和視覺語言模型測驗

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

什麼是 CLIP 和視覺語言模型?

CLIP(對比語言-圖像預訓練)是一種 2021 OpenAI 模型,它訓練圖像編碼器和文字編碼器,將匹配的圖片和標題靠近放置在一個共享向量空間中。此共享空間允許軟體使用簡單語言描述對圖像進行分類、搜尋和過濾,而 CLIP 式編碼器成為許多文字到圖像和多模式聊天系統的視覺前端。

CLIP背後的核心思想是什麼?

CLIP 將圖像和文字映射到一個共享向量空間中,以便可以直接測量它們的相似性。

CLIP 使用什麼訓練方法?

CLIP 使用對比目標:正確的圖像標題對被拉近,而不匹配的圖像標題對被推開。

零樣本分類在原始 CLIP 傳輸設定中意味著什麼?

CLIP有大量的預訓練;零樣本的範圍是目標分類評估。

CLIP 如何衡量圖像和標題是否匹配?

CLIP將每個編碼成向量並計算餘弦相似度;相似度越高意味著語義匹配越接近。

CLIP 大約使用了多少資料進行訓練?

CLIP 從互聯網上收集的大約 4 億個圖像標題對中學習,為其提供了廣泛的視覺語言知識。