語言人工智慧指南

及時調整

提示微調(Prompt tuning)透過學習一小段可訓練的向量序列(稱為軟提示,soft prompt)並將其附加在每個輸入之前,使凍結的語言模型能夠適應新任務。

聆聽此指南AI 語音敘述 · ~8 min

本頁閱讀時間 9 分鐘
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. 使用 Hugging Face PEFT 中的軟提示訓練投訴分類器
  5. 現實世界的實施
  6. 即時調優的未來
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

訓練期間僅有這些向量會發生變化,因此一個共享模型即可服務多項任務,且每項任務僅需儲存幾千個數字,而非模型的完整副本。

重點摘要

  1. 提示微調僅訓練一小段附加在輸入前的虛擬標記嵌入矩陣,而語言模型的所有權重皆保持凍結。
  2. 2021 年由 Lester、Al-Rfou 和 Constant 發表的 Google 論文顯示,一旦 T5 模型達到約 110 億個參數,提示微調在 SuperGLUE 上的表現即可媲美完整微調。
  3. 在 T5-XXL 上,一個五標記的軟提示僅包含 20,480 個參數,相較之下,完整微調該模型的副本則需要 110 億個參數。
  4. 在原始研究中,提示微調在大多數領域外問答測試中擊敗了完整微調,包括在 TextbookQA 上獲得了 12.5 分的 F1 分數提升。
  5. 對於參數小於約 10 億的模型,深度變體(如前綴微調 prefix tuning 和 P-Tuning v2)通常比僅輸入的軟提示效果更好。

深入探討

Prompt tuning is a way to teach an existing language model a new job without changing the model itself. A normal text prompt is a string of words, such as "Classify the sentiment of this review". A soft prompt is different: it is a short list of numeric vectors that live in the same space as the model's word embeddings (the vectors the model uses to represent each token), but they do not have to correspond to any real word. The model's billions of weights stay frozen. Training adjusts only the soft prompt, using ordinary gradient descent on labeled examples, until the frozen model produces the right outputs when the prompt sits in front of each input.

The mechanism has four steps. First, choose a prompt length p, typically between 5 and 100 virtual tokens, and create a matrix of p vectors, each with the same width e as the model's embeddings. Second, initialize those vectors, either randomly, by copying embeddings of common vocabulary words, or by copying embeddings of the class label words such as "positive" and "negative". Third, for each training example, convert the input text to embeddings as usual, concatenate the prompt matrix in front of them, and run the frozen model forward. Fourth, compute the task loss on the model's output and backpropagate it only into the prompt matrix. At inference time the learned prompt is attached to new inputs in exactly the same way. Because the prompt is the only task-specific artifact, a single deployed model can handle many tasks by swapping prompts per request.

The technique was introduced by Brian Lester, Rami Al-Rfou and Noah Constant of Google in "The Power of Scale for Parameter-Efficient Prompt Tuning", submitted to arXiv in April 2021 and published at EMNLP 2021. Its central finding is about scale. Using T5 models from small to XXL (11 billion parameters), the authors showed that prompt tuning lags behind full fine-tuning on small models but closes the gap as models pass a few billion parameters. They also report that GPT-3's few-shot prompting scored 71.8 on SuperGLUE against 89.3 for a fine-tuned T5-XXL, even though GPT-3 is 16 times larger. Google's February 2022 research blog summarizes the headline: on T5-XXL, prompt tuning matches model tuning while training about 20,000 parameters to steer an 11-billion-parameter model.

The paper's ablations give practical guidance. Lengthening the prompt beyond about 20 tokens generally gave only small gains, and the XXL model still gave strong results with a single-token prompt. Initializing from class-label embeddings beat random initialization on small and medium models, but the difference vanished at XXL size. A surprising detail concerned pretraining: T5's original span-corruption objective, where the model fills in masked spans with sentinel tokens, made frozen models poor prompt targets. The authors continued training T5 1.1 checkpoints with a plain language-modeling objective for up to 100,000 steps and released these "LM-adapted" checkpoints, which became the standard starting point for prompt-tuning experiments.

Prominent alternatives sit on a spectrum of how much of the model they touch. Prompt design, or hand-written prompting, trains nothing and relies on the model following instructions and a few examples. Prefix tuning, proposed by Xiang Lisa Li and Percy Liang in January 2021, learns continuous vectors that are inserted at every transformer layer rather than only at the input, training about 0.1% of the parameters of GPT-2 and BART. P-Tuning v2, by Xiao Liu and colleagues at Tsinghua (ACL 2022), applies deep prompts to every layer and reports matching fine-tuning across model sizes and harder tasks such as sequence labeling, with 0.1% to 3% of parameters tuned. LoRA, from Microsoft researchers in June 2021, instead adds small low-rank matrices to the weight updates and reports 10,000 times fewer trainable parameters than full fine-tuning of GPT-3 175B. Full fine-tuning updates every weight and usually gives the strongest per-task result, at the cost of storing a full model copy for each task.

The main appeal is robustness and economy. Lester and colleagues trained on SQuAD and tested on out-of-domain question-answering datasets from the MRQA 2019 shared task; prompt tuning beat full fine-tuning on most of them, with a 12.5-point F1 gap on TextbookQA. Transferring a paraphrase detector from the QQP dataset to MRPC, prompt tuning scored 76.3% accuracy against 73.1% for full fine-tuning. The authors interpret this as evidence that freezing the model limits overfitting to quirks of the training set. Storage is the other benefit: with a five-token prompt on T5-XXL, each task needs 20,480 parameters instead of another 11-billion-parameter copy, which the paper notes takes 42 GiB on disk.

The method has clear limits. On models below roughly a billion parameters, plain prompt tuning trails full fine-tuning by a wide margin on SuperGLUE, which is why P-Tuning v2 and prefix tuning add trainable vectors at deeper layers. Training can be slow to converge and sensitive to learning rate; the Google work used a learning rate of 0.3, far higher than typical fine-tuning rates. Learned prompts are hard to interpret, since their vectors rarely match real words; the paper found nearest-neighbor words formed loose semantic clusters, not readable instructions. Prompts also consume context: every virtual token occupies a position the input could have used. Finally, a prompt is tied to the exact model it was trained on, so upgrading the base model means retraining every prompt.

In practice the costs are modest. Training requires a forward and backward pass through the full frozen model, so GPU memory for activations is similar to fine-tuning, but optimizer state is needed only for the prompt, which removes most of the memory for gradients and Adam statistics. Hugging Face's PEFT library exposes the method through a PromptTuningConfig object where the user sets the number of virtual tokens and the initialization, either random, sampled vocabulary or a text string. Its documented example on a 559-million-parameter BLOOMZ model trains 8,192 parameters, about 0.0015% of the total, and the resulting checkpoint is a few kilobytes.

04工作範例

使用 Hugging Face PEFT 中的軟提示訓練投訴分類器

  1. 載入一個小型因果語言模型(例如 bigscience/bloomz-560m)及其 Tokenizer,並使用 transformers 程式庫。

  2. 建立一個 task_type 為 CAUSAL_LM、prompt_tuning_init 設為 TEXT,且包含初始化字串(例如 'Classify if the tweet is a complaint or no complaint.')的 PromptTuningConfig。

  3. 將 num_virtual_tokens 設為該字串的 Token 長度,以便每個虛擬 Token 都從真實的單字嵌入開始。

  4. 用 get_peft_model 包覆模型,並呼叫 print_trainable_parameters 以確認只有幾千個參數是可訓練的。

  5. 將每個帶標籤的推文格式化為輸入文字後接上標籤單字,然後以相對較高的學習率訓練幾個 Epoch。

  6. 儲存僅寫入提示詞矩陣的配接器(Adapter),並將其重新載入到未更動的基礎模型上方以進行推論。

它顯示了什麼

儲存的配接器大小僅有幾 KB,且包含大約 8,192 個數字。透過為保留(Held-out)的推文產生標籤並與給予相同文字提示詞的未微調基礎模型比較準確度,來進行檢查。

現實世界的實施

Google 於 2021 年的研究在凍結的 T5-XXL 模型(擁有 110 億個參數)上調整軟提示(soft prompts),並在 SuperGLUE 語言理解基準測試中達到與完整微調相當的表現,同時僅需訓練極小比例的權重。

支援團隊可以使用 Hugging Face PEFT 在 bigscience/bloomz-560m 上訓練一個軟提示,將推文分類為投訴或非投訴。該方法僅需訓練約 8,192 個參數,而基礎模型則有 5.59 億個參數。

為多位客戶提供服務的系統可以將一個凍結的模型保留在記憶體中,並將來自不同任務的請求進行批次處理,僅需替換附加在每個請求上的小型提示矩陣即可。

研究人員透過在單一凍結的 T5-XXL 模型上針對同一個 SuperGLUE 任務訓練五個提示,並採用多數決投票的方式建立了低成本的整合模型,其表現優於每個任務的單一提示平均水準。

來自 Google (ACL 2022) 的 SPoT 方法,首先在來源任務上學習一個提示,並將其重複用於初始化新目標任務的提示,在 SuperGLUE 上的表現達到或超越了完整微調。

即時調優的未來

Research since 2021 has focused on making soft prompts reusable and less dependent on model scale. SPoT (Vu and colleagues, ACL 2022) showed that a prompt learned on one or more source tasks can initialize a prompt for a new task, and its study covered 26 NLP tasks in 160 source and target combinations, reporting results that match or beat full fine-tuning on SuperGLUE with up to 27,000 times fewer task-specific parameters. P-Tuning v2 (ACL 2022) extended competitive results to smaller models by adding prompts at every layer. Open problems remain. Soft prompts are tied to one base model, so transferring them across model versions or architectures is an active question. Interpretability is limited because learned vectors rarely map to words. In industry, low-rank methods such as LoRA (2021) have become the more common choice for adapting open models, partly because their weights merge into the base model and add no inference tokens, while prompt tuning remains attractive where many tasks must share a single frozen model in one batch. Whether newer, larger models make soft prompts even more competitive, as the 2021 scaling result suggests, is still being studied on current architectures.

付諸實踐

為什麼這很重要

  • 速度與規模. 語言工作流程可以在不犧牲一致性的情況下更快地移動。
  • 交通與覆蓋範圍. 它擴展了跨語言和溝通方式的訪問。
  • 更明確的決策. 團隊可以花更多時間進行判斷,而自動化則可以處理重複。

留意

  • 幻覺的事實可以悄悄地進入報告、支持流程或研究成果。
  • 及時的敏感性可能會在類似的請求中產生不一致的結果。
  • 如果存取控制薄弱,敏感文字資料可能會暴露。

後續步驟

  1. 在推出之前定義輸出格式、語氣和品質標準。
  2. 當準確性很重要時,請使用可信任來源進行地面回應。
  3. 為高風險輸出保留人工審查檢查點。
  4. 追蹤故障模式並定期重新訓練提示或工作流程。

資料來源與延伸閱讀

  1. arXiv (EMNLP 2021)規模的力量:參數高效的提示調整
  2. Google Research利用學習到的軟提示引導凍結的語言模型
  3. arXiv (ACL 2021)前綴調整:優化用於生成的連續提示
  4. arXiv (ACL 2022)P-Tuning v2:提示調整在各種規模和任務中均可與微調相媲美
  5. arXiv (ACL 2022)SPoT:透過軟提示遷移實現更好的凍結模型適應
  6. arXivLoRA:大型語言模型的低秩適配。
  7. Hugging Face PEFT documentation提示調整

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

參加提示調整測驗

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

什麼是提示調整?

提示調整透過學習一小段可訓練的向量(稱為軟提示)並將其附加到每個輸入之前,使凍結的語言模型能夠適應新任務。訓練過程中僅會改變這些向量,因此一個共享模型可以服務多個任務,而每個任務僅需儲存幾千個數字,無需儲存模型的完整副本。

什麼是提示調音中的「軟提示」?

軟提示是嵌入空間中可訓練的連續向量,而不是人類可讀的單字。

即時調優在哪裡添加其可訓練向量?

提示調整僅在輸入處添加軟提示,這使得它比前綴調整更輕。

原論文強調了模型規模的哪些令人驚訝的效果?

論文的標題強調了「規模的力量」:更大的模型讓快速調整可以與完全微調相提並論。

誰引入了即時調整?

Google 的 Brian Lester、Rami Al-Rfou 和 Noah Constant 於 2021 年發表了提示調整論文。

快速調優如何能夠從一個模型提供多項任務?

只有一個小的軟提示是特定於任務的,因此一個凍結模型可以透過交換提示來承載許多任務。