語言人工智慧指南

自然語言推理與蘊涵

自然語言推理(NLI),亦稱為文本蕴含辨識,是判斷一句話(前提)是否使另一句話(假設)為真(蕴含)、假(矛盾)或既不為真也不為假(中立)的任務。

聆聽此指南AI 語音敘述 · ~7 min

本頁8 分鐘閱讀
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. 使用 NLI 檢查摘要句與其來源是否一致
  5. 現實世界的實施
  6. 自然語言推理與蘊涵的未來
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

這很重要,因為它是語言理解的標準測試,也是零樣本分類和檢查生成文本是否得到來源支持的實用工具。

重點摘要

  1. 自然語言推理將前提與假設對標記為蕴含、矛盾或中立,基於合理讀者會推斷的結果。
  2. SNLI(2015)擁有 570,000 個標記對,MultiNLI(2017)則有 433,000 個對,取自十種英語語域。
  3. 僅看假設的分類器在 SNLI 上正確率約 67%,顯示眾包資料包含可利用的偽影。
  4. 任何訓練好的 NLI 模型都可以作為零樣本分類器,將每個候選標籤轉為假設並評分蕴含。
  5. 基於 NLI 的檢查器會標記不受支持的摘要句,SummaC 透過聚合文件中句子級別的分數達到 74.4% 的平衡準確率。

深入探討

Natural language inference asks a question people answer constantly without noticing: if this sentence is true, what follows? A system receives two pieces of text. The premise is taken as true, for example "A man is playing a guitar on stage." The hypothesis is a claim to evaluate, such as "A man is performing music." The system outputs one of three labels. Entailment means a typical reader would conclude the hypothesis is true given the premise. Contradiction means the hypothesis must be false, as with "The man is asleep in bed." Neutral means the premise does not settle it, as with "The man is a famous musician." The judgment is about what a reasonable person would infer, not strict formal logic.

Modern NLI systems are text classifiers built on pretrained Transformer language models. The premise and hypothesis are joined into one input with a separator token between them, so every word of the hypothesis can attend to every word of the premise. A small classification layer on top of the model's output produces three scores, which a softmax function turns into probabilities for entailment, neutral and contradiction. The model is fine-tuned, meaning further trained, on hundreds of thousands of labeled pairs until its predictions match the human labels.

The task is older than neural networks. Early Recognizing Textual Entailment (RTE) challenges used small test sets, which limited learning-based approaches. That changed in August 2015, when Samuel Bowman and colleagues released the Stanford Natural Language Inference corpus (SNLI) with 570,000 human-labeled sentence pairs, about two orders of magnitude larger than earlier resources of its kind. SNLI premises are image captions, and crowd workers wrote a hypothesis for each label. In 2017 Adina Williams, Nikita Nangia and Bowman released MultiNLI with 433,000 examples from ten genres of written and spoken English, published at NAACL 2018. MultiNLI became a standard training set for general-purpose NLI models.

High scores soon turned out to hide shortcuts. In 2018 Suchin Gururangan and colleagues showed that a classifier that never sees the premise, only the hypothesis, labels about 67% of SNLI and 53% of MultiNLI correctly, against a chance rate of about 33%. Crowd workers left annotation artifacts: negation words, for example, correlate strongly with contradiction. In 2019 Tom McCoy, Ellie Pavlick and Tal Linzen built HANS, a test set where three tempting heuristics fail: assuming entailment when the hypothesis reuses the premise's words, when it is a contiguous piece of the premise, or when it is a complete subtree of the premise's parse. Models trained on MultiNLI, including BERT, did very poorly on it. Adversarial NLI (ANLI), released by Yixin Nie and colleagues in 2019, answered by having people write examples specifically to fool strong models in a human-and-model-in-the-loop process, producing a harder test set.

NLI is now used more as a tool than a leaderboard. Wenpeng Yin, Jamaal Hay and Dan Roth showed in 2019 that a trained NLI model can act as a zero-shot classifier: treat the text as the premise, write each candidate label as a hypothesis, and pick the label with the highest entailment score. The bart-large-mnli model on Hugging Face packages exactly this method. A second use is checking generated text. A summary sentence that is entailed by the source is supported; one that is contradicted or neutral may be a hallucination, meaning a statement the source does not back.

Compared with alternatives, NLI-based checks are cheap and transparent. A fine-tuned NLI model is far smaller than a chatbot-scale language model, runs quickly on modest hardware and returns probabilities that can be thresholded and audited. Question-answering-based metrics, which generate questions from a summary and check whether the source answers them the same way, are slower but catch different errors; the TRUE study found the two families complementary. Asking a large language model to judge consistency is flexible but costlier, and its verdicts can vary with the prompt.

The limitations are real. NLI datasets are built at sentence level, while documents spread facts across many sentences; Laban and colleagues identified this granularity mismatch as the reason early NLI-based summary checkers performed poorly, and their SummaC method, which aggregates sentence-pair scores, reached 74.4% balanced accuracy on its six-dataset benchmark. Labels also depend on background assumptions about what a reasonable reader infers, so annotators sometimes disagree. Shortcut learning can reappear in any new domain, and a model trained on captions and fiction may misjudge legal or medical text. A neutral label means only that the premise does not settle the claim; it does not mean the claim is false.

In practice, building with NLI requires a pretrained model already fine-tuned on MultiNLI or a combination of SNLI, MultiNLI and ANLI, which are freely available, plus a validation set from the target domain to set decision thresholds. Fine-tuning a new model takes a single modern GPU and the public datasets. The main ongoing cost is evaluation: checking examples where the model's labels and human judgment differ, and watching for the overlap and negation shortcuts the research literature has documented.

04工作範例

使用 NLI 檢查摘要句與其來源是否一致

  1. 使用 Hugging Face Transformers 函式庫載入一個在 MultiNLI 上微調過的公開 NLI 模型,例如 facebook/bart-large-mnli。

  2. 將來源句子作為前提:「圖書館將於週五下午 6 點關閉以進行員工培訓。」

  3. 對三個假設進行評分:「圖書館週五提早關閉。」、「圖書館週五晚間開放。」以及「培訓對所有員工皆為強制性。」

  4. 讀取每對組合的三個機率並記錄最高分的標籤。

  5. 預期第一個為蘊含,第二個為矛盾,第三個為中立,因為前提並未說明培訓是否為強制性。

  6. 使用重新表述且與第一個假設幾乎沒有共同字詞的前提來重複,以查看模型是否依賴字詞重疊。

它顯示了什麼

一個包含標籤和小機率的表格,顯示來源支持、反駁或未置可否的聲明。您可以自己為相同的配對貼上標籤來進行檢查;意見分歧(尤其是針對重新表述的前提)顯示模型可能是在使用捷徑而非理解語意。

現實世界的實施

facebook/bart-large-mnli 模型(位於 Hugging Face)會將每個標籤轉化為如「此範例為旅遊。」之類的假設並評估蘊涵關係,藉此將文字分類到它從未受過訓練的標籤中。

SummaC 於 2021 年發表,它將文件及其摘要分割成多個句子,並對每一對句子執行 NLI,以標記出包含來源未支援內容的摘要。

2022 年的 TRUE 研究發現,大規模 NLI 模型在評估來自各種生成任務的 11 個手動標註資料集的事實一致性時,表現非常出色。

研究人員使用 HANS 測試集來檢驗在 MultiNLI 上訓練的模型是真的在進行推理,還是僅僅假設高詞彙重疊就代表蘊含。

SNLI、MultiNLI 和 Adversarial NLI 等基準測試被用來比較語言模型,因為每個項目都有明確的三向標籤,讓計分變得簡單。

自然語言推理與蘊涵的未來

NLI research since 2019 has shifted from beating benchmarks toward measuring robustness and supporting other tasks. Adversarial collection, as in ANLI, showed that test sets written to fool current models stay hard longer than static ones. In evaluation, NLI models remain a common, inexpensive way to score factual consistency, and the 2022 TRUE study recommended NLI-based and question-answering-based metrics as starting points while noting that they catch different errors. Open problems include document-level inference, where the evidence for a claim is spread across many sentences; graded or disputed labels, where reasonable annotators disagree; and domain shift into specialized fields such as law or medicine, where training data is scarce. Large language models can perform NLI when prompted, which raises a practical question that research has not settled: when a small fine-tuned NLI classifier is preferable for its speed, cost and calibrated probabilities, and when a general model's broader knowledge justifies its cost. For hallucination detection, the consistent finding is that NLI provides useful evidence for human review rather than a final verdict on truth.

付諸實踐

為什麼這很重要

  • 速度與規模. 語言工作流程可以在不犧牲一致性的情況下更快地移動。
  • 交通與覆蓋範圍. 它擴展了跨語言和溝通方式的訪問。
  • 更明確的決策. 團隊可以花更多時間進行判斷,而自動化則可以處理重複。

留意

  • 幻覺的事實可以悄悄地進入報告、支持流程或研究成果。
  • 及時的敏感性可能會在類似的請求中產生不一致的結果。
  • 如果存取控制薄弱,敏感文字資料可能會暴露。

後續步驟

  1. 在推出之前定義輸出格式、語氣和品質標準。
  2. 當準確性很重要時,請使用可信任來源進行地面回應。
  3. 為高風險輸出保留人工審查檢查點。
  4. 追蹤故障模式並定期重新訓練提示或工作流程。

資料來源與延伸閱讀

  1. arXiv (EMNLP 2015)用於學習自然語言推論的大型標註語料庫
  2. arXiv (NAACL 2018)透過推論進行句子理解的廣泛覆蓋挑戰語料庫
  3. arXiv (NAACL 2018)自然語言推論資料中的標註人工痕跡
  4. arXiv (ACL 2019)錯得有理:診斷自然語言推論中的句法捷徑
  5. arXiv (ACL 2020)對抗性 NLI:自然語言理解的新基準
  6. arXiv (EMNLP 2019)零樣本文字分類基準測試:資料集、評估與蘊含方法
  7. arXiv (TACL)SummaC:重新審視基於 NLI 的摘要不一致性檢測模型
  8. arXiv (NAACL 2022)TRUE:重新評估事實一致性評估

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

參加自然語言推論與蘊含測驗

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

什麼是自然語言推理與蘊涵?

自然語言推論(NLI),也稱為識別文本蘊含,是一項決定某個句子(前提)是否使另一個句子(假設)成為真實(蘊含)、錯誤(矛盾)或兩者皆非(中立)的任務。它之所以重要,是因為它是語言理解的標準測試,也是零樣本分類以及檢查生成的文字是否有其來源支援的實用工具。

自然語言推理的三個標準標籤是什麼?

NLI 將前提-假設對分類為蘊涵、矛盾或中性。

在 NLI 中,「蘊含」是什麼意思?

蘊涵意味著前提保證了假設的真實性。

給定前提“一個人在舞台上彈吉他”,哪個假設是矛盾的?

如果一個人在舞台上彈吉他,舞台不可能是空的,因此這個假設與前提相矛盾。

NLI 資料集中的標註人工痕跡是什麼?

NLI 資料集可能包含某些模式,例如特定詞彙與某個標籤相關聯,這允許捷徑預測,但在領域外可能會失效。

哪些資料集被廣泛用於訓練和評估 NLI?

SNLI 和 MultiNLI 是專門為自然語言推理而建構的大型人工標記語料庫。