引導式測驗 · 中等 等級

資料集重複資料刪除和近似重複檢測測驗

比較精確和基於相似性的資料集重複資料刪除,防止訓練測試重疊,並審核閾值權衡和多樣性風險。

相關引導路徑資料集重複資料刪除
問題 1 的 8

What does a cryptographic hash detect most directly?

預覽每個問題8 問題

當您參加測驗時,就會出現答案和解釋。

  1. What does a cryptographic hash detect most directly?
    • Exact matching bytes, or exact normalized content when normalization is defined
    • Every paraphrase automatically
    • All examples with the same meaning
    • The label quality of a dataset
  2. What kind of similarity can MinHash over text shingles help identify?
    • Any two sentences with opposite meanings
    • Image color differences only
    • Lexical near-duplicates with overlapping sets of text features
    • Whether a dataset is unbiased
  3. Why cluster near-duplicates before making train-test partitions?
    • To guarantee every label is balanced
    • To keep related examples from leaking across partitions
    • To remove the need for a test set
    • To make all examples byte-identical
  4. What did the cited 2021 language-model deduplication study report?
    • Deduplication always improves every metric on every corpus
    • Every benchmark was fully deduplicated
    • Exact hashes identify all paraphrases
    • Deduplication reduced memorized text emission in its studied setting
  5. Why can a global similarity threshold be harmful?
    • It may remove meaningful variants or disproportionately filter rare language examples
    • It always makes models more fair
    • It identifies only exact matches
    • It guarantees no train-test overlap
  6. What should be reviewed before choosing a similarity threshold?
    • Only processing speed
    • 檔案副檔名
    • Whether the score is an integer
    • Precision and recall on manually reviewed example pairs
  7. What should an audit record after deduplication?
    • Only the final number of rows
    • A model’s final accuracy only
    • The algorithm, normalization, threshold, source, and retention decisions
    • Nothing if rows were deleted
  8. Does high embedding similarity prove two examples are interchangeable?
    • Yes, for every label and task
    • No; similarity is a review signal and task-specific meaning may differ
    • Yes, if both vectors are normalized
    • No, because embeddings cannot compare text