Guide-linked quiz · Medium Level
Dataset Deduplication and Near-Duplicate Detection Quiz
Compare exact and similarity-based dataset deduplication, prevent train-test overlap, and audit threshold tradeoffs and diversity risks.
Question 1 of 8
What does a cryptographic hash detect most directly?
Preview every question8 questions
Answers and explanations appear as you take the quiz.
- What does a cryptographic hash detect most directly?
- Exact matching bytes, or exact normalized content when normalization is defined
- Every paraphrase automatically
- All examples with the same meaning
- The label quality of a dataset
- What kind of similarity can MinHash over text shingles help identify?
- Any two sentences with opposite meanings
- Image color differences only
- Lexical near-duplicates with overlapping sets of text features
- Whether a dataset is unbiased
- Why cluster near-duplicates before making train-test partitions?
- To guarantee every label is balanced
- To keep related examples from leaking across partitions
- To remove the need for a test set
- To make all examples byte-identical
- What did the cited 2021 language-model deduplication study report?
- Deduplication always improves every metric on every corpus
- Every benchmark was fully deduplicated
- Exact hashes identify all paraphrases
- Deduplication reduced memorized text emission in its studied setting
- Why can a global similarity threshold be harmful?
- It may remove meaningful variants or disproportionately filter rare language examples
- It always makes models more fair
- It identifies only exact matches
- It guarantees no train-test overlap
- What should be reviewed before choosing a similarity threshold?
- Only processing speed
- The file extension
- Whether the score is an integer
- Precision and recall on manually reviewed example pairs
- What should an audit record after deduplication?
- Only the final number of rows
- A model’s final accuracy only
- The algorithm, normalization, threshold, source, and retention decisions
- Nothing if rows were deleted
- Does high embedding similarity prove two examples are interchangeable?
- Yes, for every label and task
- No; similarity is a review signal and task-specific meaning may differ
- Yes, if both vectors are normalized
- No, because embeddings cannot compare text
Keep testing yourself
More quizzes picked for your level.