Test porównawczy SWE i agentów kodujących

Odpowiedzi i wyjaśnienia pojawiają się w miarę rozwiązywania quizu.

1. O co prosi system zadanie SWE-bench?
   • Write a new library from a specification
   • Produce a patch that resolves a real GitHub issue in a Python repository
   • Answer multiple-choice questions about code
   • Translate code from Python to Java
2. Co to są testy FAIL_TO_PASS?
   • Tests that always fail
   • Tests the agent writes itself
   • Tests that are skipped during scoring
   • Tests that failed before the human fix and passed after it
3. Jaka jest rola testów PASS_TO_PASS?
   • Checking that the patch did not break behavior that already worked
   • Checking the code style of the patch
   • Measuring how fast the patch runs
   • Verifying the issue text is accurate
4. Czym jest SWE-bench Verified?
   • The full set of 2,294 tasks
   • A set of JavaScript tasks with screenshots
   • 500 tasks screened by human engineers to remove vague issues and unfair tests
   • A private test set never released publicly
5. Ile instancji znajduje się w SWE-bench Lite?
   • 164
   • 300
   • 500
   • 2,294
6. Why can the same model get very different SWE-bench scores?
   • Scores depend on the scaffold that lets the model browse, run commands and edit files
   • The tests are randomly generated each run
   • The repositories are different for each model
   • Only the model's size affects the score
7. Which is a real limitation of the original SWE-bench?
   • It has no tests
   • It uses only synthetic issues
   • It cannot be run in Docker
   • It is Python-only and drawn from about a dozen repositories
8. Does the agent see the evaluation test patch while working?
   • Yes, it is included in the issue text
   • Yes, but only the PASS_TO_PASS tests
   • No, the harness applies the test patch separately after the agent's patch
   • No, because SWE-bench has no tests