Ikibazo gihuza ibibazo · Hagati Urwego

Intebe ya SWE hamwe na Coding Agent Benchmarks Ikibazo

Gerageza uburyo imirimo ya SWE-yubatswe yubatswe, FAIL_TO_PASS na PASS_TO_PASS amanota, sisitemu ya Lite na Verified, nimpamvu scafolds ihindura ibisubizo.

Ikibazo 1 Bya 8

Niki SWE-intebe isaba sisitemu gukora?

Reba buri kibazo8 ibibazo

Ibisubizo nibisobanuro bigaragara mugihe ufata ikibazo.

  1. Niki SWE-intebe isaba sisitemu gukora?
    • Write a new library from a specification
    • Produce a patch that resolves a real GitHub issue in a Python repository
    • Answer multiple-choice questions about code
    • Translate code from Python to Java
  2. Ni ibihe bizamini bya FAIL_TO_PASS?
    • Tests that always fail
    • Tests the agent writes itself
    • Tests that are skipped during scoring
    • Tests that failed before the human fix and passed after it
  3. Ni uruhe ruhare rw'ibizamini bya PASS_TO_PASS?
    • Checking that the patch did not break behavior that already worked
    • Checking the code style of the patch
    • Measuring how fast the patch runs
    • Verifying the issue text is accurate
  4. Intebe ya SWE Yagenzuwe Niki?
    • The full set of 2,294 tasks
    • A set of JavaScript tasks with screenshots
    • 500 tasks screened by human engineers to remove vague issues and unfair tests
    • A private test set never released publicly
  5. Ni kangahe ziri muri SWE-intebe Lite?
    • 164
    • 300
    • 500
    • 2,294
  6. Why can the same model get very different SWE-bench scores?
    • Scores depend on the scaffold that lets the model browse, run commands and edit files
    • The tests are randomly generated each run
    • The repositories are different for each model
    • Only the model's size affects the score
  7. Which is a real limitation of the original SWE-bench?
    • It has no tests
    • It uses only synthetic issues
    • It cannot be run in Docker
    • It is Python-only and drawn from about a dozen repositories
  8. Does the agent see the evaluation test patch while working?
    • Yes, it is included in the issue text
    • Yes, but only the PASS_TO_PASS tests
    • No, the harness applies the test patch separately after the agent's patch
    • No, because SWE-bench has no tests