गाइड से जुड़ी प्रश्नोत्तरी · मध्यम स्तर
SWE-बेंच और कोडिंग एजेंट बेंचमार्क क्विज़
परीक्षण करें कि SWE-बेंच कार्य कैसे बनाए जाते हैं, FAIL_TO_PASS और PASS_TO_PASS स्कोरिंग, लाइट और सत्यापित उपसमुच्चय, और क्यों मचान परिणाम बदलते हैं।
प्रश्न 1 का 8
What does a SWE-bench task ask the system to do?
प्रत्येक प्रश्न का पूर्वावलोकन करें8 प्रश्न
जैसे ही आप प्रश्नोत्तरी लेते हैं उत्तर और स्पष्टीकरण दिखाई देते हैं।
- What does a SWE-bench task ask the system to do?
- Write a new library from a specification
- Produce a patch that resolves a real GitHub issue in a Python repository
- Answer multiple-choice questions about code
- Translate code from Python to Java
- What are FAIL_TO_PASS tests?
- Tests that always fail
- Tests the agent writes itself
- Tests that are skipped during scoring
- Tests that failed before the human fix and passed after it
- What is the role of PASS_TO_PASS tests?
- Checking that the patch did not break behavior that already worked
- Checking the code style of the patch
- Measuring how fast the patch runs
- Verifying the issue text is accurate
- What is SWE-bench Verified?
- The full set of 2,294 tasks
- A set of JavaScript tasks with screenshots
- 500 tasks screened by human engineers to remove vague issues and unfair tests
- A private test set never released publicly
- How many instances are in SWE-bench Lite?
- 164
- 300
- 500
- 2,294
- Why can the same model get very different SWE-bench scores?
- Scores depend on the scaffold that lets the model browse, run commands and edit files
- The tests are randomly generated each run
- The repositories are different for each model
- Only the model's size affects the score
- Which is a real limitation of the original SWE-bench?
- It has no tests
- It uses only synthetic issues
- It cannot be run in Docker
- It is Python-only and drawn from about a dozen repositories
- Does the agent see the evaluation test patch while working?
- Yes, it is included in the issue text
- Yes, but only the PASS_TO_PASS tests
- No, the harness applies the test patch separately after the agent's patch
- No, because SWE-bench has no tests