SWE-bench +

❖ SWE-bench 2026Jun 8

SWE-bench is a benchmark for evaluating large language models on real world software issues collected from GitHub. Given a codebase and an issue, a language model is tasked with generating a patch that resolves the described problem. SWE-bench Verified is a human-validated subset that more reliably evaluates AI models’ ability to solve issues. International Olympiad in Informatics (IOI) competition features standardized and automated grading.

🏆 Leaderboard

🔍
ModelSWE-benchIOIOrganizationLicenseDate
Claude Opus 59791.7AnthropicProprietary2026-07-24
DeepSeek-V4-Pro96.4DeepSeekMIT2026-08-13
GPT-5.6 Sol96.286.7OpenAIProprietary2026-07-09
Grok 4.695.6xAIProprietary2026-08-12
GLM-5.395.4Z.aiMIT2026-08-14
Claude Fable 59572.3AnthropicProprietary2026-06-09
Kimi K393.4MoonshotKimi K32026-07-16
DeepSeek-V4-Flash88.8DeepSeekMIT2026-07-31
Claude Opus 4.888.6AnthropicProprietary2026-05-28
Grok 4.586.6xAIProprietary2026-07-08
Muse Spark 1.286.649.5MetaProprietary2026-08-05
Qwen3.8-Max85.673AlibabaQwen32026-08-02
GLM-5.282.8Z.aiMIT2026-06-13
GPT-5.582.6OpenAIProprietary2026-04-23
Claude Opus 4.78247.1AnthropicProprietary2026-04-16
Muse Spark 1.182MetaProprietary2026-07-09
Claude Sonnet 579.8AnthropicProprietary2026-06-30
Gemini 3.6 Flash79.8GoogleProprietary2026-07-21
Gemini 3.1 Pro79.8GoogleProprietary2026-02-19
Qwen3.7-Max79.446.8AlibabaProprietary2026-05-19
GPT-5.479.267.8OpenAIProprietary2026-03-05
Claude Opus 4.679.236.3AnthropicProprietary2026-02-05
Kimi K2.778.2MoonshotModified MIT2026-06-12
Gemini 3 Flash76.239.1GoogleProprietary2025-12-17
Grok 4.2076.230.2xAIProprietary2026-02-17
Claude Sonnet 4.676.221.5AnthropicProprietary2026-02-17
GPT-5.3-Codex75.254.8OpenAIProprietary2026-02-05
Gemini 3 Pro74.438.8GoogleProprietary2025-11-18
GLM-5.174.432Z.aiMIT2026-04-06
Claude Opus 4.574.423.6AnthropicProprietary2025-11-24
Kimi K2.674.2MoonshotModified MIT2026-04-13
GPT-5.272.843.8OpenAIProprietary2025-12-11
GLM-572.822Z.aiMIT2026-02-11
MiniMax M2.572.86.7MiniMaxApache 2.02026-02-12
Kimi K2.570.817.7MoonshotModified MIT2026-01-26
Grok 470.626.2xAIProprietary2025-07-09
Claude Sonnet 4.570.618.5AnthropicProprietary2025-09-29
GPT-5.170.521.5OpenAIProprietary2025-11-13
GPT-570.120OpenAIProprietary2025-08-07
Claude Opus 4.170.115.2AnthropicProprietary2025-08-05
DeepSeek-V3.27014.4DeepSeekMIT2025-12-01
Qwen3-Max69.615.7AlibabaProprietary2025-09-23
Claude Haiku 4.568.86.2AnthropicProprietary2025-10-15
Claude Sonnet 4686.5AnthropicProprietary2025-05-14
Claude Opus 467.6AnthropicProprietary2025-05-14
Qwen3-Coder-480B-A35B-Instruct67AlibabaApache 2.02025-07-22
DeepSeek-V3.166DeepSeekMIT2025-08-21
Kimi K265.41.3MoonshotModified MIT2025-07-11
GPT-5 (medium)65OpenAIProprietary2025-08-07
GLM-4.564.2Z.aiMIT2025-07-28
GPT-5 mini59.8OpenAIProprietary2025-08-07
o358.4OpenAIProprietary2025-04-16
GLM-4.5-Air57.6Z.aiMIT2025-07-28
Gemini 2.5 Pro53.617.1GoogleProprietary2025-05-06
Claude 3.7 Sonnet52.8AnthropicProprietary2025-02-19
Qwen3-Coder-30B-A3B-Instruct51.6AlibabaApache 2.02025-07-30
GPT-4.148.6OpenAIProprietary2025-04-14
o4-mini455.3OpenAIProprietary2025-04-16
DeepSeek-R1-052841.4DeepSeekMIT2025-05-28
DeepSeek-V3-032438.81.7DeepSeekMIT2025-03-24
GPT-5 nano34.8OpenAIProprietary2025-08-07
Gemini 2.5 Flash28.73.9GoogleProprietary2025-04-17
GPT-4.1 mini23.9OpenAIProprietary2025-04-14
GPT-4o21.6OpenAIProprietary2024-11-20
Llama 4 Maverick Instruct21MetaLlama 42025-04-05
Gemini 2.0 Flash13.5GoogleProprietary2025-02-05
Llama 4 Scout Instruct9.1MetaLlama 42025-04-05
Qwen2.5-Coder-32B-Instruct9AlibabaApache 2.02024-11-12

SWE-bench Verified (100 turns)

Chatbot Arena + | IOI Benchmark

👋 Overview

SWE-bench tests AI systems’ ability to solve GitHub issues.

We collect 2,294 task instances by crawling Pull Requests and Issues from 12 popular Python repositories. Each instance is based on a pull request that (1) is associated with an issue, and (2) modified 1+ testing related files.

Per instance, we construct an execution environment (Docker Image) with the repository successfully installed at the commit that the Pull Request is based on. Without the Pull Request’s changes, a number of test(s) fail. After the Pull Request is merged, the same set of test(s) pass. These “Fail-to-Pass” tests are the primary signal for evaluation.

SWE-bench evaluation works as follows. Per task instance, an AI system is given the issue text. The AI system should then modify the codebase in order to resolve the described issues. When the AI system is finished, we run the aforementioned Fail-to-Pass tests to check if the issue was successfully resolved.

Code and data for the following works:

🚀 Set Up

SWE-bench uses Docker for reproducible evaluations. Follow the instructions in the Docker setup guide to install Docker on your machine. If you’re setting up on Linux, we recommend seeing the post-installation steps as well.

Finally, to build SWE-bench from source, follow these steps:

git clone git@github.com:princeton-nlp/SWE-bench.git
cd SWE-bench
pip install -e .

Test your installation by running:

python -m swebench.harness.run_evaluation \
    --predictions_path gold \
    --max_workers 1 \
    --instance_ids sympy__sympy-20590 \
    --run_id validate-gold

ℹ️ Note

If using a MacOS M-series or other ARM-based systems, add --namespace '' to the above script. By default, the evaluation script pulls images (built for Linux) from DockerHub. Adding --namespace '' will cause evaluation images to be built locally instead.

💽 Usage

Evaluate patch predictions on SWE-bench Lite with the following command:

python -m swebench.harness.run_evaluation \
    --dataset_name princeton-nlp/SWE-bench_Lite \
    --predictions_path <path_to_predictions> \
    --max_workers <num_workers> \
    --run_id <run_id>
    # use --predictions_path 'gold' to verify the gold patches
    # use --run_id to name the evaluation run
    # use --modal true to run on Modal

This command will generate docker build logs (logs/build_images) and evaluation logs (logs/run_evaluation) in the current directory.

The final evaluation results will be stored in the evaluation_results directory.

⚠️ Warning

SWE-bench evaluation can be resource intensive We recommend running on an x86_64 machine with at least 120GB of free storage, 16GB of RAM, and 8 CPU cores. We recommend using fewer than min(0.75 * os.cpu_count(), 24) for --max_workers.

If running with Docker desktop, make sure to increase your virtual disk space to ~120 free GB. Set max_workers to be consistent with the above for the CPUs available to Docker.

Support for arm64 machines is experimental.

To see the full list of arguments for the evaluation harness, run:

python -m swebench.harness.run_evaluation --help

✍️ Citation

If you find our work helpful, please use the following citations.

@inproceedings{
    jimenez2024swebench,
    title={{SWE}-bench: Can Language Models Resolve Real-world Github Issues?},
    author={Carlos E Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R Narasimhan},
    booktitle={The Twelfth International Conference on Learning Representations},
    year={2024},
    url={https://openreview.net/forum?id=VTF8yNQM66}
}

@inproceedings{
    yang2024swebenchmultimodal,
    title={{SWE}-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?},
    author={John Yang and Carlos E. Jimenez and Alex L. Zhang and Kilian Lieret and Joyce Yang and Xindi Wu and Ori Press and Niklas Muennighoff and Gabriel Synnaeve and Karthik R. Narasimhan and Diyi Yang and Sida I. Wang and Ofir Press},
    booktitle={The Thirteenth International Conference on Learning Representations},
    year={2025},
    url={https://openreview.net/forum?id=riTiq3i21b}
}
Explore Projects
Careers Developers Progress
Privacy Terms © 2026 OpenLM.ai