Benchmark

SWE-Bench

Software Engineering Benchmark

Real-world GitHub issue resolution across popular open-source repositories.

Models Tested

Claude Opus 4

Top Model

72.5%

Top Score

100 %

Max Possible

#	Model	SWE-Bench Score	Provider	Composite
1	Claude Opus 4 Anthropic	72.5%	Anthropic	97.8
2	o3 OpenAI	71.7%	OpenAI	97.1
3	o4-mini OpenAI	68.1%	OpenAI	91.5
4	Gemini 2.5 Pro Google	63.8%	Google	97.5
5	Claude Sonnet 4 Anthropic	62.3%	Anthropic	90.3
6	Grok 3 xAI	55.2%	xAI	88.8
7	DeepSeek R1 DeepSeek	49.2%	DeepSeek	92.3
8	Claude 3.5 Sonnet Anthropic	49%	Anthropic	88.4
9	Gemini 2.5 Flash Google	48.2%	Google	86.7
10	Llama 4 Maverick Meta	47.5%	Meta	87.5
11	DeepSeek V3 DeepSeek	42%	DeepSeek	86.1
12	Qwen 3 235B Alibaba	41.3%	Alibaba	84.5
13	Claude 3.5 Haiku Anthropic	40.6%	Anthropic	79.2
14	GPT-4o OpenAI	38.4%	OpenAI	89.2
15	Gemini 2.0 Flash Google	36.8%	Google	80.5
16	Llama 3.1 405B Meta	35.2%	Meta	82.1
17	Llama 3.3 70B Meta	32.8%	Meta	79.8
18	Mistral Large Mistral AI	30.1%	Mistral AI	76.5
19	Grok 2 xAI	28.3%	xAI	74.8
20	Llama 3.1 70B Meta	27.3%	Meta	74.3
21	Qwen 2.5 72B Alibaba	26.5%	Alibaba	76
22	GPT-4o Mini OpenAI	23.4%	OpenAI	75.8
23	Mixtral 8x22B Mistral AI	18.2%	Mistral AI	65.2
24	Command R+ Cohere	15.8%	Cohere	60.8
25	Phi-4 Microsoft	12.5%	Microsoft	64.5

Source: Jimenez et al., 2024