Benchmark

HumanEval

HumanEval Code Generation

Python function synthesis from docstrings — measures pass@1 accuracy.

Models Tested

Claude Opus 4

Top Model

94.5%

Top Score

100 %

Max Possible

#	Model	HumanEval Score	Provider	Composite
1	Claude Opus 4 Anthropic	94.5%	Anthropic	97.8
2	Claude Sonnet 4 Anthropic	93.7%	Anthropic	90.3
3	Gemini 2.5 Pro Google	93%	Google	97.5
4	o3 OpenAI	92.7%	OpenAI	97.1
5	DeepSeek R1 DeepSeek	92.1%	DeepSeek	92.3
6	Claude 3.5 Sonnet Anthropic	92%	Anthropic	88.4
7	o4-mini OpenAI	91.5%	OpenAI	91.5
8	Llama 4 Maverick Meta	90.8%	Meta	87.5
9	Grok 3 xAI	90.5%	xAI	88.8
10	GPT-4o OpenAI	90.2%	OpenAI	89.2
11	Gemini 2.5 Flash Google	89.8%	Google	86.7
12	DeepSeek V3 DeepSeek	89.4%	DeepSeek	86.1
13	Llama 3.1 405B Meta	89%	Meta	82.1
14	Qwen 3 235B Alibaba	88.7%	Alibaba	84.5
15	Claude 3.5 Haiku Anthropic	88.1%	Anthropic	79.2
16	Gemini 2.0 Flash Google	87.6%	Google	80.5
17	GPT-4o Mini OpenAI	87.2%	OpenAI	75.8
18	Llama 3.3 70B Meta	86.4%	Meta	79.8
19	Mistral Large Mistral AI	84.3%	Mistral AI	76.5
20	Qwen 2.5 72B Alibaba	82.4%	Alibaba	76
21	Grok 2 xAI	82.1%	xAI	74.8
22	Llama 3.1 70B Meta	80.5%	Meta	74.3
23	Mixtral 8x22B Mistral AI	75.8%	Mistral AI	65.2
24	Phi-4 Microsoft	72.4%	Microsoft	64.5
25	Command R+ Cohere	68.2%	Cohere	60.8

Source: Chen et al., 2021