🇮🇷 MIZAN: A Persian LLM Leaderboard

MIZAN: A Persian LLM Leaderboard is a comprehensive benchmark for evaluating Large Language Models (LLMs) in Persian. It combines existing datasets, translated benchmarks, and new Persian-specific data to assess LLM capabilities in understanding, generation, reasoning, and knowledge relevant to the Persian language and culture. MIZAN provides a standardized tool for researchers and developers to measure Persian LLM performance.

Search by Model Name

Filter by Model Source

All Open-Source Closed-Source

🏆 Overall Benchmark


1 🥇	Mistral Small 3.1 24B Instruct	✔️	Unknown	73.32	91.04	91.08	82.17	92.18	71.79	19.47


1 🥇	OpenAI o3	✔️	Unknown	74.68	92.60	91.38	82.17	92.18	72.07	17.65
2 🥈	Gemini 2.5 Pro	✔️	Unknown	73.32	89.12	91.08	84.01	94.28	69.93	11.52
3 🥉	GPT-5 Mini	✔️	Unknown	71.69	91.04	86.78	76.21	86.51	71.44	18.16
4	Claude 3.7 Sonnet	❌	Unknown	71.27	88.11	86.95	72.15	91.18	71.43	17.79
5	GPT-4.1	❌	Unknown	69.92	86.34	87.33	70.40	88.39	67.58	19.47
6	Gemini 2.5 Flash	✔️	Unknown	69.86	87.97	88.12	72.75	87.17	69.44	13.69
7	Gemini 2.0 Flash	❌	Unknown	68.86	84.98	83.90	69.58	86.38	70.51	17.82
8	GPT-4o	❌	Unknown	68.77	82.97	83.72	68.85	86.65	71.47	18.96
9	Gemini 2.0 Flash-Lite	❌	Unknown	66.40	84.14	82.44	64.67	81.43	69.14	16.59
10	GPT-5 Nano	✔️	Unknown	66.37	89.72	82.83	68.64	73.10	67.50	16.43
11	GPT-OSS 120B	✔️	120.0B	66.33	88.66	84.33	67.89	77.14	64.58	15.39
12	GPT-5 Mini (Minimal reasoning)	✔️	Unknown	65.60	85.61	83.53	59.48	76.86	71.79	16.32
13	GPT-4.1 Mini	❌	Unknown	65.56	83.41	84.18	61.29	77.12	68.33	19.01
14	DeepSeek-R1	✔️	671.0B	65.49	83.70	86.17	68.11	82.52	63.61	8.81
15	DeepSeek-V3	❌	671.0B	64.58	83.11	86.01	59.08	82.41	67.53	9.34
16	Gemma 3 27B IT	❌	27.4B	62.47	82.97	79.60	56.33	76.28	68.98	10.67
17	GPT-4o Mini	❌	Unknown	62.46	80.18	78.92	56.99	75.98	64.59	18.11
18	GPT-OSS 120B (Low reasoning)	✔️	120.0B	62.31	81.95	79.49	62.18	71.23	64.71	14.31
19	Qwen3-32B	❌	32.8B	62.24	80.30	76.33	56.35	76.54	67.14	16.79
20	GPT-OSS 20B	✔️	20.0B	61.96	85.86	80.46	59.58	69.65	62.85	13.35
21	Llama 3.3 70B Instruct	❌	70.6B	61.30	71.26	71.73	57.14	79.56	68.00	20.11
22	Command A	❌	111.0B	60.98	84.39	82.19	59.81	79.89	48.25	11.38
23	Gemma 3 12B IT	❌	12.2B	60.08	81.50	75.13	51.05	70.94	69.91	11.97
24	Qwen3-30B-A3B	❌	30.5B	59.39	83.26	74.31	50.98	68.80	62.56	16.41
25	Qwen3-14B	❌	14.8B	59.12	81.06	72.05	51.39	69.58	64.60	16.06
26	GPT-OSS 20B (Low reasoning)	✔️	20.0B	57.63	75.07	74.14	54.37	64.53	64.43	13.24
27	Llama 4 Scout	❌	109.0B	57.37	78.97	74.17	56.40	78.14	40.87	15.68
28	Command R Plus	❌	104.0B	57.05	70.07	68.80	48.01	73.64	62.98	18.80
29	Mistral Small 3.1 24B Instruct	❌	24.0B	55.76	75.27	72.91	47.63	68.94	56.62	13.19
30	Claude 3.5 Haiku	❌	Unknown	55.46	82.53	75.85	54.40	71.60	37.49	10.89
31	GPT-4.1 Nano	❌	Unknown	55.24	75.77	73.63	47.84	54.94	62.62	16.66
32	Gemma 3n E4B IT	❌	7.8B	54.10	75.04	68.64	46.82	59.15	65.52	9.40
33	Qwen3-8B	❌	8.2B	52.52	74.74	66.08	44.68	54.37	59.68	15.57
34	GPT-5 Nano (Minimal reasoning)	✔️	Unknown	52.16	74.45	74.43	45.57	51.94	52.41	14.18
35	Gemma 3 4B IT	❌	4.3B	49.96	74.45	66.83	38.26	48.32	62.42	9.50
36	Aya Expanse 32B	❌	32.3B	49.45	69.90	70.86	41.41	63.27	39.29	11.96
37	DeepSeek-R1 Qwen3-8B	✔️	8.2B	49.14	68.01	67.42	49.79	52.73	45.68	11.20
38	Command R v01	❌	35.0B	48.13	57.90	60.91	40.47	60.00	53.10	16.42
39	Qwen3-4B	❌	4.0B	47.91	75.77	55.99	40.25	50.33	51.21	13.89
40	Gemma 3 1B IT	❌	1.0B	32.52	54.48	43.33	28.30	25.99	36.20	6.83
41	Mistral 7B Instruct v0.3	❌	7.2B	30.39	44.05	33.98	25.53	30.15	39.17	9.44
42	Llama 3.2 3B Instruct	❌	3.2B	28.15	53.30	37.56	27.84	25.21	13.69	11.30
43	Llama 3.2 1B Instruct	❌	1.2B	20.50	36.56	29.52	19.87	24.12	4.68	8.23
44	Gemma 3 270M IT	❌	268.0M	16.94	30.80	17.71	19.48	24.84	3.73	5.10

Persian MT-Bench


1 🥇	Mistral Small 3.1 24B Instruct	✔️	Unknown	9.14	9.11	8.81	8.81	8.54	8.55	9.70	9.68	9.65	9.60	9.15	9.15	8.95	8.95	9.35	9.35	9.10	9.10	8.44	8.50	9.70	9.69

Persian IFEval


1 🥇	Mistral Small 3.1 24B Instruct	✔️	Unknown	89.67	91.04	90.77	93.79	85.94	95.65	89.80	96.50	96.77	88.52	95.00	96.83	87.50	95.65	89.80	97.90	96.77	90.16	90.16	96.83

PerMMLU


1 🥇	Mistral Small 3.1 24B Instruct	✔️	Unknown	82.17	93.75	80.00	80.00	55.00	75.00	88.57	90.00	95.00	92.82	97.83	90.00	97.78	94.38	78.00	65.08	87.34	85.65	82.41

PerCoR


1 🥇	Mistral Small 3.1 24B Instruct	✔️	Unknown	92.18	94.28	94.50	94.88	93.72	96.01	93.90	94.94	92.42	89.26	91.39	98.04	96.00	96.06	95.67	95.16	95.67	95.36	93.71	96.94	92.98	95.56	92.16	91.64	97.46	95.93	98.54	91.69	92.48	95.13	87.76	96.75	96.81	96.53	93.06	93.65	86.75	81.82	92.31	96.97	95.24	83.33	93.22	90.00	90.00

Persian NLU


1 🥇	Mistral Small 3.1 24B Instruct	✔️	Unknown	80.64	77.00	83.03	78.17	80.64	77.00	83.03	78.17	99.95	71.79


1 🥇	OpenAI o3	✔️	Unknown	80.64	77.00	83.03	78.17	80.64	77.00	83.03	78.17	100	72.07
2 🥈	GPT-5 Mini (Minimal reasoning)	✔️	Unknown	78.32	74.33	80.48	75.91	78.36	74.37	80.52	75.95	99.95	71.79
3 🥉	GPT-4o	❌	Unknown	79.88	75.80	81.08	77.58	79.88	75.80	81.08	77.58	100	71.47
4	GPT-5 Mini	✔️	Unknown	80.64	77.06	82.35	78.02	80.64	77.06	82.35	78.02	100	71.44
5	Claude 3.7 Sonnet	❌	Unknown	80.58	77.18	82.12	78.89	80.63	77.22	82.16	78.93	99.95	71.43
6	Gemini 2.0 Flash	❌	Unknown	77.40	72.96	79.69	74.93	77.44	73.00	79.73	74.97	99.95	70.51
7	Gemini 2.5 Pro	✔️	Unknown	79.40	75.48	80.64	76.79	79.40	75.48	80.64	76.79	100	69.93
8	Gemma 3 12B IT	❌	12.2B	76.65	72.36	78.57	74.40	76.69	72.40	78.61	74.44	99.95	69.91
9	Gemini 2.5 Flash	✔️	Unknown	76.11	74.13	78.80	72.63	76.11	74.13	78.80	72.63	100	69.44
10	Gemini 2.0 Flash-Lite	❌	Unknown	76.11	72.33	79.39	72.66	76.23	72.45	79.52	72.78	99.84	69.14
11	Gemma 3 27B IT	❌	27.4B	75.57	71.01	79.63	71.98	75.57	71.01	79.63	71.98	100	68.98
12	GPT-4.1 Mini	❌	Unknown	78.69	74.86	81.11	76.69	78.69	74.86	81.11	76.69	100	68.33
13	Llama 3.3 70B Instruct	❌	70.6B	73.25	70.24	78.25	69.06	73.25	70.24	78.25	69.06	100	68.00
14	GPT-4.1	❌	Unknown	80.47	76.62	80.90	78.30	80.47	76.62	80.90	78.30	100	67.58
15	DeepSeek-V3	❌	671.0B	76.70	74.12	79.91	73.46	76.70	74.12	79.91	73.46	100	67.53
16	GPT-5 Nano	✔️	Unknown	78.86	75.71	81.34	76.18	79.11	75.95	81.60	76.43	99.68	67.50
17	Qwen3-32B	❌	32.8B	78.21	75.93	77.68	75.63	78.21	75.93	77.68	75.63	100	67.14
18	Gemma 3n E4B IT	❌	7.8B	75.19	73.34	78.50	72.15	75.23	73.38	78.54	72.19	99.95	65.52
19	GPT-OSS 120B (Low reasoning)	✔️	120.0B	76.59	72.64	79.60	73.84	76.59	72.64	79.60	73.84	100	64.71
20	Qwen3-14B	❌	14.8B	74.60	69.60	77.26	71.60	74.60	69.60	77.26	71.60	100	64.60
21	GPT-4o Mini	❌	Unknown	77.13	72.89	80.37	75.32	77.13	72.89	80.37	75.32	100	64.59
22	GPT-OSS 120B	✔️	120.0B	78.43	74.75	80.40	76.03	78.43	74.75	80.40	76.03	100	64.58
23	GPT-OSS 20B (Low reasoning)	✔️	20.0B	76.97	73.12	79.18	73.83	76.97	73.12	79.18	73.83	100	64.43
24	DeepSeek-R1	✔️	671.0B	62.78	59.55	62.40	60.73	72.89	69.13	72.44	70.50	86.14	63.61
25	Command R Plus	❌	104.0B	73.84	70.68	76.59	72.64	73.84	70.68	76.59	72.64	100	62.98
26	GPT-OSS 20B	✔️	20.0B	74.22	70.56	75.66	71.08	76.74	72.96	78.23	73.50	96.71	62.85
27	GPT-4.1 Nano	❌	Unknown	74.49	71.98	77.73	70.36	74.49	71.98	77.73	70.36	100	62.62
28	Qwen3-30B-A3B	❌	30.5B	75.03	71.62	76.49	71.65	75.03	71.62	76.49	71.65	100	62.56
29	Gemma 3 4B IT	❌	4.3B	68.18	65.47	73.79	63.56	68.18	65.47	73.79	63.56	100	62.42
30	Qwen3-8B	❌	8.2B	74.11	70.10	75.29	71.73	74.11	70.10	75.29	71.73	100	59.68
31	Mistral Small 3.1 24B Instruct	❌	24.0B	59.49	66.33	67.40	59.71	59.59	66.44	67.51	59.81	99.84	56.62
32	Command R v01	❌	35.0B	61.54	65.20	67.63	52.90	61.54	65.20	67.63	52.90	100	53.10
33	GPT-5 Nano (Minimal reasoning)	✔️	Unknown	56.04	65.47	63.96	44.02	56.25	65.72	64.20	44.19	99.62	52.41
34	Qwen3-4B	❌	4.0B	68.55	64.09	71.81	64.47	68.55	64.09	71.81	64.47	100	51.21
35	Command A	❌	111.0B	80.47	76.92	81.26	78.42	80.47	76.92	81.26	78.42	100	48.25
36	DeepSeek-R1 Qwen3-8B	✔️	8.2B	70.39	65.58	71.95	66.34	70.39	65.58	71.95	66.34	100	45.68
37	Llama 4 Scout	❌	109.0B	71.52	67.03	73.84	68.73	76.51	71.71	79.00	73.53	93.47	40.87
38	Aya Expanse 32B	❌	32.3B	76.43	72.87	78.01	74.34	79.52	75.82	81.16	77.34	96.12	39.29
39	Mistral 7B Instruct v0.3	❌	7.2B	37.38	43.03	30.53	29.34	62.10	71.49	50.72	48.75	60.19	39.17
40	Claude 3.5 Haiku	❌	Unknown	75.22	71.49	76.61	73.40	75.50	71.76	76.90	73.68	99.62	37.49
41	Gemma 3 1B IT	❌	1.0B	52.00	49.08	50.48	44.58	57.18	53.97	55.51	49.02	90.94	36.20
42	Llama 3.2 3B Instruct	❌	3.2B	31.50	60.11	47.00	31.36	31.55	60.21	47.08	31.41	99.84	13.69
43	Llama 3.2 1B Instruct	❌	1.2B	7.87	13.71	12.81	7.23	21.07	36.68	34.28	19.34	37.38	4.68
44	Gemma 3 270M IT	❌	268.0M	0.22	0.10	0.22	0.13	33.33	14.81	33.33	20.51	0.65	3.73


1 🥇	Mistral Small 3.1 24B Instruct	✔️	Unknown	95.09	95.09	99.80	71.79


1 🥇	OpenAI o3	✔️	Unknown	95.09	95.09	100	72.07
2 🥈	GPT-5 Mini (Minimal reasoning)	✔️	Unknown	95.48	95.48	100	71.79
3 🥉	GPT-4o	❌	Unknown	95.87	95.87	100	71.47
4	GPT-5 Mini	✔️	Unknown	94.05	94.05	100	71.44
5	Claude 3.7 Sonnet	❌	Unknown	96.39	96.39	100	71.43
6	Gemini 2.0 Flash	❌	Unknown	96.66	96.66	100	70.51
7	Gemini 2.5 Pro	✔️	Unknown	94.46	94.46	100	69.93
8	Gemma 3 12B IT	❌	12.2B	95.02	95.02	100	69.91
9	Gemini 2.5 Flash	✔️	Unknown	94.01	94.01	100	69.44
10	Gemini 2.0 Flash-Lite	❌	Unknown	95.51	95.51	100	69.14
11	Gemma 3 27B IT	❌	27.4B	95.83	95.83	100	68.98
12	GPT-4.1 Mini	❌	Unknown	96.22	96.22	100	68.33
13	Llama 3.3 70B Instruct	❌	70.6B	94.10	94.10	100	68.00
14	GPT-4.1	❌	Unknown	96.20	96.20	100	67.58
15	DeepSeek-V3	❌	671.0B	95.51	95.51	100	67.53
16	GPT-5 Nano	✔️	Unknown	94.18	94.18	100	67.50
17	Qwen3-32B	❌	32.8B	96.18	96.18	100	67.14
18	Gemma 3n E4B IT	❌	7.8B	95.87	95.87	100	65.52
19	GPT-OSS 120B (Low reasoning)	✔️	120.0B	94.75	94.75	100	64.71
20	Qwen3-14B	❌	14.8B	96.04	96.04	100	64.60
21	GPT-4o Mini	❌	Unknown	95.90	95.90	100	64.59
22	GPT-OSS 120B	✔️	120.0B	94.92	94.92	100	64.58
23	GPT-OSS 20B (Low reasoning)	✔️	20.0B	94.39	94.39	100	64.43
24	DeepSeek-R1	✔️	671.0B	95.81	95.81	100	63.61
25	Command R Plus	❌	104.0B	94.46	94.46	100	62.98
26	GPT-OSS 20B	✔️	20.0B	94.96	94.96	100	62.85
27	GPT-4.1 Nano	❌	Unknown	93.00	93.00	100	62.62
28	Qwen3-30B-A3B	❌	30.5B	94.21	94.21	100	62.56
29	Gemma 3 4B IT	❌	4.3B	91.99	91.99	100	62.42
30	Qwen3-8B	❌	8.2B	95.55	95.55	100	59.68
31	Mistral Small 3.1 24B Instruct	❌	24.0B	94.25	94.25	100	56.62
32	Command R v01	❌	35.0B	90.09	90.09	100	53.10
33	GPT-5 Nano (Minimal reasoning)	✔️	Unknown	83.02	83.18	99.80	52.41
34	Qwen3-4B	❌	4.0B	91.75	93.24	98.40	51.21
35	Command A	❌	111.0B	13.38	80.57	16.60	48.25
36	DeepSeek-R1 Qwen3-8B	✔️	8.2B	89.43	89.43	100	45.68
37	Llama 4 Scout	❌	109.0B	9.75	97.52	10.00	40.87
38	Aya Expanse 32B	❌	32.3B	0	0	0	39.29
39	Mistral 7B Instruct v0.3	❌	7.2B	44.33	73.63	60.20	39.17
40	Claude 3.5 Haiku	❌	Unknown	32.18	87.92	36.60	37.49
41	Gemma 3 1B IT	❌	1.0B	72.33	80.37	90.00	36.20
42	Llama 3.2 3B Instruct	❌	3.2B	0	0	0	13.69
43	Llama 3.2 1B Instruct	❌	1.2B	0	0	2.00	4.68
44	Gemma 3 270M IT	❌	268.0M	0	0	99.40	3.73


1 🥇	Mistral Small 3.1 24B Instruct	✔️	Unknown	48.98	46.27	55.00	71.79


1 🥇	OpenAI o3	✔️	Unknown	48.98	46.27	55.00	72.07
2 🥈	GPT-5 Mini (Minimal reasoning)	✔️	Unknown	51.80	47.89	60.94	71.79
3 🥉	GPT-4o	❌	Unknown	54.93	52.96	59.59	71.47
4	GPT-5 Mini	✔️	Unknown	50.87	47.50	58.91	71.44
5	Claude 3.7 Sonnet	❌	Unknown	57.83	55.84	62.50	71.43
6	Gemini 2.0 Flash	❌	Unknown	58.38	56.21	63.49	70.51
7	Gemini 2.5 Pro	✔️	Unknown	46.66	43.01	54.61	69.93
8	Gemma 3 12B IT	❌	12.2B	47.64	42.06	59.97	69.91
9	Gemini 2.5 Flash	✔️	Unknown	60.48	59.46	64.38	69.44
10	Gemini 2.0 Flash-Lite	❌	Unknown	57.78	57.22	60.65	69.14
11	Gemma 3 27B IT	❌	27.4B	50.91	47.20	58.99	68.98
12	GPT-4.1 Mini	❌	Unknown	36.36	33.77	42.40	68.33
13	Llama 3.3 70B Instruct	❌	70.6B	46.24	43.38	52.99	68.00
14	GPT-4.1	❌	Unknown	30.98	28.33	37.11	67.58
15	DeepSeek-V3	❌	671.0B	47.38	43.83	55.17	67.53
16	GPT-5 Nano	✔️	Unknown	52.38	51.20	56.39	67.50
17	Qwen3-32B	❌	32.8B	44.08	42.06	48.71	67.14
18	Gemma 3n E4B IT	❌	7.8B	50.31	46.17	59.53	65.52
19	GPT-OSS 120B (Low reasoning)	✔️	120.0B	29.55	27.36	34.62	64.71
20	Qwen3-14B	❌	14.8B	49.21	45.54	57.84	64.60
21	GPT-4o Mini	❌	Unknown	3.74	3.43	4.49	64.59
22	GPT-OSS 120B	✔️	120.0B	9.06	8.13	11.04	64.58
23	GPT-OSS 20B (Low reasoning)	✔️	20.0B	41.31	38.82	47.43	64.43
24	DeepSeek-R1	✔️	671.0B	24.71	21.76	31.69	63.61
25	Command R Plus	❌	104.0B	0.32	0.24	0.48	62.98
26	GPT-OSS 20B	✔️	20.0B	12.39	11.71	13.88	62.85
27	GPT-4.1 Nano	❌	Unknown	45.21	40.48	56.40	62.62
28	Qwen3-30B-A3B	❌	30.5B	50.00	46.08	59.27	62.56
29	Gemma 3 4B IT	❌	4.3B	38.39	32.92	50.50	62.42
30	Qwen3-8B	❌	8.2B	15.88	15.53	17.65	59.68
31	Mistral Small 3.1 24B Instruct	❌	24.0B	2.82	2.78	3.04	56.62
32	Command R v01	❌	35.0B	0	0	0	53.10
33	GPT-5 Nano (Minimal reasoning)	✔️	Unknown	39.18	36.57	47.07	52.41
34	Qwen3-4B	❌	4.0B	34.27	32.83	39.51	51.21
35	Command A	❌	111.0B	49.28	45.31	58.07	48.25
36	DeepSeek-R1 Qwen3-8B	✔️	8.2B	14.04	12.44	18.26	45.68
37	Llama 4 Scout	❌	109.0B	13.49	12.35	15.87	40.87
38	Aya Expanse 32B	❌	32.3B	36.99	32.51	47.85	39.29
39	Mistral 7B Instruct v0.3	❌	7.2B	16.26	15.82	18.85	39.17
40	Claude 3.5 Haiku	❌	Unknown	1.34	1.32	1.48	37.49
41	Gemma 3 1B IT	❌	1.0B	0	0	0	36.20
42	Llama 3.2 3B Instruct	❌	3.2B	6.39	4.94	10.84	13.69
43	Llama 3.2 1B Instruct	❌	1.2B	0	0	0	4.68
44	Gemma 3 270M IT	❌	268.0M	0	0	0	3.73


1 🥇	Mistral Small 3.1 24B Instruct	✔️	Unknown	15.55	13.17	20.77	71.79


1 🥇	OpenAI o3	✔️	Unknown	15.55	13.17	20.77	72.07
2 🥈	GPT-5 Mini (Minimal reasoning)	✔️	Unknown	18.77	15.58	25.78	71.79
3 🥉	GPT-4o	❌	Unknown	21.61	19.02	26.84	71.47
4	GPT-5 Mini	✔️	Unknown	15.47	12.75	21.11	71.44
5	Claude 3.7 Sonnet	❌	Unknown	21.15	19.12	24.84	71.43
6	Gemini 2.0 Flash	❌	Unknown	20.58	18.61	24.21	70.51
7	Gemini 2.5 Pro	✔️	Unknown	15.60	14.49	17.66	69.93
8	Gemma 3 12B IT	❌	12.2B	33.52	29.14	41.67	69.91
9	Gemini 2.5 Flash	✔️	Unknown	19.87	18.13	22.95	69.44
10	Gemini 2.0 Flash-Lite	❌	Unknown	14.71	13.87	16.67	69.14
11	Gemma 3 27B IT	❌	27.4B	23.84	20.42	30.15	68.98
12	GPT-4.1 Mini	❌	Unknown	18.56	15.78	24.12	68.33
13	Llama 3.3 70B Instruct	❌	70.6B	28.77	27.33	32.22	68.00
14	GPT-4.1	❌	Unknown	12.18	10.21	16.09	67.58
15	DeepSeek-V3	❌	671.0B	26.19	21.73	34.93	67.53
16	GPT-5 Nano	✔️	Unknown	11.87	10.13	15.82	67.50
17	Qwen3-32B	❌	32.8B	13.69	11.17	18.63	67.14
18	Gemma 3n E4B IT	❌	7.8B	26.46	23.49	31.67	65.52
19	GPT-OSS 120B (Low reasoning)	✔️	120.0B	17.91	15.14	23.82	64.71
20	Qwen3-14B	❌	14.8B	23.38	18.93	32.97	64.60
21	GPT-4o Mini	❌	Unknown	27.83	22.51	38.42	64.59
22	GPT-OSS 120B	✔️	120.0B	15.43	13.01	20.39	64.58
23	GPT-OSS 20B (Low reasoning)	✔️	20.0B	23.76	19.24	32.91	64.43
24	DeepSeek-R1	✔️	671.0B	32.68	29.86	38.25	63.61
25	Command R Plus	❌	104.0B	8.61	7.58	10.65	62.98
26	GPT-OSS 20B	✔️	20.0B	12.97	10.52	18.16	62.85
27	GPT-4.1 Nano	❌	Unknown	16.91	14.96	20.50	62.62
28	Qwen3-30B-A3B	❌	30.5B	26.00	21.51	34.97	62.56
29	Gemma 3 4B IT	❌	4.3B	22.99	21.15	26.35	62.42
30	Qwen3-8B	❌	8.2B	20.81	16.73	29.44	59.68
31	Mistral Small 3.1 24B Instruct	❌	24.0B	19.43	16.82	24.52	56.62
32	Command R v01	❌	35.0B	0.12	0.14	0.11	53.10
33	GPT-5 Nano (Minimal reasoning)	✔️	Unknown	8.89	7.17	12.71	52.41
34	Qwen3-4B	❌	4.0B	21.76	17.68	30.30	51.21
35	Command A	❌	111.0B	19.01	16.76	23.07	48.25
36	DeepSeek-R1 Qwen3-8B	✔️	8.2B	14.10	12.17	18.32	45.68
37	Llama 4 Scout	❌	109.0B	1.70	1.54	2.07	40.87
38	Aya Expanse 32B	❌	32.3B	21.99	19.25	26.96	39.29
39	Mistral 7B Instruct v0.3	❌	7.2B	25.11	18.99	41.00	39.17
40	Claude 3.5 Haiku	❌	Unknown	25.68	24.84	27.66	37.49
41	Gemma 3 1B IT	❌	1.0B	20.97	18.03	26.21	36.20
42	Llama 3.2 3B Instruct	❌	3.2B	2.24	2.30	2.28	13.69
43	Llama 3.2 1B Instruct	❌	1.2B	0	0	0	4.68
44	Gemma 3 270M IT	❌	268.0M	0	0	0	3.73


1 🥇	Mistral Small 3.1 24B Instruct	✔️	Unknown	88.82	77.44	62.59	65.87	88.82	77.44	62.59	65.87	92.55	71.79


1 🥇	OpenAI o3	✔️	Unknown	88.82	77.44	62.59	65.87	88.82	77.44	62.59	65.87	100	72.07
2 🥈	GPT-5 Mini (Minimal reasoning)	✔️	Unknown	88.82	79.21	75.47	77.18	88.82	79.21	75.47	77.18	100	71.79
3 🥉	GPT-4o	❌	Unknown	90.68	82.16	73.12	76.80	90.68	82.16	73.12	76.80	100	71.47
4	GPT-5 Mini	✔️	Unknown	89.44	82.88	67.30	72.39	89.44	82.88	67.30	72.39	100	71.44
5	Claude 3.7 Sonnet	❌	Unknown	81.99	81.80	59.78	63.64	88.59	88.39	64.59	68.77	92.55	71.43
6	Gemini 2.0 Flash	❌	Unknown	77.64	66.68	64.24	64.05	77.64	66.68	64.24	64.05	100	70.51
7	Gemini 2.5 Pro	✔️	Unknown	87.58	72.30	63.74	66.93	87.58	72.30	63.74	66.93	100	69.93
8	Gemma 3 12B IT	❌	12.2B	82.61	70.05	66.37	67.76	82.61	70.05	66.37	67.76	100	69.91
9	Gemini 2.5 Flash	✔️	Unknown	85.09	72.33	60.80	64.53	86.16	73.24	61.56	65.34	98.76	69.44
10	Gemini 2.0 Flash-Lite	❌	Unknown	82.61	72.11	60.34	63.94	82.61	72.11	60.34	63.94	100	69.14
11	Gemma 3 27B IT	❌	27.4B	78.88	65.29	71.87	67.46	78.88	65.29	71.87	67.46	100	68.98
12	GPT-4.1 Mini	❌	Unknown	91.30	77.91	74.78	76.24	91.30	77.91	74.78	76.24	100	68.33
13	Llama 3.3 70B Instruct	❌	70.6B	68.32	64.15	73.86	65.21	68.32	64.15	73.86	65.21	100	68.00
14	GPT-4.1	❌	Unknown	91.30	83.97	70.58	74.87	91.30	83.97	70.58	74.87	100	67.58
15	DeepSeek-V3	❌	671.0B	82.61	73.39	69.71	71.10	82.61	73.39	69.71	71.10	100	67.53
16	GPT-5 Nano	✔️	Unknown	84.47	68.21	56.82	57.94	85.00	68.63	57.17	58.31	99.38	67.50
17	Qwen3-32B	❌	32.8B	80.75	67.99	63.38	64.90	80.75	67.99	63.38	64.90	100	67.14
18	Gemma 3n E4B IT	❌	7.8B	57.76	47.85	49.96	44.87	58.49	48.45	50.59	45.44	98.76	65.52
19	GPT-OSS 120B (Low reasoning)	✔️	120.0B	81.99	74.21	57.82	57.59	81.99	74.21	57.82	57.59	100	64.71
20	Qwen3-14B	❌	14.8B	49.69	53.64	57.73	47.55	49.69	53.64	57.73	47.55	100	64.60
21	GPT-4o Mini	❌	Unknown	85.09	74.78	64.82	67.76	85.09	74.78	64.82	67.76	100	64.59
22	GPT-OSS 120B	✔️	120.0B	84.47	77.78	58.75	59.90	84.47	77.78	58.75	59.90	100	64.58
23	GPT-OSS 20B (Low reasoning)	✔️	20.0B	75.78	47.90	45.95	43.97	75.78	47.90	45.95	43.97	100	64.43
24	DeepSeek-R1	✔️	671.0B	43.48	38.63	38.34	38.38	78.65	69.89	69.36	69.43	55.28	63.61
25	Command R Plus	❌	104.0B	67.70	60.42	55.00	53.93	67.70	60.42	55.00	53.93	100	62.98
26	GPT-OSS 20B	✔️	20.0B	80.12	63.26	51.06	52.12	81.65	64.46	52.03	53.11	98.14	62.85
27	GPT-4.1 Nano	❌	Unknown	83.23	72.91	58.49	60.77	83.23	72.91	58.49	60.77	100	62.62
28	Qwen3-30B-A3B	❌	30.5B	34.78	40.69	34.20	33.24	53.33	62.40	52.44	50.97	65.22	62.56
29	Gemma 3 4B IT	❌	4.3B	50.31	52.28	51.69	46.45	50.94	52.94	52.34	47.03	98.76	62.42
30	Qwen3-8B	❌	8.2B	63.98	54.83	59.03	53.05	63.98	54.83	59.03	53.05	100	59.68
31	Mistral Small 3.1 24B Instruct	❌	24.0B	77.64	67.75	67.68	66.84	79.11	69.04	68.97	68.10	98.14	56.62
32	Command R v01	❌	35.0B	74.53	57.71	50.71	52.21	75.47	58.43	51.34	52.87	98.76	53.10
33	GPT-5 Nano (Minimal reasoning)	✔️	Unknown	66.46	57.39	53.72	50.04	67.30	58.11	54.40	50.67	98.76	52.41
34	Qwen3-4B	❌	4.0B	65.84	54.13	56.30	53.84	66.67	54.82	57.01	54.52	98.76	51.21
35	Command A	❌	111.0B	63.35	51.13	46.86	48.64	89.47	72.21	66.18	68.69	70.81	48.25
36	DeepSeek-R1 Qwen3-8B	✔️	8.2B	15.53	11.31	21.55	14.66	34.25	24.94	47.54	32.33	45.34	45.68
37	Llama 4 Scout	❌	109.0B	3.11	1.55	1.86	1.69	83.33	41.67	50.00	45.45	3.73	40.87
38	Aya Expanse 32B	❌	32.3B	0	0	0	0	0	0	0	0	0	39.29
39	Mistral 7B Instruct v0.3	❌	7.2B	29.19	19.34	23.61	18.74	58.02	38.44	46.93	37.25	50.31	39.17
40	Claude 3.5 Haiku	❌	Unknown	0	0	0	0	0	0	0	0	0	37.49
41	Gemma 3 1B IT	❌	1.0B	2.48	2.59	2.66	1.75	40.00	41.67	42.86	28.15	6.21	36.20
42	Llama 3.2 3B Instruct	❌	3.2B	0	0	0	0	0	0	0	0	0	13.69
43	Llama 3.2 1B Instruct	❌	1.2B	0	0	0	0	0	0	0	0	0.62	4.68
44	Gemma 3 270M IT	❌	268.0M	65.84	16.46	23.91	19.50	68.83	17.21	25.00	20.38	95.65	3.73


1 🥇	Mistral Small 3.1 24B Instruct	✔️	Unknown	86.64	86.64	99.60	71.79


1 🥇	OpenAI o3	✔️	Unknown	86.64	86.64	100	72.07
2 🥈	GPT-5 Mini (Minimal reasoning)	✔️	Unknown	85.69	85.69	100	71.79
3 🥉	GPT-4o	❌	Unknown	86.12	86.12	100	71.47
4	GPT-5 Mini	✔️	Unknown	85.21	85.21	100	71.44
5	Claude 3.7 Sonnet	❌	Unknown	86.06	86.06	100	71.43
6	Gemini 2.0 Flash	❌	Unknown	84.93	84.93	100	70.51
7	Gemini 2.5 Pro	✔️	Unknown	83.58	83.58	100	69.93
8	Gemma 3 12B IT	❌	12.2B	87.68	87.68	100	69.91
9	Gemini 2.5 Flash	✔️	Unknown	83.67	83.67	100	69.44
10	Gemini 2.0 Flash-Lite	❌	Unknown	84.61	84.61	100	69.14
11	Gemma 3 27B IT	❌	27.4B	87.97	87.97	100	68.98
12	GPT-4.1 Mini	❌	Unknown	86.47	86.47	100	68.33
13	Llama 3.3 70B Instruct	❌	70.6B	82.75	82.75	100	68.00
14	GPT-4.1	❌	Unknown	86.24	86.24	100	67.58
15	DeepSeek-V3	❌	671.0B	86.42	86.42	100	67.53
16	GPT-5 Nano	✔️	Unknown	84.04	84.04	100	67.50
17	Qwen3-32B	❌	32.8B	85.51	85.51	100	67.14
18	Gemma 3n E4B IT	❌	7.8B	86.02	86.02	100	65.52
19	GPT-OSS 120B (Low reasoning)	✔️	120.0B	84.10	84.10	100	64.71
20	Qwen3-14B	❌	14.8B	85.09	85.09	100	64.60
21	GPT-4o Mini	❌	Unknown	86.72	86.72	100	64.59
22	GPT-OSS 120B	✔️	120.0B	84.12	84.12	100	64.58
23	GPT-OSS 20B (Low reasoning)	✔️	20.0B	85.21	85.21	100	64.43
24	DeepSeek-R1	✔️	671.0B	85.48	85.48	100	63.61
25	Command R Plus	❌	104.0B	83.32	83.32	100	62.98
26	GPT-OSS 20B	✔️	20.0B	84.38	84.38	100	62.85
27	GPT-4.1 Nano	❌	Unknown	87.21	87.21	100	62.62
28	Qwen3-30B-A3B	❌	30.5B	86.44	86.44	100	62.56
29	Gemma 3 4B IT	❌	4.3B	84.71	84.71	100	62.42
30	Qwen3-8B	❌	8.2B	84.98	84.98	100	59.68
31	Mistral Small 3.1 24B Instruct	❌	24.0B	83.46	83.46	100	56.62
32	Command R v01	❌	35.0B	83.76	83.76	100	53.10
33	GPT-5 Nano (Minimal reasoning)	✔️	Unknown	79.74	80.06	99.60	52.41
34	Qwen3-4B	❌	4.0B	83.63	84.31	99.20	51.21
35	Command A	❌	111.0B	0	0	5.80	48.25
36	DeepSeek-R1 Qwen3-8B	✔️	8.2B	80.21	80.21	100	45.68
37	Llama 4 Scout	❌	109.0B	9.83	96.34	10.20	40.87
38	Aya Expanse 32B	❌	32.3B	0	0	0	39.29
39	Mistral 7B Instruct v0.3	❌	7.2B	55.31	80.39	68.80	39.17
40	Claude 3.5 Haiku	❌	Unknown	25.34	86.18	29.40	37.49
41	Gemma 3 1B IT	❌	1.0B	66.78	69.14	96.60	36.20
42	Llama 3.2 3B Instruct	❌	3.2B	0	0	0	13.69
43	Llama 3.2 1B Instruct	❌	1.2B	0	0	1.80	4.68
44	Gemma 3 270M IT	❌	268.0M	0	0	100	3.73


1 🥇	Mistral Small 3.1 24B Instruct	✔️	Unknown	86.59	90.51	84.33	85.46	86.59	90.51	84.33	85.46	99.49	71.79


1 🥇	OpenAI o3	✔️	Unknown	86.59	90.51	84.33	85.46	86.59	90.51	84.33	85.46	100	72.07
2 🥈	GPT-5 Mini (Minimal reasoning)	✔️	Unknown	84.42	89.08	81.86	82.97	84.85	89.54	82.28	83.39	99.49	71.79
3 🥉	GPT-4o	❌	Unknown	87.99	91.33	85.97	87.09	87.99	91.33	85.97	87.09	100	71.47
4	GPT-5 Mini	✔️	Unknown	83.78	88.76	81.18	82.21	84.10	89.10	81.49	82.53	99.62	71.44
5	Claude 3.7 Sonnet	❌	Unknown	87.74	91.18	85.67	86.80	87.74	91.18	85.67	86.80	100	71.43
6	Gemini 2.0 Flash	❌	Unknown	88.89	91.87	87.01	88.11	88.89	91.87	87.01	88.11	100	70.51
7	Gemini 2.5 Pro	✔️	Unknown	85.57	89.93	83.13	84.26	85.57	89.93	83.13	84.26	100	69.93
8	Gemma 3 12B IT	❌	12.2B	88.12	91.26	86.16	87.26	88.12	91.26	86.16	87.26	100	69.91
9	Gemini 2.5 Flash	✔️	Unknown	82.12	88.10	79.10	80.04	82.12	88.10	79.10	80.04	100	69.44
10	Gemini 2.0 Flash-Lite	❌	Unknown	89.40	92.18	87.61	88.69	89.40	92.18	87.61	88.69	100	69.14
11	Gemma 3 27B IT	❌	27.4B	90.29	92.62	88.69	89.71	90.29	92.62	88.69	89.71	100	68.98
12	GPT-4.1 Mini	❌	Unknown	83.27	88.69	80.45	81.47	83.27	88.69	80.45	81.47	100	68.33
13	Llama 3.3 70B Instruct	❌	70.6B	87.74	91.18	85.67	86.80	87.74	91.18	85.67	86.80	100	68.00
14	GPT-4.1	❌	Unknown	85.95	90.14	83.58	84.71	85.95	90.14	83.58	84.71	100	67.58
15	DeepSeek-V3	❌	671.0B	84.67	89.44	82.09	83.19	84.67	89.44	82.09	83.19	100	67.53
16	GPT-5 Nano	✔️	Unknown	82.12	88.10	79.10	80.04	82.12	88.10	79.10	80.04	100	67.50
17	Qwen3-32B	❌	32.8B	89.53	92.26	87.76	88.83	89.53	92.26	87.76	88.83	100	67.14
18	Gemma 3n E4B IT	❌	7.8B	89.02	91.95	87.16	88.26	89.02	91.95	87.16	88.26	100	65.52
19	GPT-OSS 120B (Low reasoning)	✔️	120.0B	85.44	89.86	82.99	84.10	85.44	89.86	82.99	84.10	100	64.71
20	Qwen3-14B	❌	14.8B	83.91	89.02	81.19	82.25	83.91	89.02	81.19	82.25	100	64.60
21	GPT-4o Mini	❌	Unknown	86.08	90.22	83.73	84.86	86.08	90.22	83.73	84.86	100	64.59
22	GPT-OSS 120B	✔️	120.0B	84.04	89.09	81.34	82.41	84.04	89.09	81.34	82.41	100	64.58
23	GPT-OSS 20B (Low reasoning)	✔️	20.0B	82.76	88.22	79.89	80.87	82.76	88.22	79.89	80.87	100	64.43
24	DeepSeek-R1	✔️	671.0B	91.32	93.30	89.89	90.84	91.32	93.30	89.89	90.84	100	63.61
25	Command R Plus	❌	104.0B	89.02	91.95	87.16	88.26	89.02	91.95	87.16	88.26	100	62.98
26	GPT-OSS 20B	✔️	20.0B	81.35	87.11	78.44	79.38	82.41	88.24	79.46	80.41	98.72	62.85
27	GPT-4.1 Nano	❌	Unknown	80.46	87.27	77.16	77.91	80.46	87.27	77.16	77.91	100	62.62
28	Qwen3-30B-A3B	❌	30.5B	77.78	83.43	75.75	76.19	81.53	87.45	79.40	79.86	95.40	62.56
29	Gemma 3 4B IT	❌	4.3B	86.97	90.57	84.81	85.93	86.97	90.57	84.81	85.93	100	62.42
30	Qwen3-8B	❌	8.2B	79.82	86.96	76.42	77.08	79.82	86.96	76.42	77.08	100	59.68
31	Mistral Small 3.1 24B Instruct	❌	24.0B	68.07	82.09	62.69	59.33	68.07	82.09	62.69	59.33	100	56.62
32	Command R v01	❌	35.0B	91.70	92.33	90.95	91.45	92.53	93.16	91.77	92.27	99.11	53.10
33	GPT-5 Nano (Minimal reasoning)	✔️	Unknown	82.38	86.08	80.27	81.21	85.09	88.92	82.92	83.89	96.81	52.41
34	Qwen3-4B	❌	4.0B	58.24	68.60	56.44	54.13	69.41	81.76	67.26	64.51	83.91	51.21
35	Command A	❌	111.0B	88.76	91.79	86.87	87.97	88.76	91.79	86.87	87.97	100	48.25
36	DeepSeek-R1 Qwen3-8B	✔️	8.2B	36.27	34.35	36.96	35.35	92.21	87.32	93.95	89.87	39.34	45.68
37	Llama 4 Scout	❌	109.0B	89.66	92.34	87.91	88.98	89.66	92.34	87.91	88.98	100	40.87
38	Aya Expanse 32B	❌	32.3B	90.80	93.08	89.25	90.26	90.80	93.08	89.25	90.26	100	39.29
39	Mistral 7B Instruct v0.3	❌	7.2B	87.23	90.57	85.15	86.25	87.23	90.57	85.15	86.25	100	39.17
40	Claude 3.5 Haiku	❌	Unknown	85.57	89.61	83.14	84.34	86.34	90.42	83.89	85.10	99.11	37.49
41	Gemma 3 1B IT	❌	1.0B	50.96	60.43	55.90	47.35	50.96	60.43	55.90	47.35	100	36.20
42	Llama 3.2 3B Instruct	❌	3.2B	52.11	26.05	45.59	33.16	57.14	28.57	50.00	36.36	91.19	13.69
43	Llama 3.2 1B Instruct	❌	1.2B	9.20	4.60	7.92	5.82	58.06	29.03	50.00	36.73	15.84	4.68
44	Gemma 3 270M IT	❌	268.0M	0	0	0	0	0	0	0	0	0	3.73


1 🥇	Mistral Small 3.1 24B Instruct	✔️	Unknown	86.89	88.09	86.78	86.83	86.89	88.09	86.78	86.83	99.42	71.79


1 🥇	OpenAI o3	✔️	Unknown	86.89	88.09	86.78	86.83	86.89	88.09	86.78	86.83	100	72.07
2 🥈	GPT-5 Mini (Minimal reasoning)	✔️	Unknown	81.33	82.44	81.29	81.12	81.80	82.92	81.76	81.59	99.42	71.79
3 🥉	GPT-4o	❌	Unknown	82.54	83.84	82.65	82.39	82.54	83.84	82.65	82.39	100	71.47
4	GPT-5 Mini	✔️	Unknown	85.42	86.34	85.39	85.37	85.53	86.45	85.50	85.48	99.87	71.44
5	Claude 3.7 Sonnet	❌	Unknown	86.83	87.96	86.94	86.81	86.83	87.96	86.94	86.81	100	71.43
6	Gemini 2.0 Flash	❌	Unknown	69.44	75.67	69.80	67.94	69.44	75.67	69.80	67.94	100	70.51
7	Gemini 2.5 Pro	✔️	Unknown	76.78	81.25	77.11	76.00	76.78	81.25	77.11	76.00	100	69.93
8	Gemma 3 12B IT	❌	12.2B	65.35	72.79	65.59	64.02	65.35	72.79	65.59	64.02	100	69.91
9	Gemini 2.5 Flash	✔️	Unknown	73.34	76.91	73.68	72.30	73.34	76.91	73.68	72.30	100	69.44
10	Gemini 2.0 Flash-Lite	❌	Unknown	66.18	77.30	66.72	61.91	66.18	77.30	66.72	61.91	100	69.14
11	Gemma 3 27B IT	❌	27.4B	69.18	76.12	69.66	66.38	69.18	76.12	69.66	66.38	100	68.98
12	GPT-4.1 Mini	❌	Unknown	74.10	76.33	74.23	73.76	74.10	76.33	74.23	73.76	100	68.33
13	Llama 3.3 70B Instruct	❌	70.6B	72.63	80.62	73.14	69.81	72.63	80.62	73.14	69.81	100	68.00
14	GPT-4.1	❌	Unknown	80.69	82.74	80.78	80.56	80.69	82.74	80.78	80.56	100	67.58
15	DeepSeek-V3	❌	671.0B	74.06	79.38	74.41	72.66	74.06	79.38	74.41	72.66	100	67.53
16	GPT-5 Nano	✔️	Unknown	76.53	78.47	76.59	76.30	76.53	78.47	76.59	76.30	100	67.50
17	Qwen3-32B	❌	32.8B	71.61	77.95	71.85	70.94	71.61	77.95	71.85	70.94	100	67.14
18	Gemma 3n E4B IT	❌	7.8B	70.78	74.19	71.00	69.87	70.92	74.33	71.13	70.01	99.81	65.52
19	GPT-OSS 120B (Low reasoning)	✔️	120.0B	75.70	77.21	75.84	75.34	75.70	77.21	75.84	75.34	100	64.71
20	Qwen3-14B	❌	14.8B	70.08	75.97	70.40	68.35	70.08	75.97	70.40	68.35	100	64.60
21	GPT-4o Mini	❌	Unknown	76.47	78.14	76.70	75.73	76.57	78.25	76.80	75.83	99.87	64.59
22	GPT-OSS 120B	✔️	120.0B	76.85	78.59	77.00	76.54	76.85	78.59	77.00	76.54	100	64.58
23	GPT-OSS 20B (Low reasoning)	✔️	20.0B	69.44	71.91	69.63	68.82	69.44	71.91	69.63	68.82	100	64.43
24	DeepSeek-R1	✔️	671.0B	68.54	74.52	68.84	66.90	68.59	74.57	68.89	66.94	99.94	63.61
25	Command R Plus	❌	104.0B	68.99	75.22	69.17	68.35	68.99	75.22	69.17	68.35	100	62.98
26	GPT-OSS 20B	✔️	20.0B	73.85	76.62	73.96	73.55	73.99	76.77	74.10	73.69	99.81	62.85
27	GPT-4.1 Nano	❌	Unknown	70.01	70.90	70.16	69.64	70.01	70.90	70.16	69.64	100	62.62
28	Qwen3-30B-A3B	❌	30.5B	60.87	69.40	60.93	59.08	61.10	69.67	61.16	59.31	99.62	62.56
29	Gemma 3 4B IT	❌	4.3B	63.62	67.43	63.75	62.11	63.70	67.52	63.83	62.19	99.87	62.42
30	Qwen3-8B	❌	8.2B	65.92	72.92	65.56	61.73	66.22	73.25	65.85	62.01	99.55	59.68
31	Mistral Small 3.1 24B Instruct	❌	24.0B	74.62	82.79	74.32	74.84	74.62	82.79	74.32	74.84	100	56.62
32	Command R v01	❌	35.0B	53.71	74.30	54.28	45.22	53.71	74.30	54.28	45.22	100	53.10
33	GPT-5 Nano (Minimal reasoning)	✔️	Unknown	31.78	36.74	31.58	30.03	47.38	54.78	47.08	44.78	67.07	52.41
34	Qwen3-4B	❌	4.0B	17.26	18.56	15.64	15.50	72.78	78.26	65.93	65.34	23.72	51.21
35	Command A	❌	111.0B	50.51	55.27	43.84	45.70	80.86	88.47	70.18	73.15	62.47	48.25
36	DeepSeek-R1 Qwen3-8B	✔️	8.2B	61.83	65.00	61.81	61.25	62.55	65.76	62.53	61.97	98.85	45.68
37	Llama 4 Scout	❌	109.0B	68.22	76.16	68.09	64.76	70.34	78.52	70.20	66.77	96.99	40.87
38	Aya Expanse 32B	❌	32.3B	37.85	44.33	34.23	34.75	69.32	81.19	62.69	63.64	54.60	39.29
39	Mistral 7B Instruct v0.3	❌	7.2B	54.22	56.48	54.45	52.73	54.50	56.77	54.73	53.00	99.49	39.17
40	Claude 3.5 Haiku	❌	Unknown	2.49	1.61	1.74	1.67	90.70	58.52	63.33	60.83	2.75	37.49
41	Gemma 3 1B IT	❌	1.0B	34.34	56.18	34.40	27.90	34.36	56.22	34.42	27.92	99.94	36.20
42	Llama 3.2 3B Instruct	❌	3.2B	0	0	0	0	0	0	0	0	0	13.69
43	Llama 3.2 1B Instruct	❌	1.2B	0	0	0	0	0	0	0	0	0	4.68
44	Gemma 3 270M IT	❌	268.0M	0	0	0	0	0	0	0	0	0	3.73


1 🥇	Mistral Small 3.1 24B Instruct	✔️	Unknown	85.00	88.11	82.90	83.86	85.00	88.11	82.90	83.86	98.60	71.79


1 🥇	OpenAI o3	✔️	Unknown	85.00	88.11	82.90	83.86	85.00	88.11	82.90	83.86	100	72.07
2 🥈	GPT-5 Mini (Minimal reasoning)	✔️	Unknown	85.60	86.88	84.28	84.97	86.82	88.11	85.47	86.18	98.60	71.79
3 🥉	GPT-4o	❌	Unknown	84.80	87.18	82.90	83.77	84.80	87.18	82.90	83.77	100	71.47
4	GPT-5 Mini	✔️	Unknown	81.20	85.60	78.53	79.36	81.53	85.95	78.85	79.68	99.60	71.44
5	Claude 3.7 Sonnet	❌	Unknown	84.40	86.72	82.49	83.34	84.40	86.72	82.49	83.34	100	71.43
6	Gemini 2.0 Flash	❌	Unknown	86.80	87.87	85.51	86.18	86.80	87.87	85.51	86.18	100	70.51
7	Gemini 2.5 Pro	✔️	Unknown	81.40	84.89	78.94	79.74	81.40	84.89	78.94	79.74	100	69.93
8	Gemma 3 12B IT	❌	12.2B	87.80	88.23	86.90	87.36	87.80	88.23	86.90	87.36	100	69.91
9	Gemini 2.5 Flash	✔️	Unknown	80.20	85.45	77.26	77.97	80.20	85.45	77.26	77.97	100	69.44
10	Gemini 2.0 Flash-Lite	❌	Unknown	88.80	88.86	88.23	88.49	88.80	88.86	88.23	88.49	100	69.14
11	Gemma 3 27B IT	❌	27.4B	87.60	88.07	86.67	87.15	87.60	88.07	86.67	87.15	100	68.98
12	GPT-4.1 Mini	❌	Unknown	80.40	85.57	77.49	78.23	80.40	85.57	77.49	78.23	100	68.33
13	Llama 3.3 70B Instruct	❌	70.6B	82.80	85.05	80.80	81.59	82.80	85.05	80.80	81.59	100	68.00
14	GPT-4.1	❌	Unknown	82.60	86.13	80.22	81.10	82.60	86.13	80.22	81.10	100	67.58
15	DeepSeek-V3	❌	671.0B	76.40	84.06	72.73	72.80	76.40	84.06	72.73	72.80	100	67.53
16	GPT-5 Nano	✔️	Unknown	78.00	84.53	74.65	75.05	78.00	84.53	74.65	75.05	100	67.50
17	Qwen3-32B	❌	32.8B	87.60	87.35	87.35	87.35	87.60	87.35	87.35	87.35	100	67.14
18	Gemma 3n E4B IT	❌	7.8B	86.60	87.71	85.28	85.96	86.60	87.71	85.28	85.96	100	65.52
19	GPT-OSS 120B (Low reasoning)	✔️	120.0B	83.40	87.10	81.04	81.97	83.40	87.10	81.04	81.97	100	64.71
20	Qwen3-14B	❌	14.8B	85.40	87.42	83.65	84.49	85.40	87.42	83.65	84.49	100	64.60
21	GPT-4o Mini	❌	Unknown	83.80	85.77	81.96	82.76	83.80	85.77	81.96	82.76	100	64.59
22	GPT-OSS 120B	✔️	120.0B	79.20	83.71	76.33	76.96	79.20	83.71	76.33	76.96	100	64.58
23	GPT-OSS 20B (Low reasoning)	✔️	20.0B	80.00	83.06	77.54	78.25	80.00	83.06	77.54	78.25	100	64.43
24	DeepSeek-R1	✔️	671.0B	82.60	82.67	81.65	82.00	82.60	82.67	81.65	82.00	100	63.61
25	Command R Plus	❌	104.0B	87.20	87.26	86.54	86.83	87.20	87.26	86.54	86.83	100	62.98
26	GPT-OSS 20B	✔️	20.0B	79.60	82.75	77.08	77.92	81.39	84.61	78.81	79.68	97.80	62.85
27	GPT-4.1 Nano	❌	Unknown	79.20	84.27	76.21	76.83	79.20	84.27	76.21	76.83	100	62.62
28	Qwen3-30B-A3B	❌	30.5B	82.40	86.00	79.99	80.86	82.40	86.00	79.99	80.86	100	62.56
29	Gemma 3 4B IT	❌	4.3B	83.80	84.17	82.71	83.17	83.80	84.17	82.71	83.17	100	62.42
30	Qwen3-8B	❌	8.2B	79.80	83.84	77.08	77.77	79.80	83.84	77.08	77.77	100	59.68
31	Mistral Small 3.1 24B Instruct	❌	24.0B	66.00	79.33	60.58	56.25	66.00	79.33	60.58	56.25	100	56.62
32	Command R v01	❌	35.0B	77.80	80.54	79.72	77.77	78.43	81.19	80.36	78.40	99.20	53.10
33	GPT-5 Nano (Minimal reasoning)	✔️	Unknown	76.40	76.56	75.28	75.64	79.25	79.42	78.10	78.47	96.40	52.41
34	Qwen3-4B	❌	4.0B	67.20	78.68	62.65	59.73	68.02	79.64	63.42	60.46	98.80	51.21
35	Command A	❌	111.0B	88.00	89.45	86.62	87.39	88.00	89.45	86.62	87.39	100	48.25
36	DeepSeek-R1 Qwen3-8B	✔️	8.2B	27.40	26.28	23.13	24.14	85.62	82.11	72.29	75.45	32.00	45.68
37	Llama 4 Scout	❌	109.0B	86.00	86.18	85.16	85.54	86.17	86.35	85.33	85.71	99.80	40.87
38	Aya Expanse 32B	❌	32.3B	86.40	86.16	86.07	86.11	86.40	86.16	86.07	86.11	100	39.29
39	Mistral 7B Instruct v0.3	❌	7.2B	63.40	72.38	67.21	62.20	63.40	72.38	67.21	62.20	100	39.17
40	Claude 3.5 Haiku	❌	Unknown	74.40	75.17	73.57	73.95	84.16	85.04	83.23	83.65	88.40	37.49
41	Gemma 3 1B IT	❌	1.0B	57.60	63.53	60.98	56.54	57.60	63.53	60.98	56.54	100	36.20
42	Llama 3.2 3B Instruct	❌	3.2B	57.00	28.50	50.00	36.31	57.00	28.50	50.00	36.31	100	13.69
43	Llama 3.2 1B Instruct	❌	1.2B	7.20	3.60	6.60	4.66	54.55	27.27	50.00	35.29	13.20	4.68
44	Gemma 3 270M IT	❌	268.0M	0	0	0	0	0	0	0	0	0	3.73


1 🥇	Mistral Small 3.1 24B Instruct	✔️	Unknown	45.59	79.18	71.79


1 🥇	OpenAI o3	✔️	Unknown	45.59	79.18	72.07
2 🥈	GPT-5 Mini (Minimal reasoning)	✔️	Unknown	41.23	76.32	71.79
3 🥉	GPT-4o	❌	Unknown	17.54	56.41	71.47
4	GPT-5 Mini	✔️	Unknown	42.94	76.74	71.44
5	Claude 3.7 Sonnet	❌	Unknown	21.90	58.99	71.43
6	Gemini 2.0 Flash	❌	Unknown	37.44	68.61	70.51
7	Gemini 2.5 Pro	✔️	Unknown	49.57	78.04	69.93
8	Gemma 3 12B IT	❌	12.2B	37.44	71.21	69.91
9	Gemini 2.5 Flash	✔️	Unknown	45.12	77.95	69.44
10	Gemini 2.0 Flash-Lite	❌	Unknown	33.27	68.85	69.14
11	Gemma 3 27B IT	❌	27.4B	24.93	59.53	68.98
12	GPT-4.1 Mini	❌	Unknown	29.10	65.00	68.33
13	Llama 3.3 70B Instruct	❌	70.6B	38.29	70.91	68.00
14	GPT-4.1	❌	Unknown	16.21	55.41	67.58
15	DeepSeek-V3	❌	671.0B	7.01	49.87	67.53
16	GPT-5 Nano	✔️	Unknown	41.61	76.25	67.50
17	Qwen3-32B	❌	32.8B	12.04	51.53	67.14
18	Gemma 3n E4B IT	❌	7.8B			65.52
19	GPT-OSS 120B (Low reasoning)	✔️	120.0B	20.85	57.43	64.71
20	Qwen3-14B	❌	14.8B	11.94	50.54	64.60
21	GPT-4o Mini	❌	Unknown	7.20	47.22	64.59
22	GPT-OSS 120B	✔️	120.0B	38.58	73.78	64.58
23	GPT-OSS 20B (Low reasoning)	✔️	20.0B	23.32	60.69	64.43
24	DeepSeek-R1	✔️	671.0B	36.78	70.60	63.61
25	Command R Plus	❌	104.0B	51.66	79.97	62.98
26	GPT-OSS 20B	✔️	20.0B	42.09	74.74	62.85
27	GPT-4.1 Nano	❌	Unknown	21.52	60.52	62.62
28	Qwen3-30B-A3B	❌	30.5B	1.42	61.09	62.56
29	Gemma 3 4B IT	❌	4.3B	29.86	64.84	62.42
30	Qwen3-8B	❌	8.2B	1.90	49.54	59.68
31	Mistral Small 3.1 24B Instruct	❌	24.0B	13.08	51.12	56.62
32	Command R v01	❌	35.0B	41.99	74.01	53.10
33	GPT-5 Nano (Minimal reasoning)	✔️	Unknown	21.99	58.50	52.41
34	Qwen3-4B	❌	4.0B	20.47	56.61	51.21
35	Command A	❌	111.0B	5.88	44.59	48.25
36	DeepSeek-R1 Qwen3-8B	✔️	8.2B			45.68
37	Llama 4 Scout	❌	109.0B	16.59	52.91	40.87
38	Aya Expanse 32B	❌	32.3B	85.31	35.71	39.29
39	Mistral 7B Instruct v0.3	❌	7.2B	47.39	34.40	39.17
40	Claude 3.5 Haiku	❌	Unknown	1.52	32.22	37.49
41	Gemma 3 1B IT	❌	1.0B	15.64	47.98	36.20
42	Llama 3.2 3B Instruct	❌	3.2B	2.37	40.03	13.69
43	Llama 3.2 1B Instruct	❌	1.2B	66.35	33.78	4.68
44	Gemma 3 270M IT	❌	268.0M	75.83	21.24	3.73


1 🥇	Mistral Small 3.1 24B Instruct	✔️	Unknown	81.20	68.30	69.32	67.16	81.20	68.30	69.32	67.16	99.20	71.79


1 🥇	OpenAI o3	✔️	Unknown	81.20	68.30	69.32	67.16	81.20	68.30	69.32	67.16	100	72.07
2 🥈	GPT-5 Mini (Minimal reasoning)	✔️	Unknown	69.80	64.70	60.82	59.48	69.80	64.70	60.82	59.48	100	71.79
3 🥉	GPT-4o	❌	Unknown	75.60	65.31	66.85	63.59	75.60	65.31	66.85	63.59	100	71.47
4	GPT-5 Mini	✔️	Unknown	77.00	68.51	68.31	66.17	77.00	68.51	68.31	66.17	100	71.44
5	Claude 3.7 Sonnet	❌	Unknown	79.20	66.25	66.35	65.83	79.20	66.25	66.35	65.83	100	71.43
6	Gemini 2.0 Flash	❌	Unknown	78.40	68.19	65.18	65.19	79.03	68.74	65.71	65.72	99.20	70.51
7	Gemini 2.5 Pro	✔️	Unknown	80.20	67.49	69.17	67.12	80.20	67.49	69.17	67.12	100	69.93
8	Gemma 3 12B IT	❌	12.2B	65.80	60.06	55.60	53.15	66.60	60.79	56.27	53.80	98.80	69.91
9	Gemini 2.5 Flash	✔️	Unknown	72.80	61.60	61.34	60.40	72.80	61.60	61.34	60.40	100	69.44
10	Gemini 2.0 Flash-Lite	❌	Unknown	74.00	63.09	66.59	63.40	75.05	63.98	67.53	64.30	98.60	69.14
11	Gemma 3 27B IT	❌	27.4B	70.20	60.71	59.03	58.06	70.77	61.19	59.50	58.53	99.20	68.98
12	GPT-4.1 Mini	❌	Unknown	73.40	65.55	64.60	62.68	73.40	65.55	64.60	62.68	100	68.33
13	Llama 3.3 70B Instruct	❌	70.6B	59.40	56.08	53.29	52.78	63.60	60.05	57.06	56.51	93.40	68.00
14	GPT-4.1	❌	Unknown	74.80	64.29	66.34	62.86	74.80	64.29	66.34	62.86	100	67.58
15	DeepSeek-V3	❌	671.0B	77.00	65.62	67.52	64.26	77.00	65.62	67.52	64.26	100	67.53
16	GPT-5 Nano	✔️	Unknown	70.40	58.96	58.84	58.24	70.40	58.96	58.84	58.24	100	67.50
17	Qwen3-32B	❌	32.8B	74.20	61.67	61.26	59.90	74.50	61.92	61.50	60.14	99.60	67.14
18	Gemma 3n E4B IT	❌	7.8B	45.80	39.28	40.72	35.45	63.79	54.71	56.71	49.37	71.80	65.52
19	GPT-OSS 120B (Low reasoning)	✔️	120.0B	64.80	60.16	56.43	55.21	64.80	60.16	56.43	55.21	100	64.71
20	Qwen3-14B	❌	14.8B	63.60	62.48	52.00	52.12	63.60	62.48	52.00	52.12	100	64.60
21	GPT-4o Mini	❌	Unknown	70.40	66.79	62.59	62.65	70.54	66.93	62.72	62.78	99.80	64.59
22	GPT-OSS 120B	✔️	120.0B	72.20	64.57	63.00	61.27	72.20	64.57	63.00	61.27	100	64.58
23	GPT-OSS 20B (Low reasoning)	✔️	20.0B	67.40	58.52	59.36	57.66	68.22	59.23	60.08	58.36	98.80	64.43
24	DeepSeek-R1	✔️	671.0B	60.80	59.72	50.95	51.60	63.33	62.21	53.07	53.76	96.00	63.61
25	Command R Plus	❌	104.0B	68.20	57.30	57.17	56.06	72.40	60.83	60.69	59.51	94.20	62.98
26	GPT-OSS 20B	✔️	20.0B	67.40	58.36	59.46	57.86	69.63	60.28	61.42	59.77	96.80	62.85
27	GPT-4.1 Nano	❌	Unknown	46.20	49.18	32.62	30.48	51.91	55.26	36.65	34.24	89.00	62.62
28	Qwen3-30B-A3B	❌	30.5B	58.60	58.83	47.21	49.37	58.60	58.83	47.21	49.37	100	62.56
29	Gemma 3 4B IT	❌	4.3B	56.20	48.47	45.47	42.45	59.28	51.13	47.96	44.78	94.80	62.42
30	Qwen3-8B	❌	8.2B	56.00	53.10	47.06	48.42	57.03	54.07	47.92	49.30	98.20	59.68
31	Mistral Small 3.1 24B Instruct	❌	24.0B	65.60	58.19	56.50	54.72	70.39	62.44	60.62	58.72	93.20	56.62
32	Command R v01	❌	35.0B	41.20	38.19	21.94	16.62	47.36	43.90	25.22	19.10	87.00	53.10
33	GPT-5 Nano (Minimal reasoning)	✔️	Unknown	39.40	51.77	29.80	26.26	43.68	57.40	33.03	29.11	90.20	52.41
34	Qwen3-4B	❌	4.0B	36.80	60.15	29.70	27.67	37.40	61.12	30.19	28.12	98.40	51.21
35	Command A	❌	111.0B	61.80	62.85	54.80	56.32	61.80	62.85	54.80	56.32	100	48.25
36	DeepSeek-R1 Qwen3-8B	✔️	8.2B	66.40	59.47	57.20	57.32	67.21	60.19	57.89	58.02	98.80	45.68
37	Llama 4 Scout	❌	109.0B	62.00	55.70	56.20	52.18	65.40	58.76	59.28	55.04	94.80	40.87
38	Aya Expanse 32B	❌	32.3B	62.80	54.60	52.37	52.00	68.41	59.48	57.04	56.64	91.80	39.29
39	Mistral 7B Instruct v0.3	❌	7.2B	7.80	16.26	8.69	6.16	12.11	25.25	13.50	9.56	64.40	39.17
40	Claude 3.5 Haiku	❌	Unknown	75.00	63.27	63.47	62.32	75.00	63.27	63.47	62.32	100	37.49
41	Gemma 3 1B IT	❌	1.0B	38.00	20.19	17.56	11.96	38.31	20.35	17.70	12.05	99.20	36.20
42	Llama 3.2 3B Instruct	❌	3.2B	1.80	13.57	12.60	1.09	1.85	13.93	12.93	1.12	97.40	13.69
43	Llama 3.2 1B Instruct	❌	1.2B	0	0	0	0	0	0	0	0	0	4.68
44	Gemma 3 270M IT	❌	268.0M	0.60	0.10	0.40	0.16	25.00	4.17	16.67	6.67	2.40	3.73

Persian NLG


1 🥇	Mistral Small 3.1 24B Instruct	✔️	Unknown	24.47	33.88	26.93	19.47


1 🥇	Llama 3.3 70B Instruct	❌	70.6B	24.47	33.88	26.93	20.11
2 🥈	GPT-4.1	❌	Unknown	25.76	39.25	29.86	19.47
3 🥉	GPT-4.1 Mini	❌	Unknown	24.39	36.87	28.16	19.01
4	GPT-4o	❌	Unknown	22.76	36.55	26.79	18.96
5	Command R Plus	❌	104.0B	21.50	30.20	24.05	18.80
6	GPT-5 Mini	✔️	Unknown	26.03	38.04	29.68	18.16
7	GPT-4o Mini	❌	Unknown	19.43	34.36	23.69	18.11
8	Gemini 2.0 Flash	❌	Unknown	31.41	41.03	34.42	17.82
9	Claude 3.7 Sonnet	❌	Unknown	16.93	38.86	22.73	17.79
10	OpenAI o3	✔️	Unknown	21.26	37.32	26.03	17.65
11	Qwen3-32B	❌	32.8B	22.81	37.09	27.34	16.79
12	GPT-4.1 Nano	❌	Unknown	16.22	30.26	20.21	16.66
13	Gemini 2.0 Flash-Lite	❌	Unknown	27.82	38.23	31.10	16.59
14	GPT-5 Nano	✔️	Unknown	22.21	33.53	25.53	16.43
15	Command R v01	❌	35.0B	15.46	25.34	18.29	16.42
16	Qwen3-30B-A3B	❌	30.5B	19.97	33.93	24.17	16.41
17	GPT-5 Mini (Minimal reasoning)	✔️	Unknown	15.49	27.99	18.74	16.32
18	Qwen3-14B	❌	14.8B	19.87	34.31	24.19	16.06
19	Llama 4 Scout	❌	109.0B	15.72	26.52	18.89	15.68
20	Qwen3-8B	❌	8.2B	19.65	34.08	23.89	15.57
21	GPT-OSS 120B	✔️	120.0B	21.40	33.83	25.07	15.39
22	GPT-OSS 120B (Low reasoning)	✔️	120.0B	16.42	32.22	20.21	14.31
23	GPT-5 Nano (Minimal reasoning)	✔️	Unknown	10.16	29.42	14.33	14.18
24	Qwen3-4B	❌	4.0B	17.44	26.97	20.18	13.89
25	Gemini 2.5 Flash	✔️	Unknown	31.21	41.63	34.45	13.69
26	GPT-OSS 20B	✔️	20.0B	18.77	30.37	22.15	13.35
27	GPT-OSS 20B (Low reasoning)	✔️	20.0B	13.00	27.07	16.52	13.24
28	Mistral Small 3.1 24B Instruct	❌	24.0B	15.21	26.32	18.43	13.19
29	Gemma 3 12B IT	❌	12.2B	25.73	37.40	29.28	11.97
30	Aya Expanse 32B	❌	32.3B	9.80	34.80	14.44	11.96
31	Gemini 2.5 Pro	✔️	Unknown	29.78	41.85	33.24	11.52
32	Command A	❌	111.0B	21.72	39.39	26.84	11.38
33	Llama 3.2 3B Instruct	❌	3.2B	12.64	25.83	16.01	11.30
34	DeepSeek-R1 Qwen3-8B	✔️	8.2B	16.69	29.52	20.08	11.20
35	Claude 3.5 Haiku	❌	Unknown	8.71	36.01	13.36	10.89
36	Gemma 3 27B IT	❌	27.4B	20.42	34.57	24.60	10.67
37	Gemma 3 4B IT	❌	4.3B	17.04	30.01	20.79	9.50
38	Mistral 7B Instruct v0.3	❌	7.2B	11.27	29.83	15.26	9.44
39	Gemma 3n E4B IT	❌	7.8B	13.66	32.58	18.03	9.40
40	DeepSeek-V3	❌	671.0B	19.39	40.71	24.40	9.34
41	DeepSeek-R1	✔️	671.0B	13.42	41.00	18.41	8.81
42	Llama 3.2 1B Instruct	❌	1.2B	5.68	21.06	7.93	8.23
43	Gemma 3 1B IT	❌	1.0B	4.57	9.92	5.76	6.83
44	Gemma 3 270M IT	❌	268.0M	4.27	4.48	2.91	5.10


1 🥇	Mistral Small 3.1 24B Instruct	✔️	Unknown	20.19	47.53	16.57	11.94	10.95	5.46	7.83	19.47


1 🥇	Llama 3.3 70B Instruct	❌	70.6B	20.19	47.53	16.57	11.94	8.19	5.46	7.83	20.11
2 🥈	GPT-4.1	❌	Unknown	19.74	41.03	18.99	16.38	10.95	4.87	7.37	19.47
3 🥉	GPT-4.1 Mini	❌	Unknown	19.79	44.61	17.45	13.74	9.16	4.90	7.28	19.01
4	GPT-4o	❌	Unknown	21.00	48.06	19.05	14.12	8.61	5.29	6.89	18.96
5	Command R Plus	❌	104.0B	20.18	43.33	19.25	15.31	10.26	5.20	7.08	18.80
6	GPT-5 Mini	✔️	Unknown	17.57	37.45	16.41	13.78	8.96	4.38	6.79	18.16
7	GPT-4o Mini	❌	Unknown	20.14	47.11	18.31	11.42	8.06	5.04	6.49	18.11
8	Gemini 2.0 Flash	❌	Unknown	17.18	36.48	15.33	16.21	9.68	4.58	5.76	17.82
9	Claude 3.7 Sonnet	❌	Unknown	18.00	40.04	17.03	12.26	8.33	4.39	5.95	17.79
10	OpenAI o3	✔️	Unknown	17.45	34.94	17.01	14.47	9.94	4.86	7.50	17.65
11	Qwen3-32B	❌	32.8B	16.23	36.78	14.76	9.38	6.99	4.35	6.20	16.79
12	GPT-4.1 Nano	❌	Unknown	18.32	40.52	16.93	14.00	8.13	4.81	6.11	16.66
13	Gemini 2.0 Flash-Lite	❌	Unknown	17.50	37.41	15.93	13.26	10.03	5.01	6.52	16.59
14	GPT-5 Nano	✔️	Unknown	17.01	36.20	16.39	11.69	7.85	4.63	7.04	16.43
15	Command R v01	❌	35.0B	15.07	29.51	15.01	11.04	9.22	4.98	6.93	16.42
16	Qwen3-30B-A3B	❌	30.5B	16.18	38.22	13.38	8.61	7.71	4.42	5.87	16.41
17	GPT-5 Mini (Minimal reasoning)	✔️	Unknown	17.39	37.20	16.18	13.65	8.87	4.58	6.50	16.32
18	Qwen3-14B	❌	14.8B	15.35	35.54	12.85	8.58	7.87	4.05	5.86	16.06
19	Llama 4 Scout	❌	109.0B	16.95	39.81	14.01	11.73	7.59	3.80	5.71	15.68
20	Qwen3-8B	❌	8.2B	15.50	37.22	12.32	8.82	7.25	4.24	5.29	15.57
21	GPT-OSS 120B	✔️	120.0B	17.32	38.38	16.81	10.06	7.62	4.36	6.41	15.39
22	GPT-OSS 120B (Low reasoning)	✔️	120.0B	16.49	36.63	15.65	10.07	7.87	4.59	5.67	14.31
23	GPT-5 Nano (Minimal reasoning)	✔️	Unknown	16.25	37.28	14.26	10.20	7.66	4.73	5.40	14.18
24	Qwen3-4B	❌	4.0B	12.17	29.16	9.19	6.55	6.12	3.86	4.54	13.89
25	Gemini 2.5 Flash	✔️	Unknown	6.51	11.94	6.41	9.27	3.92	2.23	1.84	13.69
26	GPT-OSS 20B	✔️	20.0B	14.82	33.32	13.49	7.99	7.25	4.25	5.70	13.35
27	GPT-OSS 20B (Low reasoning)	✔️	20.0B	16.46	36.91	15.67	9.45	6.98	4.20	6.07	13.24
28	Mistral Small 3.1 24B Instruct	❌	24.0B	14.58	35.42	12.59	8.87	6.73	4.07	2.93	13.19
29	Gemma 3 12B IT	❌	12.2B	8.82	18.87	8.80	6.58	2.96	1.92	3.66	11.97
30	Aya Expanse 32B	❌	32.3B	9.51	22.04	7.72	9.14	5.56	2.96	1.46	11.96
31	Gemini 2.5 Pro	✔️	Unknown	2.81	5.81	2.53	3.00	2.15	0.82	0.63	11.52
32	Command A	❌	111.0B	4.35	8.65	3.72	5.81	3.11	1.45	1.05	11.38
33	Llama 3.2 3B Instruct	❌	3.2B	10.74	26.90	7.84	5.73	5.35	3.74	2.79	11.30
34	DeepSeek-R1 Qwen3-8B	✔️	8.2B	11.19	27.32	8.69	5.30	5.02	3.22	4.05	11.20
35	Claude 3.5 Haiku	❌	Unknown	5.64	11.54	4.84	6.12	4.67	2.18	1.18	10.89
36	Gemma 3 27B IT	❌	27.4B	4.40	9.33	4.46	4.36	1.97	1.33	0.87	10.67
37	Gemma 3 4B IT	❌	4.3B	3.67	6.23	4.43	3.09	3.31	1.25	1.17	9.50
38	Mistral 7B Instruct v0.3	❌	7.2B	4.73	9.51	3.48	4.17	4.42	2.39	1.89	9.44
39	Gemma 3n E4B IT	❌	7.8B	6.01	15.38	5.48	2.17	2.06	1.72	0.93	9.40
40	DeepSeek-V3	❌	671.0B	4.39	7.15	5.95	4.28	2.58	1.34	1.43	9.34
41	DeepSeek-R1	✔️	671.0B	4.09	9.02	3.28	4.44	2.79	1.48	0.71	8.81
42	Llama 3.2 1B Instruct	❌	1.2B	3.74	7.74	3.42	2.58	2.40	1.50	1.46	8.23
43	Gemma 3 1B IT	❌	1.0B	4.61	13.10	3.17	1.34	1.41	1.28	0.65	6.83
44	Gemma 3 270M IT	❌	268.0M	0.13	0.35	0	0	0	0	0.20	5.10


1 🥇	Mistral Small 3.1 24B Instruct	✔️	Unknown	25.59	16.87	21.14	20.08	50.99	9.84	11.26	19.47


1 🥇	Llama 3.3 70B Instruct	❌	70.6B	25.59	16.87	21.14	20.08	50.99	9.84	11.26	20.11
2 🥈	GPT-4.1	❌	Unknown	23.07	15.28	19.27	16.28	46.76	8.10	10.09	19.47
3 🥉	GPT-4.1 Mini	❌	Unknown	23.89	14.32	20.57	17.76	48.42	8.86	10.45	19.01
4	GPT-4o	❌	Unknown	23.40	15.98	19.47	16.39	47.48	8.25	9.53	18.96
5	Command R Plus	❌	104.0B	23.38	13.86	21.30	17.02	47.82	8.30	7.20	18.80
6	GPT-5 Mini	✔️	Unknown	24.35	16.57	20.55	17.27	49.13	8.83	9.52	18.16
7	GPT-4o Mini	❌	Unknown	23.33	14.98	19.72	17.26	46.79	9.05	10.08	18.11
8	Gemini 2.0 Flash	❌	Unknown	7.57	3.17	5.30	10.28	15.74	3.36	2.79	17.82
9	Claude 3.7 Sonnet	❌	Unknown	22.48	13.42	19.09	17.41	45.44	8.77	9.76	17.79
10	OpenAI o3	✔️	Unknown	23.70	18.58	20.08	17.27	45.07	8.93	10.39	17.65
11	Qwen3-32B	❌	32.8B	21.80	12.72	17.28	15.89	45.73	9.02	9.45	16.79
12	GPT-4.1 Nano	❌	Unknown	21.66	13.49	18.11	16.42	43.84	9.43	8.28	16.66
13	Gemini 2.0 Flash-Lite	❌	Unknown	8.58	3.82	7.50	9.86	15.14	5.68	5.71	16.59
14	GPT-5 Nano	✔️	Unknown	22.94	13.48	18.81	16.43	48.21	8.58	9.14	16.43
15	Command R v01	❌	35.0B	18.92	12.91	17.21	17.37	34.61	7.76	8.28	16.42
16	Qwen3-30B-A3B	❌	30.5B	21.78	11.90	17.94	17.18	45.00	8.37	10.34	16.41
17	GPT-5 Mini (Minimal reasoning)	✔️	Unknown	24.33	16.12	20.45	16.59	49.56	8.79	10.07	16.32
18	Qwen3-14B	❌	14.8B	21.45	13.07	16.98	15.52	45.14	8.43	8.54	16.06
19	Llama 4 Scout	❌	109.0B	19.09	10.39	15.14	12.96	42.67	6.20	5.84	15.68
20	Qwen3-8B	❌	8.2B	20.24	11.63	16.49	15.13	41.71	8.58	9.29	15.57
21	GPT-OSS 120B	✔️	120.0B	20.10	12.96	16.87	12.59	41.81	7.67	7.62	15.39
22	GPT-OSS 120B (Low reasoning)	✔️	120.0B	19.44	10.61	16.28	11.69	41.73	8.18	7.57	14.31
23	GPT-5 Nano (Minimal reasoning)	✔️	Unknown	20.99	11.27	17.01	15.73	44.64	8.05	8.69	14.18
24	Qwen3-4B	❌	4.0B	18.41	10.11	14.92	13.78	38.03	8.52	8.57	13.89
25	Gemini 2.5 Flash	✔️	Unknown	5.59	1.90	3.69	4.01	13.82	2.33	1.75	13.69
26	GPT-OSS 20B	✔️	20.0B	18.11	11.81	15.04	10.43	37.94	6.42	7.72	13.35
27	GPT-OSS 20B (Low reasoning)	✔️	20.0B	18.19	11.13	14.63	11.90	37.72	7.73	8.62	13.24
28	Mistral Small 3.1 24B Instruct	❌	24.0B	14.51	3.93	10.09	9.30	36.61	5.37	5.04	13.19
29	Gemma 3 12B IT	❌	12.2B	2.57	1.16	2.15	2.62	4.72	1.79	1.70	11.97
30	Aya Expanse 32B	❌	32.3B	14.44	6.37	10.46	10.65	33.32	5.73	5.26	11.96
31	Gemini 2.5 Pro	✔️	Unknown	3.49	1.71	2.85	3.00	7.20	1.67	1.59	11.52
32	Command A	❌	111.0B	4.86	2.32	3.97	4.72	9.72	2.47	2.39	11.38
33	Llama 3.2 3B Instruct	❌	3.2B	8.62	3.34	7.59	8.92	16.89	4.28	4.73	11.30
34	DeepSeek-R1 Qwen3-8B	✔️	8.2B	12.49	5.65	7.40	6.77	31.12	5.24	4.80	11.20
35	Claude 3.5 Haiku	❌	Unknown	6.91	3.21	5.35	8.00	13.40	3.63	3.93	10.89
36	Gemma 3 27B IT	❌	27.4B	2.00	0.98	1.45	2.60	3.45	1.51	1.57	10.67
37	Gemma 3 4B IT	❌	4.3B	1.69	0.85	1.37	1.82	3.01	1.22	1.10	9.50
38	Mistral 7B Instruct v0.3	❌	7.2B	9.02	5.22	8.29	7.56	16.46	4.86	5.19	9.44
39	Gemma 3n E4B IT	❌	7.8B	1.96	1.02	1.54	2.31	3.59	1.19	1.18	9.40
40	DeepSeek-V3	❌	671.0B	4.50	1.66	4.78	4.26	8.02	2.53	2.69	9.34
41	DeepSeek-R1	✔️	671.0B	4.14	1.95	3.46	3.97	7.98	2.44	2.32	8.81
42	Llama 3.2 1B Instruct	❌	1.2B	4.23	1.25	3.14	2.95	10.07	1.18	2.47	8.23
43	Gemma 3 1B IT	❌	1.0B	1.46	0.65	1.19	1.52	2.74	0.94	0.94	6.83
44	Gemma 3 270M IT	❌	268.0M	3.83	1.80	3.59	3.71	6.49	2.22	3.41	5.10


1 🥇	Mistral Small 3.1 24B Instruct	✔️	Unknown	13.02	11.05	7.42	20.32	19.47


1 🥇	Llama 3.3 70B Instruct	❌	70.6B	13.02	11.05	7.42	20.32	20.11
2 🥈	GPT-4.1	❌	Unknown	15.34	12.50	6.24	26.82	19.47
3 🥉	GPT-4.1 Mini	❌	Unknown	14.35	12.21	6.95	23.52	19.01
4	GPT-4o	❌	Unknown	15.43	12.84	6.60	26.39	18.96
5	Command R Plus	❌	104.0B	14.14	13.64	6.53	21.87	18.80
6	GPT-5 Mini	✔️	Unknown	13.34	11.44	6.33	21.91	18.16
7	GPT-4o Mini	❌	Unknown	13.98	12.73	6.58	22.24	18.11
8	Gemini 2.0 Flash	❌	Unknown	15.66	11.23	6.29	29.00	17.82
9	Claude 3.7 Sonnet	❌	Unknown	14.72	12.94	6.43	24.37	17.79
10	OpenAI o3	✔️	Unknown	13.71	11.48	6.36	22.93	17.65
11	Qwen3-32B	❌	32.8B	9.86	8.58	6.23	14.59	16.79
12	GPT-4.1 Nano	❌	Unknown	13.15	10.64	6.42	22.06	16.66
13	Gemini 2.0 Flash-Lite	❌	Unknown	14.30	13.26	5.32	23.88	16.59
14	GPT-5 Nano	✔️	Unknown	11.84	9.05	5.23	20.90	16.43
15	Command R v01	❌	35.0B	11.45	11.91	6.48	15.71	16.42
16	Qwen3-30B-A3B	❌	30.5B	9.04	7.79	6.10	13.09	16.41
17	GPT-5 Mini (Minimal reasoning)	✔️	Unknown	13.13	10.91	6.59	21.57	16.32
18	Qwen3-14B	❌	14.8B	9.10	8.63	5.58	12.93	16.06
19	Llama 4 Scout	❌	109.0B	9.98	8.80	4.25	16.59	15.68
20	Qwen3-8B	❌	8.2B	8.21	7.30	5.79	11.41	15.57
21	GPT-OSS 120B	✔️	120.0B	10.64	8.97	5.53	17.18	15.39
22	GPT-OSS 120B (Low reasoning)	✔️	120.0B	10.34	9.91	5.55	15.34	14.31
23	GPT-5 Nano (Minimal reasoning)	✔️	Unknown	9.97	8.62	4.59	16.44	14.18
24	Qwen3-4B	❌	4.0B	6.36	5.57	5.40	8.07	13.89
25	Gemini 2.5 Flash	✔️	Unknown	7.72	6.10	2.73	14.08	13.69
26	GPT-OSS 20B	✔️	20.0B	7.05	6.76	4.21	10.04	13.35
27	GPT-OSS 20B (Low reasoning)	✔️	20.0B	8.07	6.91	4.61	12.51	13.24
28	Mistral Small 3.1 24B Instruct	❌	24.0B	5.95	5.40	3.52	8.79	13.19
29	Gemma 3 12B IT	❌	12.2B	3.26	3.33	1.06	5.28	11.97
30	Aya Expanse 32B	❌	32.3B	6.08	6.37	2.59	9.13	11.96
31	Gemini 2.5 Pro	✔️	Unknown	2.28	2.49	0.99	3.27	11.52
32	Command A	❌	111.0B	3.97	3.55	1.61	6.62	11.38
33	Llama 3.2 3B Instruct	❌	3.2B	3.82	5.18	2.35	3.85	11.30
34	DeepSeek-R1 Qwen3-8B	✔️	8.2B	4.40	3.13	3.09	6.90	11.20
35	Claude 3.5 Haiku	❌	Unknown	5.82	5.40	2.33	9.55	10.89
36	Gemma 3 27B IT	❌	27.4B	4.66	2.38	1.21	10.21	10.67
37	Gemma 3 4B IT	❌	4.3B	1.87	1.75	0.98	2.84	9.50
38	Mistral 7B Instruct v0.3	❌	7.2B	3.53	3.18	2.19	5.13	9.44
39	Gemma 3n E4B IT	❌	7.8B	2.27	1.96	1.07	3.72	9.40
40	DeepSeek-V3	❌	671.0B	3.03	2.72	1.29	4.98	9.34
41	DeepSeek-R1	✔️	671.0B	3.35	3.14	1.39	5.43	8.81
42	Llama 3.2 1B Instruct	❌	1.2B	0.56	0.85	0.73	0.13	8.23
43	Gemma 3 1B IT	❌	1.0B	1.01	1.16	0.68	1.17	6.83
44	Gemma 3 270M IT	❌	268.0M	1.57	2.22	1.08	1.40	5.10


1 🥇	Mistral Small 3.1 24B Instruct	✔️	Unknown	8.60	4.41	8.34	13.07	19.47


1 🥇	Llama 3.3 70B Instruct	❌	70.6B	8.60	4.41	8.34	13.07	20.11
2 🥈	GPT-4.1	❌	Unknown	8.28	3.84	7.81	13.17	19.47
3 🥉	GPT-4.1 Mini	❌	Unknown	6.90	3.97	7.51	9.22	19.01
4	GPT-4o	❌	Unknown	6.57	3.47	7.32	8.91	18.96
5	Command R Plus	❌	104.0B	9.23	5.11	5.90	16.68	18.80
6	GPT-5 Mini	✔️	Unknown	7.21	3.74	6.36	11.52	18.16
7	GPT-4o Mini	❌	Unknown	6.53	3.73	6.89	8.96	18.11
8	Gemini 2.0 Flash	❌	Unknown	7.93	3.38	5.71	14.69	17.82
9	Claude 3.7 Sonnet	❌	Unknown	7.61	3.21	6.14	13.49	17.79
10	OpenAI o3	✔️	Unknown	8.49	3.79	8.11	13.56	17.65
11	Qwen3-32B	❌	32.8B	5.07	3.16	5.34	6.70	16.79
12	GPT-4.1 Nano	❌	Unknown	5.68	3.27	5.79	7.99	16.66
13	Gemini 2.0 Flash-Lite	❌	Unknown	3.42	1.98	4.19	4.07	16.59
14	GPT-5 Nano	✔️	Unknown	5.64	3.46	5.55	7.92	16.43
15	Command R v01	❌	35.0B	8.53	4.64	7.13	13.81	16.42
16	Qwen3-30B-A3B	❌	30.5B	4.83	3.10	5.12	6.27	16.41
17	GPT-5 Mini (Minimal reasoning)	✔️	Unknown	5.57	3.41	5.96	7.33	16.32
18	Qwen3-14B	❌	14.8B	4.94	3.70	5.68	5.46	16.06
19	Llama 4 Scout	❌	109.0B	5.49	3.18	5.68	7.60	15.68
20	Qwen3-8B	❌	8.2B	4.23	3.29	4.45	4.96	15.57
21	GPT-OSS 120B	✔️	120.0B	3.82	2.46	4.03	4.96	15.39
22	GPT-OSS 120B (Low reasoning)	✔️	120.0B	3.66	2.36	4.46	4.15	14.31
23	GPT-5 Nano (Minimal reasoning)	✔️	Unknown	4.62	3.38	4.95	5.52	14.18
24	Qwen3-4B	❌	4.0B	3.51	3.14	4.21	3.20	13.89
25	Gemini 2.5 Flash	✔️	Unknown	3.83	3.22	2.90	5.36	13.69
26	GPT-OSS 20B	✔️	20.0B	2.87	2.51	3.56	2.56	13.35
27	GPT-OSS 20B (Low reasoning)	✔️	20.0B	3.17	2.85	3.68	2.99	13.24
28	Mistral Small 3.1 24B Instruct	❌	24.0B	1.92	1.51	2.46	1.80	13.19
29	Gemma 3 12B IT	❌	12.2B	0.98	0.72	1.11	1.12	11.97
30	Aya Expanse 32B	❌	32.3B	3.19	2.23	2.97	4.37	11.96
31	Gemini 2.5 Pro	✔️	Unknown	1.79	0.97	1.10	3.31	11.52
32	Command A	❌	111.0B	2.36	1.24	1.99	3.84	11.38
33	Llama 3.2 3B Instruct	❌	3.2B	1.37	1.10	1.35	1.64	11.30
34	DeepSeek-R1 Qwen3-8B	✔️	8.2B	0.59	0.64	0.68	0.43	11.20
35	Claude 3.5 Haiku	❌	Unknown	3.31	2.02	2.81	5.11	10.89
36	Gemma 3 27B IT	❌	27.4B	1.01	0.72	1.01	1.29	10.67
37	Gemma 3 4B IT	❌	4.3B	0.68	0.57	0.90	0.57	9.50
38	Mistral 7B Instruct v0.3	❌	7.2B	1.67	1.82	2.04	1.16	9.44
39	Gemma 3n E4B IT	❌	7.8B	0.86	0.69	0.97	0.93	9.40
40	DeepSeek-V3	❌	671.0B	1.99	1.14	1.55	3.27	9.34
41	DeepSeek-R1	✔️	671.0B	1.98	1.12	1.59	3.25	8.81
42	Llama 3.2 1B Instruct	❌	1.2B	1.64	1.53	2.20	1.21	8.23
43	Gemma 3 1B IT	❌	1.0B	0.45	0.46	0.52	0.37	6.83
44	Gemma 3 270M IT	❌	268.0M	0.62	0.58	0.59	0.68	5.10


1 🥇	Mistral Small 3.1 24B Instruct	✔️	Unknown	16.32	38.75	21.58	19.47


1 🥇	Llama 3.3 70B Instruct	❌	70.6B	16.32	38.75	21.58	20.11
2 🥈	GPT-4.1	❌	Unknown	16.81	35.68	21.90	19.47
3 🥉	GPT-4.1 Mini	❌	Unknown	17.15	36.93	22.49	19.01
4	GPT-4o	❌	Unknown	16.51	39.82	22.40	18.96
5	Command R Plus	❌	104.0B	15.91	41.63	22.09	18.80
6	GPT-5 Mini	✔️	Unknown	15.54	33.58	20.46	18.16
7	GPT-4o Mini	❌	Unknown	15.78	39.02	21.56	18.11
8	Gemini 2.0 Flash	❌	Unknown	18.09	41.45	24.07	17.82
9	Claude 3.7 Sonnet	❌	Unknown	15.39	38.50	21.16	17.79
10	OpenAI o3	✔️	Unknown	14.95	33.43	19.78	17.65
11	Qwen3-32B	❌	32.8B	15.15	36.83	20.63	16.79
12	GPT-4.1 Nano	❌	Unknown	16.18	34.77	20.98	16.66
13	Gemini 2.0 Flash-Lite	❌	Unknown	18.10	40.16	23.81	16.59
14	GPT-5 Nano	✔️	Unknown	13.87	34.72	18.99	16.43
15	Command R v01	❌	35.0B	19.44	37.61	24.26	16.42
16	Qwen3-30B-A3B	❌	30.5B	15.24	38.39	20.84	16.41
17	GPT-5 Mini (Minimal reasoning)	✔️	Unknown	15.08	33.71	19.99	16.32
18	Qwen3-14B	❌	14.8B	14.69	37.44	20.23	16.06
19	Llama 4 Scout	❌	109.0B	15.40	34.14	20.07	15.68
20	Qwen3-8B	❌	8.2B	14.63	38.56	20.24	15.57
21	GPT-OSS 120B	✔️	120.0B	11.97	36.92	17.38	15.39
22	GPT-OSS 120B (Low reasoning)	✔️	120.0B	11.72	36.46	17.04	14.31
23	GPT-5 Nano (Minimal reasoning)	✔️	Unknown	13.90	33.59	18.83	14.18
24	Qwen3-4B	❌	4.0B	14.30	39.77	20.13	13.89
25	Gemini 2.5 Flash	✔️	Unknown	15.26	39.46	20.99	13.69
26	GPT-OSS 20B	✔️	20.0B	10.76	35.50	15.78	13.35
27	GPT-OSS 20B (Low reasoning)	✔️	20.0B	11.72	38.02	17.20	13.24
28	Mistral Small 3.1 24B Instruct	❌	24.0B	14.73	40.67	20.72	13.19
29	Gemma 3 12B IT	❌	12.2B	16.58	36.78	21.89	11.97
30	Aya Expanse 32B	❌	32.3B	13.38	39.79	19.33	11.96
31	Gemini 2.5 Pro	✔️	Unknown	15.80	35.50	20.84	11.52
32	Command A	❌	111.0B	14.13	40.17	20.05	11.38
33	Llama 3.2 3B Instruct	❌	3.2B	13.86	31.34	18.19	11.30
34	DeepSeek-R1 Qwen3-8B	✔️	8.2B	11.37	34.97	16.29	11.20
35	Claude 3.5 Haiku	❌	Unknown	17.73	34.16	22.33	10.89
36	Gemma 3 27B IT	❌	27.4B	15.88	37.36	21.32	10.67
37	Gemma 3 4B IT	❌	4.3B	16.38	35.36	21.35	9.50
38	Mistral 7B Instruct v0.3	❌	7.2B	11.81	39.23	17.08	9.44
39	Gemma 3n E4B IT	❌	7.8B	14.68	38.58	20.41	9.40
40	DeepSeek-V3	❌	671.0B	8.93	37.02	13.92	9.34
41	DeepSeek-R1	✔️	671.0B	10.82	37.27	16.07	8.81
42	Llama 3.2 1B Instruct	❌	1.2B	15.66	26.42	17.60	8.23
43	Gemma 3 1B IT	❌	1.0B	13.40	31.84	17.91	6.83
44	Gemma 3 270M IT	❌	268.0M	11.54	24.02	13.89	5.10


1 🥇	Mistral Small 3.1 24B Instruct	✔️	Unknown	22.38	35.33	24.85	19.47


1 🥇	Llama 3.3 70B Instruct	❌	70.6B	22.38	35.33	24.85	20.11
2 🥈	GPT-4.1	❌	Unknown	12.35	37.61	18.09	19.47
3 🥉	GPT-4.1 Mini	❌	Unknown	11.56	40.33	17.50	19.01
4	GPT-4o	❌	Unknown	11.35	39.10	17.17	18.96
5	Command R Plus	❌	104.0B	12.27	43.15	18.57	18.80
6	GPT-5 Mini	✔️	Unknown	9.17	38.94	14.47	18.16
7	GPT-4o Mini	❌	Unknown	11.60	39.80	17.52	18.11
8	Gemini 2.0 Flash	❌	Unknown	12.28	37.51	17.93	17.82
9	Claude 3.7 Sonnet	❌	Unknown	11.88	39.48	17.87	17.79
10	OpenAI o3	✔️	Unknown	9.50	34.12	14.38	17.65
11	Qwen3-32B	❌	32.8B	10.90	39.36	16.63	16.79
12	GPT-4.1 Nano	❌	Unknown	11.21	36.10	16.61	16.66
13	Gemini 2.0 Flash-Lite	❌	Unknown	11.91	36.54	17.44	16.59
14	GPT-5 Nano	✔️	Unknown	8.31	35.66	13.09	16.43
15	Command R v01	❌	35.0B	12.98	35.49	18.42	16.42
16	Qwen3-30B-A3B	❌	30.5B	12.13	39.23	18.05	16.41
17	GPT-5 Mini (Minimal reasoning)	✔️	Unknown	9.69	38.48	15.05	16.32
18	Qwen3-14B	❌	14.8B	11.24	40.32	17.12	16.06
19	Llama 4 Scout	❌	109.0B	13.77	38.01	19.29	15.68
20	Qwen3-8B	❌	8.2B	10.93	39.79	16.69	15.57
21	GPT-OSS 120B	✔️	120.0B	8.39	38.43	13.39	15.39
22	GPT-OSS 120B (Low reasoning)	✔️	120.0B	8.09	37.94	12.98	14.31
23	GPT-5 Nano (Minimal reasoning)	✔️	Unknown	9.29	35.03	14.25	14.18
24	Qwen3-4B	❌	4.0B	10.67	41.09	16.48	13.89
25	Gemini 2.5 Flash	✔️	Unknown	10.70	43.57	16.73	13.69
26	GPT-OSS 20B	✔️	20.0B	7.97	35.74	12.64	13.35
27	GPT-OSS 20B (Low reasoning)	✔️	20.0B	8.13	38.55	13.06	13.24
28	Mistral Small 3.1 24B Instruct	❌	24.0B	10.29	43.48	16.21	13.19
29	Gemma 3 12B IT	❌	12.2B	11.57	36.31	16.97	11.97
30	Aya Expanse 32B	❌	32.3B	10.95	39.79	16.75	11.96
31	Gemini 2.5 Pro	✔️	Unknown	10.91	34.87	16.14	11.52
32	Command A	❌	111.0B	11.17	42.09	17.23	11.38
33	Llama 3.2 3B Instruct	❌	3.2B	15.15	37.76	20.34	11.30
34	DeepSeek-R1 Qwen3-8B	✔️	8.2B	8.33	40.28	13.38	11.20
35	Claude 3.5 Haiku	❌	Unknown	13.29	35.79	18.87	10.89
36	Gemma 3 27B IT	❌	27.4B	11.25	37.17	16.73	10.67
37	Gemma 3 4B IT	❌	4.3B	10.96	37.35	16.45	9.50
38	Mistral 7B Instruct v0.3	❌	7.2B	9.22	44.02	14.81	9.44
39	Gemma 3n E4B IT	❌	7.8B	10.72	39.67	16.27	9.40
40	DeepSeek-V3	❌	671.0B	8.11	39.40	13.16	9.34
41	DeepSeek-R1	✔️	671.0B	8.49	38.21	13.60	8.81
42	Llama 3.2 1B Instruct	❌	1.2B	20.61	27.55	21.92	8.23
43	Gemma 3 1B IT	❌	1.0B	11.14	37.32	16.61	6.83
44	Gemma 3 270M IT	❌	268.0M	8.19	43.33	12.74	5.10

About MIZAN: A Persian LLM Leaderboard

MIZAN: A Persian LLM Leaderboard is designed to establish a standard and comprehensive benchmark for evaluating Large Language Models (LLMs) in the Persian language. This project combines existing datasets, translates and localizes globally recognized benchmarks, and incorporates newly developed, Persian-specific datasets. MIZAN aims to provide a multi-dimensional assessment of models' capabilities across various linguistic, knowledge-based, and reasoning tasks. Its primary goal is to offer researchers, developers, and enthusiasts a transparent and reliable view of LLM performance in the Persian language landscape.

MIZAN provides a holistic view of models' strengths and weaknesses by assessing them across a suite of key tasks, contributing to the advancement of AI research for the Persian language.

PerCoR is the first large-scale Persian benchmark for evaluating models' ability in commonsense reasoning through multi-choice sentence completion. It includes over 106,000 samples from diverse domains such as news, religion, and lifestyle, extracted from more than 40 Persian websites. Innovative methods like "segmentation by conjunctions" were used to create coherent and diverse sentences and options, while the DRESS-AF technique helped generate challenging, human-solvable distractors.

link

This dataset is a Persian-adapted and localized version of IFEval, assessing models' proficiency in accurately executing complex, multi-step instructions (Instruction Following). The translation process involved a hybrid machine-human approach, with prompts unsuitable for the Persian language being rewritten or removed.

link

PerMMLU is an expanded and localized version of the renowned MMLU benchmark, designed to measure general and specialized knowledge of models in Persian. Tailored to cover knowledge at various levels and relevant to the Iranian cultural context, it comprises three main sub-datasets:

SPK (School Persian Knowledge): Contains 5,581 multiple-choice questions from the official Iranian school curriculum (grades 4-12) across 78 diverse subjects. Data was collected from the "Paadars" educational website and subsequently cleaned.
UPK (University Persian Knowledge): Includes 7,793 multiple-choice questions from Master's and PhD entrance exams across 25 academic disciplines (e.g., medicine, engineering, humanities, arts). This data was extracted from exam booklets using OCR technology and cleaned by LLMs.
GPK (General Persian Knowledge): Consists of 1,003 multiple-choice questions on 15 topics related to general knowledge specific to Iranian society (e.g., city souvenirs, religious edicts, national laws, famous personalities, cultural idioms). This data was generated using LLMs with specific prompts and reviewed by humans.

link

This is a localized version of the MT-Bench benchmark, evaluating models on multi-turn question-answering and dialogue-based tasks. Questions involve multi-step requests or require creative responses. In the Persian version, all samples were translated and rewritten by humans, and some were expanded to 3 or 4 turns. Two new topics were also added:

Native Iranian Knowledge: Questions about cultural topics such as films, actors, and Iranian figures.
Chat-Retrieval: Involves a multi-turn dialogue where the model must extract a relevant question and answer based on the user's needs.

link

This section comprises a collection of existing Persian benchmarks for evaluating various aspects of Natural Language Understanding. Key tasks and datasets include:

Sentiment Analysis: DeepSentiPers
Text Classification: Synthetic Persian Tone, SID
Natural Language Inference (NLI): FarsTAIL
Semantic Textual Similarity (STS): Synthetic Persian STS, FarSICK
Named Entity Recognition (NER): Arman
Paraphrase Detection: FarsiParaphraseDetection, ParsiNLU
Extractive Question Answering (EQA): PQuAD
Keyword Extraction: Synthetic Persian Keywords

link

This section focuses on Natural Language Generation, covering tasks such as:

Summarization: SamSUM-fa, PnSummary
Machine Translation: TEP, MIZAN, EPOQUE
Question Generation: PersianQA

The goal is to assess the generative capabilities of models.

link

MIZAN is a significant step towards the scientific and localized evaluation of language models for Persian, aiming to serve as a valuable assessment reference for researchers, developers, and anyone interested in practical language models.

Request Model Evaluation for MIZAN

We're excited to evaluate new models for MIZAN: A Persian LLM Leaderboard! Please choose the submission path that best fits how your model can be accessed for evaluation.

Option 1: Your model is publicly available on Hugging Face Hub

If your model and its tokenizer can be loaded directly using their Hugging Face identifier (e.g., username/model_name), you can use the simplified form below to submit its key identifiers. Your submission will be added to our central tracking repository on the Hugging Face Hub. Our team will attempt to gather other necessary details from the Hub.

✨ Form for Option 1: Submit a Hugging Face Model to the Hub

Model Name (Hugging Face ID: e.g., username/model_name)

Revision/Commit (Optional, defaults to 'main' if left empty)

Option 2: Your model is NOT on Hugging Face, is private, or requires custom setup

If your model is hosted elsewhere, is private, requires specific access permissions, needs custom inference code, or involves a more complex setup for evaluation, please initiate your submission request via email.

To submit via email, please send comprehensive details to: 📧 mehran.sarmadi99@sharif.edu

Our team will review your email and work with you to facilitate the evaluation process.

Built with Gradio logo


1 🥇	Gemini 2.5 Pro	✔️	Unknown	94.28	94.28	94.50	94.88	93.72	96.01	93.90	94.94	92.42	89.26	91.39	98.04	96.00	96.06	95.67	95.16	95.67	95.36	93.71	96.94	92.98	95.56	92.16	91.64	97.46	95.93	98.54	91.69	92.48	95.13	87.76	96.75	96.81	96.53	93.06	93.65	86.75	81.82	100	100	95.24	83.33	93.22	90.00	90.00
2 🥈	OpenAI o3	✔️	Unknown	92.18	92.18	92.15	90.23	90.27	93.35	92.00	91.86	92.00	93.46	88.79	94.90	95.39	93.73	94.19	92.41	92.79	92.95	91.18	94.59	90.80	93.35	91.30	88.25	95.79	95.77	97.62	90.19	91.91	94.71	87.85	94.87	98.95	95.65	89.92	88.85	84.03	82.61	92.31	96.97	83.72	81.58	89.17	100	60.00
3 🥉	Claude 3.7 Sonnet	❌	Unknown	91.18	91.18	91.84	90.63	90.27	92.45	93.60	91.84	89.00	87.58	87.02	93.63	96.71	92.24	92.35	91.46	94.41	92.53	91.18	91.98	90.80	91.69	88.89	88.25	93.82	93.41	94.76	88.83	91.18	91.63	82.58	93.59	96.84	93.98	92.34	91.15	85.71	86.96	100	100	86.05	78.95	83.33	100	80.00
4	GPT-4.1	❌	Unknown	88.39	88.39	86.10	83.59	87.03	90.33	88.40	87.06	88.50	85.62	83.78	92.36	92.76	91.04	88.38	86.08	91.15	88.80	91.18	91.18	86.78	91.97	87.92	86.35	92.70	91.55	94.29	85.83	86.03	90.75	80.21	89.74	92.63	90.97	85.08	83.08	84.03	86.96	100	100	88.37	84.21	88.33	90.00	90.00
5	Gemini 2.5 Flash	✔️	Unknown	87.17	87.14	87.01	86.72	86.22	90.33	85.20	87.47	90.00	85.62	82.30	89.81	88.82	91.04	89.30	87.66	87.87	87.55	91.83	89.78	85.63	88.64	82.13	84.76	89.61	90.85	92.38	83.11	83.82	92.07	81.94	90.38	93.68	88.29	80.65	86.92	76.05	82.61	92.31	93.94	83.72	78.95	77.50	90.00	80.00
6	GPT-4o	❌	Unknown	86.65	78.32	84.59	83.59	85.95	88.82	89.60	84.97	85.50	82.35	82.01	91.72	93.42	86.57	88.38	88.92	94.10	88.80	85.29	88.18	81.03	88.92	85.51	81.27	91.29	90.85	92.86	84.47	84.56	87.22	81.25	90.38	91.58	89.97	81.45	84.62	78.15	78.26	92.31	96.97	76.74	71.05	85.00	80.00	90.00
7	GPT-5 Mini	✔️	Unknown	86.51	86.51	84.29	80.86	87.57	89.12	86.40	89.56	91.00	83.66	82.01	90.45	89.47	86.27	88.69	89.24	87.87	88.38	84.64	89.18	82.18	88.37	83.57	83.81	86.24	89.67	91.43	86.10	87.50	85.46	80.21	91.67	90.53	89.63	83.87	84.23	78.15	73.91	100	90.91	79.07	84.21	82.50	90.00	50.00
8	Gemini 2.0 Flash	❌	Unknown	86.38	86.38	83.99	82.81	82.43	88.52	84.00	87.47	89.50	83.66	80.24	89.81	94.08	89.55	88.99	88.92	87.54	87.55	84.26	87.78	81.61	88.37	88.89	81.27	89.04	90.85	91.90	83.92	88.24	90.31	81.25	94.23	91.58	89.30	84.27	81.54	77.73	69.57	100	96.97	74.42	78.95	81.67	100	90.00
9	DeepSeek-R1	✔️	671.0B	82.52	82.52	82.48	80.47	85.95	85.80	78.00	82.67	83.50	79.08	80.83	90.45	90.07	82.69	83.49	79.11	86.51	82.16	81.70	83.37	78.74	83.10	82.13	75.24	87.36	84.51	90.00	83.38	80.88	84.58	76.04	89.10	89.47	80.94	77.82	81.15	76.47	60.87	92.31	93.94	83.72	73.68	75.83	90.00	70.00
10	DeepSeek-V3	❌	671.0B	82.41	51.15	80.97	77.34	82.16	85.80	81.20	81.42	84.50	78.43	76.70	85.99	89.47	82.99	85.32	81.65	86.89	81.74	84.64	82.36	79.31	82.83	80.19	75.24	86.52	85.68	90.00	83.11	81.62	88.11	75.69	91.67	90.53	83.28	78.23	80.38	76.89	65.22	84.62	96.97	79.07	73.68	76.67	80.00	70.00
11	Gemini 2.0 Flash-Lite	❌	Unknown	81.43	81.43	79.76	77.73	80.00	83.38	79.60	80.38	82.00	77.12	76.70	85.99	89.47	82.99	84.10	81.96	85.57	82.16	82.03	81.76	80.46	81.16	78.74	76.83	85.11	84.74	89.52	78.47	83.82	84.58	77.08	85.26	91.58	83.95	74.60	82.31	74.79	69.57	100	100	79.07	71.05	74.17	90.00	60.00
12	Command A	❌	111.0B	79.89	79.84	76.74	76.95	76.96	83.99	78.00	78.71	82.00	77.12	76.11	87.90	84.87	81.19	82.26	80.38	85.57	82.99	85.62	81.53	74.71	80.06	79.23	73.33	83.43	84.27	90.95	77.38	80.88	84.58	72.22	83.97	89.47	79.93	73.39	77.31	69.75	65.22	92.31	84.85	74.42	65.79	71.67	80.00	60.00
13	Llama 3.3 70B Instruct	❌	70.6B	79.56	11.23	77.64	76.17	77.30	85.50	78.00	79.54	75.50	73.86	74.93	84.08	84.21	80.30	82.87	82.59	86.23	82.57	83.66	80.76	73.56	79.22	78.74	71.43	84.55	84.04	90.48	78.75	79.41	84.14	74.65	80.77	89.47	79.60	74.19	73.46	68.49	60.87	84.62	87.88	79.07	65.79	75.83	90.00	50.00
14	Llama 4 Scout	❌	109.0B	78.14	14.72	79.46	74.22	74.86	81.57	75.60	79.33	80.00	71.90	71.98	84.08	84.21	78.51	80.73	77.85	85.57	81.33	80.07	77.15	74.71	77.01	77.78	75.24	79.49	81.92	84.29	78.20	80.88	79.30	69.79	82.69	88.42	81.61	72.58	77.69	71.01	60.87	76.92	84.85	74.42	76.32	69.17	80.00	60.00
15	GPT-OSS 120B	✔️	120.0B	77.14	77.14	74.32	76.56	77.84	81.57	73.20	78.50	79.00	82.35	72.57	80.25	77.63	78.81	78.59	78.48	78.69	74.27	76.80	82.36	71.84	80.33	72.46	69.84	75.56	81.22	89.52	74.66	83.82	77.09	74.65	78.85	82.11	73.91	70.16	74.62	76.89	65.22	76.92	87.88	72.09	71.05	69.17	70.00	60.00
16	GPT-4.1 Mini	❌	Unknown	77.12	77.12	77.04	75.39	76.49	82.48	76.00	77.45	78.50	75.82	67.85	84.08	82.24	79.10	79.20	79.43	82.95	76.35	79.74	80.56	72.41	77.84	77.78	70.79	77.25	80.99	87.62	75.48	79.41	78.41	68.75	82.69	86.32	79.26	70.56	70.77	68.91	65.22	84.62	81.82	62.79	65.79	70.83	70.00	50.00
17	GPT-5 Mini (Minimal reasoning)	✔️	Unknown	76.86	76.86	76.74	71.37	77.03	78.18	77.33	77.41	80.00	75.82	68.44	80.89	82.89	78.44	80.37	79.11	83.61	77.59	76.47	79.36	70.69	77.01	74.88	71.75	75.00	81.69	83.33	75.14	83.09	79.74	68.40	79.49	86.17	79.93	72.98	74.23	72.27	73.91	69.23	78.79	72.09	71.05	67.50	80.00	40.00
18	Qwen3-32B	❌	32.8B	76.54	76.53	74.92	75.00	76.22	78.25	72.40	80.38	80.00	74.51	69.91	80.25	81.58	80.30	80.73	79.43	80.00	76.35	80.39	78.76	68.97	77.84	76.33	67.94	77.25	82.16	85.24	75.20	80.88	76.65	69.10	75.64	87.37	76.25	68.95	71.92	72.69	52.17	76.92	84.85	67.44	65.79	67.50	70.00	70.00
19	Gemma 3 27B IT	❌	27.4B	76.28	76.28	69.79	72.66	76.22	81.87	74.00	76.62	78.00	69.93	71.09	82.17	80.26	79.40	75.54	77.85	82.30	81.74	78.43	79.76	71.26	75.35	73.91	70.79	77.53	79.58	84.29	74.93	78.68	82.38	69.10	78.21	83.16	79.93	69.76	73.08	67.23	69.57	84.62	93.94	69.77	63.16	70.00	80.00	70.00
20	GPT-4o Mini	❌	Unknown	75.98	75.98	74.92	71.48	73.51	80.06	73.60	75.99	77.50	65.36	71.98	79.62	82.89	74.93	78.29	80.06	81.31	74.27	78.10	76.15	76.44	77.84	75.36	69.52	76.69	79.58	87.62	77.66	75.74	76.21	69.10	78.21	87.37	80.60	67.34	75.00	67.65	65.22	84.62	84.85	74.42	76.32	70.83	80.00	40.00
21	Command R Plus	❌	104.0B	73.64	73.64	75.23	67.58	70.54	74.62	72.40	72.23	70.00	69.93	70.21	80.89	84.87	71.94	73.39	72.47	82.62	71.78	77.12	77.15	71.84	74.79	76.81	66.98	74.16	79.34	81.90	72.21	75.00	76.65	66.32	78.21	86.32	75.25	66.53	75.00	61.34	65.22	92.31	87.88	65.12	50.00	70.83	90.00	70.00
22	GPT-5 Nano	✔️	Unknown	73.10	73.09	66.47	69.92	75.41	77.64	71.20	72.65	78.00	71.24	68.14	77.71	81.58	77.01	75.23	77.85	74.43	73.03	76.80	78.56	67.82	74.24	68.12	68.57	68.82	74.18	83.33	71.66	77.21	74.01	67.36	71.79	82.11	69.57	68.95	72.69	69.75	52.17	92.31	87.88	65.12	52.63	67.50	70.00	50.00
23	Claude 3.5 Haiku	❌	Unknown	71.60	71.60	68.88	69.14	70.00	73.11	71.20	71.82	74.50	60.13	67.55	77.71	86.84	71.94	70.64	70.57	78.03	68.88	74.84	74.35	67.24	72.85	74.88	67.62	75.84	75.59	78.10	70.03	72.79	76.65	62.02	76.28	84.04	73.24	61.69	69.23	66.81	56.52	92.31	78.79	65.12	71.05	57.50	90.00	30.00
24	GPT-OSS 120B (Low reasoning)	✔️	120.0B	71.23	71.23	72.51	69.14	69.19	75.53	66.80	68.68	79.00	78.43	65.49	74.52	74.34	75.52	74.01	74.37	73.11	67.22	71.57	76.15	64.94	70.36	71.50	65.08	64.33	75.12	82.38	69.21	75.74	74.01	69.79	71.15	80.00	65.89	63.31	68.08	73.53	60.87	100	84.85	69.77	65.79	63.33	80.00	50.00
25	Gemma 3 12B IT	❌	12.2B	70.94	70.94	65.86	65.62	72.43	76.44	70.40	73.70	76.00	67.97	66.67	75.16	76.32	72.24	75.84	71.84	72.13	72.20	68.30	73.55	62.07	71.47	71.98	66.03	70.22	73.24	77.14	73.30	73.53	71.81	64.93	75.64	82.11	71.57	60.89	65.77	63.03	65.22	76.92	87.88	69.77	73.68	65.83	80.00	80.00
26	GPT-OSS 20B	✔️	20.0B	69.65	69.63	68.58	61.33	72.97	78.25	65.60	71.19	76.50	79.74	69.91	75.16	67.76	71.64	72.17	75.24	69.51	65.98	75.16	72.09	70.69	70.08	65.70	59.05	62.36	71.60	80.00	66.49	76.47	71.37	66.67	69.87	76.84	66.22	62.75	57.31	68.91	52.17	53.85	87.88	69.77	71.05	66.67	70.00	40.00
27	Qwen3-14B	❌	14.8B	69.58	69.58	64.95	62.89	69.73	70.69	65.20	72.03	77.00	61.44	64.31	75.80	76.32	73.73	71.56	72.78	72.79	69.71	72.55	70.74	63.79	68.42	68.60	65.40	70.79	76.76	81.43	66.21	76.47	73.13	65.97	69.87	81.05	62.21	62.50	66.92	65.97	56.52	92.31	87.88	62.79	60.53	56.67	90.00	70.00
28	Mistral Small 3.1 24B Instruct	❌	24.0B	68.94	68.94	67.07	63.67	69.46	70.39	66.40	68.27	75.50	63.40	63.42	74.52	76.97	70.15	71.25	73.42	72.79	63.07	76.47	71.74	58.05	68.14	68.12	61.59	69.94	74.41	83.33	65.94	70.59	72.69	65.28	70.51	85.26	67.89	58.87	66.92	66.81	43.48	69.23	75.76	65.12	68.42	55.00	80.00	40.00
29	Qwen3-30B-A3B	❌	30.5B	68.80	68.80	64.65	66.41	72.43	73.11	66.80	67.64	76.50	70.59	65.19	80.25	73.68	71.34	70.03	70.25	68.20	70.12	71.90	67.54	63.22	67.04	65.22	61.27	73.88	73.00	79.05	69.21	72.79	68.72	66.32	68.59	80.00	61.20	61.29	65.00	65.97	47.83	76.92	87.88	65.12	65.79	60.00	80.00	40.00
30	GPT-OSS 20B (Low reasoning)	✔️	20.0B	64.53	64.53	61.93	56.64	66.22	71.90	58.80	62.63	71.00	71.24	62.24	69.43	59.87	70.45	62.69	69.30	61.97	63.49	68.63	66.93	60.92	60.39	63.29	60.95	60.67	67.84	75.71	64.85	72.79	70.04	62.50	70.51	72.63	60.54	56.85	55.38	63.45	52.17	61.54	81.82	65.12	63.16	59.17	70.00	50.00
31	Aya Expanse 32B	❌	32.3B	63.27	5.85	62.24	57.03	66.22	64.35	63.20	63.47	65.00	56.21	61.95	70.70	67.76	62.09	62.69	65.19	65.57	66.39	66.67	62.73	62.07	66.76	62.32	59.05	67.98	59.15	73.33	64.03	61.03	70.48	54.17	62.18	76.84	62.54	59.27	62.31	56.72	52.17	61.54	75.76	62.79	65.79	50.83	80.00	60.00
32	Command R v01	❌	35.0B	60.00	60.00	60.42	56.64	57.30	59.52	60.00	59.08	60.50	58.82	57.23	63.69	65.79	57.31	62.39	61.39	62.95	65.15	63.07	61.52	57.47	59.00	53.14	56.19	62.92	59.15	65.71	62.67	63.97	68.72	48.96	62.18	67.37	63.55	57.26	61.15	56.72	30.43	69.23	60.61	60.47	47.37	50.00	80.00	60.00
33	Gemma 3n E4B IT	❌	7.8B	59.15	59.15	55.29	57.81	59.46	59.82	58.80	62.84	61.50	60.13	55.16	62.42	62.50	61.19	60.86	62.97	60.66	61.41	60.46	61.32	62.64	57.34	61.35	54.29	57.58	58.22	64.76	60.49	63.97	55.95	54.51	58.97	68.42	60.20	48.79	61.54	54.20	60.87	61.54	66.67	58.14	60.53	42.50	60.00	40.00
34	GPT-4.1 Nano	❌	Unknown	54.94	54.94	53.47	49.22	58.65	59.21	55.60	54.70	60.00	52.29	50.44	63.06	66.45	55.82	51.07	59.49	56.39	51.04	60.78	56.11	59.20	55.68	52.66	49.21	54.78	58.22	63.81	54.50	61.03	57.27	50.00	50.00	62.11	48.16	48.39	56.92	50.42	34.78	53.85	57.58	48.84	47.37	40.00	70.00	40.00
35	Qwen3-8B	❌	8.2B	54.37	54.37	49.55	51.56	52.70	56.50	50.80	54.70	60.50	48.37	50.15	65.61	57.24	61.19	56.27	56.96	55.41	56.43	60.46	57.11	51.72	51.52	55.56	47.62	59.27	59.39	67.62	47.41	54.41	56.83	51.04	53.21	65.26	46.49	44.76	51.54	56.30	30.43	69.23	57.58	51.16	57.89	39.17	70.00	40.00
36	DeepSeek-R1 Qwen3-8B	✔️	8.2B	52.73	52.72	46.53	52.16	55.68	56.50	50.00	52.61	58.50	56.86	49.56	62.42	58.94	56.29	53.52	56.96	52.46	50.21	54.25	51.30	54.02	53.19	53.88	46.67	55.62	56.81	60.00	54.77	44.85	53.74	47.57	48.72	61.05	45.82	43.32	51.15	51.68	43.48	38.46	66.67	51.16	52.63	45.00	70.00	30.00
37	GPT-5 Nano (Minimal reasoning)	✔️	Unknown	51.94	51.93	53.78	47.84	56.10	54.08	45.60	52.09	55.00	55.56	48.97	53.50	60.53	52.84	54.13	53.16	53.29	53.94	56.54	52.10	44.83	51.25	49.28	46.98	50.56	55.63	58.10	49.05	60.29	55.95	48.26	45.81	60.00	48.32	47.18	52.51	50.84	39.13	61.54	54.55	39.53	55.26	40.83	50.00	40.00
38	Qwen3-4B	❌	4.0B	50.33	50.33	49.55	48.05	48.11	52.57	50.40	50.73	61.50	49.02	46.02	54.14	51.97	57.31	55.66	50.95	46.56	51.45	50.98	52.30	51.15	47.65	47.83	42.54	50.28	53.29	60.48	47.96	61.03	55.07	48.96	50.64	58.95	39.13	43.95	48.08	55.46	34.78	53.85	54.55	48.84	52.63	30.83	70.00	10.00
39	Gemma 3 4B IT	❌	4.3B	48.32	48.32	49.85	44.14	44.86	50.76	50.40	48.23	50.00	41.18	52.51	50.32	51.97	49.85	44.95	50.63	53.44	46.06	50.33	47.29	44.83	49.03	47.34	41.90	48.31	53.05	56.67	49.32	52.94	48.02	44.79	42.31	64.21	45.15	40.73	50.00	44.54	52.17	53.85	63.64	46.51	47.37	35.00	90.00	50.00
40	Mistral 7B Instruct v0.3	❌	7.2B	30.15	30.11	29.91	29.30	28.65	28.40	27.60	29.23	34.50	32.68	32.15	25.48	28.95	33.13	30.58	35.13	26.23	32.78	32.35	32.06	28.16	29.92	31.88	26.98	31.74	30.28	33.81	28.88	33.09	32.60	29.17	23.72	40.00	22.41	24.60	33.46	33.61	30.43	46.15	27.27	44.19	15.79	20.83	70.00	10.00
41	Gemma 3 1B IT	❌	1.0B	25.99	25.99	27.19	25.00	25.68	32.02	27.60	26.30	24.50	24.18	29.79	24.84	26.32	25.07	24.77	25.95	28.52	24.90	27.12	27.05	27.59	26.04	26.09	26.35	27.25	25.12	28.57	20.98	27.21	24.67	22.22	25.00	29.47	24.41	25.00	26.92	23.53	30.43	38.46	15.15	25.58	26.32	19.17	10.00	40.00
42	Llama 3.2 3B Instruct	❌	3.2B	25.21	25.17	27.79	29.30	25.95	22.36	20.00	24.22	24.00	23.53	26.84	28.03	23.68	22.99	25.99	26.90	23.28	26.97	21.24	24.65	25.86	26.32	28.99	26.03	25.56	28.87	22.38	25.61	25.00	22.03	25.00	26.92	28.42	21.07	23.39	23.08	32.35	17.39	46.15	33.33	27.91	23.68	18.33	30.00	50.00
43	Gemma 3 270M IT	❌	268.0M	24.84	24.39	25.68	24.61	24.86	29.91	27.20	23.17	24.00	25.49	24.48	28.03	25.00	23.28	20.49	25.32	30.16	21.16	26.80	24.05	26.44	27.15	19.32	23.81	26.97	24.88	22.38	23.16	21.32	26.43	29.17	24.36	26.32	25.75	24.60	25.77	17.65	26.09	23.08	24.24	23.26	21.05	24.17	20.00	20.00
44	Llama 3.2 1B Instruct	❌	1.2B	24.12	0.79	25.38	25.39	24.86	28.10	24.80	21.92	24.50	24.84	25.07	28.03	24.34	22.09	21.41	23.73	29.84	22.82	25.82	22.24	27.01	25.21	18.84	23.49	26.97	25.59	17.62	23.43	22.79	29.07	28.12	19.87	27.37	23.08	20.97	22.69	18.49	21.74	15.38	15.15	27.91	21.05	22.50	20.00	40.00