🇮🇷 MIZAN: A Persian LLM Leaderboard

MIZAN: A Persian LLM Leaderboard is a comprehensive benchmark for evaluating Large Language Models (LLMs) in Persian. It combines existing datasets, translated benchmarks, and new Persian-specific data to assess LLM capabilities in understanding, generation, reasoning, and knowledge relevant to the Persian language and culture. MIZAN provides a standardized tool for researchers and developers to measure Persian LLM performance.

Search by Model Name

Filter by Model Source

All Open-Source Closed-Source

🏆 Overall Benchmark


10	Mistral Small 3.1 24B Instruct	Unknown	71.27	88.11	87.33	78.08	91.18	71.47	20.11


1	Claude 3.7 Sonnet	Unknown	71.27	88.11	86.95	72.15	91.18	71.43	17.79
2	GPT-4.1	Unknown	69.92	86.34	87.33	70.40	88.39	67.58	19.47
3	Gemini 2.0 Flash	Unknown	68.86	84.98	83.90	69.58	86.38	70.51	17.82
4	GPT-4o	Unknown	68.77	82.97	83.72	68.85	86.65	71.47	18.96
5	DeepSeek-R1	671.0B	67.15	83.70	86.17	78.08	82.52	63.61	8.81
6	Gemini 2.0 Flash-Lite	Unknown	66.40	84.14	82.44	64.67	81.43	69.14	16.59
7	GPT-4.1 Mini	Unknown	65.56	83.41	84.18	61.29	77.12	68.33	19.01
8	DeepSeek-V3	671.0B	64.58	83.11	86.01	59.08	82.41	67.53	9.34
9	Gemma 3 27B IT	27.4B	62.47	82.97	79.60	56.33	76.28	68.98	10.67
10	GPT-4o Mini	Unknown	62.46	80.18	78.92	56.99	75.98	64.59	18.11
11	Qwen3-32B	32.8B	62.24	80.30	76.33	56.35	76.54	67.14	16.79
12	Llama 3.3 70B Instruct	70.6B	61.30	71.26	71.73	57.14	79.56	68.00	20.11
13	Command A	111.0B	60.98	84.39	82.19	59.81	79.89	48.25	11.38
14	Gemma 3 12B IT	12.2B	60.08	81.50	75.13	51.05	70.94	69.91	11.97
15	Qwen3-30B-A3B	30.5B	59.39	83.26	74.31	50.98	68.80	62.56	16.41
16	Qwen3-14B	14.8B	59.12	81.06	72.05	51.39	69.58	64.60	16.06
17	Command R Plus	104.0B	57.05	70.07	68.80	48.01	73.64	62.98	18.80
18	Mistral Small 3.1 24B Instruct	24.0B	55.76	75.27	72.91	47.63	68.94	56.62	13.19
19	Claude 3.5 Haiku	Unknown	55.46	82.53	75.85	54.40	71.60	37.49	10.89
20	GPT-4.1 Nano	Unknown	55.24	75.77	73.63	47.84	54.94	62.62	16.66
21	Qwen3-8B	8.2B	52.52	74.74	66.08	44.68	54.37	59.68	15.57
22	Gemma 3 4B IT	4.3B	49.96	74.45	66.83	38.26	48.32	62.42	9.50
23	Aya Expanse 32B	32.3B	49.45	69.90	70.86	41.41	63.27	39.29	11.96
24	Command R v01	35.0B	48.13	57.90	60.91	40.47	60.00	53.10	16.42
25	Qwen3-4B	4.0B	47.91	75.77	55.99	40.25	50.33	51.21	13.89
26	Gemma 3 1B IT	1.0B	32.52	54.48	43.33	28.30	25.99	36.20	6.83
27	Mistral 7B Instruct v0.3	7.2B	30.39	44.05	33.98	25.53	30.15	39.17	9.44
28	Llama 3.2 3B Instruct	3.2B	28.15	53.30	37.56	27.84	25.21	13.69	11.30
29	Llama 3.2 1B Instruct	1.2B	20.50	36.56	29.52	19.87	24.12	4.68	8.23
	OpenAI o4-mini	Unknown					85.51

Persian MT-Bench


10	Mistral Small 3.1 24B Instruct	Unknown	8.74	8.73	8.41	8.38	8.61	8.62	9.07	8.97	9.78	9.75	8.15	8.15	8.45	8.45	8.70	8.70	9.10	9.10	7.78	7.93	9.30	9.27

Persian IFEval


10	Mistral Small 3.1 24B Instruct	Unknown	83.37	88.11	87.75	91.48	83.08	97.83	87.76	89.66	96.77	75.61	95.08	95.24	84.62	97.83	87.76	94.48	96.77	85.37	98.36	96.83

PerMMLU


10	Mistral Small 3.1 24B Instruct	Unknown	78.08	42.50	60.00	65.00	55.00	67.50	84.76	62.50	90.00	78.97	89.13	76.00	93.33	84.27	72.00	47.62	72.68	78.00	87.58

PerCoR


10	Mistral Small 3.1 24B Instruct	Unknown	91.18	91.18	91.84	90.63	90.27	92.45	93.60	91.84	89.00	87.58	87.02	93.63	96.71	92.24	92.35	91.46	94.41	92.53	91.18	91.98	90.80	91.69	88.89	88.25	93.82	93.41	94.76	88.83	91.18	91.63	82.58	93.59	96.84	93.98	92.34	91.15	85.71	86.96	92.31	96.97	86.05	78.95	83.33	90.00	80.00

Persian NLG


10	Mistral Small 3.1 24B Instruct	Unknown	24.47	33.88	26.93	20.11


1	Llama 3.3 70B Instruct	70.6B	24.47	33.88	26.93	20.11
2	GPT-4.1	Unknown	25.76	39.25	29.86	19.47
3	GPT-4.1 Mini	Unknown	24.39	36.87	28.16	19.01
4	GPT-4o	Unknown	22.76	36.55	26.79	18.96
5	Command R Plus	104.0B	21.50	30.20	24.05	18.80
6	GPT-4o Mini	Unknown	19.43	34.36	23.69	18.11
7	Gemini 2.0 Flash	Unknown	31.41	41.03	34.42	17.82
8	Claude 3.7 Sonnet	Unknown	16.93	38.86	22.73	17.79
9	Qwen3-32B	32.8B	22.81	37.09	27.34	16.79
10	GPT-4.1 Nano	Unknown	16.22	30.26	20.21	16.66
11	Gemini 2.0 Flash-Lite	Unknown	27.82	38.23	31.10	16.59
12	Command R v01	35.0B	15.46	25.34	18.29	16.42
13	Qwen3-30B-A3B	30.5B	19.97	33.93	24.17	16.41
14	Qwen3-14B	14.8B	19.87	34.31	24.19	16.06
15	Qwen3-8B	8.2B	19.65	34.08	23.89	15.57
16	Qwen3-4B	4.0B	17.44	26.97	20.18	13.89
17	Mistral Small 3.1 24B Instruct	24.0B	15.21	26.32	18.43	13.19
18	Gemma 3 12B IT	12.2B	25.73	37.40	29.28	11.97
19	Aya Expanse 32B	32.3B	9.80	34.80	14.44	11.96
20	Command A	111.0B	21.72	39.39	26.84	11.38
21	Llama 3.2 3B Instruct	3.2B	12.64	25.83	16.01	11.30
22	Claude 3.5 Haiku	Unknown	8.71	36.01	13.36	10.89
23	Gemma 3 27B IT	27.4B	20.42	34.57	24.60	10.67
24	Gemma 3 4B IT	4.3B	17.04	30.01	20.79	9.50
25	Mistral 7B Instruct v0.3	7.2B	11.27	29.83	15.26	9.44
26	DeepSeek-V3	671.0B	19.39	40.71	24.40	9.34
27	DeepSeek-R1	671.0B	13.42	41.00	18.41	8.81
28	Llama 3.2 1B Instruct	1.2B	5.68	21.06	7.93	8.23
29	Gemma 3 1B IT	1.0B	4.57	9.92	5.76	6.83
	OpenAI o4-mini	Unknown


10	Mistral Small 3.1 24B Instruct	Unknown	20.19	47.53	16.57	11.94	10.95	5.46	7.83	20.11


1	Llama 3.3 70B Instruct	70.6B	20.19	47.53	16.57	11.94	8.19	5.46	7.83	20.11
2	GPT-4.1	Unknown	19.74	41.03	18.99	16.38	10.95	4.87	7.37	19.47
3	GPT-4.1 Mini	Unknown	19.79	44.61	17.45	13.74	9.16	4.90	7.28	19.01
4	GPT-4o	Unknown	21.00	48.06	19.05	14.12	8.61	5.29	6.89	18.96
5	Command R Plus	104.0B	20.18	43.33	19.25	15.31	10.26	5.20	7.08	18.80
6	GPT-4o Mini	Unknown	20.14	47.11	18.31	11.42	8.06	5.04	6.49	18.11
7	Gemini 2.0 Flash	Unknown	17.18	36.48	15.33	16.21	9.68	4.58	5.76	17.82
8	Claude 3.7 Sonnet	Unknown	18.00	40.04	17.03	12.26	8.33	4.39	5.95	17.79
9	Qwen3-32B	32.8B	16.23	36.78	14.76	9.38	6.99	4.35	6.20	16.79
10	GPT-4.1 Nano	Unknown	18.32	40.52	16.93	14.00	8.13	4.81	6.11	16.66
11	Gemini 2.0 Flash-Lite	Unknown	17.50	37.41	15.93	13.26	10.03	5.01	6.52	16.59
12	Command R v01	35.0B	15.07	29.51	15.01	11.04	9.22	4.98	6.93	16.42
13	Qwen3-30B-A3B	30.5B	16.18	38.22	13.38	8.61	7.71	4.42	5.87	16.41
14	Qwen3-14B	14.8B	15.35	35.54	12.85	8.58	7.87	4.05	5.86	16.06
15	Qwen3-8B	8.2B	15.50	37.22	12.32	8.82	7.25	4.24	5.29	15.57
16	Qwen3-4B	4.0B	12.17	29.16	9.19	6.55	6.12	3.86	4.54	13.89
17	Mistral Small 3.1 24B Instruct	24.0B	14.58	35.42	12.59	8.87	6.73	4.07	2.93	13.19
18	Gemma 3 12B IT	12.2B	8.82	18.87	8.80	6.58	2.96	1.92	3.66	11.97
19	Aya Expanse 32B	32.3B	9.51	22.04	7.72	9.14	5.56	2.96	1.46	11.96
20	Command A	111.0B	4.35	8.65	3.72	5.81	3.11	1.45	1.05	11.38
21	Llama 3.2 3B Instruct	3.2B	10.74	26.90	7.84	5.73	5.35	3.74	2.79	11.30
22	Claude 3.5 Haiku	Unknown	5.64	11.54	4.84	6.12	4.67	2.18	1.18	10.89
23	Gemma 3 27B IT	27.4B	4.40	9.33	4.46	4.36	1.97	1.33	0.87	10.67
24	Gemma 3 4B IT	4.3B	3.67	6.23	4.43	3.09	3.31	1.25	1.17	9.50
25	Mistral 7B Instruct v0.3	7.2B	4.73	9.51	3.48	4.17	4.42	2.39	1.89	9.44
26	DeepSeek-V3	671.0B	4.39	7.15	5.95	4.28	2.58	1.34	1.43	9.34
27	DeepSeek-R1	671.0B	4.09	9.02	3.28	4.44	2.79	1.48	0.71	8.81
28	Llama 3.2 1B Instruct	1.2B	3.74	7.74	3.42	2.58	2.40	1.50	1.46	8.23
29	Gemma 3 1B IT	1.0B	4.61	13.10	3.17	1.34	1.41	1.28	0.65	6.83
	OpenAI o4-mini	Unknown


10	Mistral Small 3.1 24B Instruct	Unknown	25.59	16.87	21.14	20.08	50.99	9.84	11.26	20.11


1	Llama 3.3 70B Instruct	70.6B	25.59	16.87	21.14	20.08	50.99	9.84	11.26	20.11
2	GPT-4.1	Unknown	23.07	15.28	19.27	16.28	46.76	8.10	10.09	19.47
3	GPT-4.1 Mini	Unknown	23.89	14.32	20.57	17.76	48.42	8.86	10.45	19.01
4	GPT-4o	Unknown	23.40	15.98	19.47	16.39	47.48	8.25	9.53	18.96
5	Command R Plus	104.0B	23.38	13.86	21.30	17.02	47.82	8.30	7.20	18.80
6	GPT-4o Mini	Unknown	23.33	14.98	19.72	17.26	46.79	9.05	10.08	18.11
7	Gemini 2.0 Flash	Unknown	7.57	3.17	5.30	10.28	15.74	3.36	2.79	17.82
8	Claude 3.7 Sonnet	Unknown	22.48	13.42	19.09	17.41	45.44	8.77	9.76	17.79
9	Qwen3-32B	32.8B	21.80	12.72	17.28	15.89	45.73	9.02	9.45	16.79
10	GPT-4.1 Nano	Unknown	21.66	13.49	18.11	16.42	43.84	9.43	8.28	16.66
11	Gemini 2.0 Flash-Lite	Unknown	8.58	3.82	7.50	9.86	15.14	5.68	5.71	16.59
12	Command R v01	35.0B	18.92	12.91	17.21	17.37	34.61	7.76	8.28	16.42
13	Qwen3-30B-A3B	30.5B	21.78	11.90	17.94	17.18	45.00	8.37	10.34	16.41
14	Qwen3-14B	14.8B	21.45	13.07	16.98	15.52	45.14	8.43	8.54	16.06
15	Qwen3-8B	8.2B	20.24	11.63	16.49	15.13	41.71	8.58	9.29	15.57
16	Qwen3-4B	4.0B	18.41	10.11	14.92	13.78	38.03	8.52	8.57	13.89
17	Mistral Small 3.1 24B Instruct	24.0B	14.51	3.93	10.09	9.30	36.61	5.37	5.04	13.19
18	Gemma 3 12B IT	12.2B	2.57	1.16	2.15	2.62	4.72	1.79	1.70	11.97
19	Aya Expanse 32B	32.3B	14.44	6.37	10.46	10.65	33.32	5.73	5.26	11.96
20	Command A	111.0B	4.86	2.32	3.97	4.72	9.72	2.47	2.39	11.38
21	Llama 3.2 3B Instruct	3.2B	8.62	3.34	7.59	8.92	16.89	4.28	4.73	11.30
22	Claude 3.5 Haiku	Unknown	6.91	3.21	5.35	8.00	13.40	3.63	3.93	10.89
23	Gemma 3 27B IT	27.4B	2.00	0.98	1.45	2.60	3.45	1.51	1.57	10.67
24	Gemma 3 4B IT	4.3B	1.69	0.85	1.37	1.82	3.01	1.22	1.10	9.50
25	Mistral 7B Instruct v0.3	7.2B	9.02	5.22	8.29	7.56	16.46	4.86	5.19	9.44
26	DeepSeek-V3	671.0B	4.50	1.66	4.78	4.26	8.02	2.53	2.69	9.34
27	DeepSeek-R1	671.0B	4.14	1.95	3.46	3.97	7.98	2.44	2.32	8.81
28	Llama 3.2 1B Instruct	1.2B	4.23	1.25	3.14	2.95	10.07	1.18	2.47	8.23
29	Gemma 3 1B IT	1.0B	1.46	0.65	1.19	1.52	2.74	0.94	0.94	6.83
	OpenAI o4-mini	Unknown


10	Mistral Small 3.1 24B Instruct	Unknown	13.02	11.05	7.42	20.32	20.11


1	Llama 3.3 70B Instruct	70.6B	13.02	11.05	7.42	20.32	20.11
2	GPT-4.1	Unknown	15.34	12.50	6.24	26.82	19.47
3	GPT-4.1 Mini	Unknown	14.35	12.21	6.95	23.52	19.01
4	GPT-4o	Unknown	15.43	12.84	6.60	26.39	18.96
5	Command R Plus	104.0B	14.14	13.64	6.53	21.87	18.80
6	GPT-4o Mini	Unknown	13.98	12.73	6.58	22.24	18.11
7	Gemini 2.0 Flash	Unknown	15.66	11.23	6.29	29.00	17.82
8	Claude 3.7 Sonnet	Unknown	14.72	12.94	6.43	24.37	17.79
9	Qwen3-32B	32.8B	9.86	8.58	6.23	14.59	16.79
10	GPT-4.1 Nano	Unknown	13.15	10.64	6.42	22.06	16.66
11	Gemini 2.0 Flash-Lite	Unknown	14.30	13.26	5.32	23.88	16.59
12	Command R v01	35.0B	11.45	11.91	6.48	15.71	16.42
13	Qwen3-30B-A3B	30.5B	9.04	7.79	6.10	13.09	16.41
14	Qwen3-14B	14.8B	9.10	8.63	5.58	12.93	16.06
15	Qwen3-8B	8.2B	8.21	7.30	5.79	11.41	15.57
16	Qwen3-4B	4.0B	6.36	5.57	5.40	8.07	13.89
17	Mistral Small 3.1 24B Instruct	24.0B	5.95	5.40	3.52	8.79	13.19
18	Gemma 3 12B IT	12.2B	3.26	3.33	1.06	5.28	11.97
19	Aya Expanse 32B	32.3B	6.08	6.37	2.59	9.13	11.96
20	Command A	111.0B	3.97	3.55	1.61	6.62	11.38
21	Llama 3.2 3B Instruct	3.2B	3.82	5.18	2.35	3.85	11.30
22	Claude 3.5 Haiku	Unknown	5.82	5.40	2.33	9.55	10.89
23	Gemma 3 27B IT	27.4B	4.66	2.38	1.21	10.21	10.67
24	Gemma 3 4B IT	4.3B	1.87	1.75	0.98	2.84	9.50
25	Mistral 7B Instruct v0.3	7.2B	3.53	3.18	2.19	5.13	9.44
26	DeepSeek-V3	671.0B	3.03	2.72	1.29	4.98	9.34
27	DeepSeek-R1	671.0B	3.35	3.14	1.39	5.43	8.81
28	Llama 3.2 1B Instruct	1.2B	0.56	0.85	0.73	0.13	8.23
29	Gemma 3 1B IT	1.0B	1.01	1.16	0.68	1.17	6.83
	OpenAI o4-mini	Unknown


10	Mistral Small 3.1 24B Instruct	Unknown	8.60	4.41	8.34	13.07	20.11


1	Llama 3.3 70B Instruct	70.6B	8.60	4.41	8.34	13.07	20.11
2	GPT-4.1	Unknown	8.28	3.84	7.81	13.17	19.47
3	GPT-4.1 Mini	Unknown	6.90	3.97	7.51	9.22	19.01
4	GPT-4o	Unknown	6.57	3.47	7.32	8.91	18.96
5	Command R Plus	104.0B	9.23	5.11	5.90	16.68	18.80
6	GPT-4o Mini	Unknown	6.53	3.73	6.89	8.96	18.11
7	Gemini 2.0 Flash	Unknown	7.93	3.38	5.71	14.69	17.82
8	Claude 3.7 Sonnet	Unknown	7.61	3.21	6.14	13.49	17.79
9	Qwen3-32B	32.8B	5.07	3.16	5.34	6.70	16.79
10	GPT-4.1 Nano	Unknown	5.68	3.27	5.79	7.99	16.66
11	Gemini 2.0 Flash-Lite	Unknown	3.42	1.98	4.19	4.07	16.59
12	Command R v01	35.0B	8.53	4.64	7.13	13.81	16.42
13	Qwen3-30B-A3B	30.5B	4.83	3.10	5.12	6.27	16.41
14	Qwen3-14B	14.8B	4.94	3.70	5.68	5.46	16.06
15	Qwen3-8B	8.2B	4.23	3.29	4.45	4.96	15.57
16	Qwen3-4B	4.0B	3.51	3.14	4.21	3.20	13.89
17	Mistral Small 3.1 24B Instruct	24.0B	1.92	1.51	2.46	1.80	13.19
18	Gemma 3 12B IT	12.2B	0.98	0.72	1.11	1.12	11.97
19	Aya Expanse 32B	32.3B	3.19	2.23	2.97	4.37	11.96
20	Command A	111.0B	2.36	1.24	1.99	3.84	11.38
21	Llama 3.2 3B Instruct	3.2B	1.37	1.10	1.35	1.64	11.30
22	Claude 3.5 Haiku	Unknown	3.31	2.02	2.81	5.11	10.89
23	Gemma 3 27B IT	27.4B	1.01	0.72	1.01	1.29	10.67
24	Gemma 3 4B IT	4.3B	0.68	0.57	0.90	0.57	9.50
25	Mistral 7B Instruct v0.3	7.2B	1.67	1.82	2.04	1.16	9.44
26	DeepSeek-V3	671.0B	1.99	1.14	1.55	3.27	9.34
27	DeepSeek-R1	671.0B	1.98	1.12	1.59	3.25	8.81
28	Llama 3.2 1B Instruct	1.2B	1.64	1.53	2.20	1.21	8.23
29	Gemma 3 1B IT	1.0B	0.45	0.46	0.52	0.37	6.83
	OpenAI o4-mini	Unknown


10	Mistral Small 3.1 24B Instruct	Unknown	16.32	38.75	21.58	20.11


1	Llama 3.3 70B Instruct	70.6B	16.32	38.75	21.58	20.11
2	GPT-4.1	Unknown	16.81	35.68	21.90	19.47
3	GPT-4.1 Mini	Unknown	17.15	36.93	22.49	19.01
4	GPT-4o	Unknown	16.51	39.82	22.40	18.96
5	Command R Plus	104.0B	15.91	41.63	22.09	18.80
6	GPT-4o Mini	Unknown	15.78	39.02	21.56	18.11
7	Gemini 2.0 Flash	Unknown	18.09	41.45	24.07	17.82
8	Claude 3.7 Sonnet	Unknown	15.39	38.50	21.16	17.79
9	Qwen3-32B	32.8B	15.15	36.83	20.63	16.79
10	GPT-4.1 Nano	Unknown	16.18	34.77	20.98	16.66
11	Gemini 2.0 Flash-Lite	Unknown	18.10	40.16	23.81	16.59
12	Command R v01	35.0B	19.44	37.61	24.26	16.42
13	Qwen3-30B-A3B	30.5B	15.24	38.39	20.84	16.41
14	Qwen3-14B	14.8B	14.69	37.44	20.23	16.06
15	Qwen3-8B	8.2B	14.63	38.56	20.24	15.57
16	Qwen3-4B	4.0B	14.30	39.77	20.13	13.89
17	Mistral Small 3.1 24B Instruct	24.0B	14.73	40.67	20.72	13.19
18	Gemma 3 12B IT	12.2B	16.58	36.78	21.89	11.97
19	Aya Expanse 32B	32.3B	13.38	39.79	19.33	11.96
20	Command A	111.0B	14.13	40.17	20.05	11.38
21	Llama 3.2 3B Instruct	3.2B	13.86	31.34	18.19	11.30
22	Claude 3.5 Haiku	Unknown	17.73	34.16	22.33	10.89
23	Gemma 3 27B IT	27.4B	15.88	37.36	21.32	10.67
24	Gemma 3 4B IT	4.3B	16.38	35.36	21.35	9.50
25	Mistral 7B Instruct v0.3	7.2B	11.81	39.23	17.08	9.44
26	DeepSeek-V3	671.0B	8.93	37.02	13.92	9.34
27	DeepSeek-R1	671.0B	10.82	37.27	16.07	8.81
28	Llama 3.2 1B Instruct	1.2B	15.66	26.42	17.60	8.23
29	Gemma 3 1B IT	1.0B	13.40	31.84	17.91	6.83
	OpenAI o4-mini	Unknown


10	Mistral Small 3.1 24B Instruct	Unknown	22.38	35.33	24.85	20.11


1	Llama 3.3 70B Instruct	70.6B	22.38	35.33	24.85	20.11
2	GPT-4.1	Unknown	12.35	37.61	18.09	19.47
3	GPT-4.1 Mini	Unknown	11.56	40.33	17.50	19.01
4	GPT-4o	Unknown	11.35	39.10	17.17	18.96
5	Command R Plus	104.0B	12.27	43.15	18.57	18.80
6	GPT-4o Mini	Unknown	11.60	39.80	17.52	18.11
7	Gemini 2.0 Flash	Unknown	12.28	37.51	17.93	17.82
8	Claude 3.7 Sonnet	Unknown	11.88	39.48	17.87	17.79
9	Qwen3-32B	32.8B	10.90	39.36	16.63	16.79
10	GPT-4.1 Nano	Unknown	11.21	36.10	16.61	16.66
11	Gemini 2.0 Flash-Lite	Unknown	11.91	36.54	17.44	16.59
12	Command R v01	35.0B	12.98	35.49	18.42	16.42
13	Qwen3-30B-A3B	30.5B	12.13	39.23	18.05	16.41
14	Qwen3-14B	14.8B	11.24	40.32	17.12	16.06
15	Qwen3-8B	8.2B	10.93	39.79	16.69	15.57
16	Qwen3-4B	4.0B	10.67	41.09	16.48	13.89
17	Mistral Small 3.1 24B Instruct	24.0B	10.29	43.48	16.21	13.19
18	Gemma 3 12B IT	12.2B	11.57	36.31	16.97	11.97
19	Aya Expanse 32B	32.3B	10.95	39.79	16.75	11.96
20	Command A	111.0B	11.17	42.09	17.23	11.38
21	Llama 3.2 3B Instruct	3.2B	15.15	37.76	20.34	11.30
22	Claude 3.5 Haiku	Unknown	13.29	35.79	18.87	10.89
23	Gemma 3 27B IT	27.4B	11.25	37.17	16.73	10.67
24	Gemma 3 4B IT	4.3B	10.96	37.35	16.45	9.50
25	Mistral 7B Instruct v0.3	7.2B	9.22	44.02	14.81	9.44
26	DeepSeek-V3	671.0B	8.11	39.40	13.16	9.34
27	DeepSeek-R1	671.0B	8.49	38.21	13.60	8.81
28	Llama 3.2 1B Instruct	1.2B	20.61	27.55	21.92	8.23
29	Gemma 3 1B IT	1.0B	11.14	37.32	16.61	6.83
	OpenAI o4-mini	Unknown

Persian NLU


10	Mistral Small 3.1 24B Instruct	Unknown	79.88	75.80	81.08	77.58	79.88	75.80	81.08	77.58	99.95	71.47


1	GPT-4o	Unknown	79.88	75.80	81.08	77.58	79.88	75.80	81.08	77.58	100	71.47
2	Claude 3.7 Sonnet	Unknown	80.58	77.18	82.12	78.89	80.63	77.22	82.16	78.93	99.95	71.43
3	Gemini 2.0 Flash	Unknown	77.40	72.96	79.69	74.93	77.44	73.00	79.73	74.97	99.95	70.51
4	Gemma 3 12B IT	12.2B	76.65	72.36	78.57	74.40	76.69	72.40	78.61	74.44	99.95	69.91
5	Gemini 2.0 Flash-Lite	Unknown	76.11	72.33	79.39	72.66	76.23	72.45	79.52	72.78	99.84	69.14
6	Gemma 3 27B IT	27.4B	75.57	71.01	79.63	71.98	75.57	71.01	79.63	71.98	100	68.98
7	GPT-4.1 Mini	Unknown	78.69	74.86	81.11	76.69	78.69	74.86	81.11	76.69	100	68.33
8	Llama 3.3 70B Instruct	70.6B	73.25	70.24	78.25	69.06	73.25	70.24	78.25	69.06	100	68.00
9	GPT-4.1	Unknown	80.47	76.62	80.90	78.30	80.47	76.62	80.90	78.30	100	67.58
10	DeepSeek-V3	671.0B	76.70	74.12	79.91	73.46	76.70	74.12	79.91	73.46	100	67.53
11	Qwen3-32B	32.8B	78.21	75.93	77.68	75.63	78.21	75.93	77.68	75.63	100	67.14
12	Qwen3-14B	14.8B	74.60	69.60	77.26	71.60	74.60	69.60	77.26	71.60	100	64.60
13	GPT-4o Mini	Unknown	77.13	72.89	80.37	75.32	77.13	72.89	80.37	75.32	100	64.59
14	DeepSeek-R1	671.0B	62.78	59.55	62.40	60.73	72.89	69.13	72.44	70.50	86.14	63.61
15	Command R Plus	104.0B	73.84	70.68	76.59	72.64	73.84	70.68	76.59	72.64	100	62.98
16	GPT-4.1 Nano	Unknown	74.49	71.98	77.73	70.36	74.49	71.98	77.73	70.36	100	62.62
17	Qwen3-30B-A3B	30.5B	75.03	71.62	76.49	71.65	75.03	71.62	76.49	71.65	100	62.56
18	Gemma 3 4B IT	4.3B	68.18	65.47	73.79	63.56	68.18	65.47	73.79	63.56	100	62.42
19	Qwen3-8B	8.2B	74.11	70.10	75.29	71.73	74.11	70.10	75.29	71.73	100	59.68
20	Mistral Small 3.1 24B Instruct	24.0B	59.49	66.33	67.40	59.71	59.59	66.44	67.51	59.81	99.84	56.62
21	Command R v01	35.0B	61.54	65.20	67.63	52.90	61.54	65.20	67.63	52.90	100	53.10
22	Qwen3-4B	4.0B	68.55	64.09	71.81	64.47	68.55	64.09	71.81	64.47	100	51.21
23	Command A	111.0B	80.47	76.92	81.26	78.42	80.47	76.92	81.26	78.42	100	48.25
24	Aya Expanse 32B	32.3B	76.43	72.87	78.01	74.34	79.52	75.82	81.16	77.34	96.12	39.29
25	Mistral 7B Instruct v0.3	7.2B	37.38	43.03	30.53	29.34	62.10	71.49	50.72	48.75	60.19	39.17
26	Claude 3.5 Haiku	Unknown	75.22	71.49	76.61	73.40	75.50	71.76	76.90	73.68	99.62	37.49
27	Gemma 3 1B IT	1.0B	52.00	49.08	50.48	44.58	57.18	53.97	55.51	49.02	90.94	36.20
28	Llama 3.2 3B Instruct	3.2B	31.50	60.11	47.00	31.36	31.55	60.21	47.08	31.41	99.84	13.69
29	Llama 3.2 1B Instruct	1.2B	7.87	13.71	12.81	7.23	21.07	36.68	34.28	19.34	37.38	4.68
	OpenAI o4-mini	Unknown


10	Mistral Small 3.1 24B Instruct	Unknown	95.87	95.87	98.40	71.47


1	GPT-4o	Unknown	95.87	95.87	100	71.47
2	Claude 3.7 Sonnet	Unknown	96.39	96.39	100	71.43
3	Gemini 2.0 Flash	Unknown	96.66	96.66	100	70.51
4	Gemma 3 12B IT	12.2B	95.02	95.02	100	69.91
5	Gemini 2.0 Flash-Lite	Unknown	95.51	95.51	100	69.14
6	Gemma 3 27B IT	27.4B	95.83	95.83	100	68.98
7	GPT-4.1 Mini	Unknown	96.22	96.22	100	68.33
8	Llama 3.3 70B Instruct	70.6B	94.10	94.10	100	68.00
9	GPT-4.1	Unknown	96.20	96.20	100	67.58
10	DeepSeek-V3	671.0B	95.51	95.51	100	67.53
11	Qwen3-32B	32.8B	96.18	96.18	100	67.14
12	Qwen3-14B	14.8B	96.04	96.04	100	64.60
13	GPT-4o Mini	Unknown	95.90	95.90	100	64.59
14	DeepSeek-R1	671.0B	95.81	95.81	100	63.61
15	Command R Plus	104.0B	94.46	94.46	100	62.98
16	GPT-4.1 Nano	Unknown	93.00	93.00	100	62.62
17	Qwen3-30B-A3B	30.5B	94.21	94.21	100	62.56
18	Gemma 3 4B IT	4.3B	91.99	91.99	100	62.42
19	Qwen3-8B	8.2B	95.55	95.55	100	59.68
20	Mistral Small 3.1 24B Instruct	24.0B	94.25	94.25	100	56.62
21	Command R v01	35.0B	90.09	90.09	100	53.10
22	Qwen3-4B	4.0B	91.75	93.24	98.40	51.21
23	Command A	111.0B	13.38	80.57	16.60	48.25
24	Aya Expanse 32B	32.3B	0	0	0	39.29
25	Mistral 7B Instruct v0.3	7.2B	44.33	73.63	60.20	39.17
26	Claude 3.5 Haiku	Unknown	32.18	87.92	36.60	37.49
27	Gemma 3 1B IT	1.0B	72.33	80.37	90.00	36.20
28	Llama 3.2 3B Instruct	3.2B	0	0	0	13.69
29	Llama 3.2 1B Instruct	1.2B	0	0	2.00	4.68
	OpenAI o4-mini	Unknown


10	Mistral Small 3.1 24B Instruct	Unknown	54.93	52.96	59.59	71.47


1	GPT-4o	Unknown	54.93	52.96	59.59	71.47
2	Claude 3.7 Sonnet	Unknown	57.83	55.84	62.50	71.43
3	Gemini 2.0 Flash	Unknown	58.38	56.21	63.49	70.51
4	Gemma 3 12B IT	12.2B	47.64	42.06	59.97	69.91
5	Gemini 2.0 Flash-Lite	Unknown	57.78	57.22	60.65	69.14
6	Gemma 3 27B IT	27.4B	50.91	47.20	58.99	68.98
7	GPT-4.1 Mini	Unknown	36.36	33.77	42.40	68.33
8	Llama 3.3 70B Instruct	70.6B	46.24	43.38	52.99	68.00
9	GPT-4.1	Unknown	30.98	28.33	37.11	67.58
10	DeepSeek-V3	671.0B	47.38	43.83	55.17	67.53
11	Qwen3-32B	32.8B	44.08	42.06	48.71	67.14
12	Qwen3-14B	14.8B	49.21	45.54	57.84	64.60
13	GPT-4o Mini	Unknown	3.74	3.43	4.49	64.59
14	DeepSeek-R1	671.0B	24.71	21.76	31.69	63.61
15	Command R Plus	104.0B	0.32	0.24	0.48	62.98
16	GPT-4.1 Nano	Unknown	45.21	40.48	56.40	62.62
17	Qwen3-30B-A3B	30.5B	50.00	46.08	59.27	62.56
18	Gemma 3 4B IT	4.3B	38.39	32.92	50.50	62.42
19	Qwen3-8B	8.2B	15.88	15.53	17.65	59.68
20	Mistral Small 3.1 24B Instruct	24.0B	2.82	2.78	3.04	56.62
21	Command R v01	35.0B	0	0	0	53.10
22	Qwen3-4B	4.0B	34.27	32.83	39.51	51.21
23	Command A	111.0B	49.28	45.31	58.07	48.25
24	Aya Expanse 32B	32.3B	36.99	32.51	47.85	39.29
25	Mistral 7B Instruct v0.3	7.2B	16.26	15.82	18.85	39.17
26	Claude 3.5 Haiku	Unknown	1.34	1.32	1.48	37.49
27	Gemma 3 1B IT	1.0B	0	0	0	36.20
28	Llama 3.2 3B Instruct	3.2B	6.39	4.94	10.84	13.69
29	Llama 3.2 1B Instruct	1.2B	0	0	0	4.68
	OpenAI o4-mini	Unknown


10	Mistral Small 3.1 24B Instruct	Unknown	21.61	19.02	26.84	71.47


1	GPT-4o	Unknown	21.61	19.02	26.84	71.47
2	Claude 3.7 Sonnet	Unknown	21.15	19.12	24.84	71.43
3	Gemini 2.0 Flash	Unknown	20.58	18.61	24.21	70.51
4	Gemma 3 12B IT	12.2B	33.52	29.14	41.67	69.91
5	Gemini 2.0 Flash-Lite	Unknown	14.71	13.87	16.67	69.14
6	Gemma 3 27B IT	27.4B	23.84	20.42	30.15	68.98
7	GPT-4.1 Mini	Unknown	18.56	15.78	24.12	68.33
8	Llama 3.3 70B Instruct	70.6B	28.77	27.33	32.22	68.00
9	GPT-4.1	Unknown	12.18	10.21	16.09	67.58
10	DeepSeek-V3	671.0B	26.19	21.73	34.93	67.53
11	Qwen3-32B	32.8B	13.69	11.17	18.63	67.14
12	Qwen3-14B	14.8B	23.38	18.93	32.97	64.60
13	GPT-4o Mini	Unknown	27.83	22.51	38.42	64.59
14	DeepSeek-R1	671.0B	32.68	29.86	38.25	63.61
15	Command R Plus	104.0B	8.61	7.58	10.65	62.98
16	GPT-4.1 Nano	Unknown	16.91	14.96	20.50	62.62
17	Qwen3-30B-A3B	30.5B	26.00	21.51	34.97	62.56
18	Gemma 3 4B IT	4.3B	22.99	21.15	26.35	62.42
19	Qwen3-8B	8.2B	20.81	16.73	29.44	59.68
20	Mistral Small 3.1 24B Instruct	24.0B	19.43	16.82	24.52	56.62
21	Command R v01	35.0B	0.12	0.14	0.11	53.10
22	Qwen3-4B	4.0B	21.76	17.68	30.30	51.21
23	Command A	111.0B	19.01	16.76	23.07	48.25
24	Aya Expanse 32B	32.3B	21.99	19.25	26.96	39.29
25	Mistral 7B Instruct v0.3	7.2B	25.11	18.99	41.00	39.17
26	Claude 3.5 Haiku	Unknown	25.68	24.84	27.66	37.49
27	Gemma 3 1B IT	1.0B	20.97	18.03	26.21	36.20
28	Llama 3.2 3B Instruct	3.2B	2.24	2.30	2.28	13.69
29	Llama 3.2 1B Instruct	1.2B	0	0	0	4.68
	OpenAI o4-mini	Unknown


10	Mistral Small 3.1 24B Instruct	Unknown	90.68	82.16	73.12	76.80	90.68	82.16	73.12	76.80	92.55	71.47


1	GPT-4o	Unknown	90.68	82.16	73.12	76.80	90.68	82.16	73.12	76.80	100	71.47
2	Claude 3.7 Sonnet	Unknown	81.99	81.80	59.78	63.64	88.59	88.39	64.59	68.77	92.55	71.43
3	Gemini 2.0 Flash	Unknown	77.64	66.68	64.24	64.05	77.64	66.68	64.24	64.05	100	70.51
4	Gemma 3 12B IT	12.2B	82.61	70.05	66.37	67.76	82.61	70.05	66.37	67.76	100	69.91
5	Gemini 2.0 Flash-Lite	Unknown	82.61	72.11	60.34	63.94	82.61	72.11	60.34	63.94	100	69.14
6	Gemma 3 27B IT	27.4B	78.88	65.29	71.87	67.46	78.88	65.29	71.87	67.46	100	68.98
7	GPT-4.1 Mini	Unknown	91.30	77.91	74.78	76.24	91.30	77.91	74.78	76.24	100	68.33
8	Llama 3.3 70B Instruct	70.6B	68.32	64.15	73.86	65.21	68.32	64.15	73.86	65.21	100	68.00
9	GPT-4.1	Unknown	91.30	83.97	70.58	74.87	91.30	83.97	70.58	74.87	100	67.58
10	DeepSeek-V3	671.0B	82.61	73.39	69.71	71.10	82.61	73.39	69.71	71.10	100	67.53
11	Qwen3-32B	32.8B	80.75	67.99	63.38	64.90	80.75	67.99	63.38	64.90	100	67.14
12	Qwen3-14B	14.8B	49.69	53.64	57.73	47.55	49.69	53.64	57.73	47.55	100	64.60
13	GPT-4o Mini	Unknown	85.09	74.78	64.82	67.76	85.09	74.78	64.82	67.76	100	64.59
14	DeepSeek-R1	671.0B	43.48	38.63	38.34	38.38	78.65	69.89	69.36	69.43	55.28	63.61
15	Command R Plus	104.0B	67.70	60.42	55.00	53.93	67.70	60.42	55.00	53.93	100	62.98
16	GPT-4.1 Nano	Unknown	83.23	72.91	58.49	60.77	83.23	72.91	58.49	60.77	100	62.62
17	Qwen3-30B-A3B	30.5B	34.78	40.69	34.20	33.24	53.33	62.40	52.44	50.97	65.22	62.56
18	Gemma 3 4B IT	4.3B	50.31	52.28	51.69	46.45	50.94	52.94	52.34	47.03	98.76	62.42
19	Qwen3-8B	8.2B	63.98	54.83	59.03	53.05	63.98	54.83	59.03	53.05	100	59.68
20	Mistral Small 3.1 24B Instruct	24.0B	77.64	67.75	67.68	66.84	79.11	69.04	68.97	68.10	98.14	56.62
21	Command R v01	35.0B	74.53	57.71	50.71	52.21	75.47	58.43	51.34	52.87	98.76	53.10
22	Qwen3-4B	4.0B	65.84	54.13	56.30	53.84	66.67	54.82	57.01	54.52	98.76	51.21
23	Command A	111.0B	63.35	51.13	46.86	48.64	89.47	72.21	66.18	68.69	70.81	48.25
24	Aya Expanse 32B	32.3B	0	0	0	0	0	0	0	0	0	39.29
25	Mistral 7B Instruct v0.3	7.2B	29.19	19.34	23.61	18.74	58.02	38.44	46.93	37.25	50.31	39.17
26	Claude 3.5 Haiku	Unknown	0	0	0	0	0	0	0	0	0	37.49
27	Gemma 3 1B IT	1.0B	2.48	2.59	2.66	1.75	40.00	41.67	42.86	28.15	6.21	36.20
28	Llama 3.2 3B Instruct	3.2B	0	0	0	0	0	0	0	0	0	13.69
29	Llama 3.2 1B Instruct	1.2B	0	0	0	0	0	0	0	0	0.62	4.68
	OpenAI o4-mini	Unknown


10	Mistral Small 3.1 24B Instruct	Unknown	86.12	86.12	99.20	71.47


1	GPT-4o	Unknown	86.12	86.12	100	71.47
2	Claude 3.7 Sonnet	Unknown	86.06	86.06	100	71.43
3	Gemini 2.0 Flash	Unknown	84.93	84.93	100	70.51
4	Gemma 3 12B IT	12.2B	87.68	87.68	100	69.91
5	Gemini 2.0 Flash-Lite	Unknown	84.61	84.61	100	69.14
6	Gemma 3 27B IT	27.4B	87.97	87.97	100	68.98
7	GPT-4.1 Mini	Unknown	86.47	86.47	100	68.33
8	Llama 3.3 70B Instruct	70.6B	82.75	82.75	100	68.00
9	GPT-4.1	Unknown	86.24	86.24	100	67.58
10	DeepSeek-V3	671.0B	86.42	86.42	100	67.53
11	Qwen3-32B	32.8B	85.51	85.51	100	67.14
12	Qwen3-14B	14.8B	85.09	85.09	100	64.60
13	GPT-4o Mini	Unknown	86.72	86.72	100	64.59
14	DeepSeek-R1	671.0B	85.48	85.48	100	63.61
15	Command R Plus	104.0B	83.32	83.32	100	62.98
16	GPT-4.1 Nano	Unknown	87.21	87.21	100	62.62
17	Qwen3-30B-A3B	30.5B	86.44	86.44	100	62.56
18	Gemma 3 4B IT	4.3B	84.71	84.71	100	62.42
19	Qwen3-8B	8.2B	84.98	84.98	100	59.68
20	Mistral Small 3.1 24B Instruct	24.0B	83.46	83.46	100	56.62
21	Command R v01	35.0B	83.76	83.76	100	53.10
22	Qwen3-4B	4.0B	83.63	84.31	99.20	51.21
23	Command A	111.0B	0	0	5.80	48.25
24	Aya Expanse 32B	32.3B	0	0	0	39.29
25	Mistral 7B Instruct v0.3	7.2B	55.31	80.39	68.80	39.17
26	Claude 3.5 Haiku	Unknown	25.34	86.18	29.40	37.49
27	Gemma 3 1B IT	1.0B	66.78	69.14	96.60	36.20
28	Llama 3.2 3B Instruct	3.2B	0	0	0	13.69
29	Llama 3.2 1B Instruct	1.2B	0	0	1.80	4.68
	OpenAI o4-mini	Unknown


10	Mistral Small 3.1 24B Instruct	Unknown	87.99	91.33	85.97	87.09	87.99	91.33	85.97	87.09	95.40	71.47


1	GPT-4o	Unknown	87.99	91.33	85.97	87.09	87.99	91.33	85.97	87.09	100	71.47
2	Claude 3.7 Sonnet	Unknown	87.74	91.18	85.67	86.80	87.74	91.18	85.67	86.80	100	71.43
3	Gemini 2.0 Flash	Unknown	88.89	91.87	87.01	88.11	88.89	91.87	87.01	88.11	100	70.51
4	Gemma 3 12B IT	12.2B	88.12	91.26	86.16	87.26	88.12	91.26	86.16	87.26	100	69.91
5	Gemini 2.0 Flash-Lite	Unknown	89.40	92.18	87.61	88.69	89.40	92.18	87.61	88.69	100	69.14
6	Gemma 3 27B IT	27.4B	90.29	92.62	88.69	89.71	90.29	92.62	88.69	89.71	100	68.98
7	GPT-4.1 Mini	Unknown	83.27	88.69	80.45	81.47	83.27	88.69	80.45	81.47	100	68.33
8	Llama 3.3 70B Instruct	70.6B	87.74	91.18	85.67	86.80	87.74	91.18	85.67	86.80	100	68.00
9	GPT-4.1	Unknown	85.95	90.14	83.58	84.71	85.95	90.14	83.58	84.71	100	67.58
10	DeepSeek-V3	671.0B	84.67	89.44	82.09	83.19	84.67	89.44	82.09	83.19	100	67.53
11	Qwen3-32B	32.8B	89.53	92.26	87.76	88.83	89.53	92.26	87.76	88.83	100	67.14
12	Qwen3-14B	14.8B	83.91	89.02	81.19	82.25	83.91	89.02	81.19	82.25	100	64.60
13	GPT-4o Mini	Unknown	86.08	90.22	83.73	84.86	86.08	90.22	83.73	84.86	100	64.59
14	DeepSeek-R1	671.0B	91.32	93.30	89.89	90.84	91.32	93.30	89.89	90.84	100	63.61
15	Command R Plus	104.0B	89.02	91.95	87.16	88.26	89.02	91.95	87.16	88.26	100	62.98
16	GPT-4.1 Nano	Unknown	80.46	87.27	77.16	77.91	80.46	87.27	77.16	77.91	100	62.62
17	Qwen3-30B-A3B	30.5B	77.78	83.43	75.75	76.19	81.53	87.45	79.40	79.86	95.40	62.56
18	Gemma 3 4B IT	4.3B	86.97	90.57	84.81	85.93	86.97	90.57	84.81	85.93	100	62.42
19	Qwen3-8B	8.2B	79.82	86.96	76.42	77.08	79.82	86.96	76.42	77.08	100	59.68
20	Mistral Small 3.1 24B Instruct	24.0B	68.07	82.09	62.69	59.33	68.07	82.09	62.69	59.33	100	56.62
21	Command R v01	35.0B	91.70	92.33	90.95	91.45	92.53	93.16	91.77	92.27	99.11	53.10
22	Qwen3-4B	4.0B	58.24	68.60	56.44	54.13	69.41	81.76	67.26	64.51	83.91	51.21
23	Command A	111.0B	88.76	91.79	86.87	87.97	88.76	91.79	86.87	87.97	100	48.25
24	Aya Expanse 32B	32.3B	90.80	93.08	89.25	90.26	90.80	93.08	89.25	90.26	100	39.29
25	Mistral 7B Instruct v0.3	7.2B	87.23	90.57	85.15	86.25	87.23	90.57	85.15	86.25	100	39.17
26	Claude 3.5 Haiku	Unknown	85.57	89.61	83.14	84.34	86.34	90.42	83.89	85.10	99.11	37.49
27	Gemma 3 1B IT	1.0B	50.96	60.43	55.90	47.35	50.96	60.43	55.90	47.35	100	36.20
28	Llama 3.2 3B Instruct	3.2B	52.11	26.05	45.59	33.16	57.14	28.57	50.00	36.36	91.19	13.69
29	Llama 3.2 1B Instruct	1.2B	9.20	4.60	7.92	5.82	58.06	29.03	50.00	36.73	15.84	4.68
	OpenAI o4-mini	Unknown


10	Mistral Small 3.1 24B Instruct	Unknown	82.54	83.84	82.65	82.39	82.54	83.84	82.65	82.39	99.87	71.47


1	GPT-4o	Unknown	82.54	83.84	82.65	82.39	82.54	83.84	82.65	82.39	100	71.47
2	Claude 3.7 Sonnet	Unknown	86.83	87.96	86.94	86.81	86.83	87.96	86.94	86.81	100	71.43
3	Gemini 2.0 Flash	Unknown	69.44	75.67	69.80	67.94	69.44	75.67	69.80	67.94	100	70.51
4	Gemma 3 12B IT	12.2B	65.35	72.79	65.59	64.02	65.35	72.79	65.59	64.02	100	69.91
5	Gemini 2.0 Flash-Lite	Unknown	66.18	77.30	66.72	61.91	66.18	77.30	66.72	61.91	100	69.14
6	Gemma 3 27B IT	27.4B	69.18	76.12	69.66	66.38	69.18	76.12	69.66	66.38	100	68.98
7	GPT-4.1 Mini	Unknown	74.10	76.33	74.23	73.76	74.10	76.33	74.23	73.76	100	68.33
8	Llama 3.3 70B Instruct	70.6B	72.63	80.62	73.14	69.81	72.63	80.62	73.14	69.81	100	68.00
9	GPT-4.1	Unknown	80.69	82.74	80.78	80.56	80.69	82.74	80.78	80.56	100	67.58
10	DeepSeek-V3	671.0B	74.06	79.38	74.41	72.66	74.06	79.38	74.41	72.66	100	67.53
11	Qwen3-32B	32.8B	71.61	77.95	71.85	70.94	71.61	77.95	71.85	70.94	100	67.14
12	Qwen3-14B	14.8B	70.08	75.97	70.40	68.35	70.08	75.97	70.40	68.35	100	64.60
13	GPT-4o Mini	Unknown	76.47	78.14	76.70	75.73	76.57	78.25	76.80	75.83	99.87	64.59
14	DeepSeek-R1	671.0B	68.54	74.52	68.84	66.90	68.59	74.57	68.89	66.94	99.94	63.61
15	Command R Plus	104.0B	68.99	75.22	69.17	68.35	68.99	75.22	69.17	68.35	100	62.98
16	GPT-4.1 Nano	Unknown	70.01	70.90	70.16	69.64	70.01	70.90	70.16	69.64	100	62.62
17	Qwen3-30B-A3B	30.5B	60.87	69.40	60.93	59.08	61.10	69.67	61.16	59.31	99.62	62.56
18	Gemma 3 4B IT	4.3B	63.62	67.43	63.75	62.11	63.70	67.52	63.83	62.19	99.87	62.42
19	Qwen3-8B	8.2B	65.92	72.92	65.56	61.73	66.22	73.25	65.85	62.01	99.55	59.68
20	Mistral Small 3.1 24B Instruct	24.0B	74.62	82.79	74.32	74.84	74.62	82.79	74.32	74.84	100	56.62
21	Command R v01	35.0B	53.71	74.30	54.28	45.22	53.71	74.30	54.28	45.22	100	53.10
22	Qwen3-4B	4.0B	17.26	18.56	15.64	15.50	72.78	78.26	65.93	65.34	23.72	51.21
23	Command A	111.0B	50.51	55.27	43.84	45.70	80.86	88.47	70.18	73.15	62.47	48.25
24	Aya Expanse 32B	32.3B	37.85	44.33	34.23	34.75	69.32	81.19	62.69	63.64	54.60	39.29
25	Mistral 7B Instruct v0.3	7.2B	54.22	56.48	54.45	52.73	54.50	56.77	54.73	53.00	99.49	39.17
26	Claude 3.5 Haiku	Unknown	2.49	1.61	1.74	1.67	90.70	58.52	63.33	60.83	2.75	37.49
27	Gemma 3 1B IT	1.0B	34.34	56.18	34.40	27.90	34.36	56.22	34.42	27.92	99.94	36.20
28	Llama 3.2 3B Instruct	3.2B	0	0	0	0	0	0	0	0	0	13.69
29	Llama 3.2 1B Instruct	1.2B	0	0	0	0	0	0	0	0	0	4.68
	OpenAI o4-mini	Unknown


10	Mistral Small 3.1 24B Instruct	Unknown	84.80	87.18	82.90	83.77	84.80	87.18	82.90	83.77	99.20	71.47


1	GPT-4o	Unknown	84.80	87.18	82.90	83.77	84.80	87.18	82.90	83.77	100	71.47
2	Claude 3.7 Sonnet	Unknown	84.40	86.72	82.49	83.34	84.40	86.72	82.49	83.34	100	71.43
3	Gemini 2.0 Flash	Unknown	86.80	87.87	85.51	86.18	86.80	87.87	85.51	86.18	100	70.51
4	Gemma 3 12B IT	12.2B	87.80	88.23	86.90	87.36	87.80	88.23	86.90	87.36	100	69.91
5	Gemini 2.0 Flash-Lite	Unknown	88.80	88.86	88.23	88.49	88.80	88.86	88.23	88.49	100	69.14
6	Gemma 3 27B IT	27.4B	87.60	88.07	86.67	87.15	87.60	88.07	86.67	87.15	100	68.98
7	GPT-4.1 Mini	Unknown	80.40	85.57	77.49	78.23	80.40	85.57	77.49	78.23	100	68.33
8	Llama 3.3 70B Instruct	70.6B	82.80	85.05	80.80	81.59	82.80	85.05	80.80	81.59	100	68.00
9	GPT-4.1	Unknown	82.60	86.13	80.22	81.10	82.60	86.13	80.22	81.10	100	67.58
10	DeepSeek-V3	671.0B	76.40	84.06	72.73	72.80	76.40	84.06	72.73	72.80	100	67.53
11	Qwen3-32B	32.8B	87.60	87.35	87.35	87.35	87.60	87.35	87.35	87.35	100	67.14
12	Qwen3-14B	14.8B	85.40	87.42	83.65	84.49	85.40	87.42	83.65	84.49	100	64.60
13	GPT-4o Mini	Unknown	83.80	85.77	81.96	82.76	83.80	85.77	81.96	82.76	100	64.59
14	DeepSeek-R1	671.0B	82.60	82.67	81.65	82.00	82.60	82.67	81.65	82.00	100	63.61
15	Command R Plus	104.0B	87.20	87.26	86.54	86.83	87.20	87.26	86.54	86.83	100	62.98
16	GPT-4.1 Nano	Unknown	79.20	84.27	76.21	76.83	79.20	84.27	76.21	76.83	100	62.62
17	Qwen3-30B-A3B	30.5B	82.40	86.00	79.99	80.86	82.40	86.00	79.99	80.86	100	62.56
18	Gemma 3 4B IT	4.3B	83.80	84.17	82.71	83.17	83.80	84.17	82.71	83.17	100	62.42
19	Qwen3-8B	8.2B	79.80	83.84	77.08	77.77	79.80	83.84	77.08	77.77	100	59.68
20	Mistral Small 3.1 24B Instruct	24.0B	66.00	79.33	60.58	56.25	66.00	79.33	60.58	56.25	100	56.62
21	Command R v01	35.0B	77.80	80.54	79.72	77.77	78.43	81.19	80.36	78.40	99.20	53.10
22	Qwen3-4B	4.0B	67.20	78.68	62.65	59.73	68.02	79.64	63.42	60.46	98.80	51.21
23	Command A	111.0B	88.00	89.45	86.62	87.39	88.00	89.45	86.62	87.39	100	48.25
24	Aya Expanse 32B	32.3B	86.40	86.16	86.07	86.11	86.40	86.16	86.07	86.11	100	39.29
25	Mistral 7B Instruct v0.3	7.2B	63.40	72.38	67.21	62.20	63.40	72.38	67.21	62.20	100	39.17
26	Claude 3.5 Haiku	Unknown	74.40	75.17	73.57	73.95	84.16	85.04	83.23	83.65	88.40	37.49
27	Gemma 3 1B IT	1.0B	57.60	63.53	60.98	56.54	57.60	63.53	60.98	56.54	100	36.20
28	Llama 3.2 3B Instruct	3.2B	57.00	28.50	50.00	36.31	57.00	28.50	50.00	36.31	100	13.69
29	Llama 3.2 1B Instruct	1.2B	7.20	3.60	6.60	4.66	54.55	27.27	50.00	35.29	13.20	4.68
	OpenAI o4-mini	Unknown


10	Mistral Small 3.1 24B Instruct	Unknown	17.54	56.41	71.47


1	GPT-4o	Unknown	17.54	56.41	71.47
2	Claude 3.7 Sonnet	Unknown	21.90	58.99	71.43
3	Gemini 2.0 Flash	Unknown	37.44	68.61	70.51
4	Gemma 3 12B IT	12.2B	37.44	71.21	69.91
5	Gemini 2.0 Flash-Lite	Unknown	33.27	68.85	69.14
6	Gemma 3 27B IT	27.4B	24.93	59.53	68.98
7	GPT-4.1 Mini	Unknown	29.10	65.00	68.33
8	Llama 3.3 70B Instruct	70.6B	38.29	70.91	68.00
9	GPT-4.1	Unknown	16.21	55.41	67.58
10	DeepSeek-V3	671.0B	7.01	49.87	67.53
11	Qwen3-32B	32.8B	12.04	51.53	67.14
12	Qwen3-14B	14.8B	11.94	50.54	64.60
13	GPT-4o Mini	Unknown	7.20	47.22	64.59
14	DeepSeek-R1	671.0B	36.78	70.60	63.61
15	Command R Plus	104.0B	51.66	79.97	62.98
16	GPT-4.1 Nano	Unknown	21.52	60.52	62.62
17	Qwen3-30B-A3B	30.5B	1.42	61.09	62.56
18	Gemma 3 4B IT	4.3B	29.86	64.84	62.42
19	Qwen3-8B	8.2B	1.90	49.54	59.68
20	Mistral Small 3.1 24B Instruct	24.0B	13.08	51.12	56.62
21	Command R v01	35.0B	41.99	74.01	53.10
22	Qwen3-4B	4.0B	20.47	56.61	51.21
23	Command A	111.0B	5.88	44.59	48.25
24	Aya Expanse 32B	32.3B	85.31	35.71	39.29
25	Mistral 7B Instruct v0.3	7.2B	47.39	34.40	39.17
26	Claude 3.5 Haiku	Unknown	1.52	32.22	37.49
27	Gemma 3 1B IT	1.0B	15.64	47.98	36.20
28	Llama 3.2 3B Instruct	3.2B	2.37	40.03	13.69
29	Llama 3.2 1B Instruct	1.2B	66.35	33.78	4.68
	OpenAI o4-mini	Unknown


10	Mistral Small 3.1 24B Instruct	Unknown	75.60	65.31	66.85	63.59	75.60	65.31	66.85	63.59	99.20	71.47


1	GPT-4o	Unknown	75.60	65.31	66.85	63.59	75.60	65.31	66.85	63.59	100	71.47
2	Claude 3.7 Sonnet	Unknown	79.20	66.25	66.35	65.83	79.20	66.25	66.35	65.83	100	71.43
3	Gemini 2.0 Flash	Unknown	78.40	68.19	65.18	65.19	79.03	68.74	65.71	65.72	99.20	70.51
4	Gemma 3 12B IT	12.2B	65.80	60.06	55.60	53.15	66.60	60.79	56.27	53.80	98.80	69.91
5	Gemini 2.0 Flash-Lite	Unknown	74.00	63.09	66.59	63.40	75.05	63.98	67.53	64.30	98.60	69.14
6	Gemma 3 27B IT	27.4B	70.20	60.71	59.03	58.06	70.77	61.19	59.50	58.53	99.20	68.98
7	GPT-4.1 Mini	Unknown	73.40	65.55	64.60	62.68	73.40	65.55	64.60	62.68	100	68.33
8	Llama 3.3 70B Instruct	70.6B	59.40	56.08	53.29	52.78	63.60	60.05	57.06	56.51	93.40	68.00
9	GPT-4.1	Unknown	74.80	64.29	66.34	62.86	74.80	64.29	66.34	62.86	100	67.58
10	DeepSeek-V3	671.0B	77.00	65.62	67.52	64.26	77.00	65.62	67.52	64.26	100	67.53
11	Qwen3-32B	32.8B	74.20	61.67	61.26	59.90	74.50	61.92	61.50	60.14	99.60	67.14
12	Qwen3-14B	14.8B	63.60	62.48	52.00	52.12	63.60	62.48	52.00	52.12	100	64.60
13	GPT-4o Mini	Unknown	70.40	66.79	62.59	62.65	70.54	66.93	62.72	62.78	99.80	64.59
14	DeepSeek-R1	671.0B	60.80	59.72	50.95	51.60	63.33	62.21	53.07	53.76	96.00	63.61
15	Command R Plus	104.0B	68.20	57.30	57.17	56.06	72.40	60.83	60.69	59.51	94.20	62.98
16	GPT-4.1 Nano	Unknown	46.20	49.18	32.62	30.48	51.91	55.26	36.65	34.24	89.00	62.62
17	Qwen3-30B-A3B	30.5B	58.60	58.83	47.21	49.37	58.60	58.83	47.21	49.37	100	62.56
18	Gemma 3 4B IT	4.3B	56.20	48.47	45.47	42.45	59.28	51.13	47.96	44.78	94.80	62.42
19	Qwen3-8B	8.2B	56.00	53.10	47.06	48.42	57.03	54.07	47.92	49.30	98.20	59.68
20	Mistral Small 3.1 24B Instruct	24.0B	65.60	58.19	56.50	54.72	70.39	62.44	60.62	58.72	93.20	56.62
21	Command R v01	35.0B	41.20	38.19	21.94	16.62	47.36	43.90	25.22	19.10	87.00	53.10
22	Qwen3-4B	4.0B	36.80	60.15	29.70	27.67	37.40	61.12	30.19	28.12	98.40	51.21
23	Command A	111.0B	61.80	62.85	54.80	56.32	61.80	62.85	54.80	56.32	100	48.25
24	Aya Expanse 32B	32.3B	62.80	54.60	52.37	52.00	68.41	59.48	57.04	56.64	91.80	39.29
25	Mistral 7B Instruct v0.3	7.2B	7.80	16.26	8.69	6.16	12.11	25.25	13.50	9.56	64.40	39.17
26	Claude 3.5 Haiku	Unknown	75.00	63.27	63.47	62.32	75.00	63.27	63.47	62.32	100	37.49
27	Gemma 3 1B IT	1.0B	38.00	20.19	17.56	11.96	38.31	20.35	17.70	12.05	99.20	36.20
28	Llama 3.2 3B Instruct	3.2B	1.80	13.57	12.60	1.09	1.85	13.93	12.93	1.12	97.40	13.69
29	Llama 3.2 1B Instruct	1.2B	0	0	0	0	0	0	0	0	0	4.68
	OpenAI o4-mini	Unknown

About MIZAN: A Persian LLM Leaderboard

MIZAN: A Persian LLM Leaderboard is designed to establish a standard and comprehensive benchmark for evaluating Large Language Models (LLMs) in the Persian language. This project combines existing datasets, translates and localizes globally recognized benchmarks, and incorporates newly developed, Persian-specific datasets. MIZAN aims to provide a multi-dimensional assessment of models' capabilities across various linguistic, knowledge-based, and reasoning tasks. Its primary goal is to offer researchers, developers, and enthusiasts a transparent and reliable view of LLM performance in the Persian language landscape.

MIZAN provides a holistic view of models' strengths and weaknesses by assessing them across a suite of key tasks, contributing to the advancement of AI research for the Persian language.

PerCoR is the first large-scale Persian benchmark for evaluating models' ability in commonsense reasoning through multi-choice sentence completion. It includes over 106,000 samples from diverse domains such as news, religion, and lifestyle, extracted from more than 40 Persian websites. Innovative methods like "segmentation by conjunctions" were used to create coherent and diverse sentences and options, while the DRESS-AF technique helped generate challenging, human-solvable distractors.

link

This dataset is a Persian-adapted and localized version of IFEval, assessing models' proficiency in accurately executing complex, multi-step instructions (Instruction Following). The translation process involved a hybrid machine-human approach, with prompts unsuitable for the Persian language being rewritten or removed.

link

PerMMLU is an expanded and localized version of the renowned MMLU benchmark, designed to measure general and specialized knowledge of models in Persian. Tailored to cover knowledge at various levels and relevant to the Iranian cultural context, it comprises three main sub-datasets:

SPK (School Persian Knowledge): Contains 5,581 multiple-choice questions from the official Iranian school curriculum (grades 4-12) across 78 diverse subjects. Data was collected from the "Paadars" educational website and subsequently cleaned.
UPK (University Persian Knowledge): Includes 7,793 multiple-choice questions from Master's and PhD entrance exams across 25 academic disciplines (e.g., medicine, engineering, humanities, arts). This data was extracted from exam booklets using OCR technology and cleaned by LLMs.
GPK (General Persian Knowledge): Consists of 1,003 multiple-choice questions on 15 topics related to general knowledge specific to Iranian society (e.g., city souvenirs, religious edicts, national laws, famous personalities, cultural idioms). This data was generated using LLMs with specific prompts and reviewed by humans.

link

This is a localized version of the MT-Bench benchmark, evaluating models on multi-turn question-answering and dialogue-based tasks. Questions involve multi-step requests or require creative responses. In the Persian version, all samples were translated and rewritten by humans, and some were expanded to 3 or 4 turns. Two new topics were also added:

Native Iranian Knowledge: Questions about cultural topics such as films, actors, and Iranian figures.
Chat-Retrieval: Involves a multi-turn dialogue where the model must extract a relevant question and answer based on the user's needs.

link

This section comprises a collection of existing Persian benchmarks for evaluating various aspects of Natural Language Understanding. Key tasks and datasets include:

Sentiment Analysis: DeepSentiPers
Text Classification: Synthetic Persian Tone, SID
Natural Language Inference (NLI): FarsTAIL
Semantic Textual Similarity (STS): Synthetic Persian STS, FarSICK
Named Entity Recognition (NER): Arman
Paraphrase Detection: FarsiParaphraseDetection, ParsiNLU
Extractive Question Answering (EQA): PQuAD
Keyword Extraction: Synthetic Persian Keywords

link

This section focuses on Natural Language Generation, covering tasks such as:

Summarization: SamSUM-fa, PnSummary
Machine Translation: TEP, MIZAN, EPOQUE
Question Generation: PersianQA

The goal is to assess the generative capabilities of models.

link

MIZAN is a significant step towards the scientific and localized evaluation of language models for Persian, aiming to serve as a valuable assessment reference for researchers, developers, and anyone interested in practical language models.

Request Model Evaluation for MIZAN

We're excited to evaluate new models for MIZAN: A Persian LLM Leaderboard! Please choose the submission path that best fits how your model can be accessed for evaluation.

Option 1: Your model is publicly available on Hugging Face Hub

If your model and its tokenizer can be loaded directly using their Hugging Face identifier (e.g., username/model_name), you can use the simplified form below to submit its key identifiers. Your submission will be added to our central tracking repository on the Hugging Face Hub. Our team will attempt to gather other necessary details from the Hub.

✨ Form for Option 1: Submit a Hugging Face Model to the Hub

Model Name (Hugging Face ID: e.g., username/model_name)

Revision/Commit (Optional, defaults to 'main' if left empty)

Option 2: Your model is NOT on Hugging Face, is private, or requires custom setup

If your model is hosted elsewhere, is private, requires specific access permissions, needs custom inference code, or involves a more complex setup for evaluation, please initiate your submission request via email.

To submit via email, please send comprehensive details to: 📧 mehran.sarmadi99@sharif.edu

Our team will review your email and work with you to facilitate the evaluation process.

Built with Gradio logo


1	Claude 3.7 Sonnet	Unknown	83.37	88.11	87.75	91.48	83.08	97.83	87.76	89.66	100	75.61	95.08	95.24	84.62	97.83	87.76	94.48	100	85.37	98.36	96.83
2	GPT-4.1	Unknown	81.40	86.34	85.12	89.43	78.46	93.48	89.12	84.83	100	78.05	86.89	95.24	81.54	93.48	90.48	91.03	100	83.74	88.52	95.24
3	Gemini 2.0 Flash	Unknown	79.17	84.98	82.46	87.78	68.75	91.30	90.48	79.17	100	76.42	96.72	95.24	70.31	91.30	90.48	85.42	100	83.74	96.72	95.24
4	Command A	111.0B	78.02	84.39	83.30	88.37	84.62	93.48	89.80	77.93	100	76.86	80.33	90.48	84.62	93.48	91.16	84.83	100	84.30	86.89	93.65
5	Gemini 2.0 Flash-Lite	Unknown	79.21	84.14	81.62	86.49	72.31	91.30	83.67	81.38	96.77	76.42	96.72	95.24	73.85	91.30	83.67	87.59	100	80.49	96.72	95.24
6	DeepSeek-R1	671.0B	77.02	83.70	81.40	87.08	73.85	97.83	85.03	78.62	100	75.61	98.36	85.71	75.38	97.83	85.71	85.52	100	80.49	100	92.06
7	GPT-4.1 Mini	Unknown	77.24	83.41	79.87	85.76	69.23	84.78	87.76	83.45	96.77	71.54	91.80	95.24	72.31	84.78	88.44	88.28	96.77	75.61	93.44	95.24
8	Qwen3-30B-A3B	30.5B	76.62	83.26	80.79	86.38	63.49	88.37	91.37	79.55	96.55	71.93	96.55	91.80	79.37	88.37	92.81	82.58	96.55	75.44	96.55	91.80
9	DeepSeek-V3	671.0B	77.02	83.11	82.06	86.93	78.46	100	85.03	80.00	100	66.67	96.72	88.89	81.54	100	86.39	85.52	100	74.80	98.36	93.65
10	Gemma 3 27B IT	27.4B	76.37	82.97	80.53	86.34	72.31	89.13	80.95	79.31	100	78.86	98.36	87.30	72.31	89.13	80.95	87.59	100	87.80	98.36	87.30
11	GPT-4o	Unknown	77.24	82.97	81.40	86.49	80.00	97.83	89.12	77.93	100	68.29	83.61	92.06	81.54	97.83	91.16	83.45	100	74.80	88.52	93.65
12	Claude 3.5 Haiku	Unknown	76.15	82.53	78.77	85.17	73.85	95.65	85.71	84.14	96.77	60.98	100	88.89	75.38	95.65	87.76	87.59	96.77	66.67	100	92.06
13	Gemma 3 12B IT	12.2B	73.96	81.50	78.77	85.17	69.23	91.30	89.12	75.86	96.77	74.80	80.33	88.89	69.23	91.30	89.12	83.45	100	82.93	83.61	90.48
14	Qwen3-14B	14.8B	74.84	81.06	78.99	84.58	72.31	89.13	89.80	77.24	93.55	74.80	70.49	88.89	76.92	89.13	91.16	84.14	93.55	76.42	75.41	95.24
15	Qwen3-32B	32.8B	73.42	80.30	78.60	84.70	62.50	91.11	89.51	77.86	96.77	61.40	93.33	88.89	68.75	91.11	90.91	85.00	100	66.67	95.00	96.83
16	GPT-4o Mini	Unknown	73.09	80.18	77.24	83.55	73.85	89.13	84.35	73.79	96.77	66.67	90.16	93.65	75.38	89.13	87.76	77.93	96.77	73.98	93.44	93.65
17	Qwen3-4B	4.0B	67.18	75.77	72.65	79.88	53.85	89.13	90.48	71.72	83.87	66.67	81.97	71.43	56.92	89.13	91.16	79.31	87.10	72.36	88.52	74.60
18	GPT-4.1 Nano	Unknown	68.27	75.77	71.99	78.85	58.46	76.09	80.27	77.93	96.77	68.29	77.05	80.95	61.54	76.09	80.95	83.45	96.77	71.54	83.61	84.13
19	Mistral Small 3.1 24B Instruct	24.0B	66.44	75.27	72.75	79.97	61.54	84.44	80.14	76.43	96.77	64.10	73.33	81.67	66.15	84.44	81.56	80.71	96.77	76.07	80.00	85.00
20	Qwen3-8B	8.2B	65.43	74.74	70.90	79.00	58.46	82.61	87.76	66.90	90.32	66.67	72.13	84.13	64.62	82.61	88.44	76.55	90.32	69.92	78.69	87.30
21	Gemma 3 4B IT	4.3B	66.52	74.45	71.77	79.15	50.77	89.13	82.99	71.72	100	68.29	75.41	73.02	55.38	89.13	83.67	79.31	100	78.05	80.33	76.19
22	Llama 3.3 70B Instruct	70.6B	62.08	71.26	65.19	73.93	65.62	80.43	83.33	73.43	96.77	53.66	72.13	58.73	65.62	80.43	84.03	76.92	96.77	59.35	75.41	63.49
23	Command R Plus	104.0B	61.15	70.07	63.80	72.74	58.46	80.00	80.14	78.72	67.74	64.23	62.30	52.38	58.46	80.00	82.88	81.56	67.74	71.54	63.93	52.38
24	Aya Expanse 32B	32.3B	61.27	69.90	68.05	75.48	52.31	76.09	85.71	71.72	100	59.35	29.51	87.30	63.08	76.09	89.80	73.79	100	73.17	36.07	88.89
25	Command R v01	35.0B	48.35	57.90	53.85	62.04	33.85	64.44	74.15	70.63	64.52	62.60	13.33	41.27	40.00	64.44	75.51	74.13	64.52	69.11	21.67	47.62
26	Gemma 3 1B IT	1.0B	43.76	54.48	47.48	58.15	13.85	76.09	76.87	58.62	87.10	58.54	18.03	30.16	13.85	76.09	76.87	64.14	90.32	68.29	22.95	31.75
27	Llama 3.2 3B Instruct	3.2B	42.45	53.30	47.92	58.15	24.62	52.17	62.59	61.38	58.06	57.72	54.10	31.75	32.31	52.17	64.63	65.52	58.06	67.48	60.66	36.51
28	Mistral 7B Instruct v0.3	7.2B	33.70	44.05	39.82	49.49	3.08	52.17	50.34	60.69	90.32	41.46	24.59	28.57	9.23	52.17	51.70	64.83	90.32	54.47	32.79	34.92
29	Llama 3.2 1B Instruct	1.2B	25.16	36.56	28.45	41.12	6.15	39.13	39.46	40.00	51.61	49.59	36.07	19.05	6.15	39.13	40.14	45.52	51.61	56.91	49.18	26.98
	OpenAI o4-mini	Unknown


1	Claude 3.7 Sonnet	Unknown	91.18	91.18	91.84	90.63	90.27	92.45	93.60	91.84	89.00	87.58	87.02	93.63	96.71	92.24	92.35	91.46	94.41	92.53	91.18	91.98	90.80	91.69	88.89	88.25	93.82	93.41	94.76	88.83	91.18	91.63	82.58	93.59	96.84	93.98	92.34	91.15	85.71	86.96	100	100	86.05	78.95	83.33	100	80.00
2	GPT-4.1	Unknown	88.39	88.39	86.10	83.59	87.03	90.33	88.40	87.06	88.50	85.62	83.78	92.36	92.76	91.04	88.38	86.08	91.15	88.80	91.18	91.18	86.78	91.97	87.92	86.35	92.70	91.55	94.29	85.83	86.03	90.75	80.21	89.74	92.63	90.97	85.08	83.08	84.03	86.96	100	100	88.37	84.21	88.33	90.00	90.00
3	GPT-4o	Unknown	86.65	78.32	84.59	83.59	85.95	88.82	89.60	84.97	85.50	82.35	82.01	91.72	93.42	86.57	88.38	88.92	94.10	88.80	85.29	88.18	81.03	88.92	85.51	81.27	91.29	90.85	92.86	84.47	84.56	87.22	81.25	90.38	91.58	89.97	81.45	84.62	78.15	78.26	92.31	96.97	76.74	71.05	85.00	80.00	90.00
4	Gemini 2.0 Flash	Unknown	86.38	86.38	83.99	82.81	82.43	88.52	84.00	87.47	89.50	83.66	80.24	89.81	94.08	89.55	88.99	88.92	87.54	87.55	84.26	87.78	81.61	88.37	88.89	81.27	89.04	90.85	91.90	83.92	88.24	90.31	81.25	94.23	91.58	89.30	84.27	81.54	77.73	69.57	100	96.97	74.42	78.95	81.67	100	90.00
5	OpenAI o4-mini	Unknown	85.51	85.51	84.29	82.81	85.41	86.10	84.00	88.31	87.00	87.58	79.65	88.54	90.13	87.16	90.52	84.81	87.87	87.14	87.58	87.98	79.89	87.53	82.61	80.32	85.11	88.50	92.86	83.11	88.97	87.22	78.82	89.74	91.58	87.63	83.47	82.69	79.41	69.57	76.92	93.94	83.72	81.58	75.83	80.00	30.00
6	DeepSeek-R1	671.0B	82.52	82.52	82.48	80.47	85.95	85.80	78.00	82.67	83.50	79.08	80.83	90.45	90.07	82.69	83.49	79.11	86.51	82.16	81.70	83.37	78.74	83.10	82.13	75.24	87.36	84.51	90.00	83.38	80.88	84.58	76.04	89.10	89.47	80.94	77.82	81.15	76.47	60.87	92.31	93.94	83.72	73.68	75.83	90.00	70.00
7	DeepSeek-V3	671.0B	82.41	51.15	80.97	77.34	82.16	85.80	81.20	81.42	84.50	78.43	76.70	85.99	89.47	82.99	85.32	81.65	86.89	81.74	84.64	82.36	79.31	82.83	80.19	75.24	86.52	85.68	90.00	83.11	81.62	88.11	75.69	91.67	90.53	83.28	78.23	80.38	76.89	65.22	84.62	96.97	79.07	73.68	76.67	80.00	70.00
8	Gemini 2.0 Flash-Lite	Unknown	81.43	81.43	79.76	77.73	80.00	83.38	79.60	80.38	82.00	77.12	76.70	85.99	89.47	82.99	84.10	81.96	85.57	82.16	82.03	81.76	80.46	81.16	78.74	76.83	85.11	84.74	89.52	78.47	83.82	84.58	77.08	85.26	91.58	83.95	74.60	82.31	74.79	69.57	100	100	79.07	71.05	74.17	90.00	60.00
9	Command A	111.0B	79.89	79.84	76.74	76.95	76.96	83.99	78.00	78.71	82.00	77.12	76.11	87.90	84.87	81.19	82.26	80.38	85.57	82.99	85.62	81.53	74.71	80.06	79.23	73.33	83.43	84.27	90.95	77.38	80.88	84.58	72.22	83.97	89.47	79.93	73.39	77.31	69.75	65.22	92.31	84.85	74.42	65.79	71.67	80.00	60.00
10	Llama 3.3 70B Instruct	70.6B	79.56	11.23	77.64	76.17	77.30	85.50	78.00	79.54	75.50	73.86	74.93	84.08	84.21	80.30	82.87	82.59	86.23	82.57	83.66	80.76	73.56	79.22	78.74	71.43	84.55	84.04	90.48	78.75	79.41	84.14	74.65	80.77	89.47	79.60	74.19	73.46	68.49	60.87	84.62	87.88	79.07	65.79	75.83	90.00	50.00
11	GPT-4.1 Mini	Unknown	77.12	77.12	77.04	75.39	76.49	82.48	76.00	77.45	78.50	75.82	67.85	84.08	82.24	79.10	79.20	79.43	82.95	76.35	79.74	80.56	72.41	77.84	77.78	70.79	77.25	80.99	87.62	75.48	79.41	78.41	68.75	82.69	86.32	79.26	70.56	70.77	68.91	65.22	84.62	81.82	62.79	65.79	70.83	70.00	50.00
12	Qwen3-32B	32.8B	76.54	76.53	74.92	75.00	76.22	78.25	72.40	80.38	80.00	74.51	69.91	80.25	81.58	80.30	80.73	79.43	80.00	76.35	80.39	78.76	68.97	77.84	76.33	67.94	77.25	82.16	85.24	75.20	80.88	76.65	69.10	75.64	87.37	76.25	68.95	71.92	72.69	52.17	76.92	84.85	67.44	65.79	67.50	70.00	70.00
13	Gemma 3 27B IT	27.4B	76.28	76.28	69.79	72.66	76.22	81.87	74.00	76.62	78.00	69.93	71.09	82.17	80.26	79.40	75.54	77.85	82.30	81.74	78.43	79.76	71.26	75.35	73.91	70.79	77.53	79.58	84.29	74.93	78.68	82.38	69.10	78.21	83.16	79.93	69.76	73.08	67.23	69.57	84.62	93.94	69.77	63.16	70.00	80.00	70.00
14	GPT-4o Mini	Unknown	75.98	75.98	74.92	71.48	73.51	80.06	73.60	75.99	77.50	65.36	71.98	79.62	82.89	74.93	78.29	80.06	81.31	74.27	78.10	76.15	76.44	77.84	75.36	69.52	76.69	79.58	87.62	77.66	75.74	76.21	69.10	78.21	87.37	80.60	67.34	75.00	67.65	65.22	84.62	84.85	74.42	76.32	70.83	80.00	40.00
15	Command R Plus	104.0B	73.64	73.64	75.23	67.58	70.54	74.62	72.40	72.23	70.00	69.93	70.21	80.89	84.87	71.94	73.39	72.47	82.62	71.78	77.12	77.15	71.84	74.79	76.81	66.98	74.16	79.34	81.90	72.21	75.00	76.65	66.32	78.21	86.32	75.25	66.53	75.00	61.34	65.22	92.31	87.88	65.12	50.00	70.83	90.00	70.00
16	Claude 3.5 Haiku	Unknown	71.60	71.60	68.88	69.14	70.00	73.11	71.20	71.82	74.50	60.13	67.55	77.71	86.84	71.94	70.64	70.57	78.03	68.88	74.84	74.35	67.24	72.85	74.88	67.62	75.84	75.59	78.10	70.03	72.79	76.65	62.02	76.28	84.04	73.24	61.69	69.23	66.81	56.52	92.31	78.79	65.12	71.05	57.50	90.00	30.00
17	Gemma 3 12B IT	12.2B	70.94	70.94	65.86	65.62	72.43	76.44	70.40	73.70	76.00	67.97	66.67	75.16	76.32	72.24	75.84	71.84	72.13	72.20	68.30	73.55	62.07	71.47	71.98	66.03	70.22	73.24	77.14	73.30	73.53	71.81	64.93	75.64	82.11	71.57	60.89	65.77	63.03	65.22	76.92	87.88	69.77	73.68	65.83	80.00	80.00
18	Qwen3-14B	14.8B	69.58	69.58	64.95	62.89	69.73	70.69	65.20	72.03	77.00	61.44	64.31	75.80	76.32	73.73	71.56	72.78	72.79	69.71	72.55	70.74	63.79	68.42	68.60	65.40	70.79	76.76	81.43	66.21	76.47	73.13	65.97	69.87	81.05	62.21	62.50	66.92	65.97	56.52	92.31	87.88	62.79	60.53	56.67	90.00	70.00
19	Mistral Small 3.1 24B Instruct	24.0B	68.94	68.94	67.07	63.67	69.46	70.39	66.40	68.27	75.50	63.40	63.42	74.52	76.97	70.15	71.25	73.42	72.79	63.07	76.47	71.74	58.05	68.14	68.12	61.59	69.94	74.41	83.33	65.94	70.59	72.69	65.28	70.51	85.26	67.89	58.87	66.92	66.81	43.48	69.23	75.76	65.12	68.42	55.00	80.00	40.00
20	Qwen3-30B-A3B	30.5B	68.80	68.80	64.65	66.41	72.43	73.11	66.80	67.64	76.50	70.59	65.19	80.25	73.68	71.34	70.03	70.25	68.20	70.12	71.90	67.54	63.22	67.04	65.22	61.27	73.88	73.00	79.05	69.21	72.79	68.72	66.32	68.59	80.00	61.20	61.29	65.00	65.97	47.83	76.92	87.88	65.12	65.79	60.00	80.00	40.00
21	Aya Expanse 32B	32.3B	63.27	5.85	62.24	57.03	66.22	64.35	63.20	63.47	65.00	56.21	61.95	70.70	67.76	62.09	62.69	65.19	65.57	66.39	66.67	62.73	62.07	66.76	62.32	59.05	67.98	59.15	73.33	64.03	61.03	70.48	54.17	62.18	76.84	62.54	59.27	62.31	56.72	52.17	61.54	75.76	62.79	65.79	50.83	80.00	60.00
22	Command R v01	35.0B	60.00	60.00	60.42	56.64	57.30	59.52	60.00	59.08	60.50	58.82	57.23	63.69	65.79	57.31	62.39	61.39	62.95	65.15	63.07	61.52	57.47	59.00	53.14	56.19	62.92	59.15	65.71	62.67	63.97	68.72	48.96	62.18	67.37	63.55	57.26	61.15	56.72	30.43	69.23	60.61	60.47	47.37	50.00	80.00	60.00
23	GPT-4.1 Nano	Unknown	54.94	54.94	53.47	49.22	58.65	59.21	55.60	54.70	60.00	52.29	50.44	63.06	66.45	55.82	51.07	59.49	56.39	51.04	60.78	56.11	59.20	55.68	52.66	49.21	54.78	58.22	63.81	54.50	61.03	57.27	50.00	50.00	62.11	48.16	48.39	56.92	50.42	34.78	53.85	57.58	48.84	47.37	40.00	70.00	40.00
24	Qwen3-8B	8.2B	54.37	54.37	49.55	51.56	52.70	56.50	50.80	54.70	60.50	48.37	50.15	65.61	57.24	61.19	56.27	56.96	55.41	56.43	60.46	57.11	51.72	51.52	55.56	47.62	59.27	59.39	67.62	47.41	54.41	56.83	51.04	53.21	65.26	46.49	44.76	51.54	56.30	30.43	69.23	57.58	51.16	57.89	39.17	70.00	40.00
25	Qwen3-4B	4.0B	50.33	50.33	49.55	48.05	48.11	52.57	50.40	50.73	61.50	49.02	46.02	54.14	51.97	57.31	55.66	50.95	46.56	51.45	50.98	52.30	51.15	47.65	47.83	42.54	50.28	53.29	60.48	47.96	61.03	55.07	48.96	50.64	58.95	39.13	43.95	48.08	55.46	34.78	53.85	54.55	48.84	52.63	30.83	70.00	10.00
26	Gemma 3 4B IT	4.3B	48.32	48.32	49.85	44.14	44.86	50.76	50.40	48.23	50.00	41.18	52.51	50.32	51.97	49.85	44.95	50.63	53.44	46.06	50.33	47.29	44.83	49.03	47.34	41.90	48.31	53.05	56.67	49.32	52.94	48.02	44.79	42.31	64.21	45.15	40.73	50.00	44.54	52.17	53.85	63.64	46.51	47.37	35.00	90.00	50.00
27	Mistral 7B Instruct v0.3	7.2B	30.15	30.11	29.91	29.30	28.65	28.40	27.60	29.23	34.50	32.68	32.15	25.48	28.95	33.13	30.58	35.13	26.23	32.78	32.35	32.06	28.16	29.92	31.88	26.98	31.74	30.28	33.81	28.88	33.09	32.60	29.17	23.72	40.00	22.41	24.60	33.46	33.61	30.43	46.15	27.27	44.19	15.79	20.83	70.00	10.00
28	Gemma 3 1B IT	1.0B	25.99	25.99	27.19	25.00	25.68	32.02	27.60	26.30	24.50	24.18	29.79	24.84	26.32	25.07	24.77	25.95	28.52	24.90	27.12	27.05	27.59	26.04	26.09	26.35	27.25	25.12	28.57	20.98	27.21	24.67	22.22	25.00	29.47	24.41	25.00	26.92	23.53	30.43	38.46	15.15	25.58	26.32	19.17	10.00	40.00
29	Llama 3.2 3B Instruct	3.2B	25.21	25.17	27.79	29.30	25.95	22.36	20.00	24.22	24.00	23.53	26.84	28.03	23.68	22.99	25.99	26.90	23.28	26.97	21.24	24.65	25.86	26.32	28.99	26.03	25.56	28.87	22.38	25.61	25.00	22.03	25.00	26.92	28.42	21.07	23.39	23.08	32.35	17.39	46.15	33.33	27.91	23.68	18.33	30.00	50.00
30	Llama 3.2 1B Instruct	1.2B	24.12	0.79	25.38	25.39	24.86	28.10	24.80	21.92	24.50	24.84	25.07	28.03	24.34	22.09	21.41	23.73	29.84	22.82	25.82	22.24	27.01	25.21	18.84	23.49	26.97	25.59	17.62	23.43	22.79	29.07	28.12	19.87	27.37	23.08	20.97	22.69	18.49	21.74	15.38	15.15	27.91	21.05	22.50	20.00	40.00