Agnes-SeaLLM-8B

Introduction

We introduce Agnes-SeaLLM-8B, a compact Large Language Model (LLM) meticulously optimized for Southeast Asian languages. Despite its efficient footprint, it delivers performance rivaling much larger open-source models, excelling in tasks such as mathematical reasoning, translation, and instruction following. Furthermore, it has been specially tuned to enhance reliability, minimize hallucinations, and provide culturally sensitive, safe responses tailored to the Southeast Asian context.

🔥 Highlights

Compact Efficiency & Rapid Deployment: Significantly smaller than mainstream LLMs, Agnes-SeaLLM-8B enables high-speed inference and low-resource deployment without sacrificing accuracy or multilingual proficiency. It is the ideal choice for edge devices and resource-constrained environments.

Top-Tier Performance in its Class: Outperforms comparable open-source models across multi-dimensional benchmarks, including academic examinations, complex instruction following, mathematics, and high-precision translation.

Superior Instruction Following: Demonstrates exceptional capability in handling multi-turn dialogues and executing complex, nuanced tasks with high fidelity.

Safe, Reliable, and Culturally Aware: Engineered to reduce hallucinations and increase sensitivity to Southeast Asian cultural nuances. This ensures safer interactions while maintaining peak performance in English and Chinese.

Balanced Multilingual Mastery: Maintains high-quality, consistent output across a broad linguistic spectrum, overcoming the "seesaw effect" often found in regional models.

【SeaExam Summary statistics】

Model	avg	M3Exam	MMLU
Agnes-SeaLLM-8B	75.32	77.47	74.13
Llama-sea-Lion-v2-8B-IT	55.86	61.11	52.91
Sailor2-20B	67.50	74.15	63.77
glm-4-9b-chat	64.35	69.36	61.53
Meta-Llama-3-70B	63.67	64.25	63.55

Agnes-SeaLLM-8B (75.32 avg) crushes 8B peers and outpaces Sailor2-20B, offering 70B-tier performance with compact, low-resource deployment.
Exceptional scores in both M3Exam and MMLU prove it bridges regional linguistic nuance with elite global reasoning.

【M3Exam Model evaluation sub-statistics】

Model	avg	M3Exam zh	M3Exam en	M3Exam id	M3Exam vi	M3Exam th
Agnes-SeaLLM-8B	76.84	83.70	93.24	72.97	70.03	64.25
Llama-sea-Lion-v2-8B-IT	59.95	52.95	78.41	54.37	54.00	60.00
Sailor2-20B	74.96	80.90	88.94	69.68	68.65	66.65
glm-4-9b-chat	71.62	85.71	89.78	63.16	66.02	53.43
Meta-Llama-3-70B	63.82	63.04	84.75	57.97	52.02	61.32

With a 71.29 average, Agnes-SeaLLM-8B delivers "70B-class" performance in an 8B footprint, outperforming the 20B Sailor2 and crushing 8B peers like Llama-3 and GLM-4.
It sets a new benchmark for Southeast Asia, achieving top-tier scores in Indonesian (72.97) and Vietnamese (70.03)**, while maintaining elite English (93.24) and Chinese (83.70) proficiency.

【MMLU Model evaluation sub-statistics】

Model	avg	MMLU zh	MMLU en	MMLU id	MMLU vi	MMLU th
Agnes-SeaLLM-8B	71.29	71.40	78.60	71.54	66.84	62.70
Llama-sea-Lion-v2-8B-IT	55.86	49.58	61.65	54.42	47.72	51.19
Sailor2-20B	67.50	64.84	72.88	64.60	59.58	56.95
glm-4-9b-chat	63.35	64.84	72.42	60.04	60.04	50.32
Meta-Llama-3-70B	58.21	63.35	60.91	72.32	62.95	62.35

【Indonesian Fine-grained discipline statistics】

Indonesian Standards	Agnes-SeaLLM-8B	Sailor2-20B	glm-4-9b-chat	Llama-sea-Lion-v2-8B	Meta-Llama-3-8B
Overall Average	73.37	64.60	60.04	54.42	54.60
abstract_algebra	64.00	48.00	34.00	28.00	32.00
anatomy	70.00	62.00	46.00	46.00	50.00
astronomy	90.00	68.00	64.00	66.00	58.00
business_ethics	68.00	60.00	56.00	64.00	56.00
clinical_knowledge	84.00	66.00	68.00	64.00	66.00
college_biology	98.00	84.00	68.00	60.00	62.00
college_chemistry	70.00	38.00	50.00	40.00	42.00
college_computer_science	74.00	54.00	52.00	36.00	38.00
college_mathematics	60.00	38.00	42.00	32.00	38.00
college_medicine	86.00	66.00	74.00	68.00	64.00
college_physics	82.00	68.00	60.00	38.00	44.00
computer_security	80.00	86.00	64.00	50.00	56.00
conceptual_physics	84.00	68.00	72.00	44.00	46.00
elementary_mathematics	92.00	78.00	78.00	72.00	80.00
high_school_biology	90.00	84.00	80.00	70.00	74.00
high_school_chemistry	94.00	50.00	62.00	54.00	50.00
high_school_mathematics	86.00	44.00	64.00	44.00	54.00
international_law	68.00	80.00	84.00	78.00	74.00
marketing	90.00	82.00	70.00	78.00	68.00
sociology	80.00	68.00	66.00	70.00	64.00
world_religions	76.00	84.00	76.00	66.00	76.00

【MMLU Fine-grained discipline statistics】

language / subject	Correct	Total	Accuracy	Percentage
chinese	2176	2850	76.35%	12.81%
chinese → abstract_algebra	36	50	72.00%	0.22%
chinese → anatomy	31	50	62.00%	0.22%
chinese → astronomy	42	50	84.00%	0.22%
chinese → business_ethics	36	50	72.00%	0.22%
chinese → clinical_knowledge	40	50	80.00%	0.22%
chinese → college_biology	46	50	92.00%	0.22%
chinese → college_chemistry	37	50	74.00%	0.22%
chinese → college_computer_science	40	50	80.00%	0.22%
chinese → college_mathematics	34	50	68.00%	0.22%
chinese → college_medicine	42	50	84.00%	0.22%
chinese → college_physics	46	50	92.00%	0.22%
chinese → computer_security	39	50	78.00%	0.22%
chinese → conceptual_physics	45	50	90.00%	0.22%
chinese → econometrics	34	50	68.00%	0.22%
chinese → electrical_engineering	36	50	72.00%	0.22%
chinese → elementary_mathematics	48	50	96.00%	0.22%
chinese → formal_logic	31	50	62.00%	0.22%
chinese → global_facts	26	50	52.00%	0.22%
chinese → high_school_biology	46	50	92.00%	0.22%
chinese → high_school_chemistry	41	50	82.00%	0.22%
chinese → high_school_computer_science	47	50	94.00%	0.22%
chinese → high_school_european_history	41	50	82.00%	0.22%
chinese → high_school_geography	42	50	84.00%	0.22%
chinese → high_school_government_and_politics	42	50	84.00%	0.22%
chinese → high_school_macroeconomics	43	50	86.00%	0.22%
chinese → high_school_mathematics	41	50	82.00%	0.22%
chinese → high_school_microeconomics	43	50	86.00%	0.22%
chinese → high_school_physics	40	50	80.00%	0.22%
chinese → high_school_psychology	47	50	94.00%	0.22%
chinese → high_school_statistics	40	50	80.00%	0.22%
chinese → high_school_us_history	36	50	72.00%	0.22%
chinese → high_school_world_history	42	50	84.00%	0.22%
chinese → human_aging	31	50	62.00%	0.22%
chinese → human_sexuality	41	50	82.00%	0.22%
chinese → international_law	34	50	68.00%	0.22%
chinese → jurisprudence	33	50	66.00%	0.22%
chinese → logical_fallacies	37	50	74.00%	0.22%
chinese → machine_learning	36	50	72.00%	0.22%
chinese → management	40	50	80.00%	0.22%
chinese → marketing	44	50	88.00%	0.22%
chinese → medical_genetics	41	50	82.00%	0.22%
chinese → miscellaneous	35	50	70.00%	0.22%
chinese → moral_disputes	34	50	68.00%	0.22%
chinese → moral_scenarios	34	50	68.00%	0.22%
chinese → nutrition	41	50	82.00%	0.22%
chinese → philosophy	25	50	50.00%	0.22%
chinese → prehistory	37	50	74.00%	0.22%
chinese → professional_accounting	32	50	64.00%	0.22%
chinese → professional_law	28	50	56.00%	0.22%
chinese → professional_medicine	39	50	78.00%	0.22%
chinese → professional_psychology	39	50	78.00%	0.22%
chinese → public_relations	34	50	68.00%	0.22%
chinese → security_studies	40	50	80.00%	0.22%
chinese → sociology	38	50	76.00%	0.22%
chinese → us_foreign_policy	43	50	86.00%	0.22%
chinese → virology	24	50	48.00%	0.22%
chinese → world_religions	36	50	72.00%	0.22%
english	2402	2850	84.28%	12.81%
english → abstract_algebra	38	50	76.00%	0.22%
english → anatomy	40	50	80.00%	0.22%
english → astronomy	43	50	86.00%	0.22%
english → business_ethics	39	50	78.00%	0.22%
english → clinical_knowledge	45	50	90.00%	0.22%
english → college_biology	49	50	98.00%	0.22%
english → college_chemistry	35	50	70.00%	0.22%
english → college_computer_science	43	50	86.00%	0.22%
english → college_mathematics	41	50	82.00%	0.22%
english → college_medicine	47	50	94.00%	0.22%
english → college_physics	43	50	86.00%	0.22%
english → computer_security	41	50	82.00%	0.22%
english → conceptual_physics	44	50	88.00%	0.22%
english → econometrics	44	50	88.00%	0.22%
english → electrical_engineering	39	50	78.00%	0.22%
english → elementary_mathematics	49	50	98.00%	0.22%
english → formal_logic	36	50	72.00%	0.22%
english → global_facts	26	50	52.00%	0.22%
english → high_school_biology	45	50	90.00%	0.22%
english → high_school_chemistry	47	50	94.00%	0.22%
english → high_school_computer_science	50	50	100.00%	0.22%
english → high_school_european_history	45	50	90.00%	0.22%
english → high_school_geography	45	50	90.00%	0.22%
english → high_school_government_and_politics	48	50	96.00%	0.22%
english → high_school_macroeconomics	42	50	84.00%	0.22%
english → high_school_mathematics	40	50	80.00%	0.22%
english → high_school_microeconomics	48	50	96.00%	0.22%
english → high_school_physics	46	50	92.00%	0.22%
english → high_school_psychology	46	50	92.00%	0.22%
english → high_school_statistics	41	50	82.00%	0.22%
english → high_school_us_history	45	50	90.00%	0.22%
english → high_school_world_history	43	50	86.00%	0.22%
english → human_aging	38	50	76.00%	0.22%
english → human_sexuality	45	50	90.00%	0.22%
english → international_law	40	50	80.00%	0.22%
english → jurisprudence	44	50	88.00%	0.22%
english → logical_fallacies	45	50	90.00%	0.22%
english → machine_learning	34	50	68.00%	0.22%
english → management	46	50	92.00%	0.22%
english → marketing	46	50	92.00%	0.22%
english → medical_genetics	46	50	92.00%	0.22%
english → miscellaneous	46	50	92.00%	0.22%
english → moral_disputes	43	50	86.00%	0.22%
english → moral_scenarios	40	50	80.00%	0.22%
english → nutrition	44	50	88.00%	0.22%
english → philosophy	40	50	80.00%	0.22%
english → prehistory	46	50	92.00%	0.22%
english → professional_accounting	37	50	74.00%	0.22%
english → professional_law	32	50	64.00%	0.22%
english → professional_medicine	43	50	86.00%	0.22%
english → professional_psychology	40	50	80.00%	0.22%
english → public_relations	34	50	68.00%	0.22%
english → security_studies	39	50	78.00%	0.22%
english → sociology	41	50	82.00%	0.22%
english → us_foreign_policy	44	50	88.00%	0.22%
english → virology	30	50	60.00%	0.22%
english → world_religions	46	50	92.00%	0.22%
indonesian	2091	2850	73.37%	12.81%
indonesian → abstract_algebra	32	50	64.00%	0.22%
indonesian → anatomy	35	50	70.00%	0.22%
indonesian → astronomy	45	50	90.00%	0.22%
indonesian → business_ethics	34	50	68.00%	0.22%
indonesian → clinical_knowledge	42	50	84.00%	0.22%
indonesian → college_biology	49	50	98.00%	0.22%
indonesian → college_chemistry	35	50	70.00%	0.22%
indonesian → college_computer_science	37	50	74.00%	0.22%
indonesian → college_mathematics	30	50	60.00%	0.22%
indonesian → college_medicine	43	50	86.00%	0.22%
indonesian → college_physics	41	50	82.00%	0.22%
indonesian → computer_security	40	50	80.00%	0.22%
indonesian → conceptual_physics	42	50	84.00%	0.22%
indonesian → econometrics	35	50	70.00%	0.22%
indonesian → electrical_engineering	32	50	64.00%	0.22%
indonesian → elementary_mathematics	46	50	92.00%	0.22%
indonesian → formal_logic	31	50	62.00%	0.22%
indonesian → global_facts	28	50	56.00%	0.22%
indonesian → high_school_biology	45	50	90.00%	0.22%
indonesian → high_school_chemistry	47	50	94.00%	0.22%
indonesian → high_school_computer_science	45	50	90.00%	0.22%
indonesian → high_school_european_history	31	50	62.00%	0.22%
indonesian → high_school_geography	42	50	84.00%	0.22%
indonesian → high_school_government_and_politics	37	50	74.00%	0.22%
indonesian → high_school_macroeconomics	38	50	76.00%	0.22%
indonesian → high_school_mathematics	43	50	86.00%	0.22%
indonesian → high_school_microeconomics	43	50	86.00%	0.22%
indonesian → high_school_physics	42	50	84.00%	0.22%
indonesian → high_school_psychology	42	50	84.00%	0.22%
indonesian → high_school_statistics	40	50	80.00%	0.22%
indonesian → high_school_us_history	38	50	76.00%	0.22%
indonesian → high_school_world_history	40	50	80.00%	0.22%
indonesian → human_aging	23	50	46.00%	0.22%
indonesian → human_sexuality	41	50	82.00%	0.22%
indonesian → international_law	34	50	68.00%	0.22%
indonesian → jurisprudence	34	50	68.00%	0.22%
indonesian → logical_fallacies	30	50	60.00%	0.22%
indonesian → machine_learning	32	50	64.00%	0.22%
indonesian → management	38	50	76.00%	0.22%
indonesian → marketing	45	50	90.00%	0.22%
indonesian → medical_genetics	44	50	88.00%	0.22%
indonesian → miscellaneous	35	50	70.00%	0.22%
indonesian → moral_disputes	40	50	80.00%	0.22%
indonesian → moral_scenarios	32	50	64.00%	0.22%
indonesian → nutrition	39	50	78.00%	0.22%
indonesian → philosophy	27	50	54.00%	0.22%
indonesian → prehistory	34	50	68.00%	0.22%
indonesian → professional_accounting	23	50	46.00%	0.22%
indonesian → professional_law	18	50	36.00%	0.22%
indonesian → professional_medicine	36	50	72.00%	0.22%
indonesian → professional_psychology	31	50	62.00%	0.22%
indonesian → public_relations	31	50	62.00%	0.22%
indonesian → security_studies	34	50	68.00%	0.22%
indonesian → sociology	40	50	80.00%	0.22%
indonesian → us_foreign_policy	34	50	68.00%	0.22%
indonesian → virology	28	50	56.00%	0.22%
indonesian → world_religions	38	50	76.00%	0.22%
thai	1897	2850	66.56%	12.81%
thai → abstract_algebra	27	50	54.00%	0.22%
thai → anatomy	27	50	54.00%	0.22%
thai → astronomy	44	50	88.00%	0.22%
thai → business_ethics	36	50	72.00%	0.22%
thai → clinical_knowledge	38	50	76.00%	0.22%
thai → college_biology	42	50	84.00%	0.22%
thai → college_chemistry	28	50	56.00%	0.22%
thai → college_computer_science	30	50	60.00%	0.22%
thai → college_mathematics	27	50	54.00%	0.22%
thai → college_medicine	39	50	78.00%	0.22%
thai → college_physics	39	50	78.00%	0.22%
thai → computer_security	41	50	82.00%	0.22%
thai → conceptual_physics	37	50	74.00%	0.22%
thai → econometrics	29	50	58.00%	0.22%
thai → electrical_engineering	36	50	72.00%	0.22%
thai → elementary_mathematics	49	50	98.00%	0.22%
thai → formal_logic	24	50	48.00%	0.22%
thai → global_facts	21	50	42.00%	0.22%
thai → high_school_biology	38	50	76.00%	0.22%
thai → high_school_chemistry	37	50	74.00%	0.22%
thai → high_school_computer_science	41	50	82.00%	0.22%
thai → high_school_european_history	38	50	76.00%	0.22%
thai → high_school_geography	37	50	74.00%	0.22%
thai → high_school_government_and_politics	35	50	70.00%	0.22%
thai → high_school_macroeconomics	43	50	86.00%	0.22%
thai → high_school_mathematics	41	50	82.00%	0.22%
thai → high_school_microeconomics	37	50	74.00%	0.22%
thai → high_school_physics	38	50	76.00%	0.22%
thai → high_school_psychology	34	50	68.00%	0.22%
thai → high_school_statistics	34	50	68.00%	0.22%
thai → high_school_us_history	33	50	66.00%	0.22%
thai → high_school_world_history	38	50	76.00%	0.22%
thai → human_aging	25	50	50.00%	0.22%
thai → human_sexuality	32	50	64.00%	0.22%
thai → international_law	33	50	66.00%	0.22%
thai → jurisprudence	28	50	56.00%	0.22%
thai → logical_fallacies	30	50	60.00%	0.22%
thai → machine_learning	31	50	62.00%	0.22%
thai → management	33	50	66.00%	0.22%
thai → marketing	36	50	72.00%	0.22%
thai → medical_genetics	36	50	72.00%	0.22%
thai → miscellaneous	37	50	74.00%	0.22%
thai → moral_disputes	29	50	58.00%	0.22%
thai → moral_scenarios	34	50	68.00%	0.22%
thai → nutrition	40	50	80.00%	0.22%
thai → philosophy	29	50	58.00%	0.22%
thai → prehistory	30	50	60.00%	0.22%
thai → professional_accounting	27	50	54.00%	0.22%
thai → professional_law	19	50	38.00%	0.22%
thai → professional_medicine	35	50	70.00%	0.22%
thai → professional_psychology	26	50	52.00%	0.22%
thai → public_relations	26	50	52.00%	0.22%
thai → security_studies	28	50	56.00%	0.22%
thai → sociology	29	50	58.00%	0.22%
thai → us_foreign_policy	30	50	60.00%	0.22%
thai → virology	26	50	52.00%	0.22%
thai → world_religions	30	50	60.00%	0.22%
vietnamese	1997	2850	70.07%	12.81%
vietnamese → abstract_algebra	35	50	70.00%	0.22%
vietnamese → anatomy	24	50	48.00%	0.22%
vietnamese → astronomy	44	50	88.00%	0.22%
vietnamese → business_ethics	32	50	64.00%	0.22%
vietnamese → clinical_knowledge	40	50	80.00%	0.22%
vietnamese → college_biology	44	50	88.00%	0.22%
vietnamese → college_chemistry	34	50	68.00%	0.22%
vietnamese → college_computer_science	36	50	72.00%	0.22%
vietnamese → college_mathematics	31	50	62.00%	0.22%
vietnamese → college_medicine	45	50	90.00%	0.22%
vietnamese → college_physics	38	50	76.00%	0.22%
vietnamese → computer_security	40	50	80.00%	0.22%
vietnamese → conceptual_physics	42	50	84.00%	0.22%
vietnamese → econometrics	37	50	74.00%	0.22%
vietnamese → electrical_engineering	36	50	72.00%	0.22%
vietnamese → elementary_mathematics	44	50	88.00%	0.22%
vietnamese → formal_logic	29	50	58.00%	0.22%
vietnamese → global_facts	24	50	48.00%	0.22%
vietnamese → high_school_biology	43	50	86.00%	0.22%
vietnamese → high_school_chemistry	44	50	88.00%	0.22%
vietnamese → high_school_computer_science	46	50	92.00%	0.22%
vietnamese → high_school_european_history	32	50	64.00%	0.22%
vietnamese → high_school_geography	38	50	76.00%	0.22%
vietnamese → high_school_government_and_politics	36	50	72.00%	0.22%
vietnamese → high_school_macroeconomics	43	50	86.00%	0.22%
vietnamese → high_school_mathematics	44	50	88.00%	0.22%
vietnamese → high_school_microeconomics	43	50	86.00%	0.22%
vietnamese → high_school_physics	38	50	76.00%	0.22%
vietnamese → high_school_psychology	40	50	80.00%	0.22%
vietnamese → high_school_statistics	41	50	82.00%	0.22%
vietnamese → high_school_us_history	34	50	68.00%	0.22%
vietnamese → high_school_world_history	38	50	76.00%	0.22%
vietnamese → human_aging	21	50	42.00%	0.22%
vietnamese → human_sexuality	33	50	66.00%	0.22%
vietnamese → international_law	31	50	62.00%	0.22%
vietnamese → jurisprudence	33	50	66.00%	0.22%
vietnamese → logical_fallacies	33	50	66.00%	0.22%
vietnamese → machine_learning	27	50	54.00%	0.22%
vietnamese → management	38	50	76.00%	0.22%
vietnamese → marketing	39	50	78.00%	0.22%
vietnamese → medical_genetics	42	50	84.00%	0.22%
vietnamese → miscellaneous	31	50	62.00%	0.22%
vietnamese → moral_disputes	33	50	66.00%	0.22%
vietnamese → moral_scenarios	28	50	56.00%	0.22%
vietnamese → nutrition	42	50	84.00%	0.22%
vietnamese → philosophy	27	50	54.00%	0.22%
vietnamese → prehistory	34	50	68.00%	0.22%
vietnamese → professional_accounting	25	50	50.00%	0.22%
vietnamese → professional_law	19	50	38.00%	0.22%
vietnamese → professional_medicine	34	50	68.00%	0.22%
vietnamese → professional_psychology	28	50	56.00%	0.22%
vietnamese → public_relations	27	50	54.00%	0.22%
vietnamese → security_studies	32	50	64.00%	0.22%
vietnamese → sociology	31	50	62.00%	0.22%
vietnamese → us_foreign_policy	36	50	72.00%	0.22%
vietnamese → virology	24	50	48.00%	0.22%
vietnamese → world_religions	34	50	68.00%	0.22%

📦 SeaExam Evaluation Package

/
├── function.py      # Main function 
├── main.py           # Main evaluation script
├── requirements.txt      # Python dependencies
├── data/                 # Test datasets (26MB)
│   ├── m3exam-chinese/
│   ├── m3exam-english/
│   ├── m3exam-indonesian/
│   ├── m3exam-thai/
│   ├── m3exam-vietnamese/
│   ├── mmlu-chinese/
│   ├── mmlu-english/
│   ├── mmlu-indonesian/
│   ├── mmlu-thai/
│   └── mmlu-vietnamese/
├── models/               # [YOU NEED TO ADD] Place your model here
│   └── Llama-SEA-LION-v2-8B-IT/  (or your model name)
├── README.md             # This file
└── quick_run.sh                # Quick start script

🔧 Prerequisites

Hardware Requirements

GPU: Minimum 2 GPUs with 24GB+ VRAM each
RAM: 64GB+ recommended
Storage: ~20GB (model + data + outputs)

Software Requirements

OS: Linux (tested on Ubuntu)
Python: 3.10 - 3.12
CUDA: 12.x
GPU Driver: Latest NVIDIA driver

📥 Installation Steps

1. Copy this folder to your server

# Example: Using scp
scp -r Seaexam/ user@your-server:/path/to/destination/

# Or using rsync
rsync -avz Seaexam/ user@your-server:/path/to/destination/Seaexam/

2. Prepare your model

Place your model files in the models/ directory:

cd Seaexam
mkdir -p models
# Copy or download your model to models/Llama-SEA-LION-v2-8B-IT/
# The model directory should contain:
# - config.json
# - tokenizer files
# - model weight files (.safetensors or .bin)

3. Create Python virtual environment (recommended)

cd Seaexam
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

4. Install dependencies

conda create -n SeaExam python=3.12
conda activate SeaExam
pip install --upgrade pip
pip install -r requirements.txt

Note: This will install PyTorch, vLLM, and other dependencies. May take 10-20 minutes.

⚙️ Configuration

Method 1: Edit main.py directly

Open main.py and modify the configuration section (lines 12-24):

# Set your model path
MODEL_NAME = "./models/Llama-SEA-LION-v2-8B-IT"  # Or use absolute path

# GPU settings
TENSOR_PARALLEL_SIZE = 2  # Number of GPUs (must match CUDA_VISIBLE_DEVICES)
os.environ["CUDA_VISIBLE_DEVICES"] = "0,1"  # GPU IDs to use

# Batch size (adjust based on your GPU memory)
BATCH_SIZE = 256

Method 2: Use environment variable

export MODEL_PATH="/path/to/your/model"
export CUDA_VISIBLE_DEVICES="0,1"
python main.py

Configuration Tips

GPU Selection

Check available GPUs: nvidia-smi
Use specific GPUs: CUDA_VISIBLE_DEVICES="0,1" (uses GPU 0 and 1)
Match TENSOR_PARALLEL_SIZE with number of GPUs in CUDA_VISIBLE_DEVICES

Memory Optimization

If you encounter OOM (Out of Memory) errors:

Reduce BATCH_SIZE (e.g., 128, 64, 32)
Reduce gpu_memory_utilization in code (default 0.95 → 0.8)
Reduce max_model_len (default 4096 → 2048)

🚀 Running the Evaluation

Quick Start

cd Seaexam
source venv/bin/activate  # If using virtual environment
python main.py

Using the run script

chmod +x quick_run.sh
./quick_run.sh

Monitor Progress

The script shows a progress bar with real-time accuracy:

Inference: 45% |████████          | 2345/5000 [10:23<12:45] Acc: 67.23%

Resume from Checkpoint

The script automatically saves progress to progress.json. If interrupted:

Just run the script again
It will resume from where it stopped

📊 Output Files

After completion, you will find:

result.txt - Detailed evaluation report with accuracy by language/subject
progress.json - Progress checkpoint (can be deleted after successful completion)

🔍 Verification

Before running the full evaluation, verify your setup:

# Check CUDA installation
nvcc --version

# Check GPU availability
nvidia-smi

# Check Python packages
pip list | grep -E "torch|transformers|vllm"

# Test model loading (Python)
python -c "from transformers import AutoTokenizer; tokenizer = AutoTokenizer.from_pretrained('./models/Llama-SEA-LION-v2-8B-IT', trust_remote_code=True); print('Model loads successfully!')"

📝 Expected Results

Total Questions: ~5000 (varies by data version)
Runtime: 1-3 hours (depends on hardware)
Output Accuracy: Model-dependent (typically 40-80%)

❓ Troubleshooting

Issue: "No module named 'vllm'"

Solution: Install dependencies: pip install -r requirements.txt

Issue: "CUDA out of memory"

Solution:

Use fewer GPUs but ensure VRAM > 24GB per GPU
Check GPU usage: nvidia-smi

Issue: "No test data files found"

Solution: Verify data/ directory structure and ensure test.json files exist

Issue: CUDA version mismatch

Solution:

# Check CUDA version
nvcc --version

# If CUDA < 12.x, update requirements.txt:
# Replace cupy-cuda12x with cupy-cuda11x
# Replace nvidia-cuda-* packages with appropriate versions

📚 Additional Information

Model Format Support

Hugging Face format (recommended)
SafeTensors format
PyTorch .bin format

Data Format

Each test.json contains:

{
  "question": "...",
  "choices": ["A", "B", "C", "D"],
  "answer": 0,  // Index of correct answer (0-3)
  "metadata": {
    "subject": "...",
    "language": "..."
  }
}

Customization

You can modify the evaluation prompt in the build_prompt() function.

📧 Support

If you encounter issues:

Check the troubleshooting section above
Verify all prerequisites are met
Check GPU/CUDA compatibility
Review error messages carefully

🔄 Version Information

Package Version: 1.0
Python: 3.10-3.12
PyTorch: 2.8.0
vLLM: 0.11.0
Transformers: 4.57.1
CUDA: 12.x

Downloads last month: 24

Safetensors

Model size

308k params

Tensor type

BF16

Model tree for Agnes-AI/Agnes-SeaLLM-8b

Base model

Qwen/Qwen3-8B-Base

Finetuned

(321)

this model

Quantizations

1 model