Agnes-SeaLLM-8B

Introduction

We introduce Agnes-SeaLLM-8B, a compact Large Language Model (LLM) meticulously optimized for Southeast Asian languages. Despite its efficient footprint, it delivers performance rivaling much larger open-source models, excelling in tasks such as mathematical reasoning, translation, and instruction following. Furthermore, it has been specially tuned to enhance reliability, minimize hallucinations, and provide culturally sensitive, safe responses tailored to the Southeast Asian context.

🔥 Highlights

【SeaExam Summary statistics】

• Agnes-SeaLLM-8B (75.32 avg) crushes 8B peers and outpaces Sailor2-20B, offering 70B-tier performance with compact, low-resource deployment.
• Exceptional scores in both M3Exam and MMLU prove it bridges regional linguistic nuance with elite global reasoning.

【M3Exam Model evaluation sub-statistics】

• With a 71.29 average, Agnes-SeaLLM-8B delivers "70B-class" performance in an 8B footprint, outperforming the 20B Sailor2 and crushing 8B peers like Llama-3 and GLM-4.
• It sets a new benchmark for Southeast Asia, achieving top-tier scores in Indonesian (72.97) and Vietnamese (70.03)**, while maintaining elite English (93.24) and Chinese (83.70) proficiency.

【MMLU Model evaluation sub-statistics】

【Indonesian Fine-grained discipline statistics】

【MMLU Fine-grained discipline statistics】

📦 SeaExam Evaluation Package

/
├── function.py      # Main function 
├── main.py           # Main evaluation script
├── requirements.txt      # Python dependencies
├── data/                 # Test datasets (26MB)
│   ├── m3exam-chinese/
│   ├── m3exam-english/
│   ├── m3exam-indonesian/
│   ├── m3exam-thai/
│   ├── m3exam-vietnamese/
│   ├── mmlu-chinese/
│   ├── mmlu-english/
│   ├── mmlu-indonesian/
│   ├── mmlu-thai/
│   └── mmlu-vietnamese/
├── models/               # [YOU NEED TO ADD] Place your model here
│   └── Llama-SEA-LION-v2-8B-IT/  (or your model name)
├── README.md             # This file
└── quick_run.sh                # Quick start script

🔧 Prerequisites

Hardware Requirements

• GPU: Minimum 2 GPUs with 24GB+ VRAM each
• RAM: 64GB+ recommended
• Storage: ~20GB (model + data + outputs)

Software Requirements

• OS: Linux (tested on Ubuntu)
• Python: 3.10 - 3.12
• CUDA: 12.x
• GPU Driver: Latest NVIDIA driver

📥 Installation Steps

1. Copy this folder to your server

2. Prepare your model

3. Create Python virtual environment (recommended)

4. Install dependencies

⚙️ Configuration

Method 1: Edit main.py directly

Method 2: Use environment variable

Configuration Tips

GPU Selection

• Check available GPUs: nvidia-smi
• Use specific GPUs: CUDA_VISIBLE_DEVICES="0,1" (uses GPU 0 and 1)
• Match TENSOR_PARALLEL_SIZE with number of GPUs in CUDA_VISIBLE_DEVICES

Memory Optimization

1. Reduce BATCH_SIZE (e.g., 128, 64, 32)
2. Reduce gpu_memory_utilization in code (default 0.95 → 0.8)
3. Reduce max_model_len (default 4096 → 2048)

🚀 Running the Evaluation

Quick Start

Using the run script

Monitor Progress

Resume from Checkpoint

• Just run the script again
• It will resume from where it stopped

📊 Output Files

• result.txt - Detailed evaluation report with accuracy by language/subject
• progress.json - Progress checkpoint (can be deleted after successful completion)

🔍 Verification

📝 Expected Results

• Total Questions: ~5000 (varies by data version)
• Runtime: 1-3 hours (depends on hardware)
• Output Accuracy: Model-dependent (typically 40-80%)

❓ Troubleshooting

Issue: "No module named 'vllm'"

Issue: "CUDA out of memory"

1. Use fewer GPUs but ensure VRAM > 24GB per GPU
2. Check GPU usage: nvidia-smi

Issue: "No test data files found"

Issue: CUDA version mismatch

📚 Additional Information

Model Format Support

• Hugging Face format (recommended)
• SafeTensors format
• PyTorch .bin format

Data Format

Customization

📧 Support

1. Check the troubleshooting section above
2. Verify all prerequisites are met
3. Check GPU/CUDA compatibility
4. Review error messages carefully

🔄 Version Information

• Package Version: 1.0
• Python: 3.10-3.12
• PyTorch: 2.8.0
• vLLM: 0.11.0
• Transformers: 4.57.1
• CUDA: 12.x