You can deploy this model and start earning money today!

ecreeth/
pallaslm-110m-sft

HuggingFaceTB/smoltalk

mmlu

gsm8k

mit

Views

No views yet

🚀 PallasLM 110M SFT (Gated MLA + Sparse MoE)

PallasLM 110M SFT is a high-efficiency Language Model trained on ClimbMix and fine-tuned on SmolTalk, MMLU, and GSM8K. It features state-of-the-art DeepSeek-V3 architectural innovations:

Gated Multi-Head Latent Attention (Gated MLA)
Fine-Grained Sparse Mixture-of-Experts (MoE) (8 experts, Top-2 routing)
Fused Muon + AdamW Optimization

📊 Model Architecture Specification

Parameter	Value
Total Parameters	110,732,160
Active Parameters per Token	39,953,280
Number of Layers	14
Embedding Dimension	384
FFN Hidden Dimension	1,536
MoE Routing	8 experts total, Top-2 active per token (Layers 4..13)
Attention Mechanism	Gated MLA (Compressed KV Rank: 64, Q Rank: 64)
Vocabulary Size	32,768 (Rust BPE / Tiktoken)
Context Length	4,096 tokens

📈 Benchmark Evaluation & Performance

Zero-Shot Chat Evaluation (`chat_eval.py`)

Across 17,590 test questions, PallasLM SFT outperforms random chance baseline across all tasks:

Task Benchmark	Total Test Questions	Model Accuracy	Random Baseline	Accuracy Above Random
🟢 ARC-Easy	2,376	30.09% (715/2376)	25.00%	+5.09%
🟠 ARC-Challenge	1,172	27.99% (328/1172)	25.00%	+2.99%
🔵 MMLU (All 57 Subjects)	14,042	28.77% (4040/14042)	25.00%	+3.77%
🌟 ChatCORE Metric	17,590 Total	0.0527	0.0000	+5.27% Centered Score

Training & Fine-Tuning Metrics

Final SFT Training Step Loss: 1.2894 (peak low: 1.2567)
Minimum Validation BPB: 0.4308 (Bits-Per-Byte)
Sustained Training Speed: 136,500 tokens/sec on NVIDIA A100

High-Throughput Inference (NVIDIA A100-SXM4-40GB)

Decode Batch Size	TTFT (ms)	TPOT (ms)	Generation Speed	MBU %	VRAM Consumption
Batch 1	~29.7 ms	~27.2 ms	36.6 tok/s	82%	0.48 GiB
Batch 16	~30.0 ms	~27.4 ms	580 tok/s	89%	0.56 GiB
Batch 64	~29.6 ms	~31.9 ms	2,011 tok/s	99%	0.96 GiB
Batch 128	~36.3 ms	~37.3 ms	🌟 3,441 tok/s	🔥 110% (L2 SRAM Cache Hit)	1.49 GiB

💻 How to Load Model Weights in PyTorch

python

import torch

# 1. Load model weights
state_dict = torch.load("model.pt", map_location="cpu")

# State dict keys format:
# - "wte.weight": (32768, 384)
# - "blocks.0..13.mixer...": Gated MLA attention parameters
# - "blocks.4..13.ffn.router.weight": MoE router (8, 384)
# - "blocks.4..13.ffn.experts.0..7...": MoE expert weights

💬 Prompt Template Format (ChatML)

During fine-tuning, the model was trained on ChatML conversation tags:

<|bos|><|user_start|>What is the capital of France?<|user_end|><|assistant_start|>

Special Token IDs:

<|bos|>: Beginning of sequence
<|user_start|>: Start of user turn
<|user_end|>: End of user turn
<|assistant_start|>: Start of assistant turn
<|assistant_end|>: End of assistant turn (EOS)

📜 Training & Fine-Tuning Setup

Pre-Training: Pre-trained on ClimbMix dataset using the fused Muon + AdamW optimizer (Muon on 2D weight matrices, AdamW on embeddings and LayerNorms).
Supervised Fine-Tuning (SFT): Fine-tuned on SmolTalk (460K rows), MMLU (auxiliary train), and GSM8K (math reasoning) for 7,049 steps.

📄 License

This model repository contains model weight parameters released under the MIT License.

ecreeth/pallaslm-110m-sft