You can deploy this model and start earning money today!

AryanNsc/
QuantMobile-17M

Name: QuantMobile-17M
Rating: 2.5 (5 reviews)

mobile_llm

Views

No views yet

QuantMobileLLM — Lightweight GPT-Style Language Model

MobileLLM is a lightweight GPT-style language model designed for efficiency, fast inference, and small deployment environments.
It’s trained on FineWeb-MINI and optimized with modern attention techniques.

🚀 Model Highlights

Architecture: Decoder-only GPT-style transformer
Parameters: ~17M (6 layers, 8 heads, 256 embedding dim)
Context Length: 512 tokens
Vocabulary Size: 50,304 tokens
Precision: Supports both fp16 and bf16
Optimized for: Small GPUs, mobile inference,

🧠 Architecture Details

Component	Value
Layers	6
Attention Heads	8
KV Heads	4
Embedding Dim	256
Context Length	512
Vocab Size	50,304
Attention Type	Multi-Query Attention
Norm Type	RMSNorm
Position Encoding	Rotary Position Embeddings (RoPE)
FFN Activation	SwiGLU (`silu`)

🔹 Key Optimizations

RMSNorm → Improves training stability over LayerNorm.
Multi-Query Attention → Reduces KV-cache size → lower memory footprint.
Rotary Embeddings (RoPE) → Better handling of long context windows.
safetensors checkpoints → Faster & safer loading.

📊 Training Setup

Property	Value
Dataset	FineWeb-MINI
Tokens Trained	~100M
Optimizer	AdamW
Learning Rate	6e-4 (cosine decay)
Warmup Steps	100
Batch Size	64 × 2 grad accum
Effective Batch Size	128
Mixed Precision	`fp16` / `bf16` (auto-detect)
Distributed Training	DDP
Logging	Weights & Biases (`wandb`)
Checkpoint Format	`.safetensors`

🧩 Model Checkpoints

Step	Filename	Format
Final	`mobile_llm_final.safetensors`	safetensors
Intermediate	`checkpoints/mobile_llm_step_<step>.safetensors`	safetensors

🔮 Roadmap

Train MobileLLM on FineWeb-MINI
Add multi-query attention
Export safetensors checkpoints
Quantized int8 & int4 inference
Expand training on FineWeb-1B

📜 License

This model is licensed under the MIT License.

🌐 Links

Github → MobileLLM training code

AryanNsc/QuantMobile-17M