PAKNeura-V3 is a state-of-the-art Mixture of Experts (MoE) language model with 671B total parameters (37B active), designed to deliver high computational efficiency with superior reasoning capabilities. As Pakistan's first open-weight AI initiative, PAKNeura-V3 represents a significant milestone in democratizing access to cutting-edge language model technology for the Pakistani community and beyond.
The model is built upon three core technical foundations:
Mixture of Experts Architecture: Leveraging 256 routed experts with 8 active experts per token, enabling efficient computation while maintaining broad knowledge coverage.
Extended Context Understanding: Native support for up to 128K tokens of context, facilitating complex long-form reasoning and document understanding.
Multilingual Capability: Optimized for English, Urdu, and Pakistani regional languages, with a vocabulary size of 129,280 tokens.
Model Architecture
PAKNeura-V3 employs a Transformer-based Mixture of Experts (MoE) architecture with the following detailed configuration:
Hyperparameter
Value
Total Parameters
671B
Active Parameters
37B
Number of Layers
61
Hidden Dimension
7,168
Attention Heads
128
Key-Value Heads
128
Intermediate Size (FFN)
18,432
MoE Intermediate Size
2,048
Number of Routed Experts
256
Number of Shared Experts
1
Top-K Experts per Token
8
Expert Group Count
8
Top-K Groups
4
KV LoRA Rank
512
Q LoRA Rank
1,536
Vocabulary Size
129,280
Max Position Embeddings
163,840
Context Length
128K tokens
Norm Type
RMSNorm (eps: 1e-6)
Activation Function
SiLU
Weight Precision
bfloat16 / FP8
RoPE Theta
10,000
Attention Mechanism
PAKNeura-V3 utilizes Multi-head Latent Attention (MLA) with KV compression for efficient inference. The model supports both MHA and MQA modes, enabling flexible deployment across different hardware configurations.
Quantization
The model supports FP8 quantization (E4M3 format) with dynamic activation scaling, enabling reduced memory footprint and faster inference on compatible hardware.
Benchmarks
PAKNeura-V3 Benchmark Results
PAKNeura-V3 demonstrates competitive performance across a comprehensive suite of academic benchmarks covering reasoning, coding, knowledge, and language understanding tasks.
Consumer GPU: 4-bit quantization on RTX 4090 (dual)
Sampling: temperature = 1.0, top_p = 0.95
Chat Template
PAKNeura-V3 supports the user, assistant, and developer roles. Example usage:
python
1from transformers import AutoTokenizer
23tokenizer = AutoTokenizer.from_pretrained("pakneura/PAKNeura-V3")45messages =[6{"role":"user","content":"What is the capital of Pakistan?"},7{"role":"assistant","content":"The capital of Pakistan is Islamabad.","reasoning_content":"Islamabad is the capital since 1963..."},8]9inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
Weights Status
Component
Status
Configuration
✅ Complete
Tokenizer
✅ Complete
Model Weights
⏳ In Progress
Inference Code
⏳ In Progress
About PAKNeura
PAKNeura is a Pakistani AI organization dedicated to developing open-weight language models and advancing AI research in Pakistan. Our mission is to make cutting-edge AI technology accessible to the Pakistani community.
