Navdyut 240M (2K Context)
Navdyut 240M is a lightweight, decoder-only transformer model trained from scratch with a 2,048-token context window. This model was explicitly designed to test the limits of "Blank Slate" isolation—training purely on synthetic datasets and code to evaluate its emergent reasoning capabilities.
Model Architecture
The model utilizes a modern transformer architecture featuring Grouped Query Attention (GQA).
- Parameters: ~240 Million
- Layers: 10
- Attention Heads: 20 (Q) / 5 (KV) - GQA Ratio 4:1
- Embedding Dimension: 1280
- Context Length: 2048 tokens
- Vocabulary Size: 8192
Training Corpus
The model was trained on exactly 14.3 Billion tokens using a proprietary blend of synthetic and logic-focused datasets. The composition is highly curated to isolate reasoning and structural generation:
- 50% General Knowledge & Synthetic Stories
- 30% Curated Python Code & Educational Programming Data
- 20% Web-Scraped Mathematical Reasoning Data
Limitations & Biases
Because this model was explicitly trained in "synthetic isolation" without access to massive real-world canonical codebases (like GitHub), it struggles with zero-shot code generation on canonical benchmarks (e.g., HumanEval). It is intended as a foundational base model for subsequent instruction tuning, routing experiments, or function-calling adaptations.
