CodVa-1-Small-IT is an instruction-tuned Large Language Model developed by Smilyai-labs — a team of high-school students passionate about AI research and development. This model is the instruction-tuned (IT) variant of our base model, CodVa-1-Small, fine-tuned to follow instructions, answer questions, and assist with coding and mathematical reasoning tasks.
Key Features
🧠 Custom Architecture — Built on our in-house architecture designed from the ground up to maximise performance on code and math benchmarks
⚡ Mixture of Experts (MoE) FFN — Sparse MoE feed-forward layers with shared experts, giving a large parameter count while keeping per-token compute low
🔄 Grouped Query Attention (GQA) — Reduces KV-cache memory footprint significantly compared to standard multi-head attention, enabling longer context at inference time
📐 RoPE with Extended Theta — Rotary positional embeddings with a high theta value (5,000,000) for improved length generalisation beyond the training context window
🔧 QK Norm — Per-head RMS normalisation on queries and keys for more stable attention across long sequences
💬 Instruction Tuned — Fine-tuned on a curated dataset of code, math, and general instruction-following examples to produce a helpful, chat-capable assistant
Architecture Details
Component
Configuration
Architecture
Custom Decoder-Only Transformer
Hidden Dimension
1536
Layers
28
Attention Heads
24 (Query) / 6 (KV)
Attention Type
Grouped Query Attention (GQA)
Positional Encoding
RoPE (θ = 5,000,000)
FFN Type
Dense SwiGLU + Sparse MoE
MoE Experts
16 routed + 2 shared
MoE Top-K
2 routed experts per token
MoE Hidden Dim
1024
MoE Frequency
Every 2 layers
Context Length
4096 tokens
Vocabulary Size
32,000 (+ special tokens)
Normalisation
RMSNorm throughout
QK Norm
✅ Enabled
Structural Bias
✅ Enabled (4 relation types)
Precision
BFloat16
Training Details
Pre-Training (CodVa-1-Small Base)
The base model was pre-trained from scratch on a large corpus of code and mathematical text, optimised for strong reasoning and programming capabilities.
Instruction Fine-Tuning (This Model)
CodVa-1-Small-IT was produced by supervised fine-tuning (SFT) of the base model on a curated instruction dataset.
💻 Code generation and completion — Writing functions, classes, scripts, and full programs across a range of programming languages
🐛 Code explanation and debugging — Understanding existing code, identifying bugs, and suggesting fixes
📐 Mathematical reasoning — Step-by-step problem solving, symbolic reasoning, and numerical computation
🤖 General instruction following — Answering questions, summarising text, and following natural language instructions
🧪 Research and experimentation — Exploring small-scale LLM behaviour, fine-tuning further, or using as a base for downstream tasks
Limitations
As with all language models — particularly smaller ones — there are important limitations to be aware of:
Factual accuracy — The model may confidently produce incorrect information. Always verify outputs for any critical use case
Context length — Performance may degrade on inputs approaching the 4096-token context limit
Mathematical reliability — While the architecture is optimised for math, the model can still make arithmetic and reasoning errors, especially on complex multi-step problems
Safety — This model has not undergone extensive red-teaming or safety fine-tuning. It should not be deployed in production applications without additional safety measures
Scale — As a small model, it will be outperformed by larger models on complex tasks. It is best suited for lightweight deployment, experimentation, and research
About Smilyai-labs
Smilyai-labs is a team of high-school students who are passionate about AI research. We design and train our own model architectures from scratch, rather than fine-tuning existing open-source models, with the goal of learning every part of the deep learning stack — from architecture design and custom CUDA kernels to dataset curation and training infrastructure.
CodVa is our flagship model series, focused on code and mathematical reasoning.
We are students building real models. Feedback, collaboration offers, and questions are very welcome.
Citation
If you use CodVa-1-Small-IT in your research or projects, please consider citing or crediting the Smilyai-labs team:
@misc{codva1small,
author = {Smilyai-labs},
title = {CodVa-1-Small-IT: An Instruction-Tuned Code and Math Language Model},
year = {2025},
howpublished = {\url{https://huggingface.co/Smilyai-labs/CodVa-1-Small-IT}},
}
License
Please refer to the repository's license file for terms of use. If you intend to use this model for commercial purposes, please contact the Smilyai-labs team directly.
