İvme (Turkish: acceleration) is a series of stupidly small language models built to punch above their weight. Ivme-Coder-v1 is the first release in a new branch of the family: a ~50M parameter decoder-only model trained from scratch on Python source code.
Model Details
Parameter
Value
Architecture
Decoder-only transformer, dense (no loops, no exotic recurrence)
Parameters
~50M
Layers
12
Hidden dim
512
FFN
SwiGLU (~2.67× expansion)
Attention heads
8, full attention (no GQA)
Context length
1024 tokens
Vocab size
16,000 (custom byte-level BPE, trained on Python source)
Positional encoding
RoPE (θ=10,000)
Normalization
RMSNorm (pre-norm)
Embeddings
Tied input/output
Biases
None
Coder's architecture deliberately mirrors the same core recipe as the Conversate line — RoPE, SwiGLU, RMSNorm, tied embeddings, Muon-optimized body weights. The point of keeping the recipe constant across both lines is to isolate what changes with domain-specific data and a narrower target distribution, not to introduce a second set of untested architectural choices at the same time.
Coder was trained on a filtered slice of bigcode/starcoderdata (Python configuration only), the same decontaminated, permissively-licensed corpus underlying the original StarCoder models. Raw files were passed through length and quality heuristics — dropping empty or oversized files, minified or generated-looking source, and files with abnormal symbol density — before being packed into fixed-length training shards.
Unlike a general-text model, code has a far narrower and more rigid target distribution — Python's syntax is small, unambiguous, and repetitive relative to open-domain English. That changes the token math: Coder deliberately targets well past strict compute-optimal scaling for a model this size, but nowhere near the scale a general-purpose model in the same parameter class would need, since there is comparatively little left to learn once syntax, common idioms, and standard-library usage patterns are well covered.
Trained on an RTX PRO 6000 Blackwell GPU on Google Colab, sustaining a stable ~517,000 tokens/second throughout training — the full run completed in under two hours.
Does it actually write code?
Short answer: it writes code-shaped text, reliably. It does not reliably write correct code. At 50M parameters and ~2.5B training tokens, Coder has clearly internalized Python's surface structure — indentation, control flow, function and class definitions, docstring conventions, common standard-library idioms — but it has not learned to reason about what a function is supposed to compute. Expect syntactically valid, plausible-looking completions that are frequently wrong on inspection: incorrect recursive logic, docstrings that don't match the implementation below them, redefinitions of the same method twice in one class, and functions that trail off into a related-but-different function rather than terminating cleanly.
This is the expected and honest ceiling for a model this size trained this way — it is closer to a highly fluent autocomplete for Python's shape than to a coding assistant, and it should be treated accordingly.
Sample output, EMA weights, temperature 0.8, top_k 50:
Prompt:def fibonacci(n):
def fibonacci(n):
# Fibonacci for 5.2
if n < 3:
return n
if n % 2 == 0:
return n
return n // 2
from collections import OrderedDict # For compatibility with Python 2.6
def reverse(self, n):
# Fibonacci for 5.2
if n > 3:
return n
if n % 2 == 0:
return n
if n % 3 == 0:
return n
return n // 4
The indentation, control-flow shape, and import placement are all locally plausible. The logic is not — this is not a working Fibonacci implementation, and the model has no way to know that. This gap between fluency and correctness is the central thing to understand about Coder before using it for anything beyond exploration.
Limitations
Base model only, not instruction tuned — completes code, does not follow instructions
Learns Python syntax and idiom well; does not reliably produce logically correct or semantically meaningful code
1024 token context window
Python only — no other languages in the training mix
No retrieval-augmented or fine-tuned variant currently exists; this is a raw pretraining checkpoint
Note on batch generation: use left-padding (tokenizer.padding_side = "left") — the model does not use an explicit attention mask over padded positions, so right-padding within a batch will give incorrect results.
What's Next
Coder v1 establishes the animal line and the basic recipe: same architecture family as the fruit line, redirected at a single narrow domain. Future directions under consideration include a larger token budget to test where the syntax-fluency ceiling actually sits for a model this size, an instruction-tuned variant for actual completion/chat use rather than raw continuation, and eventually a second animal-line entry in a different narrow domain to test how well the recipe generalizes across disciplines, not just across scale.
You can check our other upcoming models on our organization card!
