You can deploy this model and start earning money today!

crumb-ai/
abl_3_1_backloaded

Name: abl_3_1_backloaded
Availability: InStock

bigcode/the-stack-dedup-v2

mit

Views

No views yet

CRUMB `abl_3_1_backloaded`

Model Overview

abl_3_1_backloaded is a hybrid decoder-only language model from the CRUMB (Compact Recurrent-hybrid Underlying Mamba Blocks) project. It interleaves Mamba-3 selective state-space layers with GQA (Grouped-Query Attention) layers at a 3:1 Mamba-majority ratio with backloaded placement, and was pre-trained exclusively on Python source code.

It is one of eleven ablation variants trained to study how the Mamba-to-attention ratio and the placement of attention layers affect small-scale (~150M-parameter) language models on Python program synthesis.

Architecture

Property	Value
Total parameters	148,210,992 (148.2M)
`d_model`	768
`n_layers`	12
`n_heads`	12
`n_kv_heads`	4 (GQA)
`d_head`	64
`d_ff`	3072
`vocab_size`	32768
`seq_len`	4096
Tie embeddings	yes
Pos. encoding	RoPE (base = 10000)
Mamba layer type	Mamba-3 (d_state=64, expand=2, headdim=64, ngroups=1, chunk=64)

Mamba : Attention ratio — 3 : 1

9 Mamba layers + 3 GQA attention layers (3 Mamba blocks per attention block).

Placement — Backloaded

Attention layers are stacked at the end of the network. Layer order: M M M M M M M M M A A A

This placement keeps the final layers — closest to the output projection — precise and attention-based, while the earlier layers compress the input through linear-time Mamba blocks.

Training

Property	Value
Training data	Python subset of `bigcode/the-stack-dedup-v2`
Tokens seen	5,367,406,590 (~5.37 B)
Steps	163,840
Context length	4096
Training time	47 h 43 m 14 s
Final learning rate	3.00e-05

Evaluation Method

Perplexity (primary metric)

Per-token cross-entropy loss with BF16 autocast, computed over the full held-out evaluation set.

Setting	Value
Eval sequences	20,063 batches
Eval tokens	328,631,940
Implementation	`src/evaluation/perplexity.py`

Generation-based metrics

Python syntax validity — 200 free-form completions generated per model from 49 diverse Python prompts at temperature=0.8, top_k=50, max_new_tokens=128; each completion checked with ast.parse(). Implementation: src/evaluation/syntax_validity.py.
Qualitative side-by-side completions — 10 fixed prompts at temperature=0.6, top_k=50, max_new_tokens=200, identical random seed per prompt. Implementation: src/evaluation/qualitative_comparison.py.

Evaluation Results

Metric	Value
Eval loss	1.2382
Eval perplexity	3.4493
Eval time	3,102.31 s (~52 min)
Syntax validity (n=200)	63 / 200 → 31.5 %
Inference gen. time (200×128 tok)	184.11 s

Rank Summary

Out of 11 ablation configurations evaluated at the same token budget:

Rank	Model	Perplexity
1	`abl_2_1_interleaved`	3.4182
2	`abl_3_1_interleaved`	3.4359
3	`abl_3_1_backloaded`	3.4493
4	`abl_2_1_backloaded`	3.4683
5	`abl_1_1_backloaded`	3.4763
6	`abl_pure_mamba`	3.5237
7	`abl_1_1_interleaved`	3.5407
8	`abl_pure_attn`	3.5939
9	`abl_3_1_frontloaded`	3.6798
10	`abl_2_1_frontloaded`	3.7078
11	`abl_1_1_frontloaded`	3.7315

abl_3_1_backloaded is the best backloaded variant across all three ratios (PPL 3.449) and the 3rd-best model overall, narrowly behind the two interleaved leaders. This result supports the broader finding that backloaded attention is competitive with interleaved attention, and that Mamba-majority hybrids (3:1) are competitive with — but not superior to — 2:1 hybrids at this scale.

Intended Use & Limitations

Domain: Python source-code language modelling.
Base model only: no instruction tuning, no chat alignment, no safety filtering. Outputs are unconstrained code completions.
Repetitive degeneration: all base CRUMB models tend to repeat function signatures / docstrings during free-form generation; this is expected behaviour for unaligned base models.

Citation / Context

This model is part of the CRUMB Phase-1 ablation study:

Efficient Architectural Hybrids for Small-Scale Language Models in Python Program Synthesis — Department of Computer Science and Engineering, Daffodil International University. Findings documented in documents/phase1_ablation_findings.md.

How to Load