This is SmolLM2-1.7B with its tensors converted and stitched into a Gated DeltaNet configuration: 24 layers, d_model=2048, 32 states of width d_k=64 / d_v=256, one mixer per layer, no attention and no separate feed-forward anywhere in the model.

It is completely untrained in this form. The weights are the donor's, rearranged. Nothing here has taken a gradient step since the surgery.

A 135M sibling, cut the same way from SmolLM2-135M, is at kennethgrace/smollm2-135m-collapsed-gdn.

Each donor decoder layer is two sub-layers — attention, then a gated feed-forward. Both fold into a single Gated DeltaNet. Five of the donor's seven matrices transfer with no slicing and no padding:

A Gated DeltaNet's output path already is an up/gate/down arrangement — the same shape as a SwiGLU — so the feed-forward has somewhere to land, while the query and key projections are already shaped for a key-addressed memory.

All 120 transplants are bit-exact. Embeddings, the tie, and every RMSNorm come across untouched. Attention's v_proj and o_proj are homeless — the feed-forward claimed the value and output side first — and are left out. The short convolutions are initialised to a causal identity (all taps zero but the last), so a freshly collapsed layer computes its transplanted projections untouched; the default convolution init would scramble them before a single gradient step.

n_heads must divide the query rows and the feed-forward width. This donor is plain multi-head attention — 32 query heads over 32 key/value heads, so the query/key ratio is 1 — which leaves every power of two up to 2048 admissible.

32 was chosen because d_k=64 is exactly the donor's head_dim. The head partition therefore transfers intact: 32 attention heads of width 64 become 32 memory states of width 64, same numbers read against the same grouping. Each state gets its own key group, so the layout is diagonal.

This is the one thing that does not survive in the 135M sibling, where nine query heads of width 64 had to become three states of width 192 — nine does not divide a 1536-wide feed-forward. Smaller head counts here would buy wider keys at the cost of a larger recurrent state and a re-cut partition; larger ones shrink both.

This is not a transformers architecture and will not load with AutoModelForCausalLM. Generation runs in constant memory through model.step(ids, states) rather than re-reading a growing prefix.

Note that float32 weights alone are ~6 GB, so a full fine-tune with Adam wants considerably more device memory than that.

Donor: HuggingFaceTB/SmolLM2-1.7B at revision effd688a12921b4cc83e3312b6feb579f70f9c71, Apache-2.0. This derivative is released under the same license. The Gated DeltaNet implementation is lumen, MIT.