Base checkpoint: linear-moe-hub/Gated-Deltanet-1.3B
Architecture: Gated DeltaNet / linear recurrent attention
Base training data reported by the upstream model card: SlimPajama 100B-token sample
License inherited from upstream model card: Apache-2.0

Training Run

Local source path: runs/gdn1_fwe_mqar_50m_full_ft/final
Tokenizer source: runs/gdn1_fwe_mqar_50m_full_ft/final
Training mode: full fine-tuning, no LoRA/adapter
Hardware target: 8x NVIDIA H200
Sequence length: 2048
Approximate additional token budget: ~50M additional tokens
Manifest/config: configs/gdn1_memory_mix_1b.json

Intended Research Use

This checkpoint is intended for research on:

long-context associative recall
RULER/MQAR-style state tracking
recurrent-state contamination during long generation
Reference-State Reset with Rolling Replay, a GDN/RNN adaptation of the R-SWA idea

Usage

These checkpoints use the FLA Gated DeltaNet implementation. In the current Long-GDN environment, plain GatedDeltaNetForCausalLM.from_pretrained() can hit a Transformers 5.x tied-weight metadata issue. The robust path is to patch the FLA tied-weight metadata before loading.

Install/runtime requirements:

bash

pip install torch transformers safetensors huggingface_hub
# plus an FLA package/source tree that provides:
#   fla.models.gated_deltanet.GatedDeltaNetForCausalLM

CPU Example

python

import torch
from transformers import AutoTokenizer
from fla.models.gated_deltanet import GatedDeltaNetForCausalLM

repo_id = "LLM-OS-Models/gdn1-fwe-mqar-50m-full-ft"

# Transformers 5.x compatibility patch for the installed FLA class.
if isinstance(getattr(GatedDeltaNetForCausalLM, "_tied_weights_keys", None), list):
    GatedDeltaNetForCausalLM._tied_weights_keys = {
        "lm_head.weight": "model.embeddings.weight"
    }

tokenizer = AutoTokenizer.from_pretrained(repo_id, use_fast=True)
model = GatedDeltaNetForCausalLM.from_pretrained(
    repo_id,
    torch_dtype=torch.float32,
)
model.eval()

prompt = "A special magic number is 12345. What is the special magic number?"
inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=32,
        do_sample=False,
    )
print(tokenizer.decode(output[0], skip_special_tokens=True))

Single-GPU bf16 Example

python

import torch
from transformers import AutoTokenizer
from fla.models.gated_deltanet import GatedDeltaNetForCausalLM

repo_id = "LLM-OS-Models/gdn1-fwe-mqar-50m-full-ft"

if isinstance(getattr(GatedDeltaNetForCausalLM, "_tied_weights_keys", None), list):
    GatedDeltaNetForCausalLM._tied_weights_keys = {
        "lm_head.weight": "model.embeddings.weight"
    }

tokenizer = AutoTokenizer.from_pretrained(repo_id, use_fast=True)
model = GatedDeltaNetForCausalLM.from_pretrained(
    repo_id,
    torch_dtype=torch.bfloat16,
).to("cuda")
model.eval()

prompt = "Reference facts:\n- key_alpha: value_123\n\nQuestion: key_alpha?\nAnswer:"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=32,
        do_sample=False,
    )
print(tokenizer.decode(output[0], skip_special_tokens=True))

Long-GDN Local Loader

The project repository includes a more defensive loader at scripts/gdn1_common.py::load_gdn1_causal_lm. It handles the compatibility patch and older public-checkpoint key conversion used in local experiments.

python

from pathlib import Path
import torch
from transformers import AutoTokenizer
from scripts.gdn1_common import load_gdn1_causal_lm

repo_or_local_path = Path("path/to/downloaded/checkpoint")
tokenizer = AutoTokenizer.from_pretrained(repo_or_local_path, use_fast=True)
model = load_gdn1_causal_lm(repo_or_local_path, torch_dtype=torch.bfloat16).to("cuda")

Known Results

50M smoke checkpoint. MQAR likelihood versus public base: 1K 0.1406 vs 0.0938, 2K 0.0625 vs 0.0313, 4K unchanged at 0.1094, 8K 0.0781 vs 0.0625, 16K unchanged at 0.1250, 32K unchanged at 0.0625, 64K regressed to 0.0000 from 0.1250.

Caveats

Smoke checkpoint only. Not a long-context win; use mainly as a reproducible continuation baseline.

Citation Context

Relevant background papers include Gated Delta Networks, Gated DeltaNet-2, Log-Linear Attention, and Unlimited OCR / R-SWA. This checkpoint does not implement a new architecture by itself; it is part of a checkpoint-preserving full fine-tuning and inference-control study.