This is a 340M controlled-pretraining checkpoint released for the paper Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus.

It is one member of the Massive Activations HLA model release and the corresponding Hugging Face Collection. The official analysis and reproducibility code is available at StartluxLabs/Massive-Activations-HLA.

Checkpoint details

Field	Value
Model	`gdn-340m-isp-hybrid-3to1-10b`
Scale	340M
Training tokens	10B
Experiment	ISP
Full-attention layers	3, 6, 9, 12, 15, 18, 21, 24 (one-based)
Output-gating variant	GDN output gate enabled; full-attention output gate disabled
Final training step	19073
Weight format	Safetensors

Compatibility and reproducibility scope

This checkpoint loads with the public, pinned environment documented in the GitHub repository:

bash

conda create -n ma-hla python=3.12 -y
conda activate ma-hla
bash scripts/install_released_gdn_cu126.sh

Public FLA is pinned to v0.5.2, commit 9c8e42e762fce087c27b673af4922795d9edb85e. Exact A800/CUDA 12.6 package versions are recorded in requirements/released-gdn-cu126.txt.

Loading

REPO_ID = "startlux-models/gdn-340m-isp-hybrid-3to1-10b"

python

import fla.models.gated_deltanet  # registers the custom config/model
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(REPO_ID)
model = AutoModelForCausalLM.from_pretrained(
    REPO_ID,
    torch_dtype="auto",
)

For analysis, prefer the GitHub registry and scripts because they validate the FLA version and recover full-attention layer metadata consistently.

Reproduction

Use the official GitHub repository for the tested analysis workflow, PAS/ISP metrics, lifecycle atlases, installation instructions, and model registry:

Intended use and limitations

This checkpoint is a research artifact for studying massive activations, hybrid linear-attention architectures, PAS/ISP morphology, attention placement, output gating, and scale. It is not instruction-tuned, safety-tuned, or validated for production deployment. It has not been comprehensively evaluated for downstream accuracy, factuality, bias, robustness, privacy, or safety.

The checkpoint uses the custom Transformers architecture GatedDeltaNetForCausalLM (model_type="gated_deltanet").

Training data

The model was trained from scratch on open data. A dataset-level composition and sampling breakdown is not included in this release; users should not infer language or domain coverage beyond the published project materials.

Citation

Please cite the accompanying paper. The arXiv link and final BibTeX entry will be added after the preprint metadata is public.

License

The released model artifacts are available under the Apache License 2.0. See LICENSE. Third-party software and datasets retain their own licenses and terms.