The model has 2,945,914 parameters. It combines Gated DeltaNet linear attention, gated full attention, sparse top-2 MoE routing, and a shared expert in a checkpoint small enough for implementation testing and experimentation.

This is a synthetic tiny checkpoint. It is not an official Qwen model, does not contain weights from an original Qwen checkpoint, and should not be expected to match the quality or capabilities of production Qwen models.

Repository contents

hf/: the final Hugging Face checkpoint and tokenizer
example_generate.py: a minimal generation example
eval_text_generation.json: generations from the final training evaluation
artifact_metadata.json: training arguments, metrics, router usage, and the expanded configuration
qwen3_next_config_dump.json: a standalone configuration dump

Optimizer checkpoints and the full training log are intentionally omitted from the distribution package.

Important architecture distinction

This checkpoint uses:

text

Qwen3NextConfig
Qwen3NextForCausalLM
model_type: qwen3_next

It does not use the separate regular Qwen3 MoE implementation:

text

Qwen3MoeConfig
Qwen3MoeForCausalLM
model_type: qwen3_moe

The model follows the Qwen3-Next hybrid layer pattern: three Gated DeltaNet linear-attention layers followed by one gated full-attention layer. Every layer contains a sparse MoE block.

Relationship to Qwen3-Next 80B

The architecture is a deliberately scaled-down member of the same Transformers architecture family as Qwen/Qwen3-Next-80B-A3B-Instruct:

text

3 x (Gated DeltaNet -> MoE)
1 x (Gated full attention -> MoE)

The original model repeats this four-layer pattern multiple times and uses hundreds of experts. Tiny Qwen3-Next 3M keeps one four-layer cycle and eight experts so that the important execution paths remain present at a much smaller scale.

This checkpoint is pretrained only. It has not been instruction-tuned and does not reproduce Qwen3-Next 80B behavior.

Model architecture

yaml

architecture: Qwen3NextForCausalLM
model_type: qwen3_next
parameter_count: 2,945,914

model_vocab_size: 1,024
tokenizer_size: 1,003
hidden_size: 216
intermediate_size: 540
num_hidden_layers: 4

layer_types:
  - linear_attention
  - linear_attention
  - linear_attention
  - full_attention

num_attention_heads: 8
num_key_value_heads: 1
head_dim: 56
partial_rotary_factor: 0.25
rope_theta: 10,000,000

linear_num_key_heads: 4
linear_key_head_dim: 54
linear_num_value_heads: 8
linear_value_head_dim: 54
linear_conv_kernel_dim: 4

num_experts: 8
num_experts_per_tok: 2
moe_intermediate_size: 54
shared_expert_intermediate_size: 54
norm_topk_prob: true
router_aux_loss_coef: 0.01

tie_word_embeddings: true
rms_norm_eps: 1.0e-6
max_position_embeddings: 1,024

The Gated DeltaNet dimensions retain the characteristic Qwen3-Next ratios:

key width: 4 x 54 = 216, equal to the hidden size
value width: 8 x 54 = 432, twice the hidden size

Full attention uses eight query heads and one key/value head. The query width is 8 x 56 = 448, while the key/value width is 56.

MoE configuration

Each of the four decoder layers contains eight routed experts and one shared expert. Every token selects two of the eight routed experts:

yaml

routed_experts: 8
top_k: 2
shared_expert: true
normalized_top_k_weights: true

Router auxiliary loss was active during training. All experts received traffic. Final aggregate expert fractions ranged from approximately 0.060 to 0.208 depending on layer and expert; no expert was unused.

Training data

The model was trained on the full TinyStories training corpus using an independent 1% validation split:

yaml

selected_stories: 2,119,489
training_stories: 2,098,294
validation_stories: 21,195
validation_fraction: 0.01

training_blocks: 2,441,089
validation_blocks: 24,608
block_size: 256

Stories were joined into a continuous packed stream:

BOS + story 1 + EOS + BOS + story 2 + EOS + ...

The stream was split into fixed 256-token blocks without padding. Only the final incomplete block was discarded.

Tokenizer

The checkpoint uses a small custom byte-level BPE tokenizer. The base BPE vocabulary was trained with:

text

BPE()
ByteLevel(add_prefix_space=False)
base_vocab_size: 1,000
min_frequency: 2
normalizer: None

Special tokens were then added at fixed IDs:

text

<s>          -> 1000
</s>         -> 1001
<|im_start|> -> 1002

The tokenizer uses <s> as both BOS and padding, and </s> as EOS. The model configuration reserves 1,024 embedding rows while the tokenizer exposes 1,003 tokens.

Training setup

The checkpoint was trained from scratch in float32 on an NVIDIA GeForce GTX 1650:

yaml

dtype: float32
batch_size: 16
block_size: 256
training_steps: 152,568
epochs: 1.0
tokens_processed: 624,918,528

optimizer: AdamW
learning_rate: 3.0e-4
warmup_steps: 1,000
scheduler: warmup + cosine decay
minimum_learning_rate: 3.0e-5
weight_decay: 0.0
grad_clip: 1.0

training_time: 13h 48m 45s

Evaluation

The final checkpoint produced:

yaml

final_train_loss: 1.4145
validation_loss: 1.4320
validation_perplexity: 4.1871

Validation loss was computed over 16 batches, or 65,536 tokens, from the independent packed validation split. These numbers are compact checkpoint diagnostics, not general language-model benchmark results.

Example generation

With prompt Once upon, sampling seed 0 produced the following representative TinyStories-style opening:

text

Once upon a time, there was a little girl named Lily. She loved to play with
her toys and go on adventures. One day, she went outside and saw a rainbow in
the sky. It was so pretty that it made her feel even happier than before.

Sampling is stochastic. Other seeds may produce a boy, an adult, an animal, or another kind of TinyStories character. The model usually produces recognizable English, but semantic contradictions, unfinished sentences, invented words, and repetition remain possible at this size.

Usage

Install the requirements:

pip install -r requirements.txt

Run the included local example from the repository root:

python example_generate.py

To load the package from Hugging Face Hub, resolve the hf directory to a local path first. This also avoids a Transformers 5.14.1 local-subfolder issue in which generation configuration lookup may incorrectly fall back to the repository root:

python

import torch
from pathlib import Path
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "shibatch/tinyqwen3next3m"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_dir = Path(snapshot_download(
    repo_id=repo,
    allow_patterns=["hf/*"],
)) / "hf"

tokenizer = AutoTokenizer.from_pretrained(model_dir)
model = AutoModelForCausalLM.from_pretrained(
    model_dir,
    dtype=torch.float32,
    experts_implementation="batched_mm",
).to(device)
model.eval()

prompt = "Once upon"
input_ids = torch.tensor(
    [[tokenizer.bos_token_id] + tokenizer.encode(
        prompt,
        add_special_tokens=False,
    )],
    dtype=torch.long,
    device=device,
)

torch.manual_seed(0)
if device.type == "cuda":
    torch.cuda.manual_seed_all(0)

with torch.no_grad():
    output = model.generate(
        input_ids=input_ids,
        max_new_tokens=100,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        top_k=40,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
        output_router_logits=False,
    )

print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))

Loading requirements

The checkpoint requires a Transformers release containing Qwen3NextForCausalLM. It was trained and tested with:

text

transformers 5.14.1
torch 2.14.0.dev20260720+cu126

The optional flash-linear-attention and causal-conv1d packages are not required. Without them, Transformers uses its PyTorch Gated DeltaNet fallback, which is the path used to train and validate this checkpoint.

The examples explicitly select experts_implementation="batched_mm". The model's routed experts have an intermediate width of 54, while the default PyTorch grouped_mm CUDA path in some recent Torch/Transformers combinations requires expert matrix strides to be multiples of 16 bytes. Without the explicit compatible implementation, loading succeeds but the first forward pass can fail with:

RuntimeError: strides should be multiple of 16 bytes

batched_mm evaluates the same expert weights without that grouped-kernel layout restriction. experts_implementation="eager" is also a compatible, slower fallback.

Intended uses

This model is intended for:

testing Qwen3NextConfig and Qwen3NextForCausalLM
testing Gated DeltaNet fallback implementations
testing the hybrid linear/full-attention layer pattern
testing sparse top-2 MoE routing and shared experts
checking router load-balancing loss
exercising custom tokenizer loading
testing generate(), save_pretrained(), and from_pretrained()
compact inference-engine and architecture experiments

It is not intended for:

instruction following or chat
factual question answering
high-quality long-form generation
production deployment
safety-critical use
benchmark comparison with production Qwen models

Limitations

Known limitations include:

only 2.95 million parameters
small 1,003-token tokenizer
English TinyStories-only pretraining
weak factual knowledge and reasoning
no instruction tuning or chat template
occasional grammatical and semantic errors
invented words and truncated sentences
repetition and template-like stories
no quality-equivalence claim with official Qwen models
no current llama.cpp or GGUF inference support assumed for Qwen3-Next

Notes on GGUF

The checkpoint is distributed as a normal float32 Hugging Face Safetensors model. A useful GGUF build requires converter and runtime support for the full Qwen3-Next graph, including Gated DeltaNet, hybrid attention, routed experts, and the shared expert. Merely placing tensors in a GGUF container is not sufficient for compatible inference.

Citation

This is a synthetic tiny Qwen3-Next-compatible MoE checkpoint trained from scratch on TinyStories. It is intended for implementation validation, debugging, education, and small-scale architecture experiments.