You can deploy this model and start earning money today!

GeneralFY/
mini-research-on-looped-transformers2026

Name: mini-research-on-looped-transformers2026
Availability: InStock

pytorch

looped-transformer

recurrent-transformer

language-modeling

fineweb

apache-2.0

Views

No views yet

Mini Research on Looped Transformers 2026

This model repository contains the important final 100M-token checkpoints from the research codebase. The models are small causal language models whose shared Qwen3-style decoder core is reused recurrently. The release uses raw PyTorch training checkpoints so that optimizer and experiment metadata remain available.

Released checkpoints

File	Architecture / training depth	Parameters	Robust best loop	Robust best CE / PPL
`checkpoints/curriculum_8_to_16_100m.pt`	BASE, 60% of tokens at T=8 then 40% at T=16	7,342,592	16	4.381245 / 79.937
`checkpoints/curriculum_16_to_8_100m.pt`	BASE, matched reverse curriculum	7,342,592	8	4.412476 / 82.473
`checkpoints/base_fixed8_100m.pt`	BASE, fixed T=8	7,342,592	8	4.374509 / 79.400
`checkpoints/far_h0_fixed8_100m.pt`	FAR-H0, fixed T=8	8,261,184	8	4.382841 / 80.064

All runs used 99,999,744 actual training tokens, seed 42, sequence length 512, global batch 8,192 tokens, and final-output next-token cross-entropy only. The FORWARD and REVERSE curricula have identical T=8/T=16 token counts and recurrent compute; only temporal order differs.

The primary checkpoint is curriculum_8_to_16_100m.pt. Its useful inference-depth optimum moves to loop 16. The reverse control returns the optimum to loop 8.

Files required for evaluation

src/ and scripts/evaluate.py: exact model and depth-sweep implementation.
configs/: exact configuration for each released checkpoint.
tokenizer/tokenizer.json: byte-level BPE tokenizer with vocabulary 16,384.
results/: compact published metrics.
SHA256SUMS.txt: integrity hashes for all checkpoint and tokenizer artifacts.

The FineWeb token streams are not distributed. Recreate them using scripts/prepare_data.py or follow the GitHub reproduction guide.

Download and evaluate the primary checkpoint

bash

hf download GeneralFY/mini-research-on-looped-transformers2026 \
  --local-dir mini-research-on-looped-transformers2026
cd mini-research-on-looped-transformers2026

python -m venv .venv
source .venv/bin/activate
python -m pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cu128
python -m pip install -r requirements.txt

sha256sum -c SHA256SUMS.txt

python scripts/evaluate.py \
  --config configs/curriculum_8_to_16_100m.yaml \
  --checkpoint checkpoints/curriculum_8_to_16_100m.pt \
  --data-dir /path/to/data \
  --loops 1:32 \
  --validation-tokens 65536 \
  --diagnostic-tokens 65536 \
  --kl-tokens 512 \
  --output results/reproduced_100m_65k.csv

The evaluator constructs the model from the YAML configuration and loads its state dict with strict=True. These files are research artifacts rather than a transformers.AutoModel package.

Integrity

The primary checkpoint SHA256 is:

8bc102b584df978bd7619fb752ad84ca41ed1656aa403d406f1345c92f836d6b

Tokenizer SHA256:

5f2f0950743afdc9f338fc0d0d408e4214529940bc3f9d04e0218d4ec29760f8