TRM-text
TRM-text is an attention-free language model based on a Tiny Recursive Model (TRM) architecture.
Unlike Transformer-based language models, TRM-text removes self-attention entirely and replaces it with recursive computation built from causal dilated depthwise convolutions, hierarchical latent refinement, and parameter reuse.
The goal of TRM-text is to investigate whether recursive neural computation can provide competitive language modeling performance while dramatically reducing computational cost.
Overview
TRM-text explores a different scaling path from Transformers.
Instead of increasing attention heads and context interactions, TRM-text repeatedly refines hidden representations using a hierarchy of recursive processing blocks.
Key properties:
- Attention-free
- Autoregressive language modeling
- Recursive computation
- Hierarchical latent refinement
- RoPE positional encoding
- Dilated depthwise convolution mixer
- Hugging Face compatible
- safetensors support
Architecture
1Tokens
2 │
3 ▼
4Embedding
5 │
6 ▼
7Low-Level TRM
8 │
9 ▼
10Mid-Level TRM
11 │
12 ▼
13High-Level TRM
14 │
15 ▼
16Recursive Feedback
17 │
18 ▼
19LM Head
Each recursive block contains:
- RMSNorm
- RoPE
- Causal Dilated Depthwise Convolution
- SwiGLU Feed Forward Network
- Residual Recurrence
No self-attention layers are used.
Model Configuration
Current release:
1Parameters: ~15M
2
3dim = 256
4hidden_dim = 512
5
6low_steps = 6
7mid_steps = 3
8high_steps = 2
9
10cycles = 3
11
12kernel_size = 5
13
14low_dilations = [1,2,4,8]
15mid_dilations = [2,4,8,16]
16high_dilations = [4,8,16,32]
Compute Efficiency
Relative training cost:
| Architecture | Relative Cost |
|---|
| Transformer | 1200 |
| HRM | 100 |
| TRM-text | 1 |
These values represent relative compute requirements under the experimental scaling assumptions used during development.
The objective of TRM-text is to maximize efficiency through:
- parameter reuse
- recursive computation
- hierarchical refinement
- elimination of attention operations
Training
Base Pretraining
Dataset:
Tokenizer:
Objective:
Instruction Tuning
Dataset:
Format:
1### Instruction:
2...
3
4### Response:
5...
Loading
1from transformers import AutoTokenizer
2from transformers import AutoModelForCausalLM
3
4tokenizer = AutoTokenizer.from_pretrained(
5 "summerMC/TRM-text",
6 trust_remote_code=True
7)
8
9model = AutoModelForCausalLM.from_pretrained(
10 "summerMC/TRM-text",
11 trust_remote_code=True
12)
Inference
1prompt = """
2### Instruction:
3Explain artificial intelligence in simple terms.
4
5### Response:
6"""
7
8inputs = tokenizer(
9 prompt,
10 return_tensors="pt"
11)
12
13outputs = model.generate(
14 **inputs,
15 max_new_tokens=128,
16 do_sample=True,
17 temperature=0.7,
18 top_k=40
19)
20
21print(
22 tokenizer.decode(
23 outputs[0],
24 skip_special_tokens=True
25 )
26)
Research Motivation
TRM-text investigates whether recursive neural systems can replace attention mechanisms in language modeling.
Research directions:
- recursive reasoning
- hierarchical computation
- efficient language models
- attention-free architectures
- low-cost scaling laws
Limitations
Current checkpoint is experimental.
Known limitations:
- small parameter count
- limited instruction tuning
- lower capability than modern frontier models
- research-focused implementation
- benchmark coverage still limited
Intended Use
TRM-text is intended for:
- language model research
- efficient architecture experimentation
- recursive computation studies
- attention-free modeling research
Not intended for:
- safety-critical systems
- medical decision making
- legal advice
- financial advice
License
Apache-2.0
Citation
1@software{trm_text_2026,
2 title={TRM-text: Attention-Free Recursive Language Modeling},
3 author={summerMC},
4 year={2026},
5 url={https://huggingface.co/summerMC/TRM-text}
6}
