This checkpoint was improved through a staged rehabilitation and curriculum SFT process focused on reducing repetition collapse and improving short instruction-following behavior.

Model

Base: summerMC/TRM-textv3
Architecture: TRM / recursive Transformer style causal language model
Task: causal language modeling and instruction-style text generation
Context length: 512 tokens
Precision used during training: bfloat16

Training Pipeline

The model was improved using the following process:

text

TRM-textv3-grpo
→ hard rescue SFT
→ small curriculum SFT
→ TRM-textv3.5

The curriculum stage used response-only supervised fine-tuning with short QA, explanation, translation, and code-generation style examples.

A repetition unlikelihood term was used during rescue/curriculum training to reduce repeated-token degeneration such as:

text

common common common
learning learning learning

Intended Use

This model is intended for research experiments on:

recursive Transformer language modeling
small language model rehabilitation
instruction tuning
output-head collapse recovery
repetition-collapse mitigation
experimental LLM miniaturization

Example

python

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "summerMC/TRM-textv3.5"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "User: What is Python?\nAssistant:"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
        repetition_penalty=1.25,
        no_repeat_ngram_size=4,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Recommended Generation Settings

python

do_sample = False
repetition_penalty = 1.2
no_repeat_ngram_size = 4
max_new_tokens = 64 to 128

For sampling:

python

do_sample = True
temperature = 0.6
top_p = 0.9
repetition_penalty = 1.25
no_repeat_ngram_size = 4

Limitations

This is an experimental research checkpoint.

Known limitations:

weak factual knowledge
limited reasoning ability
unstable long-form generation
possible repetition under poor decoding settings
may produce incorrect code
not suitable for production use

Notes

This model is part of the summerAI TRM research line investigating whether recursive/shared-block Transformer models can be made more language-model-like through staged rehabilitation, curriculum SFT, and output-distribution repair.