NOTE: THIS IS EXPERIMENTAL SOFTWARE. PLEASE ENJOY, BUT DO NOT USE FOR PRODUCTION CRITICAL STUFF! SilentReasoner-7B (SoftCOT)

This model is an experimental fine-tune of Qwen2.5-7B-Instruct designed to compress explicit Chain-of-Thought (CoT) reasoning into 8 continuous latent tokens.

Instead of generating 1,000+ tokens of Let's think step by step... text before answering, this model processes the logic invisibly using continuous embeddings, then jumps straight to the final answer.

It acts like a compiled executable: it skips conversational filler, writes dense/Pythonic code, and retrieves optimal algorithms instantly. 📊 Benchmarks (vs. Baseline Qwen2.5-7B-Instruct) Metric Baseline Qwen-7B SilentReasoner-7B HumanEval (Pass@1) 80.0% 85.0% ✅ AIME 2024 (Pass@1) 6.7% 10.0% ✅ AIME Total Tokens Generated 23,910 16,673 AIME Total KV Cache Used 1549 MB 1100 MB ⚡ Efficiency Gains on Reasoning Tasks

Token Reduction: 30.3% fewer tokens generated.
VRAM Reduction: 29.0% smaller KV cache footprint.
Zero Conversational Filler: Skips the "Sure, here is the code!" text entirely.

💻 How to run it

⚠️ CRITICAL: Because this model uses inputs_embeds to inject the 8 continuous latent vectors, standard model.generate(input_ids=...) or HF pipeline() will output blank text.

You must inject the latent embeddings. Please use the inference.py script provided in this repository.

pip install torch transformers peft bitsandbytes

Download the repository, open a terminal in the folder, and run: bash

python inference.py

If you want to build on top of this model, here is the core logic for injecting the latents: python

import torch from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig from peft import PeftModel import os

model_path = "your_username/SilentReasoner-7B" base_model_name = "Qwen/Qwen2.5-7B-Instruct"

bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, ) base_model = AutoModelForCausalLM.from_pretrained(base_model_name, quantization_config=bnb_config, device_map="auto") model = PeftModel.from_pretrained(base_model, model_path) model.eval()

extras = torch.load(os.path.join(model_path, "softcot_extras.pt"), map_location="cuda") latent_embeddings = torch.stack([extras["latent_embeddings"][str(i)] for i in range(8)]).unsqueeze(0).to(torch.bfloat16)

prompt = "What is 2 + 2?" text = f"Problem: {prompt}\nSolution: " input_ids = tokenizer(text, return_tensors="pt").input_ids.to("cuda")

with torch.no_grad(): # Inject latents prob_emb = model.get_input_embeddings()(input_ids) full_emb = torch.cat([prob_emb, latent_embeddings], dim=1) mask = torch.ones(1, full_emb.size(1), device="cuda", dtype=torch.long)

# Generate
outputs = model.generate(
    inputs_embeds=full_emb, 
    attention_mask=mask, 
    max_new_tokens=512, 
    do_sample=False
)

print(tokenizer.decode(outputs[0, 1:], skip_special_tokens=True))

🧠 How it works (SoftCOT Architecture)

During training, an auxiliary GRU decoder was attached to the 8 latent tokens. The model was forced to route its explicit reasoning through these 8 continuous vectors to reconstruct the Chain-of-Thought text. At inference, the auxiliary decoder is discarded. The base model has learned to use these 8 latents as a compressed "mental scratchpad", allowing it to bypass generating hundreds of visible thought tokens.