You can deploy this model and start earning money today!

rjz123/
colar-cot-r1q

Name: colar-cot-r1q
Availability: InStock

deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B

finetune

mit

Views

No views yet

CoLaR — Math CoT warm-up baseline (no latent compression), R1-Distill-Qwen-1.5B

A CoLaR (Compressed Latent Reasoning) checkpoint for Math — explicit-CoT baseline (no latent compression head), fine-tuned from deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B. The model reasons in compressed continuous latent embeddings rather than explicit chain-of-thought tokens.

Model

Base model: deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
Framework: CoLaR — Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains (arXiv:2505.16552).
Domain: Math — explicit-CoT baseline (no latent compression head)
Warm-start: base R1
Files: colar_r1_cot.ckpt, hparams.yaml

Training procedure

Trained with the CoLaR supervised fine-tuning (SFT) recipe: the frozen base LLM is adapted with q/v LoRA (rank 128, alpha 32) plus a trainable Latent Head (3-layer MLP) and an embedding-compression module. The objective is next-token cross-entropy on the answer plus an embed_modeling_loss (MSE) that reconstructs the compressed reasoning-step embeddings (each latent token summarizes about compression_factor chain-of-thought tokens). Reinforcement learning (GRPO) is disabled — this is an SFT-only checkpoint.

This checkpoint: CoT-SFT warm-up stage, 10 epochs. This is the pre-CoLaR explicit-CoT baseline (no compression head), used as the warm-start for the latent-compression stage.

Datasets

GSM8K
Hendrycks MATH

How to load

This is a PyTorch-Lightning checkpoint (weights under the top-level key state_dict) that fits the CoLaR scaffold — it is not directly AutoModel-loadable. Load the base model, splice this state_dict in with strict=False, and use the CoLaR runtime settings:

COLAR_EMB_STD=0.018   COLAR_COMPRESS=<compression_factor>   sep_token=###
TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1

See the CoLaR repository (github.com/xiaomi-research/colar) for the exact loader. The shared GSM8K warm-start ancestor is the official AlbertTan/CoLaR release.

Research artifact for latent-reasoning study (small 1–1.5B model).

rjz123/colar-cot-r1q