A 1.2B hybrid model (SSM + attention) built in two stages: knowledge distillation from a 24B MoE hybrid teacher on STEM chain-of-thought data, then supervised fine-tuning on logical inference. The first proof-weighted distillation + SFT pipeline on a non-transformer architecture.
Liquid Foundation Models run at 239 tok/s on AMD CPU and fit under 1GB of RAM. This model adds structured STEM reasoning and formal logical inference to that efficiency substrate.
LFM2.5-1.2B distilled from LFM2-24B-A2B — a 24B MoE hybrid (SSM + attention) with only 2B active parameters per token. Teacher and student share the LFM hybrid architecture, so the KL divergence transfers reasoning patterns between architecturally compatible models.
Data: 2,802 STEM CoT samples from 5 domains:
Domain
Samples
Linear Algebra
667
Differential Equations
636
Electromagnetism
580
Mathematics
576
Classical Mechanics
343
All from 0xZee. Focused subset — core mathematical reasoning domains that share the most structural overlap with logical inference.
Loss function:
Proof-Weighted Cross-Entropy (55%) — 2.5x → 1.5x on derivation tokens
Knowledge Distillation KL Divergence (45%) — T=2.0, scaled by T²
Training format:
Solve the following problem carefully and show a rigorous derivation.
Problem:
{question}
Proof:
{CoT}
Final Answer:
{response}
Stage 1 hyperparameters:
Parameter
Value
Epochs
1
Effective batch size
8
Learning rate
1.5e-5 → 1e-6 (cosine)
Temperature
2.0
Proof weight
2.5 → 1.5
Precision
bf16
Stage 2: Logical Inference SFT
Fine-tuned on KK04/LogicInference_OA — a reproduction of the LogicInference dataset (Santiago Ontañón, Google Research) formatted for instruction-following. IID split, LOGICINFERENCEe format (inference first, answer at end). 5,491 unique inference problems extended to ~54,607 instruction-response pairs.
Why logical inference on a hybrid architecture? SSM components excel at sequential state propagation — exactly what formal logical inference requires. Each premise updates a logical state, and the conclusion follows from the final state. The hybrid architecture's inductive bias naturally aligns with propositional logic chains. SFT activates this alignment explicitly.
1from transformers import AutoTokenizer, AutoModelForCausalLM
2import torch
34model_id ="reaperdoesntknow/LFM2.5-1.2B-Distilled-SFT"56tokenizer = AutoTokenizer.from_pretrained(model_id)7model = AutoModelForCausalLM.from_pretrained(8 model_id,9 torch_dtype=torch.bfloat16 if torch.cuda.is_available()else torch.float32,10 device_map="auto",11)1213# Logical inference (Stage 2)14prompt ="""### Instruction:
15Consider the following premises: If p then q. If q then r. p is true. What can we infer?
1617### Response:
18"""1920# STEM derivation (Stage 1 still works)21prompt_stem ="""Solve the following problem carefully and show a rigorous derivation.
2223Problem:
24Solve the system of linear equations: 2x + y = 5, x - y = 1.
2526Proof:
27"""2829inputs = tokenizer(prompt, return_tensors="pt").to(model.device)30with torch.no_grad():31 outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)32print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Solve the following problem carefully and show a rigorous derivation.
Problem:
[Your problem]
Proof:
Logical inference (Stage 2):
### Instruction:
[Your question or logical inference problem]
### Response:
Intended Uses
Good for: On-device logical inference and STEM reasoning, mobile/edge/IoT deployment, formal reasoning tasks, educational tutoring, embedded inference pipelines, anywhere you need structured reasoning under 1GB.
Not for: Formal proof verification, safety-critical systems, complex multi-step proofs beyond model capacity, or long-context tasks beyond 1024 tokens.
Limitations
1.2B hybrid model. The SSM components give excellent inference speed but the model has hard capacity limits. Trained on 2,802 STEM samples (smaller than the 6,122 used for Qwen3 variants). Logical inference strongest on propositional logic patterns in the training data. Complex nested quantifiers may exceed capacity. Always verify.
This model is part of the Convergent Intelligence LLC: Research Division portfolio. All models in this portfolio are developed under the Discrepancy Calculus (DISC) framework — a measure-theoretic approach to understanding and controlling the gap between what a model should produce and what it actually produces.
DISC treats training singularities (loss plateaus, mode collapse, catastrophic forgetting) not as failures to be smoothed over, but as structural signals that reveal the geometry of the learning problem. Key concepts:
Discrepancy Operator (D): Measures the gap between expected and observed behavior at each training step
Jump Sets: Boundaries where model behavior changes discontinuously — these are features, not bugs
Ghost Imprinting: Teacher knowledge that transfers to student models through weight-space topology rather than explicit distillation signal
Santiago Ontañón. "LogicInference: A Large-Scale Dataset for Logical Inference." ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models.Paper | Code
From the Convergent Intelligence Portfolio
DistilQwen Collection — Our only BF16 series. Proof-weighted distillation from Qwen3-30B-A3B → 1.7B and 0.6B on H100. Three teacher variants (Instruct, Thinking, Coder), nine models, 2,788 combined downloads. The rest of the portfolio proves structure beats scale on CPU. This collection shows what happens when you give the methodology real hardware.
