Tessera-4B
A tessera is a single tile in a mosaic — and its Greek root means "four."
Tessera-4B = four curriculum stages composed into one capable model.
Tessera-4B is a 4-stage curriculum fine-tune of Qwen/Qwen3.5-4B (instruct):
Reason → Plan → Use Tools → Align. Each stage trains on the previous stage's
merged weights and replays ~10% of earlier data to prevent forgetting.
- Base: Qwen/Qwen3.5-4B (instruct) · VLM + hybrid-attention · supports a
<think> channel
- Method: QLoRA (4-bit NF4), adapter merged between stages
- Compute: a single NVIDIA B200 (192 GB), ~8 h, ~$40 end-to-end (training + evaluation, including false starts)
- Thinking preserved: emits
<think>…</think> on hard problems, stays direct on tool calls / simple asks
Results below separate standardized benchmarks from internal proxies, and note the
evaluation issues encountered. See the full write-up for details.
Quantized variants
Ready-to-run quantizations for local / edge inference (Apple Silicon MLX + llama.cpp GGUF):
Results
Standardized benchmarks — lm-evaluation-harness (base → Tessera-4B)
| Benchmark | base | Tessera-4B | Δ |
|---|
| MMLU | 0.743 | 0.723 | −0.020 |
| ARC-Challenge (acc_norm) | 0.543 | 0.561 | +0.018 |
| HellaSwag (acc_norm) | 0.731 | 0.750 | +0.019 |
| Winogrande | 0.699 | 0.693 | −0.006 |
Tool calling — held-out, 20 cases (function selection)
| base | Tessera-4B |
|---|
| Function selection | 0.90 | 0.85 |
Internal proxy (per stage) — GSM8K-style reasoning, 200 samples
| After stage | Reasoning (proxy) |
|---|
| base | 0.375 |
| 1 Reasoning | 0.78 |
| 2 Planning | 0.775 |
| 3 Tools | 0.79 |
| 4 Align | 0.785 |
Interpretation: on standardized benchmarks, Tessera ≈ base — expected when
specializing an already-strong instruct model on a budget. We trained reasoning/agentic/tools,
not broad knowledge, so MMLU holds roughly flat (−2% is a small, normal SFT cost — we
retained general ability rather than regressing it). The large reasoning gain (0.38→0.79) is
from an internal proxy, not a standardized GSM8K-generative run, and is reported as such.
Curriculum & training
| Stage | Skill | Examples | Context | Epochs | Loss start→end |
|---|
| 1 | Reasoning | 25,000 | 8,192 | 1 | 1.03 → ~0.70 |
| 2 | Agentic | 13,000 | 8,192 | 1 | 1.27 → 0.67 |
| 3 | Tool calling | 18,000 | 8,192 | 1 | 1.16 → 0.55 |
| 4 | Instruction | 9,998 | 4,096 | 2 | 0.95 → 0.87 |
LoRA r64/α128 (1–3), r32/α64 (4); paged AdamW 8-bit; micro-batch 16 + gradient checkpointing.
Each stage's loss signature differs because each is learning something different (Stage 3's steep
drop = function calling is highly learnable; Stage 4's flat curve = a gentle, broad nudge).
10% replay kept Stage-1 reasoning stable through Stages 2–4 (no catastrophic forgetting).
Datasets (all open)
- Reasoning: OpenMathReasoning, OpenCodeReasoning, OpenThoughts3
- Agentic: Nemotron-Post-Training-v2 (chat), APIGen-MT
- Tools: ToolACE, xLAM-60k (⚠️ CC-BY-NC)
- Instruction: Tülu-3, OpenHermes-2.5, No-Robots
Evaluation methodology & pitfalls
Three eval bugs nearly produced wrong numbers — documented so others can avoid them:
--apply_chat_template tanked MMLU to ~0.26 (random) on multiple-choice loglikelihood.
Fix: no chat template for MC tasks → MMLU 0.74.- Generation cap truncated
<think> (default 256 tokens) → low GSM8K. Fix: raised to 2048.
- Tool calls are XML-style (
<function=...><parameter=...>), not JSON — a JSON parser scored
0 real calls. Fix: parse the XML form.
Still TODO (where a reasoning/tool curriculum should pull ahead): standardized
GSM8K-generative, MATH/AIME, full BFCL, τ-bench. Single run; no replay/order ablation yet.
Intended use & limitations
Research / non-commercial use (Stage 3 includes CC-BY-NC xLAM; also a Qwen3.5 derivative).
Reasoning, planning, and function calling for a small, locally-runnable model. Can hallucinate;
verify tool arguments before executing. Inherits base-model biases.
Usage
1from transformers import AutoModelForCausalLM, AutoTokenizer
2import torch
3mid = "sahilchachra/Tessera-4B-Preview"
4tok = AutoTokenizer.from_pretrained(mid)
5model = AutoModelForCausalLM.from_pretrained(mid, torch_dtype=torch.bfloat16, device_map="auto")
6msgs = [{"role":"user","content":"A train travels 60 km in 45 min. Speed in km/h?"}]
7ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
8print(tok.decode(model.generate(ids, max_new_tokens=1024)[0][ids.shape[1]:], skip_special_tokens=True))
9# May emit a <think>…</think> block before the answer.
Acknowledgements
Built on Qwen3.5-4B (Alibaba Qwen). Data: NVIDIA (OpenMath/OpenCode Reasoning, Nemotron),
Open Thoughts, Salesforce (xLAM, APIGen-MT), Team-ACE (ToolACE), Allen AI (Tülu-3),
Teknium (OpenHermes), HuggingFace H4 (No-Robots). Tooling: Unsloth, TRL, PEFT, bitsandbytes,
lm-evaluation-harness.
