Yamato-3B-v5 (r=128) — Best legal/admin specialist
About the developer — One person (FiShota) building a Japanese LM stack
from scratch on a single RTX 3090. Yamato is the legal/admin SFT specialist
in the family. See
HinoMoto-100M v15
for the from-scratch line, and
hinomoto-bench-ja
for the cultural-axis benchmark.
GitHub ·
X
Best-performing variant in the rank ablation series:
| Variant | LoRA r | Overall | New domain | avg_len |
|---|
| v1 | 16 | 46.7% | 43.3% | 612 c |
| v3 | 32 | 48.9% | 43.3% | 501 c |
| v4 | 64 | 54.3% | 53.3% | 415 c |
| v5 ⭐ | 128 | 57.6% | 50.0% | 482 c |
Same 60 hand-crafted legal Q&A samples. Only LoRA rank doubled.
At small-domain SFT scale, rank capacity scales much further than commonly
assumed (PEFT tutorials default to r=8/16). Per-doubling deltas: +2.2 / +5.4 / +3.3.
Why this monotonic gain is interesting
In a sister experiment (HinoMoto multi-axis SFT, 452 samples × 4 axes ≈ 113 per axis),
the same recipe at r=64 regressed by -4pt versus r=16 (96.0% → 92.0%).
→ rank scaling is bounded by samples-per-axis, not total samples.
- Yamato (60 / 1 = 60 per axis): under-fit at any rank, so r=128 still helps.
- HinoMoto SFT (452 / 4 = 113 per axis): saturated by r=16, larger r overfits.
Crossover threshold for 3B + LoRA: ~100 samples/axis (empirical, single-base observation).
→ Detailed analysis: HinoMoto 開発ノート #9 (note.com / GitHub).
Bench setup
- 30 items in HinoMoto-Bench-ja
yamato_legal_v02.jsonl (民法 / 刑法 / 行政手続 / 労契 / 個人情報 / 不動産登記 / 国保 / 詐欺 / 家族財産 / 年金 等)
- Pass criteria: must contain expected legal references (条文番号 / 法令名) + must NOT match anti-pattern (一律無効 / 必ず勝てます 等)
- 3-seed sampling for stochastic noise
Quants
This repo includes Q3_K_M / Q4_K_M / Q5_K_M / Q6_K plus fp16. Q4_K_M recommended
for general use; deployment task may benefit from per-task selection
(see HinoMoto 開発ノート #8 on quantization non-monotonicity).
License
MIT (inherits from Sarashina2.2-3B-instruct-v0.1).
See also
- From-scratch JP LM family: HinoMoto-100M v15
(43M params, 5-seed verified, fp32 deterministic). HinoMoto-350M Phase 1 (318M, bf16) released after Phase 1 completion.
- Bench: https://github.com/FIshota/hinomoto-bench-ja (CC BY 4.0, 418+ items, family/keigo/silence/legal/generation_gap/workplace)
- Series index: HinoMoto 開発ノート (in author profile, note.com/hinomoto_dev)
- Sister model (rank ablation): Yamato-3B-v4 (r=64, 54.3%, 同じ recipe で rank だけ違い)
- Mojo SIMD/parallel kernels: github.com/FIshota/hinomoto-mojo (matmul 400x speedup vs Python)
