Self-Forge: Judge-Gated Continual Specialization of Small Local Language Models
Author: MrWoRmMr · Status: DRAFT · rev. 2026-08-25
Draft for a repository writeup. Names/infra of the author's personal agent ecosystem are generalized. Phases 1–4 are executed with measured results below: the first specialist has been forged and evaluated against its base.
Abstract
We describe Self-Forge, an architecture in which a small, locally-hosted language model is quantized to fit commodity hardware, then continually specialized toward a narrow domain through an autonomous, judge-gated fine-tuning loop. The central claim is practical rather than theoretical: for a bounded domain, a small model that is (a) run at a healable quantization level and (b) densely specialized can match a much larger general model on the domain, on hardware costing near-zero, entirely on-premises. We take this end-to-end on 2016-era commodity hardware: a 7B base runs on an 8GB GPU (fits, reasons, refuses fabricated data at ~19 tok/s), fine-tunes via LoRA sharded across three Pascal GPUs (loss 1.9→0.4), and the resulting specialist — evaluated against its base — internalizes the target house format, eliminates the base's fabricated figures, and preserves groundedness. The result is measured, fully on-premises, at near-zero marginal cost. We report each phase's evidence, including the failure modes (a judge-gate is essential and human verification still sits above it; the method-reasoning gain is bounded by dataset size).
1. Motivation
Retrieval-augmented generation (RAG) lets an agent consult external notes but does not make the model itself more capable: knowledge stays outside the weights, and every answer is a lookup. For a domain the agent uses constantly, we want the competence internalized — the model reasoning from the domain, not retrieving it. Two constraints shape our setting: (1) sovereignty/privacy — training and inference must stay on the owner's hardware, ruling out rented cloud GPUs; (2) commodity hardware — the deployment target is an 8GB consumer GPU. The question: can a small local model be forged into a domain specialist that rivals a large general model, under these constraints?
2. The Self-Forge Loop
The approach is fine-tuning wrapped in an autonomous, verified loop:
- Ingest the domain → generate candidate training examples (Q&A, rationales).
- Judge-gate: an independent verifier admits only correct/grounded examples.
- Specialize: low-rank fine-tuning (LoRA/QLoRA) on the admitted set.
- Evaluate on a held-out set; promote the adapter only if it improves the target and does not regress a frozen "must-not-break" set.
- Repeat on a cadence (weekly → daily) until held-out performance plateaus ("topic saturated").
This mirrors human learning: test against reality, don't reinforce your own errors, consolidate periodically, unlearn bad habits.
3. Method
3.1 Quantize-then-heal. Quantization is treated as a lossy channel whose specific failures are measured (probing) and then repaired by the adapter, which compensates for quantization error on the targeted distribution. This works down to ~Q4 (a dented model the adapter can correct); below that (Q2/Q3) too much information is destroyed for a low-rank correction to recover.
3.2 Dense specialization. A small model's limited capacity, concentrated on one domain, can exceed a large generalist on that domain. Fine-tuning targets the stable method (frameworks, reasoning, style); volatile facts (prices, current figures) remain in RAG and are never baked into weights.
3.3 Controls (anti-collapse, anti-forgetting). Training a model on self-generated data risks model collapse; an external verifier (the judge) is the anchor that prevents it. Continual updates risk catastrophic forgetting; mitigations: a replay set, a library of per-domain adapters (hot-swapped rather than one growing blob), and periodic consolidation. Governance: versioned adapters with trivial rollback, shadow/canary before serving, and a drift detector that freezes learning when hallucination rises. Every step is small, verified, reversible, and human-approved.
4. Results
4.1 Inference feasibility (Phase 1)
Measured on an 8GB Maxwell-class GPU sharing memory with a 0.5GB embedder. Base model: a general 7B (Q4, ~4.7GB), pre-fine-tune baseline.
| Criterion | Result |
|---|
| Fits target GPU | ✅ ~5.2GB with embedder (< 8GB) |
| Throughput | ~18.9 tok/s |
| Domain reasoning | ✅ correct (ROE 22% > cost-of-equity 18% ⇒ price-to-book > 1) |
| Groundedness | ✅ refused to fabricate price/earnings for a nonexistent ticker |
Implication: the base is already a competent, grounded domain reasoner, so specialization is narrow (internalize method/style), not from-scratch teaching.
A companion probe stress-tested quantization at the extreme: a 22B model crushed to ~2.2-bit (IQ2) to fit the same 8GB. It did not collapse — it reasoned correctly and refused fabricated live data — but ran ~3× slower (6.4 vs 18.9 tok/s) and, on harder self-referential probes, confabulated about its own identity/config (≈7/10 grounded). Finding: aggressive quantization of a large base degrades speed and self-knowledge before core correctness; the practical winner on an 8GB target remains a small model at a healable (≥Q4) quant.
4.2 Local training path (Phase 2)
The critical bottleneck of earlier drafts — does training run at all on this hardware? — is resolved. A 7B base fine-tunes via LoRA (fp16) sharded across three Pascal-class GPUs: loss fell cleanly (3.64→2.48) over a smoke run, adapter saved, peak ~5–6GB/GPU. Two lessons: (a) the widespread belief "Pascal can't train modern PyTorch" holds only for 4-bit QLoRA (bitsandbytes); plain fp16 LoRA runs fine; (b) bleeding-edge trainer libraries broke in several ways (chunked-loss patches, collator format, a chat-template returning strings not ids), so the robust recipe is a manual torch training loop (forward→backward→step) rather than a high-level Trainer. Throughput is modest (~0.25 steps/s), making real fine-tunes minutes-to-hours, viable as batch jobs.
4.3 Dataset construction (Phase 3)
A 90-example domain dataset was built in two deliberately-separated slices:
- Persona/format (71 ex) — generated deterministically by running the agent's own valuation function across the equity index; numbers are real, format is gold, grounded by construction (no LLM, no judge). Notably ~75% of these examples teach honesty — the model admitting when a DCF is inconclusive or inapplicable.
- Method/reasoning (19 ex) — chunks from valuation textbooks → an LLM generates a Q&A → a judge filters. Here the judge-gate proved essential and measurable: naive generation was ~40% usable and produced dangerous contradictions (a 7B asserting DCF suits banks — the opposite of correct). Two guards fixed it: a retrieval-distance filter (killing off-topic chunks) and an LLM judge. A measured trade-off emerged — a strong slow cross-model judge (22B) yielded 100% clean but was impractically slow; a fast same-model judge was ~6× faster but more lenient (human inspection still caught ~24% meta/disclaimer junk it passed). Human verification remained necessary above the automated gate.
4.4 First specialist + eval-gate (Phase 4)
A 7B base was fine-tuned via LoRA (r=16, ~10M trainable params) on the 90-example dataset, 3 epochs, sharded across three Pascal GPUs (36 min, 270 steps). Training loss fell smoothly 1.885 → 0.553 → 0.408 — clean convergence, no divergence. The resulting adapter was evaluated against the raw base on the same probes, across three dimensions:
| Dimension | Base (raw 7B) | Specialist (fine-tuned) |
|---|
| Persona / format | generic markdown; fabricated numbers (invented a share price and EPS) | reproduced the agent's exact house format (structured valuation card, deterministic tone) |
| Method reasoning | correct (WACC, why-DCF-fails-for-banks) | correct — parity |
| Groundedness | refused a fabricated ticker | refused — preserved |
Verdict: success where it was trained. The fine-tune's clearest win is format/persona — the specialist internalized the house output style the base had no notion of, and in doing so stopped fabricating figures the base had invented. Crucially, fine-tuning did not damage groundedness (the primary risk — that training on generated data erodes the model's "I don't know"). Method reasoning was parity, expected given the method slice was small (19 vs 71 examples) and the base was already competent there.
One nuance validates the architecture rather than undermining it: the specialist occasionally emits imprecise figures. This is by design — the persona slice teaches the format; the real numbers are injected at inference from a deterministic function (RAG/code), never baked into weights. Volatile facts stay out of the parameters; the model supplies structure and reasoning, the code supplies truth.
5. Related Work
QLoRA (fine-tuning a 4-bit-quantized base via low-rank adapters); self-improvement / self-taught reasoning (STaR) and RLAIF (learning from self-generated, verifier-filtered data); the model-collapse literature (why an external anchor is required); catastrophic forgetting and continual-learning mitigations (replay, adapter libraries); and the RAG-vs-fine-tuning trade-off (external knowledge vs internalized capability). Our contribution is not a new algorithm but a composition: quantize-then-heal + dense specialization + a judge-gated continual loop, under strict on-premises constraints.
6. Limitations & Open Questions
- The win is format/persona, not (yet) reasoning. The specialist clearly beats the base on house-format adherence and stops the base's number-fabrication, at parity on method reasoning and groundedness. Moving method reasoning from parity to win requires a larger method dataset — the current 19 method examples are a proof, not a corpus.
- A single eval run, few probes. The verdict rests on 4 hand-inspected probes, not a held-out benchmark; a proper eval set (and comparison against the large incumbent, not just the base) is the next rigor step.
- Throughput, not feasibility, is the constraint. Training and judging on 2016-era GPUs is slow (~0.25 train steps/s; a 22B judge at ~6 tok/s), so scaling the dataset to hundreds of examples is a batch/overnight job, not interactive.
- Dataset size. 90 examples proves the pipeline; a production specialist wants substantially more method examples, bounded by textbook-chunk coverage.
- Hardware ceiling. The 8GB target caps a healable (≥Q4) model at ~9B. Larger models require larger cards or a multi-node pool.
- Whether the specialist matches the incumbent is the open empirical question the roadmap exists to answer.
- Quantization does not fix fabrication — it addresses speed/size, not a model's intrinsic tendency to hallucinate; groundedness must be verified per model.
7. Roadmap (phases)
- Lock plan ✅ · 1. Inference feasibility ✅ · 1b. base bake-off ✅ · 2. local training stack ✅ · 3. dataset (generate → judge-gate → split) ✅ · 4. first fine-tune + eval-gate ✅ · 5. close the loop (collect → gate → train → eval → cadence) — next · 6. production controls (adapter library, canary, drift kill-switch) · 7. generalize (other domains; on-device/mobile). Phases 0–4 done; the autonomous loop (5) and production controls (6) remain.
8. Availability
Intended for release under a pseudonymous repository once Phase 2+ are validated: method, evaluation harness, and the trained adapter would be public; the domain dataset and the specific loop orchestration remain private (the practical moat). Sovereignty here means owning the intelligence (local model + data + loop), not reinventing the commodity substrate (quantization formats, adapter training) — which is used as-is, and improved only where it fails this hardware specifically.
