A LoRA fine-tune of Qwen/Qwen3-8B, trained to act as the Planner component of MoE Sovereign, a local, sovereign Mixture-of-Experts LLM orchestrator. Given a user request, the model decomposes it into 1–4 subtasks and emits a single JSON array assigning each subtask to a specialist expert category or a deterministic MCP tool — nothing else. It is not a general-purpose chat/instruct model; it has one job, and the training and evaluation below are scoped to that job.

This card documents the training run in detail, including two significant failure modes from an earlier attempt, because the fixes are only meaningful in that context and because HPC-cluster SFT failures of this shape (silent, no error, normal-looking loss curve) are easy to reproduce elsewhere if the causes aren't spelled out.

Model summary


Base model	Qwen/Qwen3-8B (dense, 8.19B params, Apache 2.0)
Fine-tuning method	LoRA, r=16, α=32, dropout=0.05
LoRA targets	`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`
Trainable params	43.65M / 8.23B (0.53%)
Precision	BF16 (full precision base + LoRA; no quantization during training)
Context length trained	4,096 tokens
Formats published	safetensors (merged, BF16, ~16 GB), GGUF F16 (~16 GB), GGUF Q4_K_M (~4.7 GB, this repo)
License	Apache 2.0 (inherited from base model; no additional restrictions)

Intended use

Structured task decomposition for a multi-expert LLM orchestrator. Input: a system prompt fixing a 16-category expert taxonomy plus a tool catalog (see "Prompt format" below), followed by a user query. Output: only a JSON array, e.g.:

json

[{"task": "Explain the difference between TCP and UDP", "category": "technical_support"},
 {"task": "Convert 100 km/h to m/s", "category": "precision_tools",
  "mcp_tool": "unit_convert", "mcp_args": {"value": 100, "from_unit": "km/h", "to_unit": "m/s"}}]

The model is trained against one specific, versioned system prompt (PLANNER_SYSTEM_PROMPT in the MoE Sovereign codebase, ~2,500 tokens, 16 expert categories + a 22-tool precision catalog). Using it with a different taxonomy or system prompt is expected to degrade quality — see "Known limitations."

Training data

259,829 synthetic (query, plan) pairs, 258,529 train / 1,300 held-out eval (val_split=0.005).
5 system-prompt phrasing variants plus ~15% adversarial/negative samples (queries designed to probe category-boundary confusion, e.g. arithmetic phrased as a "technical" question that should still route to precision_tools, not technical_support).
Teacher model: meta-llama/Meta-Llama-3.1-405B-Instruct (project default for this dataset generator; Qwen/Qwen3-235B-A22B-Instruct is the documented fallback teacher — the exact teacher used for this specific merged dataset build was not independently re-verified against the original generation logs at model-card-writing time, so treat this as "default per the generation script," not as an independently re-confirmed fact).
Dataset is internal to the MoE Sovereign project and not currently published.

Training procedure

Framework: TRL SFTTrainer + PEFT LoRA + DeepSpeed ZeRO-2 (not ZeRO-3 — the 8B model fits in a single GPU's memory without parameter sharding, so ZeRO-2 is faster and, unlike ZeRO-3, composes cleanly with LoRA).
Loss masking: assistant_only_loss=True — loss is computed only over the assistant's JSON response, not over the system+user portion of the sequence. This is a native TRL 1.4 feature and is the fix for the truncation failure described below, not an incidental setting.
Schedule: 3 epochs, micro-batch 4 × grad-accum 4 × 8 GPUs = effective batch 128, cosine LR schedule, peak LR 2e-4, gradient checkpointing enabled.
Attention: eager (attn_implementation="eager") — Flash Attention 2 is not available for the ROCm 7.0 / PyTorch 2.10 build used on this cluster, so attention materializes the full O(seq²) matrix rather than a fused/tiled kernel. This is the dominant cost factor behind the per-step latency below, not model size or LoRA overhead.

Hardware: LUMI-G (EuroHPC)

Trained on LUMI-G, the GPU partition of the EuroHPC LUMI supercomputer (CSC, Finland), under EuroHPC project allocation project_465003058. One small-g node: 8× AMD Instinct MI250X GCDs (Graphics Compute Dies — each physical MI250X card exposes two GCDs as separate devices), 512 GB HBM2e, ROCm 7.0, PyTorch 2.10+rocm7.0, executed inside a Singularity container.

Training run and two prior failure modes

This model is the result of the second full training attempt against this dataset. Both failure modes below are documented here (rather than only in internal project notes) because they cost real HPC allocation and neither produced an error — both looked like a normal, converging training run right up until post-hoc inspection.

Attempt 1 — silent truncation of the training target (root-caused, not repeated here): The first run used max_seq_len=1536, carried over from an earlier, much shorter prompt format and never re-checked after the system prompt grew to ~2,500 tokens. With truncation_side="right" and no completion-only loss masking, the assistant's JSON target — which starts after the ~2,500-token system prompt — fell outside the 1,536-token window for essentially all 259,829 examples. The loss converged normally (the model was trivially learning to predict text it was already given in full, i.e. the system prompt prefix), and the run reported no errors. The resulting model passed a superficial capability check (valid JSON, plausible categories) purely because the base model already had those capabilities zero-shot — the fine-tune itself had no measurable effect. Root-caused by tokenizing real training examples post-hoc and finding the assistant span started at token ≈2,536, far past the 1,536-token cutoff.

Attempt 2, first sub-issue — GPU device-placement bug (fixed): early re-runs (with max_seq_len corrected) hit repeated out-of-memory errors that looked like a batch-size problem and were initially treated as one (leading to a QLoRA 4-bit detour that "fixed" the symptom without being the actual cause). The real cause: the training script never called torch.cuda.set_device(local_rank) before model loading. Because SFTConfig/TrainingArguments (which normally performs this via Accelerate) is only constructed after from_pretrained() runs, all 8 DeepSpeed ranks defaulted to the same device and loaded 8 copies of the model onto GPU 0 while the other 7 GPUs sat idle — visible in an OOM trace showing ~59 GB allocated on one GPU by a process that should have used ~5 GB of its own. Fixed with a single explicit torch.cuda.set_device(local_rank) call placed before any CUDA/HIP allocation. Once fixed, plain BF16 training (no quantization) ran stably — the QLoRA detour had been solving a problem that no longer existed once each rank owned its own GPU.

Attempt 2, second sub-issue — throughput: even after the fixes above, eager attention (no Flash Attention 2 on this ROCm build) kept per-step latency around 120–150 s at 8,192-token context. Reducing max_seq_len to 4,096 (still comfortably above the measured p99 of 3,628 tokens across the dataset) roughly halved the attention cost without truncating any real example. Even so, throughput meant the full 3-epoch run (6,060 steps) could not fit inside the cluster's 38-hour per-job time limit. The final run was executed as a chain of 6 SLURM jobs (afterany dependencies, each resuming from the latest checkpoint), spanning 2026-08-01 to 2026-08-10 (≈8 days 17 hours of wall-clock, cumulative queue + compute time) before the final job completed cleanly within its time limit and wrote the adapter.

Evaluation

Training-time metrics only — no independent/adversarial evaluation has been run yet against this specific checkpoint. Treat the numbers below as evidence the fine-tune converged in-distribution, not as a claim about real-world planner quality; a held-out multi-domain benchmark against the pre-fine-tune baseline is planned but not yet executed as of this card's writing.

Metric	Value	Source
Final training loss	0.00017	last logged training step (step 6,060/6,060, epoch 3.0)
Held-out eval loss	0.0235	1,300-sample held-out split, epoch ≈2.97
Held-out mean token accuracy	0.9911	same held-out split
Held-out mean token accuracy (training split, rolling)	~0.99 throughout the final epoch	training logs

The gap between the very low final training loss and the higher (but still low) held-out eval loss is consistent with normal LoRA-SFT convergence on a synthetic, template-heavy dataset, not independently confirmed to be free of overfitting on out-of-distribution phrasing — see limitations below.

Known limitations

Tied to one specific system prompt. The model was trained against a fixed, versioned ~2,500-token system prompt with a specific 16-category taxonomy and 22-tool catalog. This is not a hypothetical caveat — a taxonomy/prompt mismatch between training and serving is the documented root cause of a separate, earlier deployment failure in this project. Do not deploy this adapter behind a different system prompt without retraining or at least re-validating.
No independent evaluation yet. All numbers above are training/held-out-split metrics from the same synthetic data distribution the model was trained on. No adversarial, out-of-distribution, or human-graded evaluation has been run against this checkpoint at the time of writing.
Inherits teacher-model biases. The training data was generated by a large teacher model decomposing queries into the target taxonomy; any systematic blind spots or stylistic biases in the teacher's decompositions carry over.
Narrow by design. This is not a chat model and will not behave like one — it is trained to emit only a JSON array, nothing else, and has not been evaluated for any other task.
Quantized variants trade precision for size. The Q4_K_M GGUF in this repo (4.90 bits/weight, ~4.7 GB) is the recommended deployment format for constrained VRAM; F16 GGUF and BF16 safetensors are also published for full-precision use or further fine-tuning.

Files in this repository

File	Format	Size	Use case
`qwen3-planner-q4_k_m.gguf`	GGUF, Q4_K_M (4.90 BPW)	~4.7 GB	Recommended for local/edge inference (e.g. Ollama)
`qwen3-planner-f16.gguf`	GGUF, F16	~16 GB	Full-precision GGUF inference

The merged BF16 safetensors checkpoint is published separately at h3rb3rn/qwen3-planner-sft.

Acknowledgments

Training was performed on LUMI, a EuroHPC Joint Undertaking supercomputer hosted by CSC (Finland) and the LUMI consortium, under EuroHPC allocation project_465003058. Base model and tokenizer: Qwen3-8B (Apache 2.0).

h3rb3rn/qwen3-planner-sft-GGUF

qwen3-planner-sft