A distilled Qwen3-0.6B worker that writes the short, structured labels in the UltraCode code-intelligence pipeline: RAPTOR L0/L1 cluster labels, community labels, and link-selection picks. It replaces a 7B teacher on exactly the high-volume, prefill-bound, short-output stages where a sub-billion-parameter model is enough — running those stages on the daemon's dedicated .enrich worker at a fraction of the main LLM's per-call prefill cost.

This is a purpose-built pipeline component, not a general assistant. It only does the four label tasks below; outside that distribution its behaviour is undefined.

What it is

Base: Qwen/Qwen3-0.6B (Apache-2.0) — 28 layers, ChatML, 151 936-token vocab.
Teacher: Qwen/Qwen2.5-7B-Instruct.
Method: sequence-level knowledge distillation (SeqKD) — LoRA SFT on teacher (prompt → response) traces, 3 epochs, prompt tokens masked; then merged into the base and exported to GGUF.
Format: code-daemon-enrich-v1-Q8_0.gguf (~800 MB, Q8_0, llama.cpp). Q8_0 keeps the tiny model's logits crisp for short noun-phrase / single-token outputs.
Tokenizer: Qwen2 family — so this model can double as a speculative-decoding draft for a Qwen3 target (shared tokenizer ⇒ aligned prefixes).

What it does — the four label tasks

System prompt bucket	Output	Example
`SYS_RAPTOR_LABEL_L0` / `L1`	a cluster label `"<topic>: name1, name2, name3"`	`"AST extractor methods: extractFnDecl, extractClassDecl, harvest"`
`SYS_COMMUNITY_LABEL`	a graph-community label (noun phrase)	`"OpenVINO pipeline state"`
`SYS_LINK_SELECT`	candidate ids to link (`c<N>` picks or none)	`"c2, c5"`

All outputs are short (labels ≈ noun phrases; link-select emits ids). Long-form / multi-paragraph summaries are a different model (the long-output branch) — not this one.

Evaluation

Held-out set via a deterministic content-hash split (no RNG; the eval set is provably never in training). Metrics vs the teacher's recorded output on the same held-out prompts:

Bucket	ROUGE-L	Exact	Notes
`LINK_SELECT`	0.79	0.73	structured id-emit — the 0.6B matches the 7B on this task
`COMMUNITY`	0.40	0.21	usable label quality
`L0` labels	0.39	0.04	token metrics undersell it — a valid paraphrase scores low on exact-token overlap

For L0, an embedding-cosine check (all-MiniLM-L6-v2, student vs teacher label) gives sem-cos ≈ 0.74 — a solid paraphrase, i.e. the label bulk semantically tracks the teacher even where ROUGE-L looks weak. Honest read: LINK is a clear win; L0/COMMUNITY are deployable for display/navigation labels, with a residual gap to the 7B's exactness on L0 (the next lever is on-policy distillation / more data).

Built for speed

The stages this model serves are prefill-bound with short outputs — the regime where shrinking the model (not speculative decoding) is the right lever. In the UltraCode daemon it loads into a dedicated .enrich worker that co-resides with the main LLM, so label stages run on the 0.6B while paragraph/prose stages stay on the larger model.

Measured on a laptop RTX 5060 (8 GB), llama.cpp CUDA, Q8_0, n_ctx=8192, n_batch=2048:

What	Rate
Prefill, 850-token label prompt	~17 000 tok/s
Decode, single stream	301 tok/s
Label stage end-to-end, batched (prompt + generated)	~2 600 tok/s

The end-to-end figure is the one to plan capacity with: label prompts are short and the stage runs many of them concurrently, so wall-clock is dominated by prefill, not by decode.

Memory: 604 MiB of weights, but ~2.7 GB resident at n_ctx=8192 once the KV cache and compute buffers are allocated. Size the deployment from that number, not from the file. A smaller n_ctx reduces it proportionally — 8192 is chosen here to keep eight label sequences in flight.

Usage (llama.cpp)

bash

# ChatML; system prompt = one of the buckets above, user turn = the cluster/context.
llama-cli -m code-daemon-enrich-v1-Q8_0.gguf -c 8192 \
  -p '<|im_start|>system
Write a short label for this code/doc cluster. Format: "<topic phrase>: <name1>, <name2>, <name3>".<|im_end|>
<|im_start|>user
extractFnDecl, extractClassDecl, harvest, namedChild — an AST walker module.<|im_end|>
<|im_start|>assistant
'

Greedy decoding (temperature 0) is recommended — the outputs are factual labels.

Training data

Teacher (prompt, response) traces generated by Qwen2.5-7B-Instruct running the UltraCode knowledge-graph pipeline over a mixed multi-language code corpus (Zig, C#, TypeScript, Kotlin, Python, JavaScript). ~4,800 enrich-bucket traces after dedup. No third-party labeled dataset is used.

License & attribution

Apache-2.0 — matches the Qwen3-0.6B base and the Qwen2.5-7B-Instruct teacher (both Alibaba / Qwen team, Apache-2.0). Not legal advice. Base and teacher © the Qwen team; please also honour their model cards.

faxenoff/code-daemon-enrich-v1

code-daemon-enrich-v1