Five GGUF quantizations of a LoRA-rank-32 fine-tune for schema-conditioned e-commerce query extraction. Training used two epochs, a cosine learning-rate schedule, and assistant-only loss. No F16 GGUF is shipped because llama.cpp can dequantize dynamically at inference time.
This is a compact extraction model; Q4_K_M is the practical default and Q6_K is the higher-fidelity option.
Required prompt format
Inference must use the same chat structure and prompt contract used for training and evaluation. Reasoning was not applicable (non-reasoning model); temperature was 0, top_p was 1, and the output allowance was 4096 tokens.
System message:
text
1You extract structured filters from e-commerce search queries.
2Return only one valid JSON object, with no markdown or explanation.
3Your output must validate against the supplied JSON Schema: include every required key, preserve nesting, do not add keys, and keep arrays as arrays.
4Fill values only when stated or clearly implied by the query. Use JSON null for a required scalar field whose value is not available in the query.
5Preserve the exact spelling and capitalization of every JSON key.
The schema is value-free: it contains keys and allowed JSON types, but never target values. Every object key is required, additional properties are forbidden, and scalar types permit JSON null. If the query lacks a required scalar value, emit JSON null (not Python None, a missing key, or the string "null"). The dataset's meta_prompt column is not used.
Apply the base model's chat template; do not concatenate raw text outside that template. For Qwen3.5, disable thinking/reasoning.
Full held-out GGUF evaluation
All rows use the same 1,095-example held-out split, exact prompt construction, llama.cpp CUDA backend, 64 concurrent requests, temperature 0, and 4096 maximum generated tokens. Quant runs were concurrent, so wall-clock latency is workload-dependent and should not be interpreted as an isolated speed benchmark.
Quant
Size MiB
Strict JSON
Schema valid
Exact
Leaf F1
Key F1
Null accuracy
Truncated
Q8_0
506.5
99.73%
99.00%
20.73%
84.98%
99.07%
99.00%
1
Q6_K
482.3
99.73%
99.00%
20.55%
84.84%
99.01%
99.02%
1
Q5_K_M
400.6
99.63%
98.90%
19.09%
84.23%
98.99%
99.00%
2
Q4_K_M
379.4
99.27%
98.08%
20.82%
84.50%
98.53%
98.49%
1
Q3_K_M
339.0
99.54%
97.72%
19.00%
83.78%
98.68%
98.28%
1
merged BF16 reference
—
99.54%
98.90%
20.82%
84.89%
—
—
—
Q3_K_M prioritizes size and may materially damage instruction/schema adherence. Select using schema validity and Leaf F1, not file size alone.
Metric definitions
Strict JSON: the complete response parses as exactly one JSON value without code fences, commentary, or repair.
Schema valid: the parsed object satisfies the per-example Draft 2020-12 schema: required keys, types, nesting, arrays, and no extra keys.
Exact: case-sensitive equality to the full gold JSON; array order is significant.
Leaf F1: macro average of per-example precision/recall F1 over flattened (JSON path, typed value) pairs. This gives partial credit while still checking keys, types, and case-sensitive values.
Key F1: macro F1 over flattened JSON paths, ignoring values; it measures structural adherence.
Null accuracy: macro per-example accuracy on gold-null paths; non-null examples contribute 100% by definition, so interpret it with the other metrics.
Truncated: responses that hit the 4096-token limit. No repaired or “recoverable” JSON is counted as strict JSON.
Machine-readable results are in evaluation-results.json. The accompanying PDF includes methodology and analysis.
Then send the system and user messages above through chat mode. For production, validate every response against the supplied schema and retry or reject failures.
Limitations
This model extracts only fields represented by the supplied schema and reflects the source dataset's annotation quality. Exact match is deliberately harsh: capitalization or a single value error fails the entire example. Results apply to this held-out dataset and prompt; other schemas, languages, llama.cpp versions, chat templates, or sampling settings may behave differently. Do not treat inferred attributes as verified product facts.
