LoRA adapter for unsloth/Qwen3.5-4B, fine-tuned on the
Seasoned Advice dataset:
500 bilingual (Turkish / English) cooking and food-science conversations from
Seasoned Advice, each assistant turn carrying a reasoning trace.
Trained on both language splits (1,000 examples), language layers only, r=32, 2 epochs.
The model answers with a <think> reasoning block followed by the answer, in Turkish or English.
Unsloth's official Qwen3.5 (4B) Vision example notebook, kept for reference — the fine-tuning notebook above was adapted from it. Not this project's own work.
The stock template, inherited verbatim from unsloth/Qwen3.5-4B. Kept for reference and as the baseline the hand-written one is diffed against.
They render identical output (see the verification below), so which one is active makes no
difference to the model — the second is here to show what was inherited versus what was authored.
The hand-written template emits the Qwen3.5 ChatML grammar the adapter was trained through —
inventing new delimiters would produce tokens the model has never seen, so the template is custom
code against a fixed wire format, not a custom format:
system / developer (leading ones merged into one turn), user, assistant, tool. An unknown role raises.
Content shapes
Plain strings and the [{"type": "text", "text": ...}] part lists the training rows use. Image and video parts render as <|vision_start|><|image_pad|><|vision_end|>, since the base is a VL checkpoint.
Reasoning
Read from a reasoning_content field or parsed out of an inline <think>...</think>. Following Qwen3.5, only the turn after the most recent user question keeps its reasoning; older traces are dropped so they don't accumulate. add_generation_prompt opens a <think> block, and enable_thinking=False emits a closed empty pair to suppress it.
Tool calling
A tools list renders a <tools> manual into the system turn. Calls use the <tool_call><function=name><parameter=x> XML form — the form this model family was trained on, not the JSON-in-<tool_call> variant seen in other Qwen templates. Results come back as <tool_response> folded into a user turn, with consecutive results sharing one turn. A user turn that is only a <tool_response> doesn't count as a new question for the reasoning rule above.
test_chat_template.py verifies this by rendering the same message sets
through this template and the stock one from unsloth/Qwen3.5-4B, then diffing the output byte for
byte: 18 hand-built cases (system merging and placement, multi-turn, both content shapes, tool calls
and responses, thinking on/off, vision parts) plus all 1,000 training rows in both generation modes.
All 2,018 renders match, so the adapter's behaviour is unchanged. Rendering runs under Jinja's
ImmutableSandboxedEnvironment, the same sandbox transformers applies.
Two behaviours deliberately do not copy the stock template. Both are asserted as differences in the
test, so neither can quietly regress:
An unrecognised role raises rather than being silently dropped from the conversation.
Three or more leading system turns are all merged; the stock template merges the first two and
discards the rest.
python test_chat_template.py
Limitations
Narrow domain, 1,000 examples. Source answers are community-written and not fact-checked —
do not rely on this model for food-safety decisions.
The adapter makes the model answer more briefly. It averages ~860 characters against the base
model's ~3,300, closely matching the ~990 of the human reference answers it was trained on. That is
what it was trained to do, and it is why it scores above its base on proximity to the reference. It
is also why a judge scoring for comprehensiveness ranks it below its base: in the round robin below
it places 4th of 5 and loses to its own base 10–85–5. If you want the most thorough cooking answer
available at this size, use the base model. If you want an answer in the register of the source
corpus, use this one.
Licence
CC BY-SA 4.0, inherited from the Stack Exchange source content.
📊 Turkish MMLU Benchmark Results
Evaluated on the alibayram Turkish MMLU benchmark over the full dataset (6200 questions across 62 sections). Scoring reuses the official olcum.py prompt and cevap_dogru_mu function verbatim; inference was run with transformers/unsloth (4-bit, greedy decoding, thinking disabled).
Base vs. Fine-tuned
Model
Correct
Accuracy
Base — unsloth/Qwen3.5-4B
4168/6200
67.23%
Fine-tuned — qwen3.5-4b-seasoned-advice-lora
4173/6200
67.31%
Difference
+0.08
On the same benchmark, my fine-tuned model ranks #21 of 68 and the base model ranks #22 of 68 among all listed models.
Interpretation
The fine-tuned model performs on par with its base (67.31% vs. 67.23%, a +0.08 pt gap — 5 of 6200 questions, within run-to-run noise). This is the expected outcome: the LoRA adapter was trained on the bilingual (Turkish and English) seasoned-advice conversational dataset, a domain unrelated to Turkish MMLU's academic and professional knowledge. The meaningful result is that domain specialization did not degrade the model's general knowledge — there is no catastrophic forgetting, and a few professional-services sections (e.g. Business Management, Emergency & Disaster Management, HR) even improved by a few points. For an advice-tuned model, a general-knowledge benchmark like this serves as a regression check rather than a target metric.
Full leaderboard ranking — my two models (⭐) inserted into the official 68-model table
ℹ️ Reference rows are from the official leaderboard (run via Ollama, Q4_K_M). My two ⭐ rows were run via transformers (4-bit); the runtimes differ slightly, so read cross-pipeline comparisons as indicative rather than exact.
Turkish MMLU measures general knowledge. This measures the thing the adapter was actually trained for, on 100 held-out question/answer pairs (50 English, 50 Turkish) scraped fresh from cooking.stackexchange.com and verified to have zero overlap with the 500 pairs used in training (see dataset provenance). Thinking mode is left on — the model's trained persona — and the <think> block is stripped before the answer is scored. 5 models across 3 families were compared. Every model in it supports Turkish officially, which is a requirement rather than a preference: half the benchmark is Turkish, and a model without that support does not answer worse so much as fail to stop — one candidate dropped from an earlier draft averaged 8 seconds per English question and 69 per Turkish one, running to the token ceiling every time. Every model answered the same 100 questions, loaded in 4-bit at the same 2624-token budget with greedy decoding, and the judge ran a full round robin — all 10 pairs × 100 questions = 1,000 judgements, candidate order randomized per pair.
#
Model
Family
Params
Thinking
Judge win rate
W–L–T
Reference proximity
Mean answer
Forced close
1
Qwen3.5-4B
Alibaba
4B
yes
83.3%
325–65–10
0.581
3326 ch
61%
2
gemma-3-4b-it
Google
4B
no
53.6%
209–181–10
0.587
3310 ch
0%
3
Qwen3-4B-Instruct-2507
Alibaba
4B
no
52.0%
199–184–17
0.531
4738 ch
0%
4
⭐ qwen3.5-4b-seasoned-advice-lora
Alibaba (this work)
4B
yes
36.5%
139–242–19
0.590
862 ch
38%
5
Phi-4-mini-instruct
Microsoft
3.8B
no
23.5%
89–289–22
0.592
3925 ch
0%
Ordered by judge win rate, which excludes ties. Reference proximity is mean cosine similarity to the human reference answer — a measure of how close a model's content and register sit to the source corpus, not of how good its answer is; the judge column carries the quality claim. Mean answer length counts the final answer only, with any reasoning block stripped. Forced close is the share of rows whose reasoning ran past its 1600-token budget and had the block closed for it; non-thinking models are 0% by definition.
Interpretation
The two metrics rank the models almost in reverse, and that is the finding. On the judge the fine-tune places 4 of 5 at 36.5%, and its own base tops the table at 83.3%, beating it 85–10–5 head to head. On proximity to the human reference the order flips: the fine-tune places 2 at 0.590, above the base at 0.581.
Answer length explains the split. The human references average 987 characters; the fine-tuned model writes 862, almost exactly matching them, while the base writes 3326 — more than three times the reference. The adapter learned the corpus it was given: Stack Exchange answers are short and direct, and after 1,000 examples the model answers that way too. That is precisely what raises its proximity score and what the judge marks down, scoring as it does on practical correctness, food-safety accuracy, actionable specificity and tone — criteria a longer answer has more room to satisfy.
This is not the judge preferring length. Across all 961 decisive verdicts the longer answer won 50.2% of the time (two-sided binomial p = 0.95), indistinguishable from a coin flip, and the per-pair direction reverses rather than leaning one way. Length does not order this table at all: the wordiest model placed third, the second-wordiest placed last, and the two that finished first and second sit in the middle of the range. The fine-tune's placing is a real judgement of its answers, not an artefact of their size.
So the adapter did not fail to learn; it learned a target this judge does not reward. Whether that trade is worth making depends on the goal: it is a loss if the aim is the most comprehensive cooking answer available, and closer to a success if the aim is to answer in the voice of the source corpus. Read with the Turkish MMLU result above, the picture is consistent — the adapter changed how the model answers without damaging what it knows.
On mixing reasoning and non-reasoning models. 2 of these 5 run with a thinking step and 3 do not, and each was left in its default configuration. Ranking them together is normal practice — public leaderboards do the same — but it measures models as they ship, not at equal inference compute. The judge never sees the reasoning, only the final answer, so extra thinking earns credit solely through the answer it produces.
On the proximity metric. The embedding model reads at most 512 tokens, so the wordier models are compared on their opening rather than their whole answer; the share of each model's text that reached the encoder is listed below. An earlier version of this run left the encoder at its default 128 tokens, which cut the longest answers to about a seventh of their length and reversed the proximity ranking — the figures here are from the corrected pass. Proximity is reported beside the judge for this reason, never as a standalone ranking.
Reference proximity by language, and how much text was encoded
Model
English
Turkish
Difference
Encoded
Phi-4-mini-instruct
0.605
0.578
-0.027
72%
⭐ qwen3.5-4b-seasoned-advice-lora
0.602
0.578
-0.025
94%
gemma-3-4b-it
0.527
0.647
+0.120
62%
Qwen3.5-4B
0.543
0.619
+0.076
57%
Qwen3-4B-Instruct-2507
0.512
0.550
+0.038
41%
Ordered by proximity. Difference is Turkish minus English. Encoded is the share of the average answer that fitted inside the embedding model's 512-token window.
Head-to-head matrix (row model's wins–losses–ties against each column)
Qwen3.5-4B
gemma-3-4b-it
Qwen3-4B-Instruct-2507
qwen3.5-4b-seasoned-advice-lora
Phi-4-mini-instruct
Qwen3.5-4B
—
77–20–3
73–26–1
85–10–5
90–9–1
gemma-3-4b-it
20–77–3
—
53–43–4
60–38–2
76–23–1
Qwen3-4B-Instruct-2507
26–73–1
43–53–4
—
58–40–2
72–18–10
qwen3.5-4b-seasoned-advice-lora
10–85–5
38–60–2
40–58–2
—
51–39–10
Phi-4-mini-instruct
9–90–1
23–76–1
18–72–10
39–51–10
—
Judging cost
Requests
1,001
Tokens
3,775,629
Cost
$1.74 USD
deepseek-v4-pro with thinking enabled, 8 concurrent workers. The request count exceeds the judgement count because failed calls are retried.
