You can deploy this model and start earning money today!

KIEFERSA/
Sophea-Qwen3.6-v1

Name: Sophea-Qwen3.6-v1
Availability: InStock

Views

No views yet

Sophea-Qwen3.6-v1

Greek and English reasoning fine-tune of Qwen3.6-35B-A3B: thinking MoE, vision retained, MTP speculative decoding

📄 Released with the paper Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See (arXiv:2608.17744). Every number on this card is measured, contextualized, and caveated in the paper; read it for the full evaluation protocol.

Creator: Kiefer SA (Sophea AI Lab, Athens)
Base model: Qwen3.6-35B-A3B (sparse MoE transformer, 36.0B total / 3.97B active, 8-of-256 routing)
Ships with: the base's full multimodal (vision) stack, untouched, and the multi-token-prediction draft head for speculative decoding
Languages: Greek and English (the reasoning trace follows the question's language)
Decoding: thinking (enable_thinking=true); the trace is the product and follows the question's language

Serve thinking. Leave enable_thinking=true so the reasoning trace is produced; the trace follows the question's language. Over an OpenAI-compatible vLLM endpoint, add --reasoning-parser qwen3 so the trace arrives in message.reasoning_content and the answer in message.content.

Intended use

Greek and English reasoning assistant: math, science, logic, and general step-by-step problem solving
Deployments that need the reasoning trace itself (auditable working, tutoring, verification), in the user's language

Trace-language steerability is partial: an explicit instruction to reason in English on a Greek question is obeyed on 44.8% of items (the reverse, 83.7%). A deployment that must force the trace language should test this axis, or use Sophea-Qwen3.6-v1.1 (53.9%). Not evaluated for safety-critical, legal, or medical use.

Fine-tuned from Qwen3.6-35B-A3B; shipped merged to dense weights.

Evaluation

Arrows mark the desirable direction for every metric: ↑ higher is better, ↓ lower is better.

Headline scorecard

Axis	Qwen3.6-35B-A3B (base)	Sophea-Qwen3.6-v1
Greek-trace fidelity, Greek questions (ratio ≥ 0.9, n=5,156) ↑	0.0% (0/5,156)	97.98% (5,052/5,156)
English traces on English questions ↑	100%	100% (n=1,900)
In-question language switches / 100 sentences ↓	16.4	0.0
Answer-channel leak ↓	15.6% (English control)	3.5%
Answer-format fallback, Greek think lane ↓	0.2–3.3% per domain	24.1%
Instructed override, el→EN / en→EL ↑	n/a (already reasons in English)	44.8% / 83.7%
Median trace length, tokens (own tokenizer) ↓	1,788	586 (3.0× fewer)
Greek NLU macro, forgetting check (vs own base) ↑	75.7	75.6 (−0.01 unrounded)
English NLU macro, forgetting check (vs own base) ↑	85.1	85.2 (+0.08 unrounded)
Pooled Greek reasoning accuracy (n=5,156) ↑	80.6	73.7

The pooled accuracy delta (−6.9) is a single-run reading inside the 7.7 pp seed-to-seed noise floor measured on this family; the paper discloses it and does not interpret it. Difficulty adaptation: the base overruns (≥3× the per-item median trace length) on 98.0% of easy items and its rate is identical on easy and hard items; the fine-tunes overrun on 0.2–12% of easy items and spend more only when the item is harder.

Per-benchmark detail

Per-domain, Greek think lane

Domain (Greek think lane)	Base acc ↑	Sophea-Qwen3.6-v1 acc ↑	Base fallback ↓	Sophea-Qwen3.6-v1 fallback ↓
Math (two math slices)	92.9	83.3	3.3%	18.2%
Commonsense (HellaSwag + Winogrande)	82.4	75.8	0.5%	22.9%
Logic (ProofWriter, macro-recall)	42.6	37.3	0.2%	46.3%

No fine-tune in the study beats its own base on any axis of this table. Conditioning on rows that emit the requested answer line, the release reads 95.8 / 78.8 / 39.5 against the base's 96.0 / 82.8 / 42.7 (math / commonsense / logic), so a substantial share of the raw gap is format compliance, not reasoning.

Forgetting check: Titan-1 suite (non-reasoning mode)

This suite exists to answer one question: did the Greek fine-tune damage general ability? It did not. Against its own base, across 14 bilingual NLU benchmarks: −0.01 Greek / +0.08 English, statistically flat. Read the numbers below as a forgetting check, not a capability leaderboard; movements of a point or two are within per-benchmark noise on this suite.

Benchmark (Titan-1 suite)	Base	Sophea-Qwen3.6-v1
ARC-Challenge-el	91.8	92.0
ARC-Easy-el	96.9	97.1
Belebele-el	92.7	93.7
GreekMMLU	83.8	83.0
HellaSwag-el	57.6	59.4
Medical-MCQA-el	81.2	80.3
TruthfulQA-el	39.1	36.5
Winogrande-el	61.0	62.5
MMLU-el	76.9	76.2
Greek macro	75.7	75.6
ARC-Challenge (en)	96.1	95.7
ARC-Easy (en)	99.1	99.1
HellaSwag (en)	73.2	74.2
Winogrande (en)	73.2	74.2
MMLU (en)	83.9	82.6
English macro	85.1	85.2

Register control and grammaticality, judge-scored at n=46/58 (directional evidence only): register 45 → 42 of 46, grammaticality 41 → 42 of 58; neither politeness control nor grammatical Greek is lost.

Usage

Serve with vLLM (OpenAI-compatible; the weights are dense, no adapter needed):

vllm serve KIEFERSA/Sophea-Qwen3.6-v1 --served-model-name sophea-qwen3.6-v1 --reasoning-parser qwen3

Client (OpenAI SDK), thinking left ON:

python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="sophea-qwen3.6-v1",
    messages=[{"role": "user", "content": "Ένα κατάστημα πουλά στυλό προς 1,50 € το ένα. Πόσο κοστίζουν 12 στυλό;"}],
    temperature=0.7,
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},   # thinking model
)
print(resp.choices[0].message.reasoning_content)  # the trace, in the question's language
print(resp.choices[0].message.content)            # the final answer

Transformers:

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("KIEFERSA/Sophea-Qwen3.6-v1")
model = AutoModelForCausalLM.from_pretrained(
    "KIEFERSA/Sophea-Qwen3.6-v1", torch_dtype=torch.bfloat16, device_map="auto")

messages = [{"role": "user", "content": "Ένα τρένο διανύει 240 χλμ σε 3 ώρες. Ποια είναι η μέση ταχύτητά του;"}]
text = tok.apply_chat_template(messages, tokenize=False,
                               add_generation_prompt=True, enable_thinking=True)
inputs = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=2048, temperature=0.7, do_sample=True)
print(tok.decode(out[0, inputs.input_ids.shape[1]:], skip_special_tokens=False))

Steering the trace language (partial, measured per checkpoint): append "Think step by step IN ENGLISH." to a Greek question, or "Σκέψου βήμα προς βήμα ΣΤΑ ΕΛΛΗΝΙΚΑ." to an English one. This checkpoint follows the English steering suffix on 44.8% of Greek items and the Greek suffix on 83.7% of English items.

Speculative decoding (MTP)

This model ships the multi-token-prediction head: 19 mtp.* tensors (~1.7 GB, bf16) in model-mtp.safetensors, the single-layer draft stack that config.json declares through mtp_num_hidden_layers: 1. The head is the base model's own; speculative decoding verifies every drafted token against the main model, so it changes throughput only, never the output distribution.

Enable it with vLLM:

bash

vllm serve KIEFERSA/Sophea-Qwen3.6-v1 --served-model-name sophea-qwen3.6-v1 --trust-remote-code \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":2}'

Vision

The full vision stack of the multimodal base is included, byte-identical to the base: image inputs work exactly as on Qwen3.6-35B-A3B. The adaptation touched only the language decoder; vision performance has not been separately re-evaluated for this release.

License

Apache-2.0. The base model, Qwen/Qwen3.6-35B-A3B, is released under Apache-2.0 and this fine-tune inherits those terms. Verify base-model terms before use.

Citation

This model is a release artifact of arXiv:2608.17744. If you use it, please cite:

bibtex

@misc{kirouane2026thinking,
      title={Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See},
      author={Ayoub Kirouane and Christos Petrocheilos},
      year={2026},
      eprint={2608.17744},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2608.17744},
}