You can deploy this model and start earning money today!

selimaktas/
Flow-Drafter-Qwen3.5-27B-tr

Name: Flow-Drafter-Qwen3.5-27B-tr
Availability: InStock

Views

No views yet

Flow-Drafter-Qwen3.5-27B-tr

Speculative-decoding draft model for Qwen/Qwen3.5-27B (Chained-Flow, joint-VAE drafter), adapted to Turkish.

Lossless by construction: a proposed token is emitted only if the target itself would have emitted it. Outputs can still differ from unspeculated decode, because verifying K+1 positions at once changes the target's own fp16 reduction order — the observed divergence is comparable to the run-to-run difference between two unspeculated runs.

Warm-started from Flow-Drafter-Qwen3.5-27B-v2 (sha256 31baf9ad6a417b15eba6c896b367277b0696c4933036242900cfa1a6dbc5b072), not trained from scratch. Identify the parent by that content hash, not by commit — a card-only edit moves the commit without changing a byte of the weights. These weights are sha256 f58eb4fc5079ea82a36a808f0b4b0f2984066d3590484fe2db6972351333e242.

Adapted on ~12.95M tokens of Turkish teacher states generated by Qwen/Qwen3.5-27B itself, in non-thinking mode, at LR 1e-4 for 3 epochs.

Usage

bash

pip install chained-flow

# CF_DRAFTER_DIR is the only variable you need; a bare repo id works and pulls shortlist.pt with it.
# K comes from num_speculative_tokens, not an env var.
# --max-num-seqs IS REQUIRED HERE: this is a hybrid model needing one Mamba cache block per decode
# sequence, and vLLM's default 1024 does not fit ("exceeds available Mamba cache blocks"). 4B/9B
# do not need it.
CF_DRAFTER_DIR=selimaktas/Flow-Drafter-Qwen3.5-27B-tr \
vllm serve Qwen/Qwen3.5-27B --async-scheduling --max-num-seqs 64 \
  --speculative-config '{"method":"custom_class","model":"chained_flow.vllm_plugin.flow_proposer.FlowDrafterProposer","num_speculative_tokens":5}'

bash

curl localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "Qwen/Qwen3.5-27B",
  "messages": [{"role": "user", "content": "Merhaba, nasılsın?"}],
  "chat_template_kwargs": {"enable_thinking": false},
  "temperature": 0
}'

chat_template_kwargs: {"enable_thinking": false} is required — this drafter was trained on prompts carrying the <think>\n\n</think>\n\n prefix.

Served speedup

Every number below was measured under one condition: natural EOS (no ignore_eos), concurrency 1, greedy, chain proposer at K=5, async scheduling on for every arm including the baseline, 2 repeats, through vllm serve. Prompts: the chained-flow Turkish holdouts (bench_data_tr), 4 domains, 400 requests per arm, 0 errors. Aggregation is token-weighted pooled, not a mean of per-domain ratios. Under natural EOS the arms emit different token counts (169.1 tokens/request here, not a forced 256), so the rates are comparable but the token totals are not.

prompt set	base tok/s	v2 parent: tok/s	acc	speedup	-tr: tok/s	acc	speedup
tr_funccall	25.8	30.7	1.437	1.189x	64.0	3.136	2.477x
tr_instruct	26.3	26.5	1.206	1.010x	48.4	2.238	1.844x
tr_multiturn	26.1	26.0	1.193	0.994x	42.2	1.970	1.617x
tr_toolcall	25.7	31.2	1.477	1.215x	62.5	3.100	2.433x
pooled	26.0	27.8	1.328	1.070x	50.5	2.611	1.942x

26.0 → 50.5 tok/s, +94% against unspeculated decode.

Unlike the 4B and 9B siblings the English parent is not a net loss here (1.070x) — at 26 tok/s the base step is slow enough that accept 1.33 clears break-even. Adapting takes it 1.07x → 1.94x.

The candidate shortlist ships with the model and is loaded automatically; the loader refuses to run against a different one.

Caveats

Turkish prompt sets only; this card does not characterise English behaviour. English accept was measured offline during selection and moved −0.06 overall / −0.09 on free-form domains.

Files

model.safetensors — the drafter (flow experts, Markov head, path head, jointly-trained VAE)
chained_flow_tree_config.json — architecture + loss config
shortlist.pt — English+Turkish candidate shortlist (see above)
vae/ — the base VAE checkpoint