You can deploy this model and start earning money today!

jiamingshan/
qwen3-4b-dflash-official100k-recipe

Name: qwen3-4b-dflash-official100k-recipe
Availability: InStock

Views

No views yet

Qwen3-4B Official-100K DFlash E2E Recipe

This repo is meant to be runnable by an external user on a normal 8-GPU Linux machine. It does not assume the original lab Docker container or a /data/sjm/... checkout.

It contains the training/eval wrapper code used for this comparison. The prepared 100K dataset is hosted separately at:

https://huggingface.co/datasets/jiamingshan/qwen3-4b-dflash-official100k-prepared

Foolproof Quick Start

Requirements:

Linux machine with 8 NVIDIA GPUs.
Python 3.11 or newer.
CUDA-compatible PyTorch install.
Hugging Face access for Qwen/Qwen3-4B and z-lab/Qwen3-4B-DFlash-b16.

bash

git clone https://huggingface.co/jiamingshan/qwen3-4b-dflash-official100k-recipe
cd qwen3-4b-dflash-official100k-recipe

python3 -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install -r requirements-train.txt

# Only needed if your machine is not already logged in.
huggingface-cli login

./00_check_env.sh
./01_download_data.sh

# Train Qwen E2E+DFlash.
./10_train_e2e.sh

# Train Qwen frozen+DFlash.
./20_train_frozen.sh

The two training scripts print the final checkpoint paths. They look like:

outputs/<run_id>/checkpoints/step_002000

To run the GSM8K serving eval, install vLLM in an eval environment and pass the two checkpoint paths:

bash

python3 -m venv .venv-vllm
source .venv-vllm/bin/activate
pip install -U pip
pip install -r requirements-eval.txt

./30_eval_gsm8k_table.sh \
  outputs/<frozen_run_id>/checkpoints/step_002000 \
  outputs/<e2e_run_id>/checkpoints/step_002000

The requested table is written to:

requested_gsm8k_table.md

Exact Training Setup

Common:

text

TARGET_NAME_OR_PATH=Qwen/Qwen3-4B
DRAFT_NAME_OR_PATH=z-lab/Qwen3-4B-DFlash-b16
DATA_PATH=data/qwen3_dflash_official100k_prepared
STEPS=2000
WORLD_SIZE=8
GRADIENT_ACCUMULATION_STEPS=3
EFFECTIVE_GLOBAL_BATCH=24
SAMPLES_SEEN=48000
MAX_SEQ_LEN=3072
MAX_ANCHORS=512
ANCHOR_BATCH_SIZE=512
GAMMA=7
LOSS_NORMALIZATION=weighted

Qwen E2E+DFlash

text

FREEZE_TARGET_ALL=0
FREEZE_TARGET_UNUSED_TAIL=0
LR_DRAFT=1e-4
LR_TARGET=2e-6
DFLASH_LOSS_WEIGHT=1.0
LM_LOSS_WEIGHT=1.0
KL_LOSS_WEIGHT=0.0
FEATURE_ANCHOR_LOSS_WEIGHT=0.0
EOS_LOSS_WEIGHT=0.0

Total loss:

dflash_loss + lm_loss

Qwen frozen+DFlash

text

FREEZE_TARGET_ALL=1
LR_DRAFT=1e-4
LR_TARGET=0
DFLASH_LOSS_WEIGHT=1.0
LM_LOSS_WEIGHT=0.0
KL_LOSS_WEIGHT=0.0
FEATURE_ANCHOR_LOSS_WEIGHT=0.0
EOS_LOSS_WEIGHT=0.0

Total loss:

dflash_loss

Loss Definitions

dflash_loss is the DFlash weighted cross-entropy objective over static draft targets from the prepared data. It uses gamma=7 and loss_normalization=weighted.

lm_loss is next-token cross entropy from the trainable target's own lm_head, computed over loss_mask[:, 1:] from the same target forward pass that supplies DFlash hidden features.

Requested Table

This is the observed full GSM8K no-thinking table from the 2026-06-19 run:

Metric	Qwen target	Qwen target + zlab-official DFlash	Qwen frozen+DFlash	Qwen E2E+DFlash
Accuracy	1220/1319 = 0.9249	1209/1319 = 0.9166	1211/1319 = 0.9181	1207/1319 = 0.9151
Mean tok/s	141.82	672.98	687.82	687.48
Mean accept	n/a	6.222	6.324	6.345

Mean tok/s is mean single-shard throughput across 8 one-GPU shards. The aggregate 8-shard throughput is also stored in each eval run's aggregate_summary.json, but it is not the value in this table.

Files

src/train/train_joint_target_dflash.py: self-contained training script.
third_party/dflash-official: DFlash draft model code needed by training.
third_party/speculators: attention helpers used by the DFlash objective.
src/eval: GSM8K prompt, scoring, vLLM benchmark, acceptance parsing, and sharded aggregation scripts.
recipe.json: machine-readable recipe summary.