You can deploy this model and start earning money today!

charlie-li/
Qwen3-30B-A3B-Instruct-2507-ScaleSWE-Distilled-Epoch6-v3.1

Name: Qwen3-30B-A3B-Instruct-2507-ScaleSWE-Distilled-Epoch6-v3.1
Availability: InStock

safetensors

qwen3_moe

Views

No views yet

Qwen3-30B-A3B-Instruct-2507-ScaleSWE-Distilled-Epoch6-v3.1

This directory contains the Epoch 6 v3.1 ScaleSWE-Distilled SFT checkpoint.

This v3.1 release is a continued full-parameter fine-tune that starts from the prior charlie-li/Qwen3-30B-A3B-Instruct-2507-ScaleSWE-Distilled-Epoch6-v3 package and runs three additional epochs on the 70k-token CoderForge SingleFC/gitdiff trajectories (cutoff 71680). It uses the corrected cosine learning-rate schedule (smooth decay from 1e-5 after warmup to 0 by the final step). The original pretrained base model is Qwen/Qwen3-30B-A3B-Instruct-2507.

Training Summary

Base/init checkpoint: charlie-li/Qwen3-30B-A3B-Instruct-2507-ScaleSWE-Distilled-Epoch6-v3
Original pretrained model: Qwen3-30B-A3B-Instruct-2507
Architecture: Qwen3MoeForCausalLM
Source dataset: coderforge_dedup_resp_shortest_sharegpt_format_nothink_tool_role_singleFC_gitdiff_70k.v1_messages.jsonl
Framework: LLaMA-Factory v1 SFT
Fine-tuning type: full-parameter SFT, no PEFT/LoRA adapter
Template: qwen3_nothink
Attention implementation: flash_attention_2
Distributed setup: FSDP2 with Ulysses context parallelism on 4×8-GPU nodes (world size 32)
Context parallel size: 2
HSDP shard settings: mp_replicate_size=4, mp_shard_size=8 (data parallel size 16)
Sequence cutoff: 71680 tokens (70k)
Micro batch size: 1
Global batch size: 128 (8 micro-batches per optimizer step)
Batching workers: 16
Learning rate: 1e-5
Scheduler config: cosine (proper smooth decay; this is the "new_lr_scheduler" run)
Warmup ratio: 0.05
Max grad norm: 1.0
Activation checkpointing: enabled
Precision: bf16 training
Saved tensor dtype: bfloat16
Safetensors layout: 16 shards, matching Qwen/Qwen3-30B-A3B-Instruct-2507/
Tokenizer files: copied verbatim from Qwen/Qwen3-30B-A3B-Instruct-2507/ (byte-identical to the base model, instead of the trainer-round-tripped versions)
Configured epochs: 3, in total 9 based on Qwen/Qwen3-30B-A3B-Instruct-2507/
Trainer log epochs: 0 through 2 (zero-based epoch counter)
Save interval: every 100 steps
Save total limit: 5
Logging steps: 1
Final trainer log step: 801 / 801
Final logged loss: 0.14545220136642456
Original training log: logs/33_run_qwen3_30b_a3b_instruct_2507_scaleswe_llamafactory_v1_full_fsdp2_cp_4x8gpu_only_scaleswe_70k_rank0.log

Included Artifacts

Sharded model weights: model-00001-of-00016.safetensors through model-00016-of-00016.safetensors
Weight index: model.safetensors.index.json
Hugging Face model config: config.json
Generation config: generation_config.json
Tokenizer files: tokenizer.json, tokenizer_config.json, vocab.json, merges.txt (sourced from the pretrained base model)
Chat template: chat_template.jinja
Long-context (1M) reference config: config_1m.json (from the upstream base model; not active by default)

Loading

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_dir = "charlie-li/Qwen3-30B-A3B-Instruct-2507-ScaleSWE-Distilled-Epoch6-v3.1"
tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_dir,
    trust_remote_code=True,
    device_map="auto",
)