You can deploy this model and start earning money today!

charlie-li/
Qwen3-30B-A3B-Instruct-2507-ScaleSWE-Distilled-Epoch3-v3.1

Name: Qwen3-30B-A3B-Instruct-2507-ScaleSWE-Distilled-Epoch3-v3.1
Availability: InStock

safetensors

qwen3_moe

Views

No views yet

Qwen3-30B-A3B-Instruct-2507-ScaleSWE-Distilled-Epoch3-v3.1

This directory contains the Epoch 3 v3.1 ScaleSWE-Distilled SFT checkpoin.

This v3.1 release is a continued full-parameter fine-tune that starts from the prior charlie-li/Qwen3-30B-A3B-Instruct-2507-ScaleSWE-Distilled-Epoch3-v3 package and adapts the model to the longer-context (70k → 128k tokens) ScaleSWE-Distilled trajectories. It uses the corrected cosine learning-rate schedule (smooth decay from 1e-5 after warmup to 0 by the final step). The original pretrained base model is Qwen/Qwen3-30B-A3B-Instruct-2507.

Training Summary

Base/init checkpoint: charlie-li/Qwen3-30B-A3B-Instruct-2507-ScaleSWE-Distilled-Epoch3-v3
Original pretrained model: Qwen3-30B-A3B-Instruct-2507
Architecture: Qwen3MoeForCausalLM
Dataset index: dataset_index.yaml
Training config: training_config.yaml
Source dataset: scaleswe_distilled_sharegpt_format_nothink_tool_role_70k_to_128k.v1_messages.jsonl
Framework: LLaMA-Factory v1 SFT
Fine-tuning type: full-parameter SFT, no PEFT/LoRA adapter
Distributed setup: FSDP2 with Ulysses context parallelism on 2×8-GPU nodes
Context parallel size: 4
HSDP shard settings: mp_replicate_size=2, mp_shard_size=8
Sequence cutoff: 131072 tokens
Micro batch size: 1
Global batch size: 128
Learning rate: 1e-5
Scheduler config: cosine (proper smooth decay; this is the "new_lr_scheduler" run)
Warmup ratio: 0.05
Precision: bf16 training
Saved tensor dtype: bfloat16
Safetensors layout: 16 shards, matching Qwen/Qwen3-30B-A3B-Instruct-2507/
Tokenizer files: copied verbatim from Qwen/Qwen3-30B-A3B-Instruct-2507/ (byte-identical to the base model, instead of the trainer-round-tripped versions)
Configured epochs: 3
Trainer log epochs: 0 through 2 (zero-based epoch counter)
Save interval: every 50 steps
Save total limit: 5
Final trainer log step: 207 / 207
Final logged loss: 0.29044443368911743

Included Artifacts

Sharded model weights: model-00001-of-00016.safetensors through model-00016-of-00016.safetensors
Weight index: model.safetensors.index.json
Hugging Face model config: config.json
Generation config: generation_config.json
Tokenizer files: tokenizer.json, tokenizer_config.json (sourced from the pretrained base model)
Chat template: chat_template.jinja
Training log: trainer_log.jsonl
Provenance files: training_config.yaml, dataset_index.yaml

Loading

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_dir = "charlie-li/Qwen3-30B-A3B-Instruct-2507-ScaleSWE-Distilled-Epoch3-v3.1"
tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_dir,
    trust_remote_code=True,
    device_map="auto",
)