Views
No views yet
| Parameter | Value |
|---|---|
| Learning rate | 0.0002 |
| LR scheduler | SchedulerType.CONSTANT |
| Per-device batch size | 2 |
| Gradient accumulation | 4 |
| Effective batch size | 8 |
| Epochs | 2 |
| Max sequence length | 4096 |
| Optimizer | OptimizerNames.ADAMW_TORCH |
| Weight decay | 0.01 |
| Warmup ratio | 0.05 |
| Max gradient norm | 1.0 |
| Precision | bf16 |
| Loss type | nll |
| Chunked cross-entropy | yes |
| Parameter | Value |
|---|---|
| Rank (r) | 64 |
| Alpha | 512 |
| Target modules | attn.proj, down_proj, gate_proj, in_proj_a, in_proj_b, in_proj_qkv, in_proj_z, k_proj, linear_fc1, linear_fc2, o_proj, out_proj, q_proj, qkv, up_proj, v_proj |
| Quantization | 4-bit (nf4) |
| Dataset | Samples | Total tokens | Trainable tokens |
|---|---|---|---|
| allura-forge/doubao-seed2.0-claude-distill-v1-qwen3.5-format | 3,644 | 7,187,856 | 6,625,087 |
1model_name_or_path: Qwen/Qwen3.5-9B
2bf16: true
3gradient_checkpointing: true
4gradient_checkpointing_kwargs:
5 use_reentrant: false
6use_liger: true
7use_cce: true
8max_length: 4096
9learning_rate: 0.0002
10warmup_ratio: 0.05
11weight_decay: 0.01
12lr_scheduler_type: constant
13per_device_train_batch_size: 2
14gradient_accumulation_steps: 4
15optim: adamw_torch
16max_grad_norm: 1.0
17use_peft: true
18load_in_4bit: true
19lora_r: 64
20lora_alpha: 512
21lora_dropout: 0.0
22use_rslora: false
23logging_steps: 1
24disable_tqdm: false
25save_strategy: steps
26save_steps: 500
27save_total_limit: null
28report_to: wandb
29output_dir: output-fizz-v2
30data_config: data.yaml
31prepared_dataset: prepared
32num_train_epochs: 2
33saves_per_epoch: 1
34run_name: qwen35-9b-qlora-fizz1datasets:
2- path: allura-forge/doubao-seed2.0-claude-distill-v1-qwen3.5-format
3 type: conversational
4 truncation_strategy: drop
5shuffle_datasets: true
6shuffle_combined: true
7shuffle_seed: 42
8eval_split: 0.0
9split_seed: 42
10assistant_only_loss: true