Views
No views yet
| Parameter | Value |
|---|---|
| Learning rate | 0.0001 |
| LR scheduler | SchedulerType.COSINE |
| Per-device batch size | 1 |
| Gradient accumulation | 8 |
| Effective batch size | 8 |
| Epochs | 1 |
| Max sequence length | 16384 |
| Optimizer | OptimizerNames.PAGED_ADAMW_8BIT |
| Weight decay | 0.01 |
| Warmup ratio | 0.05 |
| Max gradient norm | 1.0 |
| Precision | bf16 |
| Loss type | nll |
| Chunked cross-entropy | yes |
| Parameter | Value |
|---|---|
| Rank (r) | 32 |
| Alpha | 64 |
| Dropout | 0.1 |
| Target modules | attn.proj, down_proj, gate_proj, in_proj_a, in_proj_b, in_proj_qkv, in_proj_z, k_proj, linear_fc1, linear_fc2, o_proj, out_proj, q_proj, qkv, up_proj, v_proj |
| Quantization | 4-bit (nf4) |
| Dataset | Samples | Total tokens | Trainable tokens |
|---|---|---|---|
| rpDungeon/some-revised-datasets/rosier_inf_strict_text.parquet | 10,466 | 65,084,382 | 65,084,382 |
1model_name_or_path: Qwen/Qwen3.5-9B
2bf16: true
3gradient_checkpointing: true
4gradient_checkpointing_kwargs:
5 use_reentrant: false
6use_liger: true
7use_cce: true
8neftune_noise_alpha: 5
9dataloader_num_workers: 4
10dataloader_pin_memory: true
11max_length: 16384
12learning_rate: 0.0001
13warmup_ratio: 0.05
14weight_decay: 0.01
15lr_scheduler_type: cosine
16per_device_train_batch_size: 1
17gradient_accumulation_steps: 8
18optim: paged_adamw_8bit
19max_grad_norm: 1.0
20use_peft: true
21load_in_4bit: true
22bnb_4bit_quant_type: nf4
23lora_r: 32
24lora_alpha: 64
25lora_dropout: 0.1
26logging_steps: 1
27disable_tqdm: false
28save_strategy: steps
29save_steps: 500
30save_total_limit: 3
31report_to: wandb
32output_dir: output-fujin-v2
33data_config: data.yaml
34prepared_dataset: prepared
35attn_implementation: flash_attention_2
36num_train_epochs: 1
37saves_per_epoch: 3
38run_name: qwen35-9b-qlora-v21datasets:
2- path: rpDungeon/some-revised-datasets
3 data_files: rosier_inf_strict_text.parquet
4 type: text
5 truncation_strategy: split
6shuffle_datasets: true
7shuffle_combined: true
8shuffle_seed: 42
9eval_split: 0.0
10split_seed: 42
11assistant_only_loss: false