Views
No views yet
| Parameter | Value |
|---|---|
| Learning rate | 0.0001 |
| LR scheduler | SchedulerType.COSINE |
| Per-device batch size | 2 |
| Gradient accumulation | 4 |
| Effective batch size | 8 |
| Epochs | 1 |
| Max sequence length | 4096 |
| Optimizer | OptimizerNames.PAGED_ADAMW_8BIT |
| Weight decay | 0.01 |
| Warmup ratio | 0.05 |
| Max gradient norm | 1.0 |
| Precision | bf16 |
| Loss type | nll |
| Chunked cross-entropy | yes |
| Parameter | Value |
|---|---|
| Rank (r) | 32 |
| Alpha | 64 |
| Target modules | attn.proj, down_proj, gate_proj, in_proj_a, in_proj_b, in_proj_qkv, in_proj_z, k_proj, linear_fc1, linear_fc2, o_proj, out_proj, q_proj, qkv, up_proj, v_proj |
| Quantization | 4-bit (nf4) |
| Dataset | Samples | Total tokens | Trainable tokens |
|---|---|---|---|
| ToastyPigeon/brainrot-cleaned/brainrot_chatlog.jsonl | 3,179 | 999,446 | 999,446 |
| rpDungeon/some-revised-datasets/springdragon_processed.jsonl | 2,909 | 5,542,206 | 5,542,206 |
| ToastyPigeon/disco-chat | 1,742 | 431,320 | 431,320 |
| rpDungeon/some-revised-datasets/wrecklora_text.parquet | 3,639 | 10,717,729 | 10,717,729 |
| rpDungeon/some-revised-datasets/floyd_text.parquet | 1,758 | 6,689,254 | 6,689,254 |
| Total | 13,227 | 24,379,955 | 24,379,955 |
1model_name_or_path: Qwen/Qwen3.5-9B
2bf16: true
3gradient_checkpointing: true
4gradient_checkpointing_kwargs:
5 use_reentrant: false
6use_liger: true
7use_cce: true
8neftune_noise_alpha: 10
9dataloader_num_workers: 4
10dataloader_pin_memory: true
11max_length: 4096
12learning_rate: 0.0001
13warmup_ratio: 0.05
14weight_decay: 0.01
15lr_scheduler_type: cosine
16per_device_train_batch_size: 2
17gradient_accumulation_steps: 4
18optim: paged_adamw_8bit
19max_grad_norm: 1.0
20use_peft: true
21load_in_4bit: true
22bnb_4bit_quant_type: nf4
23lora_r: 32
24lora_alpha: 64
25lora_dropout: 0
26logging_steps: 1
27disable_tqdm: false
28save_strategy: steps
29save_steps: 500
30save_total_limit: 3
31report_to: wandb
32output_dir: output-games-misc
33data_config: data.yaml
34prepared_dataset: prepared
35attn_implementation: flash_attention_2
36num_train_epochs: 1
37saves_per_epoch: 3
38run_name: qwen35-9b-qlora-games-misc1datasets:
2- path: rpDungeon/some-revised-datasets
3 data_files: springdragon_processed.jsonl
4 type: text
5 columns:
6 - text
7 truncation_strategy: split
8- path: rpDungeon/some-revised-datasets
9 data_files: wrecklora_text.parquet
10 type: text
11 truncation_strategy: split
12- path: ToastyPigeon/disco-chat
13 type: text
14 truncation_strategy: split
15- path: rpDungeon/some-revised-datasets
16 data_files: floyd_text.parquet
17 type: text
18 truncation_strategy: split
19- path: ToastyPigeon/brainrot-cleaned
20 data_files: brainrot_chatlog.jsonl
21 type: text
22 columns:
23 - text
24 truncation_strategy: truncate
25shuffle_datasets: true
26shuffle_combined: true
27shuffle_seed: 42
28eval_split: 0.0
29split_seed: 42
30assistant_only_loss: false