Views
No views yet
| Parameter | Value |
|---|---|
| Learning rate | 0.0001 |
| LR scheduler | SchedulerType.COSINE |
| Per-device batch size | 1 |
| Effective batch size | 1 |
| Epochs | 5 |
| Max sequence length | 2048 |
| Optimizer | OptimizerNames.PAGED_ADEMAMIX_8BIT |
| Weight decay | 0.01 |
| Warmup ratio | 0.05 |
| Max gradient norm | 1.0 |
| Precision | bf16 |
| Loss type | nll |
| Parameter | Value |
|---|---|
| Rank (r) | 32 |
| Alpha | 8 |
| Target modules | attn.proj, down_proj, gate_proj, in_proj_a, in_proj_b, in_proj_qkv, in_proj_z, k_proj, linear_fc1, linear_fc2, o_proj, out_proj, q_proj, qkv, up_proj, v_proj |
| rsLoRA | yes |
| Quantization | 4-bit (nf4) |
| Dataset | Samples | Total tokens | Trainable tokens |
|---|---|---|---|
| kronk_instruct_messages.jsonl | 236 | 192,732 | 139,817 |
1model_name_or_path: Qwen/Qwen3.5-9B
2bf16: true
3gradient_checkpointing: true
4gradient_checkpointing_kwargs:
5 use_reentrant: false
6use_liger: true
7max_length: 2048
8last_assistant_only_loss: true
9learning_rate: 0.0001
10warmup_ratio: 0.05
11weight_decay: 0.01
12lr_scheduler_type: cosine
13neftune_noise_alpha: 5
14aux_loss_top_prob_weight: 0.1
15per_device_train_batch_size: 1
16gradient_accumulation_steps: 1
17optim: paged_ademamix_8bit
18max_grad_norm: 1.0
19use_peft: true
20load_in_4bit: true
21bnb_4bit_quant_type: nf4
22lora_r: 32
23lora_alpha: 8
24lora_dropout: 0
25use_rslora: true
26logging_steps: 1
27disable_tqdm: false
28save_strategy: steps
29save_steps: 500
30save_total_limit: null
31report_to: wandb
32output_dir: output
33data_config: data.yaml
34prepared_dataset: prepared
35num_train_epochs: 5
36saves_per_epoch: 1
37run_name: qwen35-9b-kronk1datasets:
2- path: kronk_instruct_messages.jsonl
3 type: conversational
4 truncation_strategy: drop
5 columns:
6 - messages
7shuffle_datasets: true
8shuffle_combined: true
9shuffle_seed: 42
10eval_split: 0.0
11split_seed: 42