Views
No views yet
| Parameter | Value |
|---|---|
| Learning rate | 1e-05 |
| LR scheduler | rex (custom; max_lr=1e-5, min_lr=1e-6, warmup_ratio=0.05)_WITH_WARMUP |
| Per-device batch size | 1 |
| Gradient accumulation | 4 |
| Effective batch size | 4 |
| Epochs | 1 |
| Max sequence length | 6144 |
| Optimizer | OptimizerNames.PAGED_ADAMW_8BIT |
| Warmup ratio | 0.05 |
| Max gradient norm | 1.0 |
| Precision | bf16 |
| Loss type | nll |
| Assistant-only loss | yes |
| Chunked cross-entropy | yes |
| Parameter | Value |
|---|---|
| Rank (r) | 16 |
| Alpha | 32 |
| Target modules | .*language_model.layers.\d+.(self_attn.(q |
| Quantization | 4-bit (nf4) |
| Dataset | Samples | Total tokens | Trainable tokens |
|---|---|---|---|
| writing_critique.jsonl | 1,586 | 1,317,233 | 599,216 |
| instruct.jsonl | 962 | 933,867 | 838,340 |
| marvin_style_bible.jsonl | 2,549 | 11,096,548 | 10,492,459 |
| rp_generation_mistral.jsonl | 255 | 825,572 | 375,698 |
| rp_analysis.jsonl | 244 | 725,115 | 177,861 |
| rp_generation_final.jsonl | 129 | 513,974 | 240,019 |
| Total | 5,725 | 15,412,309 | 12,723,593 |
1model_name_or_path: google/gemma-4-31B-it
2data_config: data.yaml
3prepared_dataset: prepared_packed
4output_dir: gemma4-31b-it-glimmer-rp-r16a32
5chat_template_path: chat_template_with_channel.jinja
6attn_implementation: flex_attention
7bf16: true
8gradient_checkpointing: true
9gradient_checkpointing_kwargs:
10 use_reentrant: false
11use_cce: true
12chunked_mlp: true
13chunked_mlp_chunks: 16
14dataloader_num_workers: 2
15dataloader_pin_memory: true
16model_parallel: true
17max_memory:
18 0: 16GiB
19 1: 24GiB
20max_length: 6144
21per_device_train_batch_size: 1
22gradient_accumulation_steps: 4
23pad_to_multiple_of: 128
24use_peft: true
25load_in_4bit: true
26bnb_4bit_quant_type: nf4
27lora_r: 16
28lora_alpha: 32
29lora_dropout: 0.0
30use_rslora: false
31lora_target_modules: .*language_model\.layers\.\d+\.(self_attn\.(q|k|v|o)_proj|mlp\.(gate|up|down)_proj)$
32learning_rate: 1.0e-05
33lr_scheduler_type: constant_with_warmup
34warmup_ratio: 0.05
35weight_decay: 0.0
36max_grad_norm: 1.0
37optim: paged_adamw_8bit
38num_train_epochs: 1
39saves_per_epoch: 2
40save_total_limit: 4
41rolling_save_steps: 30
42rolling_save_total_limit: 1
43assistant_only_loss: true
44full_mask_reasoning: true
45logging_steps: 1
46disable_tqdm: false
47report_to: wandb
48run_name: g4-31b-it-glimmer-rp1datasets:
2- path: rp_generation_final.jsonl
3 type: chat
4 truncation_strategy: split
5- path: rp_generation_mistral.jsonl
6 type: chat
7 truncation_strategy: split
8- path: instruct.jsonl
9 type: chat
10 truncation_strategy: split
11- path: rp_analysis.jsonl
12 type: chat
13 truncation_strategy: split
14- path: writing_critique.jsonl
15 type: chat
16 truncation_strategy: split
17- path: marvin_style_bible.jsonl
18 type: chat
19 truncation_strategy: split
20shuffle_datasets: true
21shuffle_combined: true
22shuffle_seed: 42
23eval_split: 0
24split_seed: 42
25assistant_only_loss: true