See axolotl config

axolotl version: 0.13.0.dev0

yaml

1base_model: Qwen/Qwen3-8B 2# Automatically upload checkpoint and final model to HF 3# hub_model_id: username/custom_model_name 4hub_model_id: tuandunghcmut/Qwen3-8B-Private 5 6 7plugins: 8 - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin 9strict: false 10 11 12 13chat_template: qwen3 14datasets: 15 # - path: trendmicro-ailab/Primus-Seed 16 # type: chat_template 17 # split: train[:20%] 18 # split: train 19 # field_messages: conversations 20 # message_property_mappings: 21 # role: from 22 # content: value 23 - path: trendmicro-ailab/Primus-Reasoning 24 type: chat_template 25 # split: train[:20%] 26 split: train 27 split_thinking: true 28 chat_template: qwen3 29 field_messages: messages 30 message_property_mappings: 31 role: role 32 content: content 33 34 35val_set_size: 0.075 36output_dir: ./outputs/out2 37dataset_prepared_path: last_run_prepared 38 39# sequence_len: 2048 40sequence_len: 3072 41sample_packing: true 42eval_sample_packing: true 43 44 45load_in_4bit: true 46adapter: qlora 47# lora_r: 16 48# lora_alpha: 32 49 50# lora_r: 32 51# lora_alpha: 64 52 53 54lora_r: 64 55lora_alpha: 128 56 57lora_target_modules: 58 - q_proj 59 - k_proj 60 - v_proj 61 - o_proj 62 - down_proj 63 - up_proj 64lora_mlp_kernel: true 65lora_qkv_kernel: true 66lora_o_kernel: true 67 68wandb_project: 69wandb_entity: 70wandb_watch: 71wandb_name: 72wandb_log_model: 73 74gradient_accumulation_steps: 2 75micro_batch_size: 4 76num_epochs: 30 77optimizer: adamw_torch_4bit 78lr_scheduler: cosine 79learning_rate: 0.00002 80 81bf16: auto 82tf32: true 83 84gradient_checkpointing: offload 85gradient_checkpointing_kwargs: 86 use_reentrant: false 87resume_from_checkpoint: 88logging_steps: 1 89flash_attention: true 90 91warmup_ratio: 0.1 92evals_per_epoch: 4 93saves_per_epoch: 1 94weight_decay: 0.01 95special_tokens: 96 97# save_first_step: true # uncomment this to validate checkpoint saving works with your config 98

Qwen3-8B-Private

• Loss: 0.9940
• Memory/max Active (gib): 8.07
• Memory/max Allocated (gib): 8.07
• Memory/device Reserved (gib): 10.78

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 2e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_TORCH_4BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 471
• training_steps: 4710

Training results

Framework versions

• PEFT 0.17.1
• Transformers 4.56.1
• Pytorch 2.7.1+cu126
• Datasets 4.0.0
• Tokenizers 0.22.1