See axolotl config

axolotl version: 0.13.0.dev0

yaml

1base_model: meta-llama/Llama-3.1-8B 2tokenizer_type: AutoTokenizer 3trust_remote_code: true 4strict: false 5 6is_llama_derived_model: true 7 8chat_template: chatml 9 10plugins: 11 - axolotl.integrations.liger.LigerPlugin 12 13special_tokens: 14 pad_token: "<|eot_id|>" 15 16datasets: 17 - path: nvidia/Llama-Nemotron-Post-Training-Dataset 18 name: SFT 19 split: chat 20 type: chat_template 21 field_messages: input 22 message_property_mappings: 23 role: role 24 content: content 25 field_output: output 26 27train_on_inputs: false 28 29sequence_len: 8192 30eval_sequence_len: 8192 31pad_to_sequence_len: true 32sample_packing: true 33sample_packing_group_size: 100000 34sample_packing_bin_size: 200 35group_by_length: true 36 37flash_attn: true 38 39micro_batch_size: 1 40gradient_accumulation_steps: 8 41num_epochs: 5 42 43learning_rate: 2.0e-5 44optimizer: adamw_torch_fused 45adam_beta1: 0.9 46adam_beta2: 0.999 47adam_epsilon: 1.0e-8 48 49lr_scheduler: cosine 50warmup_steps: 100 51weight_decay: 0.0 52 53bf16: true 54tf32: true 55gradient_checkpointing: true 56activation_offloading: false 57 58val_set_size: 0.01 59eval_strategy: steps 60eval_steps: 100 61 62save_strategy: steps 63save_steps: 100 64save_total_limit: 3 65save_only_model: false 66save_safetensors: true 67load_best_model_at_end: true 68metric_for_best_model: eval_loss 69greater_is_better: false 70 71logging_steps: 10 72 73output_dir: ./outputs/Llama-3.1-8B-nemotron-5epochs/ 74seed: 42 75 76use_wandb: true 77wandb_project: "llama31_base_nemotron" 78wandb_name: "llama31-8b-base-nemotron" 79

outputs/Llama-3.1-8B-nemotron-5epochs/

• Loss: 0.1268
• Memory/max Active (gib): 61.65
• Memory/max Allocated (gib): 61.65
• Memory/device Reserved (gib): 88.96

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 2e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 100
• training_steps: 1425

Training results

Framework versions

• Transformers 4.57.1
• Pytorch 2.9.0+cu130
• Datasets 4.3.0
• Tokenizers 0.22.1