See axolotl config

axolotl version: 0.13.0.dev0

yaml

1base_model: mistralai/Mistral-Nemo-Instruct-2407 2 3##uncomment for 2 GPU. More than two require more settings. 4#deepspeed: deepspeed_configs/zero1.json 5 6# Model quantization for qLoRA 7bnb_config_kwargs: 8 bnb_4bit_compute_dtype: bfloat16 9 bnb_4bit_quant_type: nf4 10 bnb_4bit_use_double_quant: true 11 12seed: 42 # do not change 13val_set_size: 0.01 # Use 1% of the dataset for validation; no pre-split in dataset 14## For other datasets set to ratio based on dataset size, 100k - 0.01, ..., 100 - 0.05 15datasets: 16 - path: TeamPV/distractors-onr-v2 17 split: train 18 type: chat_template 19 conversation: messages # Your dataset has 'messages' field 20 21chat_template: tokenizer_default # Use model's built-in chat template 22 23eval_sample_packing: false # Only 70b model can handle this 24eval_batch_size: 14 # TUNE THIS to achieve ~70+ GB CRAM usage on H100 (often same value as micro_batch_size in pre-trainer config) 25evals_per_epoch: 5 26# early_stopping_patience: 3 27 28 29# Tokenization 30sequence_len: 3000 # CRITICAL to check 31pad_to_sequence_len: true 32sample_packing: false # this will make small models go insane. 33 34special_tokens: 35 pad_token: "</s>" 36 37# LoRA/DoRA 38adapter: lora 39lora_r: 32 # 70B will require 128. Memory cost, workarounds exist. 40lora_alpha: 64 # 2x r 41lora_dropout: 0.05 42lora_target_modules: # This is basic full coverage. For LLAMA use unsloth. 43 - q_proj 44 - k_proj 45 - v_proj 46 - o_proj 47 - up_proj 48 - down_proj 49 - gate_proj 50peft_use_dora: false # 2x slower training, but allowed to drop r x4 51output_dir: /model_out/mistral-nemo-12b_sft # change this 52use_tensorboard: true 53 54# Training 55micro_batch_size: 9 # TUNE THIS to achieve ~70+ GB VRAM usage on H100 56gradient_accumulation_steps: 1 # Not worth it under 12B on h100. 70B will be mandatory. 57num_epochs: 4 # SFT is 4-5 58learning_rate: 0.00005 59lr_scheduler: cosine 60warmup_ratio: 0.10 61 62# Optimizer 63# optimizer: adamw_torch_fused 64optimizer: adamw_bnb_8bit 65bf16: true 66fp16: false 67tf32: true # H100 parameter 68 69# Attention 70flash_attention: true 71 72# Memory 73gradient_checkpointing: true 74gradient_checkpointing_kwargs: 75 use_reentrant: false 76 77# Checkpointing 78save_first_step: true 79saves_per_epoch: 2 80save_total_limit: 10 81load_best_model_at_end: true 82 83# Logging 84logging_steps: 50 85 86# HuggingFace Hub upload 87hub_model_id: TeamPV/mistral-nemo-onr-sft # ALWAYS CHANGE 88hub_strategy: every_save # Options: end, every_save, checkpoint, all_checkpoints 89hf_use_auth_token: true 90

mistral-nemo-onr-sft

• Loss: 1.0149
• Memory/max Active (gib): 77.11
• Memory/max Allocated (gib): 77.11
• Memory/device Reserved (gib): 77.96

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 5e-05
• train_batch_size: 9
• eval_batch_size: 14
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 4393
• training_steps: 43939

Training results

Framework versions

• PEFT 0.17.1
• Transformers 4.57.1
• Pytorch 2.8.0+cu128
• Datasets 4.0.0
• Tokenizers 0.22.1