See axolotl config

axolotl version: 0.13.0.dev0

yaml

1# Axolotl config for finetuning Dolphin 2.9.2 Qwen2-72B 2# Optimized for QLoRA on single H100 SXM 80GB on Vast.ai 3 4base_model: cognitivecomputations/dolphin-2.9.2-qwen2-72b 5# Or use local path if you have it downloaded: 6# base_model: ./dolphin-2.9.2-qwen2-72b 7 8# Model loading configuration 9trust_remote_code: true 10load_in_8bit: false 11load_in_4bit: true # QLoRA requires 4-bit quantization 12 13# Dataset configuration - Discord DMs in chat format 14chat_template: chatml # Dolphin uses ChatML format 15datasets: 16 - path: discord_training_data.jsonl # Your Discord DM training data 17 type: chat_template 18 field_messages: conversations # Field containing the conversation list 19 message_property_mappings: 20 role: from 21 content: value 22 23dataset_prepared_path: 24val_set_size: 0.05 25output_dir: ./outputs/dolphin-qwen2-72b-finetune 26 27# Sequence and packing configuration 28sequence_len: 2048 # Adjust based on your data (max 32768 for Qwen2) 29sample_packing: true # Efficient training by packing multiple samples 30eval_sample_packing: true 31 32# LoRA configuration 33adapter: qlora 34lora_model_dir: 35lora_r: 32 # Rank - higher = more parameters (8, 16, 32, 64) 36lora_alpha: 64 # Typically 2x lora_r 37lora_dropout: 0.05 38lora_target_linear: true # Target all linear layers 39 40# Weights & Biases logging (optional) 41wandb_project: # Set to your W&B project name 42wandb_entity: 43wandb_watch: 44wandb_name: 45wandb_log_model: 46 47# Training hyperparameters 48gradient_accumulation_steps: 4 # H100 SXM can handle more, reduced from 8 49micro_batch_size: 2 # H100 80GB has enough VRAM for batch size 2 50num_epochs: 6 # Total 6 epochs (already did 2) 51optimizer: adamw_torch_fused # Fast optimizer 52lr_scheduler: cosine 53learning_rate: 0.0002 # Conservative for large models 54 55# Precision settings 56bf16: auto # Use bfloat16 if available 57tf32: true # Enable TF32 for faster training on Ampere+ GPUs 58 59# Memory optimization 60gradient_checkpointing: true 61gradient_checkpointing_kwargs: 62 use_reentrant: false 63flash_attention: true # Faster and more memory efficient attention 64 65# Training schedule 66warmup_ratio: 0.1 67evals_per_epoch: 4 68saves_per_epoch: 2 # Increased for Vast.ai - save more frequently in case of interruption 69logging_steps: 1 70weight_decay: 0.0 71 72# Checkpointing 73resume_from_checkpoint: # Can't resume from LoRA, will train from scratch 74# save_first_step: true # Uncomment to validate checkpoint saving 75 76# IMPORTANT FOR VAST.AI: Upload to HuggingFace Hub for persistence 77# Vast.ai storage is not persistent after instance stops! 78# REPLACE "your-username" with your actual HuggingFace username: 79hub_model_id: LainBear/dolphin-qwen2-72b-lainbear 80hub_strategy: every_save # Auto-upload after each checkpoint 81hub_private_repo: false # Public repo 82 83# Special tokens (uncomment if needed) 84# special_tokens: 85# bos_token: "<|im_start|>" 86# eos_token: "<|im_end|>" 87# pad_token: "<|im_end|>" 88 89

dolphin-qwen2-72b-lainbear

• Loss: 5.5116
• Memory/max Active (gib): 49.84
• Memory/max Allocated (gib): 49.84
• Memory/device Reserved (gib): 70.01

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 0.0002
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 56
• training_steps: 564

Training results

Framework versions

• PEFT 0.17.1
• Transformers 4.57.0
• Pytorch 2.7.1+cu126
• Datasets 4.0.0
• Tokenizers 0.22.1