See axolotl config

axolotl version: 0.15.0

yaml

1# ============================================================ 2# ЮрТэг — SFT Fine-Tuning Config для RunPod/Axolotl 3# Модель: Qwen 2.5 1.5B Instruct 4# Датасет: SuperPuperD/yurteg-legal-sft (635 train + 95 val) 5# ============================================================ 6 7# --- Модель --- 8base_model: Qwen/Qwen2.5-1.5B-Instruct 9model_type: AutoModelForCausalLM 10tokenizer_type: AutoTokenizer 11 12# --- Квантизация (QLoRA 4-bit) --- 13load_in_8bit: false 14load_in_4bit: true 15 16# --- Датасет --- 17datasets: 18 - path: SuperPuperD/yurteg-legal-sft 19 type: chat_template 20 chat_template: tokenizer_default 21 field_messages: messages 22 roles_to_train: 23 - assistant 24 train_on_eos: last 25 data_files: 26 - train.jsonl 27 28# Валидация — 13% от train (≈82 записи, аналогично нашему val.jsonl) 29val_set_size: 0.13 30 31# --- Результат --- 32output_dir: ./outputs/yurteg-qwen2.5-1.5b-sft 33 34# --- LoRA (максимальное качество) --- 35adapter: qlora 36lora_r: 64 37lora_alpha: 128 38lora_dropout: 0.05 39lora_target_linear: true 40 41# --- Длина и упаковка --- 42sequence_len: 4096 43sample_packing: true 44pad_to_sequence_len: true 45train_on_inputs: false 46 47# --- Обучение (агрессивные настройки) --- 48micro_batch_size: 2 49gradient_accumulation_steps: 8 50num_epochs: 8 51optimizer: adamw_bnb_8bit 52lr_scheduler: cosine 53learning_rate: 0.0001 54warmup_ratio: 0.1 55weight_decay: 0.01 56 57# --- Точность --- 58bf16: auto 59tf32: true 60 61# --- Оптимизация памяти --- 62gradient_checkpointing: true 63flash_attention: true 64 65# --- Логирование и сохранение --- 66logging_steps: 5 67evals_per_epoch: 4 68saves_per_epoch: 1 69save_total_limit: 3 70 71# --- Воспроизводимость --- 72seed: 42 73 74# --- Специальные токены (Qwen) --- 75special_tokens: 76 pad_token: "<|endoftext|>" 77

outputs/yurteg-qwen2.5-1.5b-sft

• Loss: 0.3193
• Ppl: 1.3762
• Memory/max Active (gib): 16.76
• Memory/max Allocated (gib): 16.76
• Memory/device Reserved (gib): 19.41

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 0.0001
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 16
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 12
• training_steps: 128

Training results

Framework versions

• PEFT 0.18.1
• Transformers 5.3.0
• Pytorch 2.8.0+cu128
• Datasets 4.5.0
• Tokenizers 0.22.2