# ─────────────────────────────────────────────────────────────────────────────
# Axolotl config: Qwen2.5-7B-Instruct + LoRA  (NO quantization)
# GPU: 4× RTX 2080 (32GB total) — DeepSpeed ZeRO-3 shards 14GB model across GPUs
#
# WHY Qwen2.5-7B-Instruct (not Math):
#   The Math variant is over-specialized for \boxed{} problem solving and breaks
#   under FP16 inference on Turing. The general Instruct base is robust,
#   has strong math knowledge from pretraining, and is the correct base for
#   a conversational tutoring task.
#
# WHY no quantization:
#   Full FP16 LoRA has no quantization noise in activations → cleaner gradients
#   → better adapter quality. With 4 GPUs and ZeRO-3 this is feasible.
#
# Launch with:
#   deepspeed --num_gpus 4 --module axolotl.cli.train training/qwen25_7b_lora_4gpu.yaml
# ─────────────────────────────────────────────────────────────────────────────

base_model: Qwen/Qwen2.5-7B-Instruct
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

# ── Data ──────────────────────────────────────────────────────────────────────
datasets:
  - path: final_ft_dataset/train.jsonl
    type: alpaca
    field_instruction: instruction
    field_input: input
    field_output: output

test_datasets:
  - path: final_ft_dataset/val.jsonl
    type: alpaca
    field_instruction: instruction
    field_input: input
    field_output: output
    max_samples: 20        # eval on 20 examples — fast, still meaningful signal

dataset_prepared_path: training/.prepared_qwen25_7b
val_set_size: 0          # don't carve eval from train.jsonl; use test_datasets above

sequence_len: 1024        # reduced from 2048 to fit 11GB GPUs; covers ≥95% of dataset
sample_packing: false
train_on_inputs: false      # CRITICAL: loss on output tokens only

# ── LoRA (no quantization) ────────────────────────────────────────────────────
adapter: lora               # plain LoRA, not qlora
lora_r: 32
lora_alpha: 64
lora_dropout: 0.05
lora_target_linear: true    # all linear layers
lora_fan_in_fan_out: false

# Disable Axolotl fused LoRA kernels — designed for Ampere+, cause OOM on Turing (RTX 2080)
lora_qkv_kernel: false
lora_mlp_kernel: false
lora_o_kernel: false

# ── Training ──────────────────────────────────────────────────────────────────
output_dir: /scratch/atharv.johar/AOLM/outputs/qwen25_7b_4gpu_lora_r32

num_epochs: 3
micro_batch_size: 1         # per GPU; effective batch = 1 × 4 GPUs × 8 accum = 32
gradient_accumulation_steps: 8
optimizer: adamw_torch_fused
lr_scheduler: cosine
learning_rate: 2e-4
weight_decay: 0.0
max_grad_norm: 1.0
warmup_ratio: 0.03

gradient_checkpointing: true

# ── Precision ─────────────────────────────────────────────────────────────────
bf16: false
fp16: true                  # Turing has FP16 tensor cores; no BF16 hardware
tf32: false

# ── DeepSpeed ZeRO-3 ─────────────────────────────────────────────────────────
deepspeed: training/deepspeed_zero3.json

# ── Logging & checkpointing ───────────────────────────────────────────────────
logging_steps: 10
eval_steps: 100           # evaluate every 100 steps on the 20-sample val set
save_steps: 100
save_total_limit: 3

wandb_project: aolm-ft
wandb_run_id: qwen25-7b-lora-r32-4gpu-noquant
wandb_watch: "false"

seed: 42

scratch/atharv.johar/AOLM/outputs/qwen25_7b_4gpu_lora_r32

This model is a fine-tuned version of Qwen/Qwen2.5-7B-Instruct on the final_ft_dataset/train.jsonl dataset. It achieves the following results on the evaluation set:

Loss: 0.0749
Ppl: 1.0778
Memory/max Active (gib): 6.56
Memory/max Allocated (gib): 5.98
Memory/device Reserved (gib): 9.54

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0002
train_batch_size: 1
eval_batch_size: 1
seed: 42
distributed_type: multi-GPU
num_devices: 4
gradient_accumulation_steps: 8
total_train_batch_size: 32
total_eval_batch_size: 4
optimizer: Use adamw_torch_fused with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 10
training_steps: 363
mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Ppl	Active (gib)	Allocated (gib)	Reserved (gib)
No log	0	0	2.0092	7.4574	4.68	4.68	8.83
0.1003	0.8273	100	0.0996	1.1047	4.75	4.75	9.31
0.0641	1.6536	200	0.0750	1.0779	4.75	4.75	9.65
0.0295	2.4798	300	0.0748	1.0777	4.75	4.75	9.71
0.0257	3.0	363	0.0749	1.0778	6.56	5.98	9.54

Framework versions

PEFT 0.19.1
Transformers 5.5.0
Pytorch 2.8.0+cu128
Datasets 4.5.0
Tokenizers 0.22.2

joharatharv/AOLM_final_finetuned

scratch/atharv.johar/AOLM/outputs/qwen25_7b_4gpu_lora_r32

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

joharatharv/
AOLM_final_finetuned