# 1. Base Model Configuration
base_model: Qwen/Qwen3.5-9B
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

# 2. Precision Settings (LoRA only, NO Quantization)
load_in_8bit: false
load_in_4bit: false
strict: false

# 3. LoRA Adapter Configuration
adapter: lora
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_linear: true

# 4. Dataset Configuration
datasets:
  - path: data/processed/diagnosis.train.jsonl
    type: chat_template
    chat_template: qwen3_5

test_datasets:
  - path: data/processed/diagnosis.val.jsonl
    type: chat_template
    chat_template: qwen3_5
    split: train

val_set_size: 0
# Cache the tokenized dataset so re-runs skip preprocessing.
dataset_prepared_path: last_run_prepared

# 5. Training Hyperparameters
sequence_len: 4096
# sample_packing triggers a bug in axolotl's qwen3_5 monkeypatch (get_cu_seqlens
# calls .view() on a non-contiguous position_ids tensor from packing -> RuntimeError)
# -> disabled, same workaround as the gemma config.
sample_packing: false
eval_sample_packing: false
pad_to_sequence_len: false

# SFT Hyperparameters
gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 3
# paged_adamw_8bit halves optimizer-state VRAM vs 32bit, negligible LoRA quality impact.
optimizer: paged_adamw_8bit
lr_scheduler: cosine
learning_rate: 1e-4
# ~560 total steps (1494 examples / eff. batch 8 * 3 epochs); 0.1 ratio ~= 56 warmup steps.
warmup_ratio: 0.1

# 6. Hardware and Performance Optimization
bf16: auto
fp16: false
tf32: true   # free matmul speedup on Ampere+ (RTX PRO 6000 / A6000), no quality cost.
flash_attention: true

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false

# Liger fused kernels: big VRAM win on the large-vocab LM head + faster RMSNorm/GLU.
# Compatible with sample_packing: false and sdp/flash attention (doesn't touch attention).
plugins:
  - axolotl.integrations.liger.LigerPlugin
liger_rms_norm: true
liger_glu_activation: true
liger_fused_linear_cross_entropy: true

# Dataloader throughput.
dataloader_num_workers: 2
dataloader_pin_memory: true
dataloader_prefetch_factor: 8

# 7. Logging and Output
output_dir: results/qwen3.5-9b-lora-out
logging_steps: 10
save_steps: 100
eval_steps: 100

results/qwen3.5-9b-lora-out

This model is a fine-tuned version of Qwen/Qwen3.5-9B on the data/processed/diagnosis.train.jsonl dataset. It achieves the following results on the evaluation set:

Loss: 0.5346
Ppl: 1.7068
Memory/max Active (gib): 34.5
Memory/max Allocated (gib): 34.5
Memory/device Reserved (gib): 38.37

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0001
train_batch_size: 2
eval_batch_size: 2
seed: 42
gradient_accumulation_steps: 4
total_train_batch_size: 8
optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 53
training_steps: 539

Training results

Training Loss	Epoch	Step	Validation Loss	Ppl	Active (gib)	Allocated (gib)	Reserved (gib)
No log	0	0	0.7821	2.1861	31.92	31.92	32.03
0.5638	0.5563	100	0.5557	1.7432	31.97	31.97	41.47
0.4805	1.1113	200	0.5384	1.7133	31.97	31.97	43.31
0.4766	1.6676	300	0.5294	1.6980	31.97	31.97	43.31
0.4032	2.2225	400	0.5337	1.7052	31.97	31.97	41.94
0.3978	2.7789	500	0.5346	1.7067	31.97	31.97	43.31
0.3990	2.9958	539	0.5346	1.7068	34.5	34.5	38.37

Framework versions

PEFT 0.19.1
Transformers 5.10.2
Pytorch 2.10.0+cu128
Datasets 4.8.5
Tokenizers 0.22.2

Giordanopsouza/qwen3.5-9b-clinical-lora-v2-think

results/qwen3.5-9b-lora-out

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

Giordanopsouza/
qwen3.5-9b-clinical-lora-v2-think