base_model: Qwen/Qwen3-0.6B-Base
# Automatically upload checkpoint and final model to HF
# hub_model_id: username/custom_model_name

plugins:
  - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin
strict: false

chat_template: qwen3
datasets:
  - path: timarni/mmlu-stem-alpaca
    type: alpaca
    split: train

val_set_size: 0.15
output_dir: ./outputs/qwen3_mmlu_alpaca_lr_5e-5
dataset_prepared_path: last_run_prepared

sequence_len: 4096 #2048
sample_packing: true
eval_sample_packing: true
pad_to_sequence_len: true

wandb_project: mnlp_project
wandb_entity: tim-arni
wandb_watch:
wandb_name: qwen3-0.6B-mmlu_alpaca_style_lr_5e-5
wandb_log_model:

gradient_accumulation_steps: 2
micro_batch_size: 1
num_epochs: 5
optimizer: adamw_torch
lr_scheduler: cosine
learning_rate: 0.00005 # 0.0002

bf16: auto
tf32: true

gradient_checkpointing: offload
gradient_checkpointing_kwargs:
  use_reentrant: false
resume_from_checkpoint:
logging_steps: 1
flash_attention: true

warmup_steps: 10
evals_per_epoch: 4
saves_per_epoch: 1
weight_decay: 0.0
special_tokens:

outputs/qwen3_mmlu_alpaca_lr_5e-5

This model is a fine-tuned version of Qwen/Qwen3-0.6B-Base on the timarni/mmlu-stem-alpaca dataset. It achieves the following results on the evaluation set:

Loss: 0.2293

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-05
train_batch_size: 1
eval_batch_size: 1
seed: 42
distributed_type: multi-GPU
num_devices: 2
gradient_accumulation_steps: 2
total_train_batch_size: 4
total_eval_batch_size: 2
optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 10
num_epochs: 5.0

Training results

Training Loss	Epoch	Step	Validation Loss
0.5211	0.0952	1	0.5254
0.3878	0.2857	3	0.2026
0.0918	0.5714	6	0.1485
0.108	0.8571	9	0.1240
0.116	1.0952	12	0.1226
0.0992	1.3810	15	0.1217
0.0803	1.6667	18	0.2010
0.0557	1.9524	21	0.1384
0.0627	2.1905	24	0.1467
0.0315	2.4762	27	0.1556
0.0454	2.7619	30	0.2070
0.0118	3.0	33	0.2289
0.0461	3.2857	36	0.2317
0.0082	3.5714	39	0.2292
0.029	3.8571	42	0.2290
0.0138	4.0952	45	0.2299
0.0178	4.3810	48	0.2293

Framework versions

Transformers 4.51.3
Pytorch 2.5.1+cu121
Datasets 3.5.1
Tokenizers 0.21.1

timarni/MNLP_M2_mcqa_model

outputs/qwen3_mmlu_alpaca_lr_5e-5

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

timarni/
MNLP_M2_mcqa_model