base_model: Qwen/Qwen2.5-3B-Instruct
load_in_8bit: false
load_in_4bit: false
strict: false

# Liger Kernel
plugins:
  - axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_layer_norm: true
liger_fused_linear_cross_entropy: true

# Dataset (Qwen2.5 는 thinking 모드 없음 → enable_thinking 불필요)
chat_template: qwen_25
datasets:
  - path: train_sl_v1.jsonl
    type: chat_template
    field_messages: messages
    message_property_mappings:
      role: role
      content: content
dataset_prepared_path: last_run_prepared
val_set_size: 0.03

# Output
output_dir: ./outputs/qwen2.5-3b-sl

# Full finetuning (no adapter)
adapter:
lora_model_dir:

# Sequence (SL 데이터 max ~493 chars / ~800 tokens → 1024 로 충분)
sequence_len: 1024
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true

# Attention
flash_attention: true

# W&B
wandb_project: wizl-base-m
wandb_entity:
wandb_watch:
wandb_name: 3b-base-sl-v1
wandb_log_model:

# Hub
hub_model_id: jadechoi/wizl_base_sl-3b

# Training hyperparams (1 x H200)
micro_batch_size: 24                 # 3B + seq 1024 → H200 에서 여유 → 배치 더 크게
gradient_accumulation_steps: 4       # 1GPU × 24bs × 4accum ≈ effective batch 96
num_epochs: 3
optimizer: adamw_torch_fused
lr_scheduler: cosine
learning_rate: 2e-5
warmup_ratio: 0.05
weight_decay: 0.01

# Precision
bf16: true
fp16: false
tf32: true
gradient_checkpointing: false        # 3B + 짧은 seq → 비활성화로 속도 확보

# Logging & Saving
logging_steps: 1
evals_per_epoch: 1
saves_per_epoch: 1

# DeepSpeed ZeRO-2
deepspeed: ds_config.json

special_tokens:

wizl_base_sl-3b

This model is a fine-tuned version of Qwen/Qwen2.5-3B-Instruct on the train_sl_v1.jsonl dataset. It achieves the following results on the evaluation set:

Loss: 0.0406
Memory/max Mem Active(gib): 133.37
Memory/max Mem Allocated(gib): 133.37
Memory/device Mem Reserved(gib): 135.45

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 24
eval_batch_size: 24
seed: 42
distributed_type: multi-GPU
gradient_accumulation_steps: 4
total_train_batch_size: 96
optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 2
training_steps: 24

Training results

Training Loss	Epoch	Step	Validation Loss	Mem Active(gib)	Mem Allocated(gib)	Mem Reserved(gib)
No log	0	0	4.1721	52.38	52.38	54.4
0.4398	0.9412	8	0.2842	133.37	133.37	135.45
0.0877	1.8235	16	0.0899	133.37	133.37	135.45
0.0432	2.7059	24	0.0406	133.37	133.37	135.45

Framework versions

Transformers 4.55.2
Pytorch 2.6.0+cu124
Datasets 4.0.0
Tokenizers 0.21.4