You can deploy this model and start earning money today!

philipperen55/
Qwen2.5-32B-style-CLM

Name: Qwen2.5-32B-style-CLM
Availability: InStock

generated_from_trainer

Qwen/Qwen2.5-32B

adapter

apache-2.0

Views

No views yet

See axolotl config

axolotl version: 0.5.0

yaml

# /home/az/Bureau/FINE_TUNING_VASTAI_CLM+SFT_SUR_BASE_MODEL/01_DOCKER_finetune/axolotl_config.yaml

base_model: Qwen/Qwen2.5-32B
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
trust_remote_code: true

seed: 42

load_in_8bit: false
load_in_4bit: false

# Dataset CLM pur
datasets:
  - path: philipperen55/dataset41CLM
    data_files: dataset41CLM.jsonl
    type: completion
    field: text

dataset_prepared_path: /workspace/prepared_data
val_set_size: 0.01
output_dir: /workspace/output

# Séquence et packing
sequence_len: 2048
sample_packing: true
pad_to_sequence_len: true

# LoRA
adapter: lora
lora_r: 64
lora_alpha: 128
lora_dropout: 0.05
lora_target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
  - gate_proj
  - up_proj
  - down_proj

# Training
gradient_accumulation_steps: 2
micro_batch_size: 8 # 108073MiB / 143771MiB pour 6
num_epochs: 1
learning_rate: 8e-6
lr_scheduler: constant_with_warmup

warmup_ratio: 0.10
optimizer: adamw_torch

# Précision
bf16: true
fp16: false
tf32: true

# Optimisations
flash_attention: false
gradient_checkpointing: true

# Logging et sauvegardes
logging_steps: 10
save_steps: 200
save_total_limit: 6


eval_strategy: steps      
eval_steps: 200     


# WandB
wandb_project: Qwen2.5-32B-style-CLM

# Hub
hub_model_id: philipperen55/Qwen2.5-32B-style-CLM
push_to_hub: true

Qwen2.5-32B-style-CLM

This model is a fine-tuned version of Qwen/Qwen2.5-32B on the None dataset. It achieves the following results on the evaluation set:

Loss: 1.8861

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 8e-06
train_batch_size: 8
eval_batch_size: 8
seed: 42
gradient_accumulation_steps: 2
total_train_batch_size: 16
optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: constant_with_warmup
lr_scheduler_warmup_steps: 128
num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.0008	1	4.7707
2.2894	0.1554	200	2.2169
1.9807	0.3108	400	1.9656
1.9387	0.4662	600	1.9219
1.9196	0.6216	800	1.9039
1.9094	0.7770	1000	1.8933
1.9043	0.9324	1200	1.8861

Framework versions

PEFT 0.13.2
Transformers 4.46.1
Pytorch 2.3.1+cu121
Datasets 3.0.1
Tokenizers 0.20.3