You can deploy this model and start earning money today!

philipperen55/
Qwen2.5-14B-CPT-lora

Name: Qwen2.5-14B-CPT-lora
Availability: InStock

generated_from_trainer

Qwen/Qwen2.5-14B

adapter

apache-2.0

Views

No views yet

See axolotl config

axolotl version: 0.5.0

yaml

# /home/az/Bureau/FINE_TUNING_VASTAI_CLM+SFT_SUR_BASE_MODEL/02-fineTune-CLM/axolotl_config.yaml

base_model: Qwen/Qwen2.5-14B 
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
trust_remote_code: true

seed: 42

load_in_8bit: false
load_in_4bit: false

# Dataset CLM pur
datasets:
  - path: philipperen55/dataset50CFT
    data_files: dataset50CFT.jsonl
    type: completion
    field: text

dataset_prepared_path: /workspace/prepared_data
val_set_size: 0.003
output_dir: /workspace/output

# Séquence et packing
sequence_len: 2048
sample_packing: false
pad_to_sequence_len: false
group_by_length: true
train_on_eos: true

# cela fonctionne pour Qwen, il met automatiquement le eot # refaire un test pour ministal, et sans doute mettre append_concat_token: true !!!!
append_concat_token: false
add_eos_token: false

# LoRA
adapter: lora
lora_r: 128
lora_alpha: 256
lora_dropout: 0.05
lora_target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
  - gate_proj
  - up_proj
  - down_proj
  
# Training
micro_batch_size: 3
gradient_accumulation_steps: 32
num_epochs: 1
learning_rate: 2e-4
lr_scheduler: constant_with_warmup 

warmup_ratio: 0.03
optimizer: adamw_torch_fused

# Précision
bf16: true
fp16: false
tf32: true

weight_decay: 0.0
max_grad_norm: 1.0

flash_attention: true
sdp_attention: false

overrides_of_model_config:
  attn_implementation: flash_attention_2  # forcer FA2 jusqu’au modèle
  use_cache: false

  
gradient_checkpointing: false #true

# Logging et sauvegardes
logging_steps: 10
save_steps: 500
save_total_limit: 1
save_only_model: true
save_safetensors: true


eval_strategy: steps      
eval_steps: 500   


# WandB
wandb_project: Qwen2.5-14B-CPT-lora

# Hub
hub_model_id: philipperen55/Qwen2.5-14B-CPT-lora
push_to_hub: true
hub_strategy: every_save

Qwen2.5-14B-CPT-lora

This model is a fine-tuned version of Qwen/Qwen2.5-14B on the None dataset. It achieves the following results on the evaluation set:

Loss: 1.9034

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0002
train_batch_size: 3
eval_batch_size: 3
seed: 42
gradient_accumulation_steps: 32
total_train_batch_size: 96
optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: constant_with_warmup
lr_scheduler_warmup_steps: 110
num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.0003	1	1.9523
1.9179	0.1352	500	1.9364
1.9242	0.2704	1000	1.9267
1.9127	0.4056	1500	1.9215
1.9229	0.5408	2000	1.9142
1.9044	0.6761	2500	1.9114
1.9005	0.8113	3000	1.9072
1.9068	0.9465	3500	1.9034

Framework versions

PEFT 0.13.2
Transformers 4.46.1
Pytorch 2.3.1+cu121
Datasets 3.0.1
Tokenizers 0.20.3