You can deploy this model and start earning money today!

philipperen55/
Qwen2.5-7B-dataset50CFT-dataset60SFT_lora

Name: Qwen2.5-7B-dataset50CFT-dataset60SFT_lora
Availability: InStock

generated_from_trainer

philipperen55/Qwen2.5-7B-CFT

adapter

Views

No views yet

See axolotl config

axolotl version: 0.5.0

yaml

base_model: philipperen55/Qwen2.5-7B-CFT
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
trust_remote_code: true

load_in_8bit: false
load_in_4bit: false


datasets:
  - path: philipperen55/dataset60SFT 
    ds_type: json
    data_files: dataset60SFT.jsonl
    type: input_output
  

train_on_inputs: false
add_eos_token: true

  
dataset_prepared_path: /workspace/prepared_data
val_set_size: 0.01
output_dir: /workspace/output


sequence_len: 2048
sample_packing: false
pad_to_sequence_len: false
group_by_length: true



special_tokens:
  eos_token: "<|endoftext|>"
  pad_token: "<|endoftext|>"


adapter: lora
lora_r: 32
lora_alpha: 64
lora_dropout: 0.05
lora_target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
  - gate_proj
  - up_proj
  - down_proj


gradient_accumulation_steps: 16
micro_batch_size: 4 # 32/2 = 98497MiB / 143771MiB et  16/4 plante out of memory sur H200 pour 14b !!!
num_epochs: 3
learning_rate: 2e-5
lr_scheduler: cosine
warmup_ratio: 0.03
optimizer: adamw_torch_fused
weight_decay: 0.0
max_grad_norm: 1.0


bf16: true
fp16: false
tf32: true


flash_attention: true
sdp_attention: false

overrides_of_model_config:
  attn_implementation: flash_attention_2  # forcer FA2 jusqu’au modèle
  use_cache: false
  

gradient_checkpointing: false

# --- Reproductibilité ---
seed: 42


logging_steps: 5
save_steps: 50
eval_strategy: steps 
eval_steps: 50
save_total_limit: 1


wandb_project: Qwen2.5-7B-dataset50CFT-dataset60SFT_lora


hub_model_id: philipperen55/Qwen2.5-7B-dataset50CFT-dataset60SFT_lora
push_to_hub: true
hub_strategy: every_save

Qwen2.5-7B-dataset50CFT-dataset60SFT_lora

This model is a fine-tuned version of philipperen55/Qwen2.5-7B-CFT on the None dataset. It achieves the following results on the evaluation set:

Loss: 1.2183

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 4
eval_batch_size: 4
seed: 42
gradient_accumulation_steps: 16
total_train_batch_size: 64
optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 10
num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.0087	1	1.3461
1.2403	0.4360	50	1.2547
1.2362	0.8719	100	1.2353
1.1999	1.3079	150	1.2263
1.2661	1.7439	200	1.2200
1.1829	2.1798	250	1.2224
1.2015	2.6158	300	1.2183

Framework versions

PEFT 0.13.2
Transformers 4.46.1
Pytorch 2.3.1+cu121
Datasets 3.0.1
Tokenizers 0.20.3