You can deploy this model and start earning money today!

philipperen55/
Qwen2.5-14B-v3-3690-dataset56SFT

Name: Qwen2.5-14B-v3-3690-dataset56SFT
Availability: InStock

generated_from_trainer

philipperen55/Qwen2.5-14B-style-MERGED-BF16-v3-3690

adapter

Views

No views yet

See axolotl config

axolotl version: 0.5.0

yaml

base_model: philipperen55/Qwen2.5-14B-style-MERGED-BF16-v3-3690
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
trust_remote_code: true

load_in_8bit: false
load_in_4bit: false


datasets:
  - path: philipperen55/dataset56SFT 
    ds_type: json
    data_files: dataset56SFT.jsonl
    type: input_output
  

train_on_inputs: false
add_eos_token: true

  
dataset_prepared_path: /workspace/prepared_data
val_set_size: 0.01
output_dir: /workspace/output


sequence_len: 2048
sample_packing: false
pad_to_sequence_len: false
group_by_length: true



special_tokens:
  eos_token: "<|endoftext|>"
  pad_token: "<|endoftext|>"


adapter: lora
lora_r: 32
lora_alpha: 64
lora_dropout: 0.05
lora_target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
  - gate_proj
  - up_proj
  - down_proj


gradient_accumulation_steps: 32
micro_batch_size: 2 # 32/2 = 98497MiB / 143771MiB et  16/4 plante out of memory !!!
num_epochs: 3
learning_rate: 3e-5
lr_scheduler: cosine
warmup_ratio: 0.03
optimizer: adamw_torch_fused
weight_decay: 0.0
max_grad_norm: 1.0


bf16: true
fp16: false
tf32: true


flash_attention: true
sdp_attention: false

overrides_of_model_config:
  attn_implementation: flash_attention_2  # forcer FA2 jusqu’au modèle
  use_cache: false
  

gradient_checkpointing: false

# --- Reproductibilité ---
seed: 42


logging_steps: 5
save_steps: 50
eval_strategy: steps 
eval_steps: 50
save_total_limit: 1


wandb_project: Qwen2.5-14B-v3-3690-dataset56SFT_00


hub_model_id: philipperen55/Qwen2.5-14B-v3-3690-dataset56SFT
push_to_hub: true
hub_strategy: every_save

Qwen2.5-14B-v3-3690-dataset56SFT

This model is a fine-tuned version of philipperen55/Qwen2.5-14B-style-MERGED-BF16-v3-3690 on the None dataset. It achieves the following results on the evaluation set:

Loss: 1.1214

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 3e-05
train_batch_size: 2
eval_batch_size: 2
seed: 42
gradient_accumulation_steps: 32
total_train_batch_size: 64
optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 10
num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.0084	1	1.2921
1.1494	0.4193	50	1.1616
1.1403	0.8386	100	1.1435
1.1422	1.2579	150	1.1320
1.1027	1.6771	200	1.1270
1.1006	2.0964	250	1.1228
1.0844	2.5157	300	1.1220
1.106	2.9350	350	1.1214

Framework versions

PEFT 0.13.2
Transformers 4.46.1
Pytorch 2.3.1+cu121
Datasets 3.0.1
Tokenizers 0.20.3