base_model: philipperen55/Qwen3-14B-datasetCPT70-axolotl
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
trust_remote_code: true

load_in_8bit: false
load_in_4bit: false


datasets:
  - path: philipperen55/datasetSFT73 
    ds_type: json
    data_files: datasetSFT73.jsonl
    type: input_output
  

train_on_inputs: false
add_eos_token: false

  
dataset_prepared_path: /workspace/prepared_data
val_set_size: 0.05
output_dir: /workspace/output


sequence_len: 2048
sample_packing: false
pad_to_sequence_len: false
group_by_length: true



special_tokens:
  eos_token: "<|endoftext|>"
  pad_token: "<|endoftext|>"


adapter: lora
lora_r: 32
lora_alpha: 64
lora_dropout: 0.05
lora_target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
  - gate_proj
  - up_proj
  - down_proj


gradient_accumulation_steps: 4
micro_batch_size: 4 
num_epochs: 3
learning_rate: 1e-4
lr_scheduler: cosine
warmup_ratio: 0.05
optimizer: adamw_torch_fused
weight_decay: 0.0
max_grad_norm: 1.0


bf16: true
fp16: false
tf32: true


attn_implementation: flash_attention_2


overrides_of_model_config:
  use_cache: false
  

gradient_checkpointing: true


seed: 42
#mettre 24 si ya plus de 24 vspu, sinon mettre 16 si ya 24vcpu
dataset_num_proc: 16


logging_steps: 5
save_steps: 50
eval_strategy: steps 
eval_steps: 50
save_total_limit: 1


wandb_project: datasetSFT73


hub_model_id: philipperen55/Qwen3-14B-datasetCPT70-axolotl-datasetSFT73_lora2
push_to_hub: true
hub_strategy: every_save

Qwen3-14B-datasetCPT70-axolotl-datasetSFT73_lora2

This model is a fine-tuned version of philipperen55/Qwen3-14B-datasetCPT70-axolotl on the philipperen55/datasetSFT73 dataset. It achieves the following results on the evaluation set:

Loss: 1.5262
Ppl: 4.6006
Memory/max Active (gib): 42.73
Memory/max Allocated (gib): 42.73
Memory/device Reserved (gib): 68.63

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0001
train_batch_size: 4
eval_batch_size: 4
seed: 42
gradient_accumulation_steps: 4
total_train_batch_size: 16
optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 14
training_steps: 288

Training results

Training Loss	Epoch	Step	Validation Loss	Ppl	Active (gib)	Allocated (gib)	Reserved (gib)
No log	0	0	1.7369	5.6799	38.13	38.13	45.34
1.4807	0.5208	50	1.5701	4.8072	39.13	39.13	49.81
1.4321	1.0417	100	1.5327	4.6306	39.13	39.13	84.9
1.4647	1.5625	150	1.5214	4.5785	39.13	39.13	59.41
1.4300	2.0833	200	1.5329	4.6318	39.13	39.13	71.87
1.4937	2.6042	250	1.5245	4.5931	39.13	39.13	71.34
1.0888	3.0	288	1.5262	4.6006	42.73	42.73	68.63

Framework versions

PEFT 0.19.1
Transformers 5.9.0
Pytorch 2.10.0+cu128
Datasets 4.8.5
Tokenizers 0.22.2