base_model: Qwen/Qwen3-14B-Base

trust_remote_code: true
tokenizer_use_fast: true

load_in_8bit: false
load_in_4bit: false


datasets:
  - path: philipperen55/datasetCPT70axolotlRandomized
    split: train
    data_files: datasetCPT70axolotlRandomized.jsonl
    type: completion
    field: text


val_set_size: 0.0001

dataset_prepared_path: prepared_cpt
output_dir: outputs_cpt

sequence_len: 2048
pad_to_sequence_len: true
sample_packing: true
eval_sample_packing: true
attn_implementation: flash_attention_2
excess_length_strategy: truncate
train_on_inputs: true 
add_eos_token: true

# FULL finetune CPT
micro_batch_size: 2
gradient_accumulation_steps: 8
num_epochs: 1


optimizer: adamw_8bit
learning_rate: 4e-5
weight_decay: 0.01
lr_scheduler: constant_with_warmup
warmup_ratio: 0.01
max_grad_norm: 1.0

fp16: false
bf16: true
tf32: true
gradient_checkpointing: false

logging_steps: 50
eval_steps: 1000
save_steps: 5000
save_total_limit: 1
save_only_model: true

seed: 42
#mettre 24 si ya plus de 24 vspu, sinon mettre 16 si ya 24vcpu
dataset_num_proc: 24

# WandB
wandb_project: qwen3_14b_cpt_full_axolotl

# Hub
hub_model_id:  
push_to_hub: false #mon script upload de façon plus sûre
hub_strategy: every_save

outputs_cpt

This model is a fine-tuned version of Qwen/Qwen3-14B-Base on the philipperen55/datasetCPT70axolotlRandomized dataset. It achieves the following results on the evaluation set:

Loss: 1.8920
Ppl: 6.6325
Memory/max Active (gib): 137.59
Memory/max Allocated (gib): 137.59
Memory/device Reserved (gib): 139.39

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 4e-05
train_batch_size: 2
eval_batch_size: 2
seed: 42
gradient_accumulation_steps: 8
total_train_batch_size: 16
optimizer: Use OptimizerNames.ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: constant_with_warmup
lr_scheduler_warmup_steps: 467
training_steps: 46749

Training results

Training Loss	Epoch	Step	Validation Loss	Ppl	Active (gib)	Allocated (gib)	Reserved (gib)
No log	0	0	2.0274	7.5945	33.35	33.35	33.44
1.9949	0.0214	1000	1.9929	7.3368	65.62	65.62	139.36
1.9786	0.0428	2000	1.9805	7.2465	65.62	65.62	139.39
1.9634	0.0642	3000	1.9727	7.1902	65.62	65.62	139.39
1.9636	0.0856	4000	1.9664	7.1452	65.62	65.62	139.39
1.9498	0.1069	5000	1.9614	7.1090	65.62	65.62	139.39
1.9595	0.1283	6000	1.9569	7.0775	65.62	65.62	139.49
1.9473	0.1497	7000	1.9514	7.0387	65.62	65.62	139.39
1.9472	0.1711	8000	1.9498	7.0273	65.62	65.62	139.39
1.9413	0.1925	9000	1.9471	7.0086	65.62	65.62	139.39
1.9363	0.2139	10000	1.9445	6.9900	65.62	65.62	139.39
1.9367	0.2353	11000	1.9371	6.9383	65.62	65.62	139.49
1.9306	0.2567	12000	1.9346	6.9213	65.62	65.62	139.39
1.9305	0.2781	13000	1.9323	6.9054	65.62	65.62	139.39
1.9193	0.2995	14000	1.9301	6.8899	65.62	65.62	139.39
1.9237	0.3208	15000	1.9281	6.8764	65.62	65.62	139.39
1.9206	0.3422	16000	1.9269	6.8685	65.62	65.62	139.49
1.9254	0.3636	17000	1.9246	6.8524	65.62	65.62	139.39
1.9142	0.3850	18000	1.9218	6.8334	65.62	65.62	139.39
1.9212	0.4064	19000	1.9209	6.8270	65.62	65.62	139.39
1.9099	0.4278	20000	1.9188	6.8129	65.62	65.62	139.39
1.9175	0.4492	21000	1.9174	6.8031	65.62	65.62	139.49
1.9009	0.4706	22000	1.9177	6.8050	65.62	65.62	139.39
1.9144	0.4920	23000	1.9151	6.7873	65.62	65.62	139.39
1.9087	0.5134	24000	1.9125	6.7699	65.62	65.62	139.39
1.9039	0.5347	25000	1.9122	6.7682	65.62	65.62	139.39
1.8991	0.5561	26000	1.9096	6.7504	65.62	65.62	139.49
1.8998	0.5775	27000	1.9097	6.7513	65.62	65.62	139.39
1.9083	0.5989	28000	1.9077	6.7376	65.62	65.62	139.39
1.9060	0.6203	29000	1.9067	6.731	65.62	65.62	139.39
1.9066	0.6417	30000	1.9037	6.7106	65.62	65.62	139.39
1.8953	0.6631	31000	1.9025	6.7026	65.62	65.62	139.49
1.9043	0.6845	32000	1.9026	6.7032	65.62	65.62	139.39
1.8964	0.7059	33000	1.9007	6.6908	65.62	65.62	139.39
1.9115	0.7273	34000	1.9010	6.6924	65.62	65.62	139.39
1.8899	0.7486	35000	1.8997	6.6839	65.62	65.62	139.39
1.8995	0.7700	36000	1.9001	6.6867	65.62	65.62	139.49
1.8852	0.7914	37000	1.8983	6.6744	65.62	65.62	139.39
1.8880	0.8128	38000	1.8981	6.6732	65.62	65.62	139.39
1.8923	0.8342	39000	1.8962	6.6603	65.62	65.62	139.39
1.8901	0.8556	40000	1.8951	6.6531	65.62	65.62	139.39
1.8925	0.8770	41000	1.8964	6.6620	65.62	65.62	139.49
1.8947	0.8984	42000	1.8936	6.6435	65.62	65.62	139.39
1.8887	0.9198	43000	1.8942	6.6471	65.62	65.62	139.39
1.8871	0.9412	44000	1.8934	6.6419	65.62	65.62	139.39
1.8853	0.9625	45000	1.8932	6.6405	65.62	65.62	139.39
1.8899	0.9839	46000	1.8913	6.6281	65.62	65.62	139.49
1.8791	1.0000	46749	1.8920	6.6325	137.59	137.59	139.39

Framework versions

Transformers 5.9.0
Pytorch 2.10.0+cu128
Datasets 4.8.5
Tokenizers 0.22.2

philipperen55/Qwen3-14B-datasetCPT70-axolotl

outputs_cpt

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

philipperen55/
Qwen3-14B-datasetCPT70-axolotl