base_model: Qwen/Qwen3-0.6B
trust_remote_code: true
strict: false

chat_template: qwen3

datasets:
  - path: ./dataset_bpln.jsonl
    type: chat_template
    field_messages: conversations
    message_property_mappings:
      role: from
      content: value
    roles:
      user: ["human"]
      assistant: ["gpt"]
      system: ["system"]

dataset_prepared_path: ./process
val_set_size: 0.05 
output_dir: ./outputs/out

sequence_len: 512
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true

wandb_project: Carapicu
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

load_in_8bit: false
load_in_4bit: false

gradient_accumulation_steps: 4
micro_batch_size: 4
num_epochs: 4 

optimizer: adamw_torch
lr_scheduler: cosine
learning_rate: 2e-5  
weight_decay: 0.01  

warmup_ratio: 0.05

bf16: true
fp16: false
tf32: true

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false

flash_attention: true  

logging_steps: 1
evals_per_epoch: 2  
saves_per_epoch: 1
save_total_limit: 2

special_tokens:
  eos_token: "<|im_end|>"

outputs/out

This model is a fine-tuned version of Qwen/Qwen3-0.6B on the [./dataset_bpln.jsonl] dataset. It achieves the following results on the evaluation set:

Loss: 1.5021
Ppl: 4.4913
Memory/max Active (gib): 4.8
Memory/max Allocated (gib): 4.8
Memory/device Reserved (gib): 7.25

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 3e-05
train_batch_size: 2
eval_batch_size: 2
seed: 42
gradient_accumulation_steps: 4
total_train_batch_size: 8
optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 12
training_steps: 252

Training results

Training Loss	Epoch	Step	Validation Loss	Ppl	Active (gib)	Allocated (gib)	Reserved (gib)
No log	0	0	2.4143	11.1814	2.57	2.57	2.61
1.5314	0.5059	32	1.5519	4.7203	4.8	4.8	7.91
0.8367	1.0	64	1.4883	4.4298	4.8	4.8	7.28
1.1617	1.5059	96	1.4841	4.4111	4.8	4.8	7.25
1.3480	2.0	128	1.4672	4.3369	4.8	4.8	7.25
0.9976	2.5059	160	1.5048	4.5030	4.8	4.8	7.25
1.1338	3.0	192	1.5013	4.4876	4.8	4.8	7.25
1.1199	3.5059	224	1.5028	4.4941	4.8	4.8	7.25
1.0614	3.9486	252	1.5021	4.4913	4.8	4.8	7.25

Framework versions

Transformers 5.8.0
Pytorch 2.11.0+cu128
Datasets 4.5.0
Tokenizers 0.22.2

FORMAS/Carapicu-Qwen3-0.6B-SFT

outputs/out

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

FORMAS/
Carapicu-Qwen3-0.6B-SFT