# ============================================
# 🔥 POTI-WP - MODO ESTÁVEL MAS FORTE 🔥
# ============================================

base_model: unsloth/llama-3-8b-bnb-4bit
model_type: LlamaForCausalLM
load_in_8bit: false
load_in_4bit: true
strict: false

datasets:
  - path: mrmsoares/poti-wp-instruct
    type: alpaca
    shards: 1

dataset_prepared_path: last_run_prepared
val_set_size: 0.02
output_dir: ./outputs/poti-wp-lora-chernobyl

adapter: qlora

sequence_len: 2048
sample_packing: true
pad_to_sequence_len: true
eval_sample_packing: false

lora_r: 64
lora_alpha: 128
lora_dropout: 0.05
lora_target_linear: true

# Desabilita os kernels otimizados que estão dando problema
lora_mlp_kernel: false
lora_qkv_kernel: false
lora_o_kernel: false

micro_batch_size: 2
gradient_accumulation_steps: 8

num_epochs: 4
optimizer: adamw_torch_fused
lr_scheduler: cosine
learning_rate: 2e-4
train_on_inputs: false
group_by_length: false

bf16: auto
fp16: false
tf32: true

gradient_checkpointing: true

flash_attention: true

torch_compile: false

dataloader_num_workers: 2
dataloader_pin_memory: true

logging_steps: 1
warmup_steps: 10
evals_per_epoch: 2
saves_per_epoch: 1

weight_decay: 0.01
seed: 42
neftune_noise_alpha: 5

outputs/poti-wp-lora-chernobyl

This model is a fine-tuned version of unsloth/llama-3-8b-bnb-4bit on the mrmsoares/poti-wp-instruct dataset. It achieves the following results on the evaluation set:

Loss: 12.6679
Ppl: 317384.6004
Memory/max Active (gib): 12.45
Memory/max Allocated (gib): 12.45
Memory/device Reserved (gib): 16.26

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0002
train_batch_size: 2
eval_batch_size: 2
seed: 42
gradient_accumulation_steps: 8
total_train_batch_size: 16
optimizer: Use adamw_torch_fused with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 10
training_steps: 16

Training results

Training Loss	Epoch	Step	Validation Loss	Ppl	Active (gib)	Allocated (gib)	Reserved (gib)
No log	0	0	12.6570	313965.2955	11.18	11.18	11.46
0.5264	0.4324	2	12.6579	314226.4989	12.45	12.45	16.26
0.4617	0.8649	4	12.6410	308956.2535	12.45	12.45	16.26
0.3077	1.2162	6	12.6273	304764.1832	12.45	12.45	16.26
0.206	1.6486	8	12.6059	298315.9366	12.45	12.45	16.26
0.2138	2.0	10	12.5923	294282.2715	12.45	12.45	16.26
0.1438	2.4324	12	12.6695	317914.7353	12.45	12.45	16.26
0.1409	2.8649	14	12.6859	323160.2306	12.45	12.45	16.26
0.1006	3.2162	16	12.6679	317384.6004	12.45	12.45	16.26

Framework versions

PEFT 0.18.1.dev0
Transformers 4.57.1
Pytorch 2.8.0+cu128
Datasets 4.4.2
Tokenizers 0.22.1

mrmsoares/poti-wp-llama3-8b-lora

outputs/poti-wp-lora-chernobyl

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

mrmsoares/
poti-wp-llama3-8b-lora