You can deploy this model and start earning money today!

jackryder00/
nadia_loud

peft

safetensors

llama

generated_from_trainer

chuanli11/Llama-3.2-3B-Instruct-uncensored

adapter

4-bit

bitsandbytes

Views

No views yet

See axolotl config

axolotl version: 0.4.1

yaml

# CONFIG EXPERIMENTO 4: DOBLE O NADA (LR ALTO, 5 ÉPOCAS)
base_model: chuanli11/Llama-3.2-3B-Instruct-uncensored
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer
trust_remote_code: true
special_tokens:
  pad_token: "<|eot_id|>"

datasets:
  - path: /root/nadia_finetune/dataset_v3.jsonl
    type: alpaca

val_set_size: 0.05
sequence_len: 4096
load_in_4bit: true
adapter: qlora
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj

learning_rate: 2e-5 # <<< Usamos la tasa de aprendizaje "agresiva" que funcionó bien
num_epochs: 5       # <<< Aumentamos las épocas para ver si mejora o sobreajusta

micro_batch_size: 1
gradient_accumulation_steps: 4
optimizer: paged_adamw_8bit
lr_scheduler: cosine
warmup_steps: 100

output_dir: /root/nadia_finetune/lora-output-exp4-doble-o-nada # <<< Nueva carpeta de salida
save_strategy: steps
save_steps: 200
logging_steps: 10

root/nadia_finetune/lora-output-exp4-doble-o-nada

This model is a fine-tuned version of chuanli11/Llama-3.2-3B-Instruct-uncensored on the None dataset. It achieves the following results on the evaluation set:

Loss: 2.9337

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 1
eval_batch_size: 1
seed: 42
gradient_accumulation_steps: 4
total_train_batch_size: 4
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 100
num_epochs: 5

Training results

Training Loss	Epoch	Step	Validation Loss
4.2265	0.9825	42	5.2438
3.6223	1.9883	85	4.0555
3.0948	2.9942	128	3.2039
2.8313	4.0	171	2.9670
2.7834	4.9123	210	2.9337

Framework versions

PEFT 0.13.2
Transformers 4.45.2
Pytorch 2.3.1+cu121
Datasets 3.0.1
Tokenizers 0.20.1

jackryder00/nadia_loud

root/nadia_finetune/lora-output-exp4-doble-o-nada

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

jackryder00/
nadia_loud