See axolotl config

axolotl version: 0.4.1

yaml

1base_model: concedo/KobbleTinyV2-1.1B 2model_type: LlamaForCausalLM 3tokenizer_type: LlamaTokenizer 4 5load_in_8bit: false 6load_in_4bit: true 7strict: false 8 9datasets: 10 - path: NobodyExistsOnTheInternet/AlpacaToxicQA 11 type: alpaca 12 - path: Fischerboot/freedom-rp-alpaca-shortend 13 type: alpaca 14dataset_prepared_path: 15val_set_size: 0.05 16output_dir: ./outputs/32r 17 18adapter: qlora 19lora_model_dir: 20 21sequence_len: 4096 22sample_packing: true 23eval_sample_packing: false 24pad_to_sequence_len: true 25 26lora_r: 32 27lora_alpha: 16 28lora_dropout: 0.05 29lora_target_modules: 30lora_target_linear: true 31lora_fan_in_fan_out: 32 33wandb_project: 34wandb_entity: 35wandb_watch: 36wandb_name: 37wandb_log_model: 38 39gradient_accumulation_steps: 4 40micro_batch_size: 2 41num_epochs: 4 42optimizer: paged_adamw_32bit 43lr_scheduler: cosine 44learning_rate: 0.0002 45 46train_on_inputs: false 47group_by_length: false 48bf16: auto 49fp16: 50tf32: false 51 52gradient_checkpointing: true 53early_stopping_patience: 54resume_from_checkpoint: 55local_rank: 56logging_steps: 1 57xformers_attention: 58flash_attention: true 59 60warmup_steps: 10 61evals_per_epoch: 4 62saves_per_epoch: 1 63debug: 64deepspeed: 65weight_decay: 0.0 66fsdp: 67fsdp_config: 68special_tokens: 69

outputs/32r

• Loss: 1.3368

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 0.0002
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 4

Training results

Framework versions

• PEFT 0.11.1
• Transformers 4.42.3
• Pytorch 2.1.2+cu118
• Datasets 2.19.1
• Tokenizers 0.19.1